---
title: "bidux API Modernization Features"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{bidux API Modernization Features}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = FALSE
)
```

## Introduction

The bidux package has been modernized with enhanced API consistency, better error handling, and optional tibble support. **This vignette is the definitive guide to modern API usage in bidux 0.4.0+**.

### What's New in 0.4.0

| Feature | Status | Details |
|---------|--------|---------|
| **Flat data_story API** | ✅ **REQUIRED** | Use `new_data_story(hook, context, tension, resolution)` |
| **Nested data_story format** | ❌ **REMOVED** | `variables`/`relationships` parameters removed in 0.4.0 |
| **Telemetry presets** | ✅ **STABLE** | Use `bid_telemetry_presets("moderate")` for easier configuration |
| **S3 classes and methods** | ✅ **STABLE** | Enhanced print, summary, and conversion methods |
| **Tibble integration** | ✅ **OPTIONAL** | Automatic tibble output when package available |

### Migration from 0.3.x to 0.4.0

As of 0.4.0, the nested data_story format has been removed. Update your code:

```r
# OLD (removed in 0.4.0)
story <- new_data_story(
  context = "Dashboard usage dropped",
  variables = list(hook = "Engagement declining"),
  relationships = list(resolution = "Analyze telemetry")
)

# REQUIRED (0.4.0+)
story <- new_data_story(
  hook = "Engagement declining",
  context = "Dashboard usage dropped 30%",
  tension = "Unknown if UX or user needs",
  resolution = "Analyze telemetry to identify friction"
)
```

```{r setup}
library(bidux)
# Optional tibble support
if (requireNamespace("tibble", quietly = TRUE)) {
  library(tibble)
}
```

## Enhanced Data Story Creation

The `new_data_story()` function provides a structured way to create behavioral insights. **As of bidux 0.4.0, the flat API is the ONLY supported approach** for creating data stories.

### Flat API (Required in 0.4.0+)

The flat API accepts four intuitive arguments that map directly to storytelling elements:

```{r data_story_recommended}
# RECOMMENDED: Create a structured data story using the modern flat API
story <- new_data_story(
  hook = "Users struggle with complex form layouts leading to abandonment",
  context = "User Interface Cognitive Load Analysis for complex form layouts with 40% improvement potential",
  tension = "Current forms overwhelm users with cognitive load and lack clear guidance",
  resolution = "Implement step-by-step form progression with visual progress indicators"
)

# Enhanced print method
print(story)

# Access story elements directly using the flat API
story$hook
story$context
story$tension
story$resolution

# Story is automatically structured for further BID framework usage
```

### Legacy Nested Format (Removed in 0.4.0)

The nested format using `variables` and `relationships` parameters was **REMOVED** in bidux 0.4.0. Code using this format will now error.

### Migration from Nested to Flat Format

If you're upgrading from 0.3.x, update your code from the old nested format to the flat format:

**Old nested format (no longer supported):**
```{r migration_old, eval = FALSE}
# OLD (removed in 0.4.0) - THIS WILL ERROR
story <- new_data_story(
  context = "Dashboard usage dropped",
  variables = list(hook = "User engagement declining"),
  relationships = list(resolution = "Analyze telemetry")
)
```

**Required flat format (0.4.0+):**
```{r migration_new, eval = FALSE}
# REQUIRED (0.4.0+)
story <- new_data_story(
  hook = "User engagement declining",
  context = "Dashboard usage dropped 30%",
  tension = "Don't know if UX or user needs",
  resolution = "Analyze telemetry"
)
```

## Modern BID Framework Interpretation

The enhanced `bid_interpret()` function captures user needs and central questions with comprehensive validation and modern S3 methods. **Always use the flat API for data_story objects (required in 0.4.0+).**

```{r interpret}
# Basic interpretation with modern flat API (RECOMMENDED)
result <- bid_interpret(
  central_question = "How can we reduce cognitive load in our signup form?",
  data_story = new_data_story(
    hook = "Users abandon signup forms at 60% rate",
    context = "Current form has 15 required fields",
    tension = "Users feel overwhelmed and leave",
    resolution = "Simplify form using progressive disclosure"
  ),
  quiet = FALSE
)

# View results
print(result)

# Enhanced interpretation with user personas
interpretation <- bid_interpret(
  central_question = "What drives user engagement in our dashboard?",
  data_story = new_data_story(
    hook = "Daily active users declining despite new features",
    context = "Rich dashboard with multiple visualizations",
    tension = "Users aren't discovering valuable insights",
    resolution = "Guide attention to high-value content"
  ),
  user_personas = list(
    list(
      name = "Data Analyst",
      goals = "Find patterns quickly",
      pain_points = "Too many visualizations",
      technical_level = "Advanced"
    )
  ),
  quiet = TRUE
)

# Enhanced S3 print method shows summary
print(interpretation)
```

**Note:** While `bid_interpret()` still accepts plain list format for `data_story` for backward compatibility, using `new_data_story()` with the flat API is required for proper validation and consistency.

## Structured Dashboard Layout with bid_structure

Transform insights into actionable dashboard structure with the enhanced `bid_structure` function that automatically selects optimal layout.

```{r structure}
# Follow correct BID framework order: Interpret → Notice → Anticipate → Structure
notice_result <- result |>
  bid_notice(
    problem = "Users struggle with cognitive overload in signup forms",
    evidence = "60% abandonment rate and user feedback surveys",
    quiet = TRUE
  )

# Anticipate potential cognitive biases affecting form completion
anticipate_result <- notice_result |>
  bid_anticipate(
    bias_mitigations = list(
      cognitive_overload = "Progressive form completion with clear step indicators",
      choice_overload = "Offer quick signup option with minimal required fields"
    ),
    quiet = TRUE
  )

# Structure creates actionable dashboard recommendations
structured <- bid_structure(
  previous_stage = anticipate_result,
  concepts = c("Progressive Disclosure", "Cognitive Load Theory"),
  quiet = TRUE
)

# View concept-grouped suggestions
print("Structured recommendations:")
print(structured$suggestions)

# Summary provides overview of all recommendations
summary(structured)
```

## Complete Workflow Example

Demonstrate the full modernized BID framework workflow with consistent API patterns using the **flat API throughout** (required in 0.4.0+).

```{r workflow}
# Step 1: Interpret user needs and context (using flat API)
ui_analysis <- bid_interpret(
  central_question = "How can we improve landing page conversion without damaging user trust?",
  data_story = new_data_story(
    hook = "High-pressure sales tactics may be reducing user trust",
    context = "Landing page with multiple persuasive elements",
    tension = "Need conversions but not at expense of user experience",
    resolution = "Balance persuasion with trust-building elements"
  ),
  user_personas = list(
    list(
      name = "Potential Customer",
      goals = "Evaluate product value",
      pain_points = "Feels pressured by aggressive sales tactics",
      technical_level = "Basic"
    )
  ),
  quiet = TRUE
)

# Step 2: Notice specific problems
problem_analysis <- ui_analysis |>
  bid_notice(
    problem = "Aggressive persuasion tactics creating user pressure",
    evidence = "Lower conversion rates and negative user feedback on pressure tactics",
    quiet = TRUE
  )

# Step 3: Anticipate cognitive biases affecting user trust and decision-making
interaction_analysis <- problem_analysis |>
  bid_anticipate(
    bias_mitigations = list(
      social_proof = "Display trust signals and authentic user testimonials",
      scarcity_bias = "Avoid artificial urgency tactics that undermine trust"
    ),
    quiet = TRUE
  )

# Step 4: Structure actionable recommendations
recommendations <- bid_structure(
  previous_stage = interaction_analysis,
  concepts = c("Social Proof", "Trust Signals", "Progressive Disclosure"),
  quiet = TRUE
)

# Step 5: Validate design decisions with experimental design
validation_analysis <- bid_validate(
  previous_stage = recommendations,
  include_exp_design = TRUE,
  quiet = TRUE
)

# Step 6: Create comprehensive data story from complete workflow using flat API
optimization_story <- new_data_story(
  hook = "High-pressure persuasive elements creating user trust concerns",
  context = "Complete BID Framework Analysis: Landing Page Trust-Based Conversion Optimization across all 5 stages",
  tension = "Need to balance conversion goals with user trust - current approach damages long-term relationships",
  resolution = "Apply validated BID framework recommendations balancing persuasion with trust-building, monitoring both conversion rates and user trust metrics"
)

# View complete analysis results
print("=== Complete BID Framework Workflow Results ===")
print(optimization_story)
print("\n=== Stage Summaries ===")
summary(recommendations)
summary(validation_analysis)
```

## Error Handling and Validation

The modernized API provides clear, descriptive error messages for better debugging.

```{r validation, error = TRUE}
# Clear error messages for invalid inputs
try(bid_interpret()) # Missing required parameter
try(bid_interpret(central_question = 123)) # Wrong type for central_question
try(bid_interpret(central_question = "test", data_story = "invalid")) # Invalid data_story

# Descriptive validation errors for data story creation (flat API)
try(new_data_story()) # Missing required parameter (context)
try(new_data_story(context = "")) # Invalid empty context
```

### Removed Features

As of 0.4.0, the nested format is no longer supported and will produce an error:

```{r removed_features, eval = FALSE}
# Nested format is no longer supported (removed in 0.4.0)
try(new_data_story(
  context = "Dashboard analysis",
  variables = list(metric = "engagement"),
  relationships = list(trend = "declining")
))
# Error: The nested data_story format (variables/relationships) was removed in bidux 0.4.0.
# Please use the flat API: new_data_story(hook, context, tension, resolution)
```

## Telemetry Sensitivity Presets (New in 0.3.2)

The `bid_telemetry_presets()` function provides pre-configured threshold sets for easier telemetry analysis configuration, eliminating the need for manual threshold tuning.

### Available Presets

```{r telemetry_presets}
# Three sensitivity levels available:
# - "strict": Detects even minor issues (for critical apps or new dashboards)
# - "moderate": Balanced default (appropriate for most applications)
# - "relaxed": Only detects major issues (for mature, stable dashboards)

# Get strict sensitivity thresholds
strict_thresholds <- bid_telemetry_presets("strict")
print(strict_thresholds)

# Get moderate sensitivity (default)
moderate_thresholds <- bid_telemetry_presets("moderate")

# Get relaxed sensitivity
relaxed_thresholds <- bid_telemetry_presets("relaxed")
```

### Using Presets with Telemetry Analysis

```{r telemetry_usage, eval = FALSE}
# Use strict preset for critical application
issues <- bid_ingest_telemetry(
  "telemetry.sqlite",
  thresholds = bid_telemetry_presets("strict")
)

# Use relaxed preset for mature dashboard
issues <- bid_ingest_telemetry(
  "telemetry.sqlite",
  thresholds = bid_telemetry_presets("relaxed")
)

# Override specific thresholds while using a preset as base
custom_thresholds <- bid_telemetry_presets("moderate")
custom_thresholds$unused_input_threshold <- 0.03 # More sensitive to unused inputs
issues <- bid_ingest_telemetry("telemetry.sqlite", thresholds = custom_thresholds)
```

### Threshold Comparison

| Threshold | Strict | Moderate | Relaxed | Description |
|-----------|--------|----------|---------|-------------|
| unused_input_threshold | 0.02 | 0.05 | 0.10 | Min usage rate to flag input |
| delay_threshold_secs | 20 | 30 | 60 | Seconds before flagging delay |
| error_rate_threshold | 0.05 | 0.10 | 0.20 | Error rate to flag problems |
| navigation_threshold | 0.10 | 0.20 | 0.30 | Min visit rate for pages |
| rapid_change_window | 15 | 10 | 5 | Time window for confusion detection (sec) |
| rapid_change_count | 4 | 5 | 7 | Changes in window to flag confusion |

**Recommendation:** Start with `"moderate"` preset and adjust based on your application's maturity and criticality. Use `"strict"` for new features or critical workflows, and `"relaxed"` for well-established interfaces.

## Tibble Integration

When the tibble package is available, all functions can return modern tibble objects.

```{r tibble}
# Check if tibble is available
if (requireNamespace("tibble", quietly = TRUE)) {
  # BID framework functions return tibbles by default when available
  # Using flat API
  result_tbl <- bid_interpret(
    central_question = "How can we test tibble integration?",
    data_story = new_data_story(
      hook = "Package supports modern tibble output",
      context = "Enhanced data handling with tibble package",
      tension = "Need to verify integration works correctly",
      resolution = "Test and validate tibble functionality"
    ),
    quiet = TRUE
  )

  cat("Result class:", class(result_tbl), "\n")

  # Works seamlessly with dplyr if available
  if (requireNamespace("dplyr", quietly = TRUE)) {
    library(dplyr)

    # Filter and analyze BID framework results
    interpret_summary <- result_tbl %>%
      select(stage, central_question, hook) %>%
      filter(!is.na(central_question))

    print(interpret_summary)
  }
} else {
  cat("Tibble package not available, using base data.frame\n")
}
```

## API Consistency Features

All core functions now follow consistent patterns for better usability.

```{r consistency}
# Consistent parameter patterns across functions
bid_interpret_params <- c("previous_stage", "central_question", "data_story", "user_personas", "quiet")
bid_structure_params <- c("previous_stage", "concepts", "telemetry_flags", "quiet")

# Common parameters:
# - previous_stage: Chain stages together through the BID pipeline
# - quiet: control informational messages
# - ...: pass additional named arguments through to the stage metadata

# All functions return objects with:
# - S3 classes for proper method dispatch
# - Timestamp columns for tracking
# - Consistent attribute structures
# - Modern data types (tibble when available)
```

## Breaking Changes in 0.4.0

As of version 0.4.0, the nested data_story format has been removed. This is a breaking change that requires updating existing code.

### What Was Removed

The following parameters and features are no longer supported:

1. **Nested data_story format**: The `variables` and `relationships` parameters in `new_data_story()` have been removed
2. **Layout auto-selection based on nested format**: Removed (flat API only)
3. **Layout-specific bias mitigations**: Removed in favor of explicit bias mitigation strategies

### Required Migration Steps

For codebases with existing nested data_story usage:

**Step 1: Identify all `new_data_story()` calls with variables/relationships**

Search your codebase for:
- `variables = list(`
- `relationships = list(`

**Step 2: Extract nested values and flatten**

Example migration:

```{r migration_strategy, eval = FALSE}
# OLD CODE (removed in 0.4.0) - THIS WILL ERROR
story <- new_data_story(
  context = "User study results",
  variables = list(
    hook = "Users struggle with feature discovery",
    metric = "task_completion_rate",
    value = 0.42
  ),
  relationships = list(
    resolution = "Redesign navigation with progressive disclosure"
  )
)

# NEW CODE (required in 0.4.0+)
story <- new_data_story(
  hook = "Users struggle with feature discovery",
  context = "User study results: 42% task completion rate",
  tension = "Low discoverability preventing user success",
  resolution = "Redesign navigation with progressive disclosure",
  # Optional metadata for extra fields
  metric = "task_completion_rate",
  value = 0.42
)
```

### Backward Compatibility for Other Features

All other enhancements from 0.3.x maintain backward compatibility. Only the nested data_story format was removed.

## Summary

The modernized bidux API (0.4.0+) provides:

### Core Features

- **Flat data_story API (REQUIRED)**: Intuitive `new_data_story(hook, context, tension, resolution)` is the only supported format
- **Telemetry sensitivity presets**: Pre-configured threshold sets (`strict`, `moderate`, `relaxed`) via `bid_telemetry_presets()`
- **Enhanced S3 classes** with comprehensive method implementations
- **Optional tibble support** for modern data handling
- **Robust parameter validation** with clear error messages
- **Consistent function interfaces** across all core functions

### Breaking Changes in 0.4.0

- **Nested data_story format** (variables/relationships): **REMOVED in 0.4.0**
- **Layout auto-selection based on nested format**: **REMOVED in 0.4.0**
- **Layout-specific bias mitigations**: **REMOVED in 0.4.0**

Code using these features will now produce errors and must be updated.

### Best Practices for 0.4.0+

1. **Always use flat API for data stories**: `new_data_story(hook, context, tension, resolution)`
2. **Use telemetry presets**: Start with `bid_telemetry_presets("moderate")` instead of manual thresholds
3. **Leverage S3 methods**: Use `print()`, `summary()`, and `as_tibble()` for enhanced output
4. **Follow modern patterns**: Embrace pipe workflows and tidy data principles

### Migration from 0.3.x

If upgrading from 0.3.x:

1. Search your codebase for `variables = list(` and `relationships = list(`
2. Convert all nested format calls to flat format (see migration examples above)
3. Test your code to ensure all data_story objects use the flat API
4. Remove any layout-specific bias mitigation code

These improvements make the package more robust, user-friendly, and suitable for modern R workflows with a cleaner, more maintainable API.