(MOT-4277) feat(harness): expand benchmark execution insights

[ytallo]

Summary

• prioritizes efficiency with lifecycle-aware scenario baselines, outcome guardrails, and exact execution totals
• opens a per-scenario history modal with cost, token, duration, function-call, and error trends across workflow executions
• marks scenario contract boundaries so new, changed, refactored, and removed scenarios do not distort comparisons
• adds read-only execution conversations and error-focused diagnostics, and removes the standalone trends page

Why

The benchmark dashboard should make harness efficiency changes visible without treating scenario churn or failed outcomes as improvements. Reviewers can now move from the suite overview to one scenario history and then to the complete execution evidence.

Validation

• node --test .github/benchmark-site/*.test.cjs (19 passed)
• dashboard publisher tests (5 passed)
• cargo test --manifest-path harness/tests/e2e/Cargo.toml (59 passed)
• cargo fmt --manifest-path harness/tests/e2e/Cargo.toml --check
• local Firefox validation with retained execution data, including metric switching, hover values, and detail links

Fixes MOT-4277

Summary by CodeRabbit

• New Features
  • Added an efficiency overview with cost, token, duration, function-call, and error metrics, including trends, deltas, sparklines, and comparison guardrails.
  • Added scenario history views with interactive charts, metric tabs, contract-aware comparisons, and execution details.
  • Added session transcript dialogs with readable conversation views, tool-call details, filtering, error highlighting, and read-only access.
  • Expanded execution summaries to show total tokens and function calls.
• Documentation
  • Documented transcript, efficiency, comparison, and scenario-version behavior.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
workers	Ready	Preview	Jul 30, 2026 6:20pm
workers-tech-spec	Ready	Preview	Jul 30, 2026 6:20pm

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: aed053ee-6cee-41c3-a556-9aaf763538d3

📥 Commits

Reviewing files that changed from the base of the PR and between 24b97e6 and d2545e3.

📒 Files selected for processing (24)

• .github/benchmark-site/README.md
• .github/benchmark-site/app.js
• .github/benchmark-site/execution-data.js
• .github/benchmark-site/execution-data.test.cjs
• .github/benchmark-site/execution-transcript.js
• .github/benchmark-site/execution-transcript.test.cjs
• .github/benchmark-site/execution.html
• .github/benchmark-site/execution.js
• .github/benchmark-site/index.html
• .github/benchmark-site/overview.js
• .github/benchmark-site/sample-executions.js
• .github/benchmark-site/styles.css
• .github/benchmark-site/trends.html
• .github/scripts/publish_harness_e2e_dashboard.py
• .github/scripts/tests/test_publish_harness_e2e_dashboard.py
• harness/tests/e2e/README.md
• harness/tests/e2e/src/report.rs
• harness/tests/e2e/src/scenarios/direct_answer.rs
• harness/tests/e2e/src/scenarios/mod.rs
• harness/tests/e2e/src/scenarios/persistent_state.rs
• harness/tests/e2e/src/scenarios/reactive_automation/mod.rs
• harness/tests/e2e/src/scenarios/security_review.rs
• harness/tests/e2e/src/scenarios/shell_coder_sandbox.rs
• harness/tests/e2e/src/suite.rs

---

📝 Walkthrough

Walkthrough

The PR adds versioned E2E scenario contracts, publishes scenario-level efficiency metrics, and introduces efficiency and scenario-history views. Execution details now provide filterable transcript dialogs, while dashboard markup, styling, fixtures, tests, and documentation are updated.

Changes

Scenario metrics and reporting

Layer / File(s)	Summary
Versioned scenario contracts and reports harness/tests/e2e/..., .github/scripts/...	Scenario versions are validated, propagated into reports, and published alongside contract fingerprints, averages, samples, and execution totals.
Execution metric aggregation and comparison .github/benchmark-site/execution-data.js, .github/benchmark-site/execution-data.test.cjs	Scenario metrics, efficiency totals, rolling windows, contract-scoped baselines, trends, rows, and time series are computed and tested.

Dashboard views

Layer / File(s)	Summary
Efficiency overview and scenario history .github/benchmark-site/index.html, .github/benchmark-site/overview.js, .github/benchmark-site/sample-executions.js, .github/benchmark-site/styles.css, .github/benchmark-site/README.md	The overview renders efficiency cards, deltas, sparklines, guardrails, execution totals, and contract-aware scenario history charts and tables.
Session transcript experience .github/benchmark-site/execution-transcript.js, .github/benchmark-site/execution.js, .github/benchmark-site/execution.html, .github/benchmark-site/styles.css	Execution details normalize transcript messages and tool calls into a filterable modal with contextual rendering and error indicators.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Overview
  participant ExecutionData
  participant DetailEndpoint
  participant ScenarioHistoryDialog
  Overview->>DetailEndpoint: fetch execution detail
  DetailEndpoint-->>Overview: scenario runs and totals
  Overview->>ExecutionData: buildEfficiencyOverview(executions)
  ExecutionData-->>Overview: baselines, deltas, and trends
  Overview->>ScenarioHistoryDialog: render scenario history

Loading

Possibly related PRs

• iii-hq/workers#620: Both update the reactive_automation scenario configuration.
• iii-hq/workers#635: Both modify ScenarioSpec plumbing and suite reporting.
• iii-hq/workers#637: Both modify the benchmark dashboard implementation.

Poem

A bunny hops through metrics bright,
With contract versions tucked just right.
Charts bloom, transcripts softly gleam,
Tool calls march across the screen.
“Efficiency!” I cheer with delight—
The dashboard’s ready for review tonight.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/harness-e2e-efficiency-history

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

skill-check — worker

0 verified, 50 skipped (no docs/).

Layer	Result
structure	✓
vale	✓
ai	✓
render	✓

Four for four. Nicely done.