(MOT-4277) feat(harness): add E2E benchmark dashboard

[ytallo]

Fixes MOT-4277

Summary

• publish a daily Harness E2E dashboard backed by gh-pages
• show workflow attempts in a scenario matrix, including the quality percentage reached for passing scenarios and clear failure states
• retain 100 execution summaries and 30 complete structured reports with dedicated detail pages
• keep the temporal quality trends as a separate view and record failed, cancelled, and incomplete executions

Why

The team needs a durable execution-by-execution view of Harness E2E quality so regressions, failures, and metric improvements are visible across daily deployments.

Validation

• dashboard data tests: 11 passed
• collector and publisher contract tests: 8 passed
• Python collectors and publisher compile successfully
• HTML structure, local assets, JavaScript syntax, CSS structure, and local HTTP routes validated
• git diff --check passed

Summary by CodeRabbit

• New Features

  • Added a benchmark dashboard with KPI summaries, trends, scenario health, filters, and execution history.
  • Added execution detail pages showing results, failures, retries, costs, runtimes, and available diagnostic data.
  • Added automated daily benchmark runs and publishing to GitHub Pages, including retained execution summaries and reports.
  • Added preview data and clear empty or unavailable states when benchmark information is missing.

• Documentation

  • Expanded guidance for dashboard usage, local previews, benchmark schedules, scoring, and data retention.

• Tests

  • Added coverage for dashboard rendering, filtering, benchmark collection, publishing, and workflow behavior.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
workers	Ready	Preview	Jul 30, 2026 1:01am
workers-tech-spec	Ready	Preview	Jul 30, 2026 1:01am

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 16186168-8e08-4895-a20e-4196e3b46b3e

📥 Commits

Reviewing files that changed from the base of the PR and between 04b0270 and 6a31ff6.

📒 Files selected for processing (27)

• .github/benchmark-site/README.md
• .github/benchmark-site/app.js
• .github/benchmark-site/dashboard-data.js
• .github/benchmark-site/dashboard-data.test.cjs
• .github/benchmark-site/execution-data.js
• .github/benchmark-site/execution-data.test.cjs
• .github/benchmark-site/execution.html
• .github/benchmark-site/execution.js
• .github/benchmark-site/index.html
• .github/benchmark-site/overview.js
• .github/benchmark-site/sample-data.js
• .github/benchmark-site/sample-executions.js
• .github/benchmark-site/styles.css
• .github/benchmark-site/trends.html
• .github/scripts/collect_harness_e2e_benchmarks.py
• .github/scripts/discover_changed_workers.py
• .github/scripts/publish_harness_e2e_dashboard.py
• .github/scripts/tests/test_collect_harness_e2e_benchmarks.py
• .github/scripts/tests/test_discover_changed_workers.py
• .github/scripts/tests/test_publish_harness_e2e_dashboard.py
• .github/workflows/_harness-e2e.yml
• .github/workflows/ci.yml
• .github/workflows/harness-e2e-benchmark.yml
• .github/workflows/harness-e2e-daily.yml
• .github/workflows/harness-e2e-main.yml
• .github/workflows/harness-e2e-nightly.yml
• harness/tests/e2e/README.md

---

📝 Walkthrough

Walkthrough

Changes

Harness E2E benchmark dashboard

Layer / File(s)	Summary
Benchmark metric collection and normalization .github/scripts/collect_harness_e2e_benchmarks.py, .github/benchmark-site/dashboard-data.js, .../*.test.cjs	Collects validated quality, efficiency, reliability, snapshot, and execution data, then normalizes and tests filtering, aggregation, missing-report, and attempt behavior.
Execution history and retention .github/scripts/publish_harness_e2e_dashboard.py, .github/benchmark-site/execution-data.js, .../execution-data.test.cjs, .github/scripts/tests/test_publish_harness_e2e_dashboard.py	Merges execution history, supports legacy snapshots, publishes manifests, retains detail files, and preserves distinct attempts and availability states.
Static dashboard pages and renderers .github/benchmark-site/*.html, .github/benchmark-site/*.js, .github/benchmark-site/styles.css, .github/benchmark-site/README.md	Adds overview, trends, and execution-detail pages with charts, matrices, filters, diagnostics, preview data, safe links, responsive styling, and local usage documentation.
Scheduled execution and Pages publication .github/workflows/*, .github/scripts/discover_changed_workers.py, harness/tests/e2e/README.md	Adds daily execution and Pages publication workflows, reusable benchmark inputs and artifacts, Node test execution, change detection updates, and dashboard documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Poem

I’m a rabbit with benchmarks to see,
Charts hop from commit to history.
Runs stack neatly, details stay bright,
Daily workflows publish by night.
Metrics, retries, and costs align—
Carrots applaud this dashboard design!

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/harness-e2e-benchmarks

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

skill-check — worker

0 verified, 49 skipped (no docs/).

Layer	Result
structure	✓
vale	✓
ai	✓
render	✓

Four for four. Nicely done.