Skip to content

(MOT-4299) Add native local Harness E2E dashboard #749

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ytallo merged 3 commits into from
Aug 7, 2026
Merged

(MOT-4299) Add native local Harness E2E dashboard #749

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6a4dd0f
(MOT-4299) feat(e2e): add local execution comparison
ytallo Aug 7, 2026
5e7ac9f
(MOT-4299) refactor(e2e): move local dashboard into runner
ytallo Aug 7, 2026
f05abb8
(MOT-4299) refactor(e2e): isolate local dashboard layers
ytallo Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 34 additions & 11 deletions .github/benchmark-site/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,48 @@ This static shell replaces the generic benchmark-action index at
for metric trends. `executions.js` indexes workflow attempts, and
`runs/<execution-id>.json` supplies the retained execution report.

Import the default local report and serve the real dashboard from the repository
root:
Start the local dashboard from the repository root:

```bash
python3 .github/scripts/serve_harness_e2e_dashboard.py
cargo build --locked --manifest-path harness/Cargo.toml -p harness-e2e
harness/target/debug/harness-e2e dashboard
```

Pass files or directories to import other local executions:
The dashboard can now execute one or more scenarios against the Harness already
running at `III_URL`. It discovers registered provider/model pairs from that
stack and scenario ids from the same E2E binary. The primary form only asks for an
optional label, a subject model, and scenarios; URL, judge override, run count,
and technical retries remain under **Advanced options** with safe defaults. Use
**Refresh catalog** after restarting the Harness or changing its URL. The binary
runs only one experiment at a time, streams its log, indexes the resulting
`results.json`, and keeps run metadata and logs under
`target/harness-e2e-local-runs/`.

The dashboard executes itself as an isolated child process, so changing and
restarting the Harness never recompiles the E2E client. `serve` is an alias for
`dashboard`; neither command has a Cargo fallback.

`local-runner.js` owns the browser-side execution controls and is loaded only
when `executions.js` declares `mode: "local"`. The Pages publisher always emits
`mode: "published"`, so the published dashboard keeps using only its static
history and never calls the loopback execution APIs.

The execution label is optional and intentionally descriptive only. The local
dashboard does not inspect or record Harness code changes: restart or modify the
Harness however you want, run another experiment, then select any two execution
rows and open **Compare selected**. The comparison always remains available;
different subjects, run counts, scenario sets, and behavioral contracts are
shown as warnings instead of blocking the comparison.

The listener is deliberately restricted to loopback. Over SSH, forward it with:

```bash
python3 .github/scripts/serve_harness_e2e_dashboard.py \
harness/target/e2e-reactive-fix/results.json \
target/harness-e2e-glm-5.2
ssh -L 4173:127.0.0.1:4173 user@host
```

Imports accumulate in `target/harness-e2e-dashboard-local`. Reimporting the same
report is idempotent. Use `--reset` to start a new local history, or `--host`
and `--port` to change the default `127.0.0.1:4173` listener. The command only
reads existing reports; it does not run E2E scenarios.
Use `--listen 127.0.0.1:PORT` to select another port and `--runs-dir` to select
another local history. The WebSocket URL is accessed on the host by the runner
and does not need a browser-side port forward.

To preview the sample fixtures instead, serve `.github/benchmark-site` directly:

Expand Down
97 changes: 97 additions & 0 deletions .github/benchmark-site/compare.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Compare two Harness E2E executions.">
<meta name="color-scheme" content="dark light">
<title>Compare Harness E2E executions</title>
<link rel="stylesheet" href="./styles.css">
</head>
<body>
<a class="skip-link" href="#main">Skip to comparison</a>
<div class="ambient ambient-one" aria-hidden="true"></div>
<div class="ambient ambient-two" aria-hidden="true"></div>

<header class="topbar">
<a class="brand" href="./index.html" aria-label="Harness E2E dashboard">
<span class="brand-copy"><strong>iii</strong><span>Harness benchmarks</span></span>
</a>
<nav class="topbar-actions" aria-label="Comparison actions">
<a class="button" href="./index.html">← All executions</a>
</nav>
</header>

<main id="main" class="page-shell compare-shell">
<section class="page-heading" aria-labelledby="page-title">
<div>
<div class="eyebrow"><span class="live-dot" aria-hidden="true"></span>Local E2E</div>
<h1 id="page-title">Execution comparison</h1>
<p>Execution B minus execution A. Different scenarios and contracts remain visible.</p>
</div>
</section>

<section id="compare-empty" class="empty-state" hidden>
<div class="empty-icon" aria-hidden="true">⌁</div>
<h2>Select two existing executions</h2>
<p>Return to the execution dashboard and mark any two rows for comparison.</p>
</section>

<div id="compare-content" class="compare-content" hidden>
<section class="panel" aria-labelledby="selected-heading">
<div class="panel-heading">
<div>
<div class="section-kicker">Selection</div>
<h2 id="selected-heading">A versus B</h2>
</div>
</div>
<div id="compare-selection" class="compare-selection-grid"></div>
<ul id="compare-warnings" class="compare-warning-list"></ul>
</section>

<section class="panel executions-panel" aria-labelledby="overall-heading">
<div class="panel-heading">
<div>
<div class="section-kicker">Whole execution</div>
<h2 id="overall-heading">Overall delta</h2>
<p class="trend-description">Positive and negative deltas always mean B minus A.</p>
</div>
</div>
<div id="compare-metrics" class="compare-metric-grid"></div>
</section>

<section class="panel executions-panel" aria-labelledby="scenario-heading">
<div class="panel-heading">
<div>
<div class="section-kicker">Per scenario</div>
<h2 id="scenario-heading">Scenario deltas</h2>
<p class="trend-description">Changed contracts are compared but explicitly marked.</p>
</div>
</div>
<div class="table-wrap">
<table class="compare-table">
<thead>
<tr>
<th scope="col">Scenario</th>
<th scope="col">Execution A</th>
<th scope="col">Execution B</th>
<th scope="col">Δ score</th>
<th scope="col">Δ tokens</th>
<th scope="col">Δ cost</th>
<th scope="col">Δ time</th>
<th scope="col">Contract</th>
</tr>
</thead>
<tbody id="compare-scenarios"></tbody>
</table>
</div>
</section>
</div>
</main>

<script>window.HARNESS_EXECUTIONS = window.HARNESS_EXECUTIONS || null;</script>
<script src="./execution-data.js"></script>
<script src="./executions.js"></script>
<script src="./compare.js"></script>
</body>
</html>
199 changes: 199 additions & 0 deletions .github/benchmark-site/compare.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
(function renderHarnessExecutionComparison() {
"use strict";

const api = window.HarnessExecutionData;
const manifest = window.HARNESS_EXECUTIONS || { executions: [] };
const history = {
executions: (manifest.executions || []).map(api.normalizeExecution),
};
const parameters = new URLSearchParams(window.location.search);
const left = api.findExecution(history, parameters.get("left") || "");
const right = api.findExecution(history, parameters.get("right") || "");
const elements = {
content: document.querySelector("#compare-content"),
empty: document.querySelector("#compare-empty"),
metrics: document.querySelector("#compare-metrics"),
scenarios: document.querySelector("#compare-scenarios"),
selection: document.querySelector("#compare-selection"),
warnings: document.querySelector("#compare-warnings"),
};

function escapeHtml(value) {
return String(value ?? "")
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&#039;");
}

function number(value, digits = 1) {
return typeof value === "number" && Number.isFinite(value)
? new Intl.NumberFormat("en-US", { maximumFractionDigits: digits }).format(value)
: "—";
}

function percent(value) {
return typeof value === "number" ? `${number(value, 1)}%` : "—";
}

function currency(value) {
return typeof value === "number"
? new Intl.NumberFormat("en-US", {
style: "currency",
currency: "USD",
minimumFractionDigits: value < 1 ? 3 : 2,
maximumFractionDigits: value < 1 ? 3 : 2,
}).format(value)
: "—";
}

function duration(value) {
if (typeof value !== "number") return "—";
if (Math.abs(value) < 60) return `${number(value, 1)}s`;
const sign = value < 0 ? "−" : "";
const absolute = Math.abs(value);
return `${sign}${Math.floor(absolute / 60)}m ${String(Math.round(absolute % 60)).padStart(2, "0")}s`;
}
Comment on lines +51 to +57

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Normalize the rounded duration before splitting minutes and seconds.

For example, 119.9 renders as 1m 60s. Round the absolute duration first. Then derive both minute and second fields from that rounded value.

Proposed fix 
   function duration(value) {
     if (typeof value !== "number") return "—";
     if (Math.abs(value) < 60) return `${number(value, 1)}s`;
     const sign = value < 0 ? "−" : "";
-    const absolute = Math.abs(value);
-    return `${sign}${Math.floor(absolute / 60)}m ${String(Math.round(absolute % 60)).padStart(2, "0")}s`;
+    const seconds = Math.round(Math.abs(value));
+    return `${sign}${Math.floor(seconds / 60)}m ${String(seconds % 60).padStart(2, "0")}s`;
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
function duration(value) {
if (typeof value !== "number") return "—";
if (Math.abs(value) < 60) return `${number(value, 1)}s`;
const sign = value < 0 ? "−" : "";
const absolute = Math.abs(value);
return `${sign}${Math.floor(absolute / 60)}m ${String(Math.round(absolute % 60)).padStart(2, "0")}s`;
}
function duration(value) {
if (typeof value !== "number") return "—";
if (Math.abs(value) < 60) return `${number(value, 1)}s`;
const sign = value < 0 ? "−" : "";
const seconds = Math.round(Math.abs(value));
return `${sign}${Math.floor(seconds / 60)}m ${String(seconds % 60).padStart(2, "0")}s`;
}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/benchmark-site/compare.js around lines 51 - 57, Update duration to
round the absolute value before calculating minutes and seconds, then derive
both fields from that rounded duration so values such as 119.9 normalize to 2m
00s instead of producing 60 seconds.


function date(value) {
return value && !Number.isNaN(Date.parse(value))
? new Intl.DateTimeFormat("en-US", {
month: "short",
day: "numeric",
year: "numeric",
hour: "2-digit",
minute: "2-digit",
}).format(new Date(value))
: "Unknown date";
}

function status(value) {
return {
passed: "Passed",
quality_advisory: "Quality advisory",
hard_gate_failed: "Hard gate failed",
technical_failed: "Technical failure",
infra_failed: "Infrastructure failure",
incomplete: "Incomplete",
cancelled: "Cancelled",
}[value] || "Unknown";
}
Comment on lines +71 to +81

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Render the active local execution status.

The local presenter emits status: "running" for active executions. This function maps that status to Unknown in the selection card. Add the running label.

Based on supplied upstream contract, the local presenter emits status: "running" for active executions.

Proposed fix 
       technical_failed: "Technical failure",
       infra_failed: "Infrastructure failure",
       incomplete: "Incomplete",
+      running: "Running",
       cancelled: "Cancelled",
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
function status(value) {
return {
passed: "Passed",
quality_advisory: "Quality advisory",
hard_gate_failed: "Hard gate failed",
technical_failed: "Technical failure",
infra_failed: "Infrastructure failure",
incomplete: "Incomplete",
cancelled: "Cancelled",
}[value] || "Unknown";
}
function status(value) {
return {
passed: "Passed",
quality_advisory: "Quality advisory",
hard_gate_failed: "Hard gate failed",
technical_failed: "Technical failure",
infra_failed: "Infrastructure failure",
incomplete: "Incomplete",
running: "Running",
cancelled: "Cancelled",
}[value] || "Unknown";
}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/benchmark-site/compare.js around lines 71 - 81, Update the status
mapping in status() to include the running execution state with the appropriate
active-status label, so local executions are not rendered as Unknown while in
progress. Preserve all existing status mappings and fallback behavior.


function subjectSummary(execution) {
const subjects = (execution.subjects || []).map(
(subject) => `${subject.provider || "unknown"}/${subject.model || "unknown"}`,
);
return subjects.length === 1 ? subjects[0] : `${subjects.length} subjects`;
}

function selectionCard(execution, side) {
const label = execution.label || date(execution.completed_at);
return `
<article class="compare-selection-card">
<span>Execution ${side}</span>
<h2>${escapeHtml(label)}</h2>
<div class="compare-selection-meta">
<small>${escapeHtml(date(execution.completed_at))}</small>
<small>${escapeHtml(subjectSummary(execution))}</small>
<small>${escapeHtml(status(execution.status))}</small>
<small>${number(execution.requested_runs, 0)} run${execution.requested_runs === 1 ? "" : "s"}</small>
</div>
<a class="text-link" href="./execution.html?id=${encodeURIComponent(execution.id)}">Open diagnostic detail →</a>
</article>`;
}

function signed(value, formatter) {
if (typeof value !== "number") return "—";
if (value === 0) return formatter(0);
return `${value > 0 ? "+" : ""}${formatter(value)}`;
}

function renderMetric(definition, comparison) {
const values = definition.values(comparison);
let deltaClass = "";
if (typeof values.delta === "number" && values.delta !== 0) {
const improved = definition.lowerIsBetter ? values.delta < 0 : values.delta > 0;
deltaClass = improved ? "compare-delta-improved" : "compare-delta-regressed";
}
return `
<article class="compare-metric-card">
<span>${escapeHtml(definition.label)}</span>
<div class="compare-metric-values">
<strong>${definition.format(values.left)}</strong>
<small>→ ${definition.format(values.right)}</small>
</div>
<div class="compare-delta ${deltaClass}">${signed(values.delta, definition.format)} B−A</div>
</article>`;
}

function blockingFailures(execution) {
return ["hard_gate_failures", "technical_failures", "missing_reports"].reduce(
(total, field) => total + Number(execution.totals?.[field] || 0),
0,
);
}

function sideScenario(value) {
if (!value) return '<span class="text-incomplete">Not run</span>';
const score = typeof value.score === "number" ? ` · score ${number(value.score, 1)}` : "";
const passRate =
typeof value.passRate === "number" ? ` · ${number(value.passRate * 100, 1)}% pass` : "";
return `<span class="table-status status-${value.status === "passed" ? "pass" : value.status === "quality_advisory" ? "advisory" : "fail"}">${escapeHtml(status(value.status))}</span>${score}${passRate}`;
}

if (!left || !right || left.id === right.id) {
elements.empty.hidden = false;
return;
}

const comparison = api.compareExecutions(left, right);
elements.content.hidden = false;
elements.selection.innerHTML =
selectionCard(comparison.left, "A") + selectionCard(comparison.right, "B");
elements.warnings.innerHTML = comparison.warnings
.map((warning) => `<li>${escapeHtml(warning)}</li>`)
.join("");
elements.warnings.hidden = comparison.warnings.length === 0;

const metricDefinitions = [
{ label: "Pass rate", format: percent, lowerIsBetter: false, values: (item) => item.totals.scenario_pass_rate },
{ label: "Quality score", format: number, lowerIsBetter: false, values: (item) => item.totals.average_score },
{
label: "Blocking failures",
format: (value) => number(value, 0),
lowerIsBetter: true,
values: (item) => ({
left: blockingFailures(item.left),
right: blockingFailures(item.right),
delta: blockingFailures(item.right) - blockingFailures(item.left),
}),
},
{ label: "Tokens", format: (value) => number(value, 0), lowerIsBetter: true, values: (item) => item.totals.total_tokens },
{ label: "Function calls", format: (value) => number(value, 0), lowerIsBetter: true, values: (item) => item.totals.function_calls },
{ label: "Cost", format: currency, lowerIsBetter: true, values: (item) => item.totals.total_cost_usd },
{ label: "Runtime", format: duration, lowerIsBetter: true, values: (item) => item.totals.wall_time_seconds },
];
elements.metrics.innerHTML = metricDefinitions
.map((definition) => renderMetric(definition, comparison))
.join("");

elements.scenarios.innerHTML = comparison.scenarios
.map((row) => `
<tr>
<th scope="row">
<div class="compare-scenario-name">
<span>${escapeHtml(row.subjectLabel)}</span>
<strong>${escapeHtml(row.scenarioId.replaceAll("_", " "))}</strong>
</div>
</th>
<td>${sideScenario(row.left)}</td>
<td>${sideScenario(row.right)}</td>
<td>${signed(row.deltas.score, (value) => number(value, 1))}</td>
<td>${signed(row.deltas.tokens, (value) => number(value, 0))}</td>
<td>${signed(row.deltas.cost_usd, currency)}</td>
<td>${signed(row.deltas.duration_seconds, duration)}</td>
<td><span class="comparison-contract comparison-contract-${escapeHtml(row.contract)}">${escapeHtml(row.contract)}</span></td>
</tr>`)
.join("");
})();
Loading
Loading