Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions .github/benchmark-site/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,24 @@ The execution index retains 100 workflow attempts. The latest 30 also retain the
complete structured `results.json` content, including prompts, transcripts,
session ids, gates, criteria, failures, retries, usage, and traces. The UI loads
those reports only on the detail page and renders transcript-heavy sections only
when expanded. Diagnostic logs, stack files, and credentials remain in
access-controlled Actions artifacts.
when expanded. Each run opens its transcript in a read-only dialog patterned
after the Harness chat, with message and error filters, paired function calls
and results, and recovered errors expanded by default. Diagnostic logs, stack
files, and credentials remain in access-controlled Actions artifacts.

Each full execution summary also carries compact per-scenario averages for
tokens, wall time, cost, function calls, function-call errors, sessions, and
turns. Tokens mean input plus output; cache-read tokens are already represented
in input usage and are not added again. The execution table also exposes exact
total tokens and function calls for every retained full report.

Efficiency is the primary overview. Its cards show the current operational suite
totals, while deltas use only successful scenarios with the same subject,
scenario id, and behavioral contract fingerprint. New and changed scenarios
collect five comparable executions before receiving a trend verdict. Removed
scenarios remain visible as historical rows and never count as an efficiency
gain. Contract changes start a new baseline instead of joining incompatible data.
Select any scenario in the efficiency table to compare that scenario execution
by execution. The modal switches between cost, tokens, duration, function calls,
and function errors, marks contract boundaries, and links each point to its full
execution details.
Loading
Loading