Fix SkippyBench eval reproducibility

.agents/skills/skippy-bench/SKILL.md

@@ -106,6 +106,10 @@ Optional future packs are intentionally not wired yet:
 
 Keep `sync`/`install` opt-in. Do not make normal `just build` or `cargo build`
 download external harnesses, datasets, or Docker images.
+`eval sync` checks out the fetched upstream ref directly, and `eval run` records
+the resolved harness SHA as `harness_commit` in `run.json`. Preserve both
+behaviors so benchmark evidence remains reproducible even when definitions use
+floating upstream refs.
 
 Terminal-Bench should be installed with `uv tool install --python 3.12
 terminal-bench`; Python 3.14 currently breaks the `tb` Typer CLI. Treat Docker
@@ -149,8 +153,9 @@ path, not a Skippy dataset or harness rewrite.
 
 For TTFT/FTTT, use metrics-server correlation rather than harness-only timing.
 `skippy-bench eval run` and `skippy-bench chat-corpus` create/finalize a
-metrics-server run and fail if the metrics report cannot be exported. The
-target endpoint must be emitting OTLP for the same run id. Debug telemetry is
-required for per-token spans such as `stage.openai_decode_token`; without it,
-the JSON report will still include a telemetry block explaining why TTFT/FTTT
-was unavailable.
+metrics-server run. `eval run` keeps harness success independent from a
+finalization/export failure and records telemetry as unavailable; `chat-corpus`
+still fails when its metrics report cannot be exported. The target endpoint
+must be emitting OTLP for the same run id. Debug telemetry is required for
+per-token spans such as `stage.openai_decode_token`; without it, the JSON report
+will still include a telemetry block explaining why TTFT/FTTT was unavailable.

crates/skippy-bench/README.md

@@ -143,6 +143,10 @@ fails before launching the native harness.
 `~/.cache/mesh-llm/skippy-bench/harnesses/` by default. Use `--cache-root` to
 override that location. Use `--dry-run` with `sync` or `run` to inspect the
 commands without cloning, pulling Docker images, or launching a benchmark.
+For an existing harness clone, `sync` fetches the configured upstream ref and
+checks out the fetched commit directly, so a stale local branch cannot leave the
+cache behind upstream. Each run records that resolved commit in
+`run.json` as `harness_commit` for reproducible benchmark evidence.
 Before launching native harness traffic, `eval run` enforces the same required
 tool checks as `eval doctor`, including Docker container-start readiness for
 Docker-backed evals.
@@ -194,10 +198,13 @@ local-model path. Set `SWE_BENCH_PRO_USE_LOCAL_DOCKER=0` when running the full
 harness in a different environment such as Modal.
 
 Every `eval run` writes `run.json` under the run directory with command status,
-raw artifact paths, wall-clock duration, and normalized metrics where the
-harness exposes them. `speed-bench` records request counts, latency,
+the resolved harness commit, raw artifact paths, wall-clock duration, and
+normalized metrics where the harness exposes them. `speed-bench` records request counts, latency,
 prompt/completion/total token counts, prompt and completion tok/s, and draft
-acceptance rate when the server returns llama.cpp-compatible `timings`.
+acceptance rate when the server returns llama.cpp-compatible `timings`. Because
+the upstream SPEED-Bench script does not expose an authorization argument,
+SkippyBench launches it through a small adapter that adds the bearer token from
+`--api-key` without modifying the upstream harness.
 SWE-Bench Pro records OpenAI usage tokens and client-side tok/s when the
 upstream flow produces them.
 Terminal-Bench records pass rate, resolved/unresolved task counts, token totals
@@ -214,9 +221,11 @@ run id. SkippyBench finalizes and fetches
 adds a `telemetry` block to `run.json`. When the target emits debug telemetry,
 SkippyBench derives TTFT/FTTT from the first request span to the first
 `stage.openai_decode_token` span, plus request and generation latency
-aggregates. If the target endpoint is not emitting the requested run id, or if
-debug token spans are disabled, the telemetry block records that status rather
-than filling misleading values.
+aggregates. A finalization or report-fetch failure marks telemetry unavailable
+without changing the native harness result in `report.success`. If the target
+endpoint is not emitting the requested run id, or if debug token spans are
+disabled, the telemetry block records that status rather than filling
+misleading values.
 
 Optional packs intentionally not wired yet: