[justinchuby]

The defect

acc0_w16_worker_split.py's pre-registered rule says "Compare w=16 against w=8". verdict() and report() each computed narrow, wide = min(widths), max(widths).

I ran it with --widths 2,8,16 to get an extra descriptive column. That silently re-based every verdict and the entire Amdahl calibration onto w=2. Nothing in the output said so — the header line, the threshold banner and the verdict text were byte-identical in shape to a correct run.

What it changed

Re-scored from the identical archived datasets, 7 trusted launches per arm, no new host time:

	published (w=2 baseline)	pre-registered (w=8 baseline)
blocktime=0 verdict	DISPATCH-BOUND	DISPATCH-BOUND (unchanged)
blocktime=500 (shipped) verdict	DISPATCH-BOUND + IMBALANCE-BOUND	IMBALANCE-BOUND only
serial excess over Amdahl, bt=0	+0.086	−0.009
serial excess over Amdahl, bt=500	+0.082	−0.064
recoverable-in-principle, bt=500	0.146 of the window	0.064 of the window

The sign of the headline term flips. I had named that +0.086 "serial excess" as this lane's largest recoverable term and its highest-value remaining target. It does not exist. At w=8 → w=16 the dispatcher's serial share is below what Amdahl predicts. The only term the rule attributes at the shipped blocktime is the straggler wait, 0.064 of the window — which redirects the lane at #2017's lane-versus-chunk question rather than at the dispatcher.

DISPATCH-BOUND now misses at the shipped blocktime by 0.009 on a 0.10 threshold. That is a miss, not a near-hit to argue around.

WAKE-BOUND fires on neither baseline at either blocktime, so the one genuinely surprising result survives intact: my own futex-park hypothesis stays falsified at sessions=1, and the shipped 500 µs window's cost — full parking, 345.9 parks per worker — is 2.2% of the window in wake latency.

Why w=2 is wrong mechanically, not merely by the rule

The Amdahl calibration takes the narrow width's work_frac as "serial/parallel with no defect" and divides the parallel part by the width ratio. It is only as good as its baseline, and it is sharply sensitive to it.

At w=2 the pool spawns one worker — reserve_single_group_headroom keeps a CPU for the dispatcher and the dispatcher owns a shard inline. The mean worker is busy 0.997 of the window because there is barely a barrier to be serial at: one worker cannot wait for a straggler, and the dispatcher's shard runs concurrently with it rather than serially to it. That extrapolates to a predicted serial term of 0.021 at w=16. Nothing real clears that bar, so anything observed reads as "excess".

At w=8 the mean worker is busy 0.909 — a seven-participant barrier with a measurable dispatch gap. Predicts 0.167; observed 0.103.

Same dataset, opposite conclusion. This is not a threshold that wants tuning; w=8 was pre-registered before the first run for exactly this reason.

The fix

RULE_NARROW = 8 / RULE_WIDE = 16 are now constants, documented as clauses of the rule:

• the pair is no longer derived from --widths;
• extra widths are still measured and printed, under an explicit descriptive only, NOT used by the rule or the decomposition banner;
• a run missing either member prints REPORT NOTHING (the pre-registered rule compares w=16 against w=8; this run measured 2,16) instead of substituting what it has;
• every run prints # verdict pair: w8 vs w16 (pre-registered; not taken from --widths), so the provenance is in the output rather than in the reader's memory.

--binary is no longer required for --replay. It was demanded and never used — and awkward re-scoring is how a re-based verdict survives.

Tests, and they are proved against the defect

--self-test runs on a synthetic dataset built so the w=8 rise is +0.09 (below threshold) and the w=2 rise is +0.17 (above). DISPATCH must not fire under --widths 8,16, 2,8,16, 16,8 or 2,16,8,4; IMBALANCE must; the scored pair must be exactly {8, 16}. It also covers both REPORT NOTHING paths and the n_trusted floor.

Mutation-proved by restoring the original min/max:

AssertionError: (2, 8, 16): scored on [2, 16], not the pre-registered pair

It runs in CI. hostlock.yml already path-filters on crates/onnx-runtime-ep-cpu/benches/** and already owns driver conformance for that directory, so the new step lives there. It discovers harnesses advertising --self-test by behaviour rather than by a list — the same argument that file already makes about its own path filter, that the harness which most needs the check is the one that does not exist yet. The self-tests are arithmetic on synthetic records: no binary, no host load, no lock, safe on a shared runner.

Local: --self-test passes; both archived datasets replay to the corrected verdicts; the discovery loop runs clean; YAML parses.

Record

docs/benchmarks/2026-08-26-acc0-w16-baseline-correction.md carries the corrected decomposition for both arms, the mechanical argument against w=2, and the corrected priority list.

Method note

This is the third defect of one shape this week: a harness selecting an arm or a baseline by a parameter nobody re-checked — PIN-ON, a string the code never emits (#2177); a OnceLock that latches the first arm (#1736); and now a rule baseline taken from argv. In every case the output looked exactly like a valid measurement. The general defence is the one applied here: anything the pre-registered rule names must be pinned in code and printed in the output, and the harness must have a self-test that fails against the defect rather than exercising the happy path.

I will post the corrections to #2071 and #1801 whether or not this merges.

Refs #2071 #1801 #2017

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.71%. Comparing base (bb08c5a) to head (57c24ad).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2245      +/-   ##
==========================================
+ Coverage   80.64%   80.71%   +0.06%     
==========================================
  Files         432      433       +1     
  Lines      220114   221629    +1515     
  Branches   220114   221629    +1515     
==========================================
+ Hits       177512   178883    +1371     
- Misses      36717    36821     +104     
- Partials     5885     5925      +40     

Flag	Coverage Δ
cli-ort-linux	72.51% <ø> (ø)
cli-ort-windows	72.10% <ø> (+0.09%)	⬆️
mlas	85.90% <ø> (ø)
offline	80.83% <ø> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 17 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justinchuby]

Independent review (Opus), and what it changed

One real find, which I have fixed, plus one hardening. Seven other lines of enquiry were traced and came back clean; I am recording those too, because "I checked and it was fine" is evidence and an unrecorded check gets repeated.

The find — SHOULD-FIX, and it was the important half

The self-test guarded verdict() only. But the defect lived in report() as well, and report() is the sole caller of barrier_decomposition() — the Amdahl calibration whose sign flipped from +0.086 to −0.009. So a future edit that re-based only report() would print a correct VERDICT: line above a wrong serial excess, and the CI step this PR adds would stay green.

That is the same failure class the PR is about, one level up: I tested the half that produces the label and not the half that produces the number.

Fixed in 57c24ad90. The self-test now drives the whole report through contextlib.redirect_stdout and asserts the pinned pair reached the calibration — the decomposition header must read w8/w16, and the Amdahl line must divide the parallel time by ratio 2, not the 8 a w=2 baseline gives.

Mutation-proved. Reverting only report()'s pair:

  File "acc0_w16_worker_split.py", line 547, in self_test
    report(rows, widths)
  File "acc0_w16_worker_split.py", line 379, in report
    print(f"{k:>18} {vals[narrow]:>10.3f} {vals[wide]:>10.3f} "
KeyError: 2

Before this commit that mutation passed, because the test never called report().

There is also a structural property worth naming: verdict() returns its median map keyed only by the pinned pair, so report() now cannot decompose on any other width without failing loudly rather than printing a plausible wrong number.

The hardening — CI discovery

Discovery matched the bare literal "--self-test", which is quote-style-dependent in one direction and over-broad in the other: a harness registering the flag with single quotes would be silently skipped (the "assigned, never executed" shape this workflow's own header rails against), while a file merely mentioning the flag in prose would be selected and then fail the job for not accepting it. Now matched on the argparse registration, either quote style. Verified by extracting the step's script out of the workflow YAML with a parser and running it verbatim, rather than re-typing it.

No current false positive existed — the reviewer enumerated every .py in both directories and only this harness matches — so this is prophylactic.

Traced and clean

• Pinning is complete. Every remaining min(/max( in the file is over per-worker lists, a modal-holder count, a clamp, or a sorted() inside an assertion message. No pair, median, decomposition or printed statistic derives from --widths ordering. one_launch's order = widths if launch % 2 == 0 else reversed(widths) is launch-parity anti-bias, not a comparison pair.
• The fixture genuinely straddles the threshold, recomputed by hand from the constants: pinned rise 0.173 − 0.082 = 0.091 < 0.10 so DISPATCH must not fire; re-based rise 0.173 − 0.003 = 0.170 > 0.10 so it would. The test is not vacuous.
• Every number in the doc and both commit messages reproduces exactly from a replay of the two archived datasets, including the "published" w=2 column re-derived by simulating the defect.
• The doc's Amdahl argument is arithmetically sound against barrier_decomposition's own ratio = wide/narrow. One caveat the reviewer caught and I am recording rather than silently keeping: the doc writes "0.997 extrapolates to 0.021", but 0.997 alone recomputes to 0.024 — the 0.021 comes from the full-precision median 0.99730. Both figures are the tool's own output; the rounding in the prose is display-only.
• set -euo pipefail interactions are safe: the missing nullglob is covered by [ -f "$f" ] || continue, and found=$((found + 1)) is an assignment, not the ((found++)) arithmetic-command gotcha.
• Path filters do trigger this job on both pull_request and push to main — the PR touches crates/onnx-runtime-ep-cpu/benches/** and .github/workflows/hostlock.yml.
• report()'s narrow, wide = pair cannot see a None: it is behind if not m: return, and m is non-empty only when rule_pair returned a pair.
• Only stdlib imports; the sibling acc0_gap_matrix has no import-time side effects, so the CI self-test runs cleanly from the repo root with no lock and no host load.

Marking ready for review. This is not one of the required checks and must not become one; Host lock will run on this PR because of the path filter.