bench: add a decode-shaped CPU benchmark harness (gap, warm-up, park/wake)

[justinchuby]

Revived after a long stall. The branch was 469 commits behind main; this brings it current, and the merge was resolved so that the branch is now purely additive — 2260 insertions, 0 deletions against main.

What this is, and what it is not

A model-level decode-gap harness: it runs a real ONNX model through the native session with a configurable inter-token gap distribution, and reports the steady-state wall/CPU/RSS/context-switch/pool-counter picture around it.

It is not a replacement for crates/onnx-runtime-ep-cpu/benches/decode_gap_park_ab.rs, which is already on main. That one is a synthetic harness over the decode SPMD pool. This one drives task_runtime::pool through a real graph. Those are two different pools with different park/spin policies, and — as decode_gap_park_ab.rs's own docs say — a conclusion from one must not be quoted for the other. Saying so here because the two names are confusingly similar and I expect to be the person who confuses them.

The merge decision worth flagging

My original branch extracted ~520 lines out of bench_generic.rs into a shared module. Over 469 commits, bench_generic.rs had moved on considerably. I took main's version wholesale rather than re-applying my refactor.

The consequence is a real cost and I would rather state it than hide it: model_io.rs duplicates some helpers that also exist in bench_generic.rs. I chose duplication over refactoring a file that several other lanes are actively editing. If the duplication becomes a maintenance problem the de-duplication is a separate, reviewable change; folding it into a revival PR would have made this one impossible to review and would have conflicted with everyone.

Exactly one compile error survived 469 commits of drift: PoolCounters gained straggler_waits and straggler_yields. That is a good sign for the interface, not for my branch hygiene.

The finding that came out of the smoke run, and the change it forced

The first end-to-end run reported:

cpu:           19.070 s over steady window  (7.85 cpu-s per wall-s)
native pool:   0.00 dispatches/iter  0.00 parks/iter  0.00 spin-hits/iter

Eight cores of work, and a row of zeroes from the instrument that is supposed to explain it. That is precisely what a dead counter looks like, so I chased it as one before pushing.

It was not dead. An fp32 MatMul fans out on rayon, not on task_runtime, so the zero was true — the harness was faithfully reporting a pool the model never touched.

The defect is that I could not tell those apart from the output. The pool counters answer "how hard did task_runtime work"; they cannot answer "did this model use task_runtime at all"; and the two questions produce byte-identical output. Only one of them means the harness is broken.

So the run now attributes its own routes through dispatch_ledger, and annotates a zero-dispatch row with which case produced it:

native pool:   0.00 dispatches/iter  0.02 parks/iter  0.00 spin-hits/iter  0 slot-exhausted
ATTRIBUTION:   this model never dispatched to task_runtime -- the pool row above is a
               true zero, not a dead counter. Routes below say what ran instead.
route:         MatMulF32 -> Native (f32, 8 calls, up to 8 threads)

Recording is enabled for a single attribution inference and never during the timed window, because record_with builds an Observation per dispatch.

Both branches are proved, not asserted:

condition	output
ledger on, 8-layer fixture	MatMulF32 -> Native (f32, 8 calls, up to 8 threads)
ledger on, 4-layer fixture	MatMulF32 -> Native (f32, 4 calls, up to 2 threads)
enable() commented out	zero dispatches AND zero routes recorded ... this is a dead instrument, not a measurement -- do not quote the pool row

The third row is the whole point. Without it, the dead case and the true-zero case print the same thing, and I would have pushed a harness that cannot distinguish them.

A side benefit that matters for this campaign specifically: the route line makes every run state on the record whether it was pure native, MLAS, or an ORT fallback. That was previously an assumption carried in the operator's head rather than a column in the output.

Other properties it already had, and which the smoke run exercised

• A null A/A arm. --arm null runs the same configuration twice and prints null_b_over_a with the explicit note that any A/B ratio inside this band is noise. Measured 0.968x on a small fixture and 0.992x on a large one.
• It refuses to report a steady state it did not observe. At 32 iterations: steady window: NOT REACHED (series never settled) plus WARNING: no steady state ... Every number above describes the warm-up transient. Raise --iters. At 512 iterations it detects the window and reports from it.
• Parity against ORT before the timed loop, with the ORT session built, used and dropped before timing starts — a co-resident ORT session spin-waits long after its last op and depresses a native arm measured beside it.

Validation

• cargo test -p onnx-genai-bench --features bench-native --lib — 32 passed, 0 failed
• cargo clippy -p onnx-genai-bench --features bench-native --bin bench_decode_gap -- -D warnings — clean
• cargo fmt --all -- --check — clean
• End-to-end runs under scripts/hostlock.sh, both fixtures, both ledger states

No timing claim is made from any of these runs. They are correctness and instrument-liveness checks; the numbers quoted above are shown only to demonstrate that the instrument distinguishes cases, and the fixtures were synthetic MatMul chains, not a model anyone should benchmark.

Known, not mine

cargo clippy -p onnx-genai-bench --features bench-native --bin compare -- -D warnings fails on collapsible_if at compare.rs:845. Verified identical on an unmodified tree — that feature combination is evidently not gated in CI. Not introduced here.

[justinchuby]

Pushed a correction to my own numbers, before this merges rather than after.

The 4→8 budget scaling stall reported in §41.6 does not reproduce, and I have retracted it.

Re-measuring budget 4 with the identical binary and flags:

session	p50 samples (ms)
A	1.858, 1.847, 1.861, 1.871
B	2.766, 2.761, 2.756, 3.992

Four samples inside 1% within a session — tighter than the null control band — and 1.48x apart across sessions. Notably the steady-state detector does not save you here: each reading is genuinely steady, they are just steady at different values.

The cause is the budget's own affinity confinement. select_budget_cpus is deterministic — budget 4 always confines to CPUs [0, 2, 4, 6] — and sampling /proc/stat during the slow session showed those exact cores 22-34% busy with co-tenant work at load average 8-21. A budgeted process cannot migrate away from a busy core, because migration is precisely what the budget gave up.

So the original comparison put a quiet-session budget-4 reading against a busy-session budget-8 reading and read the difference as parallel-decomposition behaviour. Measured within one session, 4→8 is 1.57x — sublinear, unremarkable, not a stall.

Three things now written down in a new §41.7 that were not before:

1. A narrower budget is more volatile on a shared host, not less — the opposite of the natural intuition. Budget 4 gives up 28 of 32 logical CPUs and keeps 4 it must share, having removed the scheduler's usual remedy. Budget 16 spans more cores and moved only 0.910 → 0.889 between the same two sessions.
2. Budgeted processes collide deterministically. Every one picks the same low-numbered physical cores rather than spreading. Correct and intended single-tenant; close to worst-case multi-tenant — and this campaign's own benchmark host is multi-tenant.
3. Budget sweeps are only interpretable if every cell is measured in one session, interleaved. Across sessions you are comparing co-tenant weather.

I also tried to isolate it by pinning to apparently-idle CPUs (taskset -c 11,17,20,31) and got worse results — 1.866 then 3.296 — because 11, 17 and 31 are SMT siblings of busy physical cores 10, 16 and 30. Picking logical CPUs without regard to SMT pairing hands you cores you do not own, which is the lesson #1232 already encoded in select_budget_cpus and which I re-learned by hand.

I have kept the original table exactly as measured rather than quietly editing the numbers, because how the figure dissolved is more useful than the figure would have been.

This does not affect the harness itself, the park-threshold result, the thread census, or the concurrency finding — the concurrency matrix was measured at a single budget within one session, which is the shape §41.7 now says is required.

[justinchuby]

Second correction, same session, same reason — I kept measuring and the number did not survive.

Retracted: the claimed nested-dispatch regression ("111 of 915 dispatches declined at 16 dispatchers, where #1377 measured zero").

That was a misreading on my part. nested_dispatch_slot_pressure issues roughly 25-30 dispatches in total, so 915 was never a figure it could produce. The full output makes the shape obvious:

 dispatchers        wall    dispatches    declined     inline
           1    1866.5us             1           0          0
           2     210.3us             2           0         13
           4     442.5us             4           0         16
           8     797.3us             8           0         74
          16    1577.9us            14           2        104
totals: 2 of 29 dispatches declined for want of a slot

Re-measured over four runs it declines 0, 1, 2 and 6 of ~25-30, with every decline in the 16-dispatcher row and none below it. #1377's zero sits inside that range rather than being contradicted by it.

The variation has the same cause as the budget-4 swing in §41.7: when co-tenants hold cores, workers free their slots later, so more nested dispatches take the inline fallback. That fallback is the designed behaviour — this is the mechanism working, not a regression.

Worth naming the common thread in both retractions, because it is the real lesson of this phase. The harness I built here fixed within-run honesty — steady-state detection, real park/wake counters, thread attribution — and it did that well enough that individual readings are now tight to within 1%. It does nothing about across-run honesty on a shared host, and the tightness is actively misleading there: four samples inside 1% look authoritative right up until the next session produces four different samples inside 1%.

Both of the claims I retracted today were of that shape. Neither would have been caught by a better detector. They were caught by re-measuring something I had already written down.

Ledger and PR description updated. The findings that stand — the 200-500us park threshold (independently confirmed by the micro-benchmark), the thread census resolving the ~2x-budget question, zero ORT threads, and the concurrency p90 collapse — were all measured within a single session at a single budget, which is the shape §41.7 now says is required.

[justinchuby]

Item 2 outcome: a real defect found, a fix built and measured, and not shipped. Recorded in a new §41.11.

The adaptive spin window is structurally pinned at MAX_SPIN. worker_loop has two doubling sites and one halving site:

• ran > 0 after a drain doubles it (~line 367)
• a spin hit doubles it (~397)
• a park halves it (~408)

The sequence after any park is: halve (500 → 250us), then the futex wake returns because work arrived, the worker drains it, ran > 0, double (250 → 500us). Net zero, every time. In any workload that does work at all the window never leaves MAX_SPIN. The comment at the halving site — "shrink it so a mostly-idle process converges on parking rather than on burning a core" — describes an intent that the very next statement undoes. It also explains why §41.4's idle CPU saturates at ~6.5 CPU-ms ≈ 16 workers × MAX_SPIN.

Running work is not evidence a spin window was well-sized; only a spin hit is. So the ran > 0 doubling is the wrong one, and removing it should let a large-gap workload converge to MIN_SPIN in five iterations.

It does not. Two binaries interleaved in one session (the rule I just wrote in §41.7), sleep gaps, budget 16:

gap	arm	p50 (ms)	cpu/wall	parks/iter
100us	base	0.844 / 0.846	12.61 / 12.84	4.16 / 2.04
100us	fix	0.877 / 0.899	12.91 / 12.68	3.41 / 7.19
2000us	base	0.974 / 1.050	5.78 / 5.45	16.92 / 19.04
2000us	fix	0.942 / 0.911	5.60 / 5.06	17.16 / 19.11

5% slower at 100us, 7% faster at 2ms, both inside the 0.81-1.24 null band, and CPU moved ~3% where I predicted ~44%.

The prediction failed for a checkable reason, which is the part worth keeping: parks/iter is ~17 in both arms at a 2ms gap. The workers park either way. The window only governs how long they spin first, and that is empirically worth ~0.4 CPU-ms per iteration rather than the ~8ms that 16 × MAX_SPIN implies — the spin loop exits early on the epoch check far more often than it runs to the window's end.

So: defect real, inference from it wrong, change reverted. I would rather log a bounded negative result than ship a scheduling change whose entire effect is inside the noise band.

The attempt did establish where idle CPU actually lives, from the per-thread census at a 2ms gap:

pool	threads	total CPU over the run
task runtime (nxrt-task-*)	15	~3.45 s
decode SPMD (onnx-genai-deco)	16	0.28 s
prefill Rayon (nxgn-prefill-*) + main	17	0.05 s

The task runtime is 12x the decode pool. Future idle-CPU work belongs there — and the prefill pool, the one that looked alarming in §41.5 for being 16 wide and anonymous, is 1.5% of the total. Worth naming (#1398), not worth making lazy.

[justinchuby]

Phase 22 (measurement only): where the single-session CPU actually goes

Single-session utilisation at budget 16 is ~9.2–9.9 CPU-s per wall-s out of 16.
This run attributes that number by phase and thread. Host: AMD EPYC 9V74,
1 socket, 1 NUMA node, 16 physical cores (32 SMT), two L3 domains (CPUs 0–15
and 16–31, 32 MiB each). Model gemm_nbits_llama3_8b_mlp_t1.onnx, 35 MB of int4
weights, gap 100 µs, 400 iters, --native-threads 16, pure native, no fallback.

0. Retraction first

In the previous update I reported that forcing the flat fan-out onto Rayon was
~1.46x faster than the task runtime at budget 16 (3 of 4 reps). That result
was wrong and I am withdrawing it. It was host weather, not routing.

The A/A null control (identical binaries, separate processes, interleaved) sizes
the band at 0.85–1.07x on p50 and 9.48–10.11 on cpu-per-wall. Re-running
the A/B with alternating arm order and discarding reps where the host collapsed
(cpu/wall fell to 2–4, load average 7.6–10.3):

rep	order	task runtime p50	forced Rayon p50	ratio
1	RT first	1.820 ms	1.900 ms	RT 1.04x
2	RAY first	1.756 ms	1.905 ms	RT 1.08x
3	RT first	1.756 ms	1.949 ms	RT 1.11x
8	RAY first	2.125 ms	1.984 ms	RAY 1.07x

Every clean rep is inside the null band. Routing is latency-neutral for this
shape. The original 1.46x came from a run where the second arm happened to land
in a quiet window. The only reason this was caught is the null control — the same
control discipline that the "avoid tuning to a paired-ORT artifact" instruction
exists to enforce. Treat the earlier number as retracted.

1. Routing is neutral on latency but not on cost

Same workload, two binaries differing only in MIN_ROUTED_FAN_OUT_WIDTH (16 vs 17):

arm	decode-Rayon CPU	nxrt-task CPU	total CPU	p50	vol / invol ctxsw per iter
task runtime	800 ms	6830 ms	7.84 s	2.055 ms	19.1 / 0.5
forced Rayon	8080 ms	0.0 ms	8.27 s	1.979 ms	47.7 / 43.3

Two things worth keeping:

• An idle task-runtime pool costs exactly 0.0 ms of CPU. All 15 workers park
  and stay parked. There is no idle-spin leak in the task runtime, which rules
  out the "unused pool burns CPU" hypothesis I had been carrying.
• The task runtime does the same work for 5% less CPU and 87x fewer
  involuntary context switches. #1363's routing is defensible on cost even
  though it does not move p50.

2. Thread inventory: 47 workers for a 16-core budget

--census-phases attributes every thread to the phase that created it:

phase	created	identity
native-session-load	+16 unnamed	global Rayon pool (build_global)
timed-session-warm	+16 onnx-genai-decode-*	pinned decode pool (build_decode_pool, matmul_nbits.rs:3999)
timed-session-warm	+15 nxrt-task-*	task runtime (+ dispatcher = 16 lanes)

That is three pools, each sized to the budget — 47 worker threads on 16 cores,
which is the ~2x-budget unnamed-thread question from an earlier phase, now closed.
The 16 unnamed threads are the global Rayon pool; Rayon does not name
build_global workers, so they inherit the process name. #1398 names them
nxgn-prefill-*, which makes this visible in any future census.

ONNX_GENAI_PROFILE_OPS=1 shows one MatMulNBits per iteration at 99.95% of
the forward pass, and the counters show 1.01 dispatches/iter. So the task
runtime performs 100% of the model work, and the decode pool's 800 ms is
not model work: 15 of its 16 workers block as a pass-through layer while one
becomes the dispatcher.

3. Utilisation decomposition at budget 16

Of the 16 CPUs available (9.15 cpu-s per wall-s observed):

component	cpu/wall	share of the 16
nxrt-task workers (useful)	7.99	50%
decode-pool pass-through (idle)	0.94	6%
global Rayon + main (idle)	0.21	1%
unused	6.86	43%

Per-iteration: each task worker is busy ~1.14 ms of a 2.055 ms iteration — a
56% duty cycle. The rest is fork-join, not model work: 15.1 parks/iter
(every worker parks every iteration), and 99.7% of dispatches wait on a
straggler with 5.75 yields/iter. Per-worker CPU spreads 430–500 ms, so the
slowest worker does ~16% more than the fastest.

So the headline "9.4 of 16" is really 8.0 useful + 1.15 overhead + 6.9 idle,
and the dominant single loss is the fork-join straggler tail, not park cost.

4. Hypotheses tested and rejected

• L3/CCD locality. The 8→16 wall (b=8 p50 2.10 ms vs b=16 2.05 ms — doubling
  cores buys ~2%) looked like it might be the CCD boundary, since budget 8 lands
  entirely inside L3 domain 0 while budget 16 spans both, and the weights (35 MB)
  exceed one 32 MiB L3. Tested directly at equal core count: 8 cores inside one
  L3 (0,2,4,6,8,10,12,14) vs 8 cores split across both
  (0,2,4,6,16,18,20,22), ONNX_GENAI_CPU_DECODE_AFFINITY=off + taskset.
  Clean reps were identical (3.657 vs 3.619; 2.219 vs 2.191). Rejected.
• Idle pool spin in the task runtime. Rejected — 0.0 ms, see above.
• Raw DRAM bandwidth. 35 MB per 2.05 ms is only ~17 GB/s, ~1.07 GB/s per
  core, well under what a single core can pull. Not obviously saturated, but see
  the caveat below.

5. What is still open

The single MatMulNBits GEMV does not scale past ~8 workers, and I have not yet
proven why. The straggler tail (99.7% of dispatches) and the 56% duty cycle are
the leading candidates, ahead of memory bandwidth. The next step is finer or
dynamic partitioning of the output rows to cut the tail, measured with the
straggler counters from #1411.

Caveat on every number here: the host is shared and was under load average
7.6–10.3 for this session, with several reps collapsing to 2–4 cpu/wall. All
comparisons are interleaved, alternate arm order, and are reported against a
measured null band; single-rep differences inside 0.85–1.07x mean nothing. The
b=8 vs b=16 comparison in particular deserves a re-run on a quiet host before
anything is built on it.

No code change in this phase — measurement only.

[justinchuby]

Caution

RETRACTED — 2026-08-19. The central claim below, that SMT siblings give a 1.26x single-session p50 win, is wrong, and so is the "9.4 of 16" baseline it is built on. Both were measured on a loaded host.

Re-measured on a genuinely quiet host (load average 1.21), interleaved with alternating arm order:

arm	p50	cpu/wall
physical-core (budget 16)	1.177 ms	12.3-12.6
SMT (32 lanes, affinity off)	1.219 ms	22.0-23.2

SMT is 1.03x slower, at 1.8x the CPU. The original 1.26x was an artifact of co-tenancy: when other jobs already occupy the sibling lanes, physical-core confinement buys you nothing (the sibling steals the core regardless), so taking all 32 CPUs simply captured more of a contended machine. That is a property of the host that day, not of the scheduler.

The "9.4 of 16" utilization figure is likewise the contended state. A quiet host reaches 12.3-12.8.

No SMT policy should be implemented on the basis of this comment. See the phase-24 comment below for the replacement analysis, which also explains the bistable fast/slow regime that made these numbers unstable in the first place.

Original text preserved unedited below for the record.

---

Phase 22b: the single-session "9.4 of 16" is memory latency, not the scheduler

Follow-up to the phase-22 attribution above. This resolves why one session cannot
fill the 16-core budget. Same host and model as before (EPYC 9V74, 16 physical
cores, gemm_nbits_llama3_8b_mlp_t1.onnx, 35 MB int4 weights, gap 100 µs, pure
native, no fallback/MLAS).

1. The fan-out itself scales almost perfectly to width 8

ONNX_GENAI_CPU_TASK_THREADS sets the task-runtime width independently of the
decode budget, so the budget (affinity, pool sizes, routing) can be held at 16
while only the fan-out width moves. Routing stays on the task runtime at every
width (calls=1, so the hot-fan-out branch is not taken). One clean session:

task width	p50	speedup	implied GB/s
1	16.175 ms	1.00x	2.2
2	8.132 ms	1.99x	4.3
4	4.133 ms	3.91x	8.5
8	2.132 ms	7.59x	16.4
16	1.945 ms	8.32x	18.0

Fitting wall = S + P/N on widths 1 and 2 gives S = 0.089 ms — a 0.55% serial
fraction — and that fit then predicts the measured points: w=4 → 4.110
(measured 4.133), w=8 → 2.100 (measured 2.132). At w=16 it predicts 1.094 but
measures 1.945.

So scaling is 95% efficient through width 8 and collapses to 52% at 16.
Serial preprocessing/postprocessing, dependency barriers and task grain are all
excluded as the cause: a 0.55% serial fraction cannot produce this.

2. It is not a bandwidth ceiling

Per-lane throughput is a flat ~2.05 GB/s (8.5/4, 16.4/8), and the total pins near
18 GB/s, which looks like a bandwidth wall. It is not — running two concurrent
sessions breaks straight through it:

config	total lanes	throughput	cpu/wall	GB/s
1 session x w8	8	469/s	~7.0	16.4
1 session x w16	16	518/s	9.64	18.1
2 sessions x w8	16	544/s	9.03	19.0
2 sessions x w16	32	750/s	14.55	26.3

26.3 GB/s comfortably exceeds the apparent 18 GB/s "ceiling", and utilisation
reaches 91% of the 16-core budget. A bandwidth-saturated workload cannot do
that. Note also that 2x w8 (16 lanes) is no better than 1x w16 (16 lanes) — what
buys throughput is more lanes than cores, not how they are grouped.

3. The mechanism is memory-latency stalls, and SMT recovers them

An M=1 GEMV streams 35 MB of int4 weights with almost no arithmetic per byte, so
each lane spends most of its time waiting on global loads — the same
latency-bound/Long-Scoreboard regime the profiling skill describes for decode
GEMVs. A lane leaves its core stalled roughly 44% of the time (measured directly
in the census above: workers are busy 1.14 ms of a 2.055 ms iteration, a 56% duty
cycle). Only a second runnable thread per core recovers those cycles.

Allowing the 16 SMT siblings confirms it. Four interleaved reps, alternating arm
order, single session:

rep	budget16/width16	budget32/width32	ratio
1	2.726 ms (8.94 cpu/wall)	1.959 ms (15.10)	1.39x
2	1.940 ms (9.33)	1.627 ms (16.93)	1.19x
3	1.977 ms (9.57)	1.529 ms (18.25)	1.29x
4	1.882 ms (10.21)	1.540 ms (18.03)	1.22x

Median 1.26x, and every rep is outside the measured null band (0.85–1.07x) in
the same direction, holding even at load average 12. Intermediate widths move
monotonically (b16/w24 → 1.93 ms, b16/w32 → 1.73 ms), so this is oversubscription
recovering stall cycles, not a threshold artifact.

This is exactly what resolve_width() already says in its own comment —
"latency-bound work likes siblings, issue-bound work does not" — but the decode
budget's physical-core affinity confines the process to 16 CPUs, so the task
runtime can never reach a sibling no matter what width is asked for.

4. What this rules out

Everything I had been chasing:

• Routing (task runtime vs Rayon) — inside the null band; retracted above.
• Idle-pool spin — an idle task-runtime pool costs 0.0 ms.
• Park/wake latency — p50 is flat (1.77–2.09 ms) across gaps of 0 to 4000 µs
  while parks go 11.5 → 17.8 and spin-hits go ~5 → 0.
• Spin-window tuning — the adaptive window converges to ~2xMIN_SPIN (~40 µs)
  because it doubles on work and halves on a failed spin, so spin is only ~3.5%
  of task CPU. Cutting it saves little and cannot touch p50.
• Serial pre/post and task grain — 0.55% serial fraction.
• L3/CCD locality — 8 cores inside one L3 vs split across both were identical.

The 9.4-of-16 figure is therefore close to the honest ceiling for a single
latency-bound decode GEMV on a physical-core budget, not a scheduler defect.
The machine reaches 91% utilisation and 26 GB/s — but only with concurrent
sessions, which is a serving-shape property, not something the scheduler can
manufacture for one stream.

5. Consequence, and what I am not doing yet

There is a real trade here, and it is shape-dependent:

• For latency, one session wants siblings: 1.26x on p50, at ~1.8x the CPU
  (9.6 → 18.0 cpu/wall). Efficiency-negative, latency-positive.
• For throughput per CPU, the physical-core budget with concurrent sessions
  wins outright: 750/s at 14.55 cpu/wall (51.5 iter/s per CPU) versus 662/s at
  18.7 (35 iter/s per CPU).

That means the right answer is a shape-aware budget policy, not a blanket
change — and it directly touches the physical-core affinity semantics from #1232
and the budget/SMT warning in #1384. I am not opening that change on this
evidence alone: the host has been at load average 7–12 all session, several reps
collapsed outright, and a policy that regresses issue-bound prefill to help
latency-bound decode would be a bad trade. It needs a quiet host and a prefill
control arm before it becomes a PR.

Residual overheads worth fixing independently, from the attribution above: three
pools totalling 47 worker threads for a 16-core budget, and the decode pool
burning 800 ms as a pure pass-through while the task runtime does 100% of the
model work.

Measurement only — no code change in this phase.

[justinchuby]

Phase 23 — global Rayon ownership audit (mandate item 2), and a correction

The mandate asked me to stop the global Rayon pool being "eagerly constructed from probes/metadata" and to quantify which calls trigger it. I did the attribution. The premise turns out to be wrong, and I want that on the record before anyone acts on it.

Who actually builds it

I named the global pool's workers (nxgn-prefill-*, from #1398), combined that with the lazy-decode-pool change (#1434), and captured a one-shot backtrace at the first width probe.

The first construction is genuine first use, not a probe:

onnx_runtime_ep_cpu::provider::CpuExecutionProvider::copy_from_host
onnx_runtime_session::executor::state::Executor::build_with_cuda_requirement
onnx_runtime_session::executor::state::Executor::build
onnx_runtime_session::InferenceSession::from_parts

copy_host_bytes does the weight upload with par_chunks_mut once the copy is over host_copy_parallel_min_bytes(). That is real parallel work on the global pool at session load. The codebase is already careful here — copy_host_bytes deliberately checks size before calling host_copy_workers(), with a comment saying exactly why — so the small-copy path is already probe-free. There is no metadata/probe path constructing it that I can find.

I also tested the "probe builds it" hypothesis directly by making the fan-out width query non-constructing. Thread count did not move (54 both ways). The hypothesis is falsified.

Is it dead weight?

Per workload, at a 16-core budget, nxgn-prefill-* CPU:

workload	global Rayon	task runtime
GEMM decode t=1	16 thr, 0.0 ms	4960 ms
RoPE t=1	16 thr, 0.0 ms	0.0 ms
KV-cat p=2047	16 thr, 0.0 ms	0.0 ms
MoE t=512 (prefill)	16 thr, 177,160 ms	0.0 ms

So it is idle in decode and is the entire prefill workhorse. Deleting it or making its bounding lazy would be a serious prefill regression. It is also load-bearing in a subtler way: build_global() fails if the pool already exists, so the explicit-budget bounding must run before first use — which is why it sits in EpFactory::initialize(). That eager call is a no-op unless a budget is set, which I confirmed: with no budget, nxgn-prefill is 0 threads.

The real remaining thread-count item

At default (no explicit budget), a decode-only process carries 54 threads, and RAYON_NUM_THREADS bisects them:

RAYON_NUM_THREADS	threads
unset	54
4	26
1	23

So 31 of the 54 are the global pool, spawned at one-worker-per-logical-CPU because that is Rayon's default — even though host_copy_workers() clamps the only thing that uses it at load to 8. We build 32 workers to run an 8-way memcpy, and in a decode-only process they then do nothing forever.

What I am not doing

The obvious fix is to give the default path the same bounded, named global pool the explicit-budget path gets (physical-core width), which would take default 54 → 38 and hand us nxgn-prefill-* attribution for free. I am not opening that PR on this evidence, because it changes process-wide prefill parallelism from 32 to 16 by default, and I have no prefill A/B on a quiet host to show that is free. That is a policy change and the standing rule is no policy claim until controlled. Logging it as a proposal.

Where the target stands

Target was "fewer than 47 workers for a 16-core budget without throughput regression". #1434 delivers 47 → 31 by removing the decode pool, with latency neutral inside the A/A null band and total CPU down ~2%. The remaining 16 are the prefill pool, and the evidence above says they should stay.

[justinchuby]

Phase 24 — the unexplained ~1.7x drift is exact-fit barrier fragility, and two of my earlier claims were wrong

The host went genuinely quiet for the first time this campaign (load average 1.21). That immediately falsified two things I had claimed, and then explained the drift we have been chasing since phase 18.

Correction 1: the "quiet band" was never quiet

I had been treating cpu_per_wall ≈ 9.5–10.1 as the quiet-host signature and rejecting anything outside it. On an actually-quiet host the same binary reports 12.3–12.8. So the "single-session utilization is 9.4 of 16" headline — which I have quoted for several phases and which motivated a lot of this lane — is the contended state, not a property of the scheduler. Real quiet-host utilization is ~12.5/16.

Correction 2: the SMT win does not exist

Phase 22b claimed SMT siblings cut single-session decode p50 by a median 1.26x. Re-run interleaved with alternating order on the quiet host:

arm	p50	cpu/wall
physical-core (budget 16)	1.177 ms	12.3–12.6
SMT (32 lanes, affinity off)	1.219 ms	22.0–23.2

SMT is 1.03x slower at 1.8x the CPU. The earlier 1.26x was an artifact of measuring on a loaded host: when co-tenants occupy the siblings, physical-core confinement does not protect you (the sibling steals the core anyway), so grabbing all 32 CPUs simply captured more machine. I withdraw the SMT proposal for mandate item 3. The existing physical-core default is correct, and I am not opening the workload-class SMT experiment — the evidence says there is nothing to win.

The actual mechanism: the barrier is an exact fit with zero slack

Identical runs, same binary, same flags, are bimodal: p50 either ~1.17 ms (cpu/wall ~12.5) or ~1.85 ms (cpu/wall ~9.6). A 1.58x spread between runs of the same binary — this is the "unexplained ~1.7x drift".

It is not noise, and it is stable within a run. It correlates cleanly with one counter, straggler_yields/iter (#1411):

regime	p50	straggler yields/iter
fast	1.17–1.18	0.28 – 1.51
slow	1.80–1.99	2.45 – 7.59

straggler_waits stays pinned at ~1.0 in both — every dispatch waits, as any fork-join does. It is the yields that explode, i.e. one lane is consistently late.

Why: at a 16-core budget the task runtime runs 15 worker threads plus the calling thread = exactly 16 runnable lanes on exactly 16 budget CPUs. There is no spare lane. mpstat during a slow window showed a co-tenant at 100% on CPU 11 (the SMT sibling of budget CPU 10) in one case and at 50% on budget CPU 14 in another. Either way a single contended lane halves, and an equal-chunk barrier makes the slowest lane set the pace for all 16.

The width sweep confirms it

Same conditions, interleaved:

task width	rep1	rep2	rep3	median	behaviour
16 (= budget)	1.819	1.171	1.862	1.819	bistable: produces both the best and the worst result
15	1.243	1.290	1.622	1.290	mostly stable
14	1.319	1.316	1.380	1.319	deterministic, never bad
12	1.508	1.519	1.505	1.505	stable but under-parallel

Width 14 gives up 13% against width 16's best case and beats its median by 1.38x, with a much tighter p90. Two spare lanes absorb the caller and any co-tenant, and the bistability disappears entirely.

What I am proposing, and what I am not doing

The finding argues the default task-runtime width should leave headroom below the CPU budget rather than matching it exactly. But that is a change to default budget/width policy, it touches the semantics in #1232/#1384, and the honest trade depends on whether the deployment is exclusive: on a dedicated machine width==budget is genuinely the fastest configuration.

Standing rule is no policy claim until controlled, and one intermittently-quiet shared host is not a controlled environment for a default-policy change. So I am logging this as a proposal with its evidence, not opening the PR. What it does justify immediately is cheap and safe: straggler_yields/iter (#1411) is a reliable online detector for this state, and any future measurement in this campaign should report it and reject reps where it exceeds ~2.

This also retroactively explains a lot of the campaign's rep-to-rep variance, and it means several earlier phase numbers were measured in the slow regime without knowing it.

[justinchuby]

Phase 25 — the width-headroom proposal, put under deterministic control. Verdict: no PR.

Phase 24 ended with a width sweep suggesting that a barrier exactly as wide as the CPU budget is fragile, and that leaving one or two lanes of headroom removes a ~1.6x bistability. That was measured against ambient co-tenancy, which is not an experiment. This is the controlled version, and it does not support opening a PR.

Design

The confound in phase 24 was that contention was whatever the host happened to be doing. So:

• Budget fixed at 16 CPUs for every cell (--native-threads 16), so the CPU set and the affinity mask are identical across arms. Only ONNX_GENAI_CPU_TASK_THREADS varies, which moves the barrier width independently of the budget. Verified: 32 threads at width 16, 30 at width 14, same 16-CPU confinement.
• Contention is deliberate, not ambient: a busy loop pinned with taskset to CPU 14, a budget lane. Self-terminating, so the arm boundaries are exact.
• Every rep runs all four widths in both conditions, alternating order between reps, and includes an A/A null control — the same cell twice — to size the band at that moment.
• 10 reps, 1500 iterations each, decode shape (qwen3_0p6b_qkv_t1).

Script is at width_experiment.sh in my scratch tree; it is deterministic and re-runnable by anyone with the harness.

The null control rejected 3 of 10 reps

Null A/A ratios: 1.072, 1.009, 1.034, 1.177, 1.472, 1.062, 1.274, 1.131, 1.012, 1.071. Reps 4, 5 and 7 are discarded — at a null of 1.47 nothing smaller than 1.5x is observable. Host load ran 11 to 21 throughout, driven by other agents' sweeps.

Result on the seven surviving reps

width	spinner OFF	spinner ON
16	0.2399	0.2847
15	0.2474	0.2777
14	0.2581	0.2558
12	0.2184	0.2371

With the deliberate spinner, p50 is monotonic in width — every lane of headroom helps, 1.20x from 16 to 12. That is the predicted signature and it is the first time the effect has appeared under a controlled cause rather than ambient noise.

Without the spinner there is no ordering at all (width 12 fastest, width 14 slowest). The reason is not that the prediction failed but that "spinner off" is not "uncontended" on this host: at load 11-21 the ambient arm is contended too, just non-deterministically. The uncontended arm the comparison actually needs was never available.

I also falsified my own mechanism

A single run early in this phase looked like a clean structural explanation — at width 16 workers almost always parked (15.41 parks/iter, 1.07 spin-hits), while at width 14 they stayed spinning and caught the next dispatch (6.27 parks, 9.23 spin-hits). That would have been a satisfying story: the caller competes with a worker for the last CPU, so workers get descheduled and must take the expensive park/wake path.

Aggregated over all 10 reps it does not survive:

width	parks/iter	spin-hits/iter	park share
16	14.39	1.56	90%
15	14.11	0.01	100%
14	13.31	0.06	100%
12	11.34	0.00	100%

Parks simply scale with worker count and spin-hits are ~0 at every width. The 9.23 spin-hits figure was itself a regime artifact — the same bistability this phase set out to explain, showing up inside my evidence for explaining it. The spin/park mechanism is withdrawn. I have a reproducible effect under controlled contention and no confirmed mechanism for it.

Verdict

Per the standing rule — no policy without control, and no PR unless a runtime-detectable policy beats both the contended and the quiet case without regressing budget semantics — this does not qualify, on three independent counts:

1. The quiet dedicated-host arm is unobtainable here, so the tradeoff cannot be quantified. A narrower default would be a real cost on an exclusive machine, which is a legitimate deployment.
2. The mechanism is unknown, so any policy would be curve-fitting to one host's contention pattern.
3. The effect (1.11x at width 14, 1.20x at width 12) is only just outside the per-rep null band, on a host where the band reached 1.47.

It stays an experiment. What it does establish, and what I would defend: the effect is real and has a controllable cause — a deterministic spinner on one budget lane reproduces it monotonically, which ambient noise never did cleanly. If a dedicated host becomes available, the missing arm is one script invocation away.

[github-actions]

✅ Benchmarks — No Regression

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
✅	matmul/small_generic_f32_threads=8/1x256x256	33.34 µs	38.27 µs	+14.8%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	49.58 µs	56.21 µs	+13.4%
✅	kv_cache/alloc_dealloc_pages	36.17 µs	40.17 µs	+11.1%
✅	gather/medium_f32_threads=1-internal/32768	3.60 µs	3.86 µs	+7.3%
✅	matmul/small_generic_f16_threads=8/1x256x256	27.07 µs	28.06 µs	+3.7%
✅	gather/medium_bf16_threads=1-internal/32768	2.22 µs	2.27 µs	+2.3%
✅	matmul/medium_generic_f16_threads=1/32x512x512	28.22 µs	28.85 µs	+2.2%
✅	qwen3_sampling_processors/top_k_top_p_fast	610.01 µs	621.38 µs	+1.9%
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	42.78 µs	43.46 µs	+1.6%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	77.96 µs	79.15 µs	+1.5%
✅	add/medium_f16_threads=1-internal/262144	96.69 µs	98.16 µs	+1.5%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.21 ms	5.29 ms	+1.5%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.85 ms	1.87 ms	+1.0%
✅	add/medium_bf16_threads=1-internal/262144	94.88 µs	95.61 µs	+0.8%
✅	gather/small_f32_threads=1-internal/4096	615.2 ns	619.6 ns	+0.7%
✅	matmul/small_generic_bf16_threads=8/1x256x256	28.82 µs	29.00 µs	+0.6%
✅	reduce_mean/small_f32_threads=1-internal/4096	13.85 µs	13.93 µs	+0.6%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	495.96 µs	497.09 µs	+0.2%
✅	grammar_masking/llguidance_compute_mask/32	68.85 µs	69.00 µs	+0.2%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.13 ms	2.14 ms	+0.2%
✅	logit_processing/seven_processor_chain_per_step	297.79 µs	297.69 µs	-0.0%
✅	matmul/small_generic_f32_threads=1/1x256x256	34.03 µs	34.01 µs	-0.1%
✅	matmul/small_generic_bf16_threads=1/1x256x256	28.45 µs	28.37 µs	-0.3%
✅	add/medium_f32_threads=1-internal/262144	22.69 µs	22.61 µs	-0.4%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.27 ms	3.26 ms	-0.5%
✅	gather/medium_f16_threads=1-internal/32768	2.23 µs	2.22 µs	-0.6%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	353.13 µs	350.22 µs	-0.8%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	8.72 ms	8.64 ms	-0.9%
✅	reduce_mean/large_f32_threads=1-internal/262144	918.55 µs	908.85 µs	-1.1%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	361.69 µs	356.97 µs	-1.3%
✅	add/small_bf16_threads=1-internal/1024	415.3 ns	409.8 ns	-1.3%
✅	add/large_bf16_threads=1-internal/4194304	1.61 ms	1.59 ms	-1.4%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	76.38 µs	74.85 µs	-2.0%
✅	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	554.85 µs	535.91 µs	-3.4%
✅	reduce_mean/medium_f32_threads=1-internal/65536	236.37 µs	226.62 µs	-4.1%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.34 ms	1.28 ms	-4.5%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.79 ms	3.62 ms	-4.6%
✅	matmul/medium_generic_f32_threads=8/32x512x512	938.15 µs	892.12 µs	-4.9%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	519.75 µs	493.49 µs	-5.1%
✅	gather/large_f16_threads=1-internal/131072	11.68 µs	10.98 µs	-6.0%
✅	add/small_f32_threads=1-internal/1024	192.5 ns	178.8 ns	-7.1%
✅	gather/small_bf16_threads=1-internal/4096	474.2 ns	438.2 ns	-7.6%
✅	matmul/medium_generic_f16_threads=8/32x512x512	30.60 µs	28.13 µs	-8.1%
✅	matmul/small_generic_f16_threads=1/1x256x256	30.20 µs	27.65 µs	-8.4%
✅	tokenization/encode_tokens_per_second	390.07 µs	354.47 µs	-9.1%
✅	qwen3_sampling_processors/top_k_partial_selection	144.03 µs	129.80 µs	-9.9%
✅	gather/small_f16_threads=1-internal/4096	488.8 ns	438.9 ns	-10.2%
✅	sampling_latency/min_p_per_token	231.67 µs	207.55 µs	-10.4%
✅	add/large_f16_threads=1-internal/4194304	1.71 ms	1.51 ms	-11.8%
✅	add/large_f32_threads=1-internal/4194304	655.35 µs	575.25 µs	-12.2%
✅	tokenization/decode_tokens_per_second	6.63 ms	5.70 ms	-14.1%
🟢	sampling_latency/top_k_per_token	57.46 µs	48.44 µs	-15.7%
🟢	qwen3_sampling_processors/top_k_full_sort_baseline	2.34 ms	1.97 ms	-15.8%
🟢	sampling_latency/top_p_per_token	422.39 µs	352.90 µs	-16.5%
🟢	add/small_f16_threads=1-internal/1024	556.5 ns	460.5 ns	-17.3%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	50.89 µs	42.01 µs	-17.5%
🟢	gather/large_f32_threads=1-internal/131072	28.60 µs	22.88 µs	-20.0%
🟢	sampling_latency/greedy_per_token	3.78 µs	3.01 µs	-20.5%
🟢	gather/large_bf16_threads=1-internal/131072	13.46 µs	10.19 µs	-24.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.30 3.27 4.58 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.20%. Comparing base (f66e842) to head (7cbaec1).
⚠️ Report is 43 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1395      +/-   ##
==========================================
+ Coverage   80.32%   81.20%   +0.87%     
==========================================
  Files         426      429       +3     
  Lines      204769   214755    +9986     
  Branches   204769   214755    +9986     
==========================================
+ Hits       164484   174395    +9911     
+ Misses      34664    34593      -71     
- Partials     5621     5767     +146     

Flag	Coverage Δ
cli-ort-linux	72.51% <ø> (?)
cli-ort-windows	72.01% <ø> (ø)
mlas	85.90% <ø> (?)
offline	81.35% <ø> (+0.80%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 79 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justinchuby]

Status: moving this back to draft, and recording precisely what is and is not still true

This branch is 312 commits behind main and red on eight lanes. It has been open since 15 Aug. Rather than leave it sitting in the queue looking like a merge candidate, here is an accounting.

What has been superseded

• The park/wake axis landed elsewhere. crates/onnx-runtime-ep-cpu/benches/decode_gap_park_ab.rs is on main and covers configurable inter-token gaps, warmup beyond pool construction, A/A null arms, and per-worker straggler attribution at EP level. That is the instrument the campaign has actually been using.
• The bench_generic.rs → model_io.rs extraction is no longer applicable. This branch moves 520 lines out of bench_generic.rs; that file has been modified many times since, including by me (host_sib=, contention fields). Rebasing the extraction would produce a large, mostly-mechanical conflict resolution that nobody can review as a unit. If the extraction is still wanted it should be proposed fresh, on its own, against current main.

What is genuinely still unlanded

A model-level gap harness. decode_gap_park_ab is EP-level and synthetic: it drives the decode pool directly. Nothing on main runs a real session with a configurable inter-token busy/sleep gap distribution. That axis is still uncovered, and the reason it matters has not gone away — a zero-gap loop is the one workload on which parking cannot show its benefit, so tuning a blocktime default against it measures the wrong thing.

What must not be merged from here under any circumstances

docs/benchmarks/2026-08-15-cpu-ep-vs-ort-attention-moe.md — 378 lines of measured claims taken before either contention instrument existed.

That is not a formality. Since these numbers were taken:

• foreign_% (perf(cpu): bound the decode dispatcher's yield loop and park it instead #1808) showed that a "quiet host" assumption was doing load-bearing work in results I had already published.
• sib_% (perf(hostmon): measure the SMT siblings the confined set is blind to #1874) showed that foreign_% is blind by construction to a co-tenant on the SMT sibling of a decode core: a budget of N pins N physical cores, so the siblings are outside the confined set. A verified 100%-occupancy hog on one sibling cost 68% of decode throughput while foreign_% read 1.1.
• Applying both screens to my own published CPU EP: the decode pool picks worker CPUs positionally, so it cannot avoid a loaded CPU (cost retracted — see comment) #1865 figure took it from 1.4–1.9% to 0.34% against a 0.17% A/A floor — and the same unscreened launch set showed a 13.8% difference between an arm and itself.

Everything in that document was measured in the regime that produced the 13.8% self-difference. Three of its conclusions were already retracted in-branch (cb62fe0f8, 6d56b7353, f3c2a6a23), which is the pattern rather than the exception. None of it should re-enter the docs tree without being re-taken with foreign_% and sib_% screening and an in-run A/A null. I would rather these numbers stay in a draft PR than be findable in docs/.

Disposition

Back to draft. Not closing it: the harness design and its interface notes are the starting point for the model-level gap work, which I still own. When that lands it will be a fresh PR against current main, carrying the harness and no pre-instrument numbers.

[justinchuby]

Out of draft. Independent (Opus) review done; it found two real defects, both of which would have published a misleading line, and both are fixed in fa2f19fbc and c0ee1e400.

1. The concurrent arm called a good measurement dead

Route attribution was only wired into the single-session path, so --sessions N > 1 left routes empty. Any fp32 model routes MatMul to rayon rather than task_runtime, so the pool row is legitimately zero — and with no routes recorded the report fell through to:

ATTRIBUTION: zero dispatches AND zero routes recorded ... this is a dead
             instrument, not a measurement -- do not quote the pool row.

The run was fine. The verdict was wrong. A false "do not quote this" is no better than a false number, and it is worse than the original silence because it sounds authoritative.

Attribution now runs on one session before the barrier region, not inside it: the ledger takes a mutex per observation, so recording across N synchronised threads would serialise the exact contention that arm exists to measure. Verified — --sessions 2 now prints route: MatMulF32 -> Native (f32, 4 calls, up to 2 threads) and the true-zero attribution.

2. Per-iteration counters were divided by the wrong interval

Counters are sampled every --steady-window iterations, so the snapshot at or before the steady boundary is generally earlier than the boundary and the counter window is wider than the sample window. Dividing counter deltas by the sample count overstated dispatches/iter, parks/iter, spin-hits/iter and both context-switch rates — and parks_per_iter is the park/wake figure this harness exists to produce.

Worse: when steady state is reached before the first interior snapshot, no snapshot qualifies and the window falls back to iteration zero, so the "steady" counters silently span the whole run. The comment there claimed "the steady window's counters never include warm-up", which was precisely wrong in that case.

Neither is fixable by sampling every iteration — that perturbs what is being measured. So both are measured and stated instead: per-iteration figures divide by the interval the counters actually span, the report prints that interval beside the sample count, and the fall-back case prints a warning. That warning fires on a real run, so the case is not hypothetical:

counter window: 512 iters, same span as the samples above
WARNING: steady state was reached before the first interior counter snapshot, so
         every counter and cpu figure above spans the whole run and INCLUDES the
         warm-up transient. Lower --steady-window to separate them.

3. Two defects in my own new text, found by running it rather than reading it

The counter-window line originally said "so the two differ" and then printed 512 and 512. A message asserting a discrepancy next to two equal numbers teaches the reader to stop reading messages. It is now conditional, and when the windows do differ it names both divisors.

Both new messages also told the reader to lower --snapshot-every. There is no such flag — that is the internal parameter name; the knob is --steady-window. Advice that cannot be followed is not advice.

Review items that did not survive

The reviewer traced and cleared: gap busy-wait/sleep correctness and seeded jitter; /proc/self/stat CPU, per-task context-switch summing and VmHWM peak RSS; the hardcoded CLK_TCK of 100; steady-state detection under tiny --iters; and the dispatch_ledger reset/enable/dropped() ordering across the two null arms.

Validation

• cargo test -p onnx-genai-bench --features bench-native --lib — 32 passed, 0 failed
• cargo clippy -p onnx-genai-bench --features bench-native --bin bench_decode_gap -- -D warnings — clean
• cargo fmt --all -- --check — clean
• End-to-end under scripts/hostlock.sh: single and concurrent arms, both ledger states, both counter-window cases

Still no timing claim from any of it — these are correctness and instrument-liveness runs on synthetic MatMul fixtures.