cpu: report requested vs realized decode width so a t=N row can be verified

[justinchuby]

Closes #1763.

A published t=N decode row is a label, not a measurement. Nothing outside decode_spmd.rs could check whether the requested width was the width that actually ran, while at least three paths silently reduce it — reserve_single_group_headroom, reserve_split_headroom, and the single-CPU cpuset branch of build_from_env that drops decode to the flat path. All three report only through report_spmd_fallback, which is tracing::debug! / NXRT_CALIB_DEBUG-gated and therefore invisible in a default benchmark run.

Found by roy while chasing an unrelated flat t=1 vs t=2 timing.

What this adds

pub fn decode_width() -> DecodeWidth;              // { requested, realized, path }
impl DecodeWidth { pub fn is_as_requested(&self) -> bool }

Three properties are load-bearing rather than incidental:

It cannot build the pool. decode_width() reads POOLS.get(), not pools(). A diagnostic that forces the thing it measures would change which path the process takes and when — the read has to be inert.

It cannot pass vacuously. is_as_requested() is false unless both widths are Some and equal. A harness that asserts before the first decode step fails, rather than quietly reporting success against an unresolved state.

requested is the unclamped request. This one was a genuine defect in my first draft, and it is worth being explicit about because it is the exact bug the type exists to prevent. resolve_persistent_decode_threads_with_override already clamps to available_parallelism(), so recording the width build_from_env receives would compare the reduced width against itself — a request for 8 lanes inside a 2-CPU cpuset would report requested=2, realized=2, satisfied. It now records decode_thread_budget(), which is what the user actually asked for. The mutation suite below is what caught it; the first version of these tests passed happily with the bug in place.

Also fixed

Three early returns previously left DECODE_PATH_LABEL at "unresolved" when decode was definitively flat — the =0 opt-out, the Adaptive + explicit-affinity branch, and total == 0. Each is a decision, not an unresolved state, and leaving them unlabelled is the same silent-mislabel class this PR removes.

Why not the reads that already exist

active_decode_worker_count() returns the Rayon width when no pool is active, so it cannot tell "flat path, 16 Rayon threads" from "SPMD pool, 16 workers", and reports no requested width. decode_path_label() carries the path but not the width. pools() is consumed externally, but only as a boolean.

Tests

The existing end-to-end arm now goes through the production pools() entry instead of calling build_from_env beside it. That matters: the previous form built a local pool, so it verified a parallel construction that could agree while production disagreed.

A new arm requests 8 lanes inside a 2-CPU cpuset — the reduction made deterministic — and asserts only that the read never claims the request was honoured when it was not. The realized width is deliberately not pinned; that is a policy detail, and pinning it would make this test a change-detector for tuning decisions it has no opinion about. It skips rather than fails where no process-wide affinity mask exists (non-Linux).

Verified by mutation. Each mutant is killed by this suite and survives without it:

Mutant	Result
realized parrots requested	killed
requested recorded post-clamp (the defect above)	killed
is_as_requested() true when the width is unresolved	killed

Observed on this host: requested=8 in a 2-CPU cpuset → allowed=2 realized=Some(2) path=spmd-pool, is_as_requested() == false. Unconfined budgets 2/4/8 → requested == realized == budget, path=spmd-pool, with budget-1 spawned threads plus the dispatcher's shard (the #1746 invariant).

Not in scope

No behaviour change to dispatch, sizing, or affinity policy — this PR only makes the existing outcome observable. No performance claim; nothing here is on a decode hot path.

@roy offered to take the harness-side assertion once this lands, which is the natural split.

Validation

• cargo test -p onnx-runtime-ep-cpu --lib — 1628 passed, 0 failed (1624 on main + 4 new)
• CI offline-linux package set — exit 0, 233 suites ok
• cargo fmt --all --check — clean
• cargo clippy -p onnx-runtime-ep-cpu --all-targets -D warnings — clean, also under --features mlas and --no-default-features
• --target aarch64-unknown-linux-gnu clippy --all-targets — clean
• All 11 Rust quality scripts — pass

[justinchuby]

Adversarial review (Opus) — verdict

No blocking issues. Four SHOULD-FIX, one NIT. All five addressed in 7a37a14f2.

The review independently traced the host-variation paths that could have made the new reduced-width arm vacuous — confinement landing on 1 CPU falling through to the flat path, select_budget_cpus clamping to the pre-confined mask so bound_process_to_decode_budget cannot re-widen, and default_threads() clamping to available so build_from_env receives Some(2) under a 2-CPU cpuset while decode_thread_budget() still returns the unclamped Some(8) — and confirmed the arm asserts on a genuinely reduced pool.

Fixed

1. build_from_env latching REQUESTED_WIDTH without populating POOLS (SHOULD-FIX, latent public-API footgun). build_from_env is pub; a direct call latched a requested width while leaving POOLS unbuilt, so a later pools() could build a different width and decode_width() would report a request no realized pool matched — "the t=N label is a lie, reintroduced through the front door", which is a fair characterisation. The latch moved into pools(), where the pool is actually realized, so the two cannot desynchronize.

2. ONNX_GENAI_CPU_DECODE_SCHEDULE inherited by the child (SHOULD-FIX). The spawn cleared DECODE_AFFINITY_ENV but not this one, and the arms assert an exact path label. Confirmed load-bearing rather than theoretical: under --features mlas with ONNX_GENAI_CPU_DECODE_SCHEDULE=steal, the arm fails without the fix —

requested=Some(2) realized=Some(1) path=work-stealing-pool

— and passes with it. (Worth noting for its own sake: Steal is #[cfg(feature = "mlas")], so this is invisible in a default build.)

3 + 4. Two doc comments that disagreed with the code (SHOULD-FIX ×2). Both concerned the default-policy case. The .or(threads) fallback means a default run latches the resolved default, so is_as_requested() is meaningful on a default-configured run rather than trivially false — which is the intent, so the docs were wrong and the code was right. Corrected both.

5. Stream-of-consciousness aside in the REQUESTED_WIDTH doc (NIT). Correctly identified as an editing artifact rather than deliberate pedagogy. Removed.

Confirmed sound, no change needed

• CSV robustness. path is one of a fixed set of &'static str labels, none containing a comma, and println! puts it at end-of-line. assert_eq!(fields.len(), 7) cannot misfire.
• 0 ↔ None mapping. Unambiguous: requested is never Some(0) (decode_thread_budget() filters > 0), realized is never Some(0) (total_workers() >= 1, rayon::current_num_threads() >= 1).
• Skip path. Correct on non-Linux and when sched_setaffinity rejects a mask. The vacuity guard holds: any host that could confine to a subset of {0,1} has available_parallelism() <= 2 < REQUEST(8), so the parent skips before spawning rather than running a no-op confinement.
• Flat-path realized. Rayon-scope-dependent, but documented as such. Flagged only as a future consideration if a harness reads it from a worker thread.

Re-validation after the fixes

All three mutants still killed (realized parroting requested; requested recorded post-clamp; is_as_requested true when unresolved).

Latest main was merged into this branch (b88a58cb6) and the full gate re-run on the merged state: 1628 EP lib tests pass, CI offline-linux set exit 0 (234 suites), fmt clean, clippy -D warnings clean under default / --features mlas / --no-default-features / aarch64-unknown-linux-gnu, all 11 Rust quality scripts pass.

[codecov]

Codecov Report

❌ Patch coverage is 76.79558% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.71%. Comparing base (9463302) to head (b88a58c).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/onnx-runtime-ep-cpu/src/decode_spmd.rs	76.79%	36 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1764      +/-   ##
==========================================
+ Coverage   80.67%   80.71%   +0.04%     
==========================================
  Files         411      411              
  Lines      198661   198803     +142     
  Branches   198661   198803     +142     
==========================================
+ Hits       160267   160467     +200     
+ Misses      32972    32912      -60     
- Partials     5422     5424       +2     

Flag	Coverage Δ
cli-ort-linux	72.47% <ø> (ø)
cli-ort-windows	72.06% <ø> (+0.09%)	⬆️
mlas	85.20% <ø> (-0.14%)	⬇️
offline	80.86% <76.79%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-runtime-ep-cpu/src/decode_spmd.rs	88.87% <76.79%> (-1.31%)	⬇️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.