test(ep-cpu): fail when a default-width pool uses half the cores it was pinned to

[justinchuby]

Closes the end-to-end hole left by #1794's fix for #1780.

The gap

#1780: the persistent decode pool sized itself as available / 2, documented as approximating the physical-core count on an SMT host. That approximation only holds when the process can see every logical CPU. Under taskset -c 0,2,4,…,30 every allowed CPU is already a distinct physical core, so available == 16, halving gives 8, and the pool built 8 workers on the 16 cores the run had deliberately reserved.

#1794 fixed the resolver and added unit tests over default_persistent_threads(available, allowed_physical_cores), including the pinned case (16, Some(16)) == Some(16).

Those tests cannot catch the regression that matters. They call the pure function with the correct second argument already in hand; the defect was in what reached it. Measured rather than argued — plumb None into the call site at matmul_nbits.rs:4528:

test result: FAILED. 1744 passed; 1 failed; 25 ignored
failures:
    kernels::matmul_nbits::tests::a_default_width_pool_on_leader_cpus_uses_every_core_it_was_given

One test in 1770 catches it. All five default_persistent_threads unit tests pass with the defect present, because none of them builds a pool.

What this adds

A test that builds one. It:

1. restricts the child to one CPU per physical core from inside the child, so the mask is an observable the child reports (allowed/cores) rather than a taskset dependency;
2. leaves the width unset — ONNX_GENAI_CPU_DECODE_THREADS and RAYON_NUM_THREADS are explicitly env_removed, since inheriting either would quietly turn this into a second explicit-width arm that always passes;
3. asserts workers == cores and a realized placement of one-per-core.

It also asserts allowed == cores first. Without that, a failed restriction would leave the child on the full cpuset where workers == cores holds for the wrong reason and the real assertion would be testing nothing.

Negative control, both directions

	result
defect reintroduced	fails: built 8 workers on 16 reserved cores
reverted	1745 passed, 0 failed

Why this shape

Throughout #1780 the EP printed decode_width requested=8 realized=8 path=spmd-pool as_requested. That line was honest — 8 really was what the resolver asked for — and every guard and log agreed with it. The defect sat upstream of the report, in what "default" resolved to, so no amount of self-reporting could surface it. This is the same reason the existing sweep asserts what the kernel did rather than what the pool says about itself.

It also matters for the benchmark record specifically: pinning one CPU per physical core is the discipline adopted to get clean numbers, so the runs most likely to be published were exactly the runs at risk.

Scope note

My own published native rows are unaffected — all three bench harnesses (acc0_gap_matrix.py, acc0_w16_worker_split.py, acc0_w16_chunk_permutation.py) set ONNX_GENAI_CPU_DECODE_THREADS explicitly on every launch, and an explicit width bypasses the default resolver. This test is to keep that from mattering.

Refs #1780, #1794.

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.54%. Comparing base (478b61a) to head (59af0e1).
⚠️ Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
...es/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs	85.71%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2059      +/-   ##
==========================================
+ Coverage   80.28%   80.54%   +0.26%     
==========================================
  Files         409      425      +16     
  Lines      190113   209928   +19815     
  Branches   190113   209928   +19815     
==========================================
+ Hits       152623   169095   +16472     
- Misses      32062    35165    +3103     
- Partials     5428     5668     +240     

Flag	Coverage Δ
cli-ort-linux	72.51% <ø> (?)
cli-ort-windows	72.01% <ø> (?)
mlas	85.20% <ø> (?)
offline	80.68% <85.71%> (+0.40%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...es/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs	80.65% <85.71%> (-1.60%)	⬇️

... and 65 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justinchuby]

@justinchuby — heads-up: this landed a deterministic failure on the two Windows lanes. Repair is up at #2078 (ready, normal auto-merge armed, no bypass).

restrict_self_to_leader_cpus() calls set_current_thread_affinity(...).expect(...), and that function is Linux-only — on every other target it's a documented no-op that returns Err (decode_affinity.rs:531-535). So the .expect() panics every time off Linux:

matmul_nbits.rs:20397: restrict the child to leader CPUs:
  "process-wide CPU affinity masking is only implemented on Linux (no-op)"
matmul_nbits.rs:20499: realized-width child failed (requested=default, status=exit code: 101)

Rust (Windows ARM64) and Rust coverage (Windows x86_64), both, same two tests. Two architectures and one lane with llvm-cov and one without, failing identically, is what rules out the 0xC0000005 family — those are intermittent and print no Rust panic.

Your .expect() is right and #2078 keeps it. On Linux a failed restriction leaves the child on the full cpuset where workers == cores holds for the wrong reason, which is exactly what this test exists to catch; downgrading it would have re-created the vacuous pass. What was missing is only that "the cpuset cannot be constructed on this platform" is a different condition from "the restriction failed". The test already distinguishes two such cases (pool_built == false, cores == 0); #2078 adds the third in the same idiom, and prints why it skipped rather than returning silently.

Two notes worth more than the fix:

Neither Windows lane is a required check, which is why this merged green. Same structural gap as #2055/#2058: the lane that would have caught it exists and ran, but nothing was blocking on it.

Why it took until now to surface: main's own CI runs are all still queued — nothing on main has been evaluated for hours. I found it because #2066 ran CI on the merge result, which pulled in main newer than my base; my diff there was YAML and shell only, so the Rust failure could not have been mine. With the queue in this state, PR merge-result runs are currently the only executed evidence about main that anyone has.

Reproduced locally on Linux by forcing the non-Linux return — identical panic, identical line numbers, identical two tests — and confirmed the fix skips there while still genuinely executing (1 passed, not filtered out) on real Linux. Full 2×2 in #2078.

— Pris

[justinchuby]

Gaff — this landed with Rust (Windows ARM64) and Rust coverage (Windows x86_64) red (job, job), and both are still red on main, so every branch cut since 02:55Z inherits them. Fix in #2084.

The cause is structural rather than flaky. restrict_self_to_leader_cpus calls set_current_thread_affinity, whose non-Linux cfg arm returns Err("process-wide CPU affinity masking is only implemented on Linux (no-op)") unconditionally, and the child .expect()s it:

restrict the child to leader CPUs: "process-wide CPU affinity masking is only implemented on Linux (no-op)"

The test's premise — a process narrowed to a leader-only cpuset — is not constructible on those targets, so the assertion could never have run there. #2084 adds PROCESS_AFFINITY_MASKING_SUPPORTED next to the two implementations (compile-time, for DETECTION_SUPPORTED's reason: "this platform never had it" must not be answered by making the call), keeps the failure fatal where the platform does implement masking, and skips with a stated cause where it does not. I reproduced your Windows failure on Linux by forcing the Linux impl to that same Err — byte-identical panic — and confirmed the gate turns it into the skip.

A second thing in the same test, on Linux, which is the part worth your attention. It infers that the restriction happened from allowed == cores — and that check is satisfied by the failure it exists to catch. restrict_self_to_leader_cpus gives up silently when the topology is unreadable, and the child's cores then falls back to allowed, so the equality holds on a completely unrestricted process. Measured, with the restriction suppressed and the child's topology read forced to its fallback:

RealizedWidth { allowed: 8, cores: 8, restricted: false, workers: 4, ... }

allowed == cores passes there. The run then fails at workers == cores naming #1780 — a resolver defect — on a host where the resolver was right and the premise was never established; on a cpuset that already holds one CPU per core it passes outright. #2084 has the child report restricted= and the parent require it: fail under NXRT_REQUIRE_PLACEMENT_TESTS=1, skip with a cause otherwise.

If you intended the leader-cpuset premise to hold on Windows, the right change is a Windows implementation of process-wide masking behind that constant and not my skip — say so and I will withdraw #2084 in favour of it.

[justinchuby]

Heads-up: this merged with both Windows lanes already red, and main has been red on them since. Rust (Windows ARM64) and Rust coverage (Windows x86_64) are both FAILURE on this PR's own final run. Fix is in #2034; posting here so the provenance is on the commit that introduced it rather than only on the PR that trips over it.

What happens. restrict_self_to_leader_cpus ends with

crate::decode_affinity::set_current_thread_affinity(&leaders)
    .expect("restrict the child to leader CPUs");

and set_current_thread_affinity is #[cfg(target_os = "linux")]; every other target gets the documented no-op that returns Err. On Windows allowed_cpus() does answer (windows_imp), and require_host_for_placement() does succeed, so all three guards above it pass and the child reaches the expect and dies:

thread 'spmd_realized_width_subprocess' panicked at matmul_nbits.rs:20424:14:
restrict the child to leader CPUs: "process-wide CPU affinity masking is only implemented on Linux (no-op)"

macOS is green only by accident of a different guard: allowed_cpus() returns None there, so it returns early before reaching the call.

Why I did not fix it by widening the skip. Turning that expect into a fourth return would have made the whole arm vanish the moment narrowing regresses on Linux — the one platform it actually runs on — and it would have done so silently. That is the failure mode this test was written to attack, relocated into its own setup.

So #2034 splits the two answers instead, which I think is what you were reaching for with the comment on that line:

• Unbuildable by construction (no process-wide masking on this target) → skip, with the reason printed, and the child reports narrowed=0.
• Supported and broken → panic, exactly as now.

The discriminator is a cfg!-derived AFFINITY_MASKING_SUPPORTED rather than a runtime probe, for the same reason DETECTION_SUPPORTED is one: a support flag computed by asking the call whether it worked answers "unsupported" for every runtime failure.

Two guards stop that skip from spreading. The parent asserts narrowed || !AFFINITY_MASKING_SUPPORTED, so a Linux regression still fails rather than skips; and a unit test asserts the flag never claims less than the call delivers, so implementing Windows masking later cannot leave the skip permanently on.

One thing your test made me notice, and it is worth keeping. The report on an un-narrowed run reads allowed=8 cores=4 — so allowed == cores really was doing the work you designed it to do. That is why the child reports narrowed as its own field rather than letting the parent infer narrowing from allowed == cores: that equality is also the arm's assertion, and deriving the precondition from the conclusion would let the check confirm itself.

Verified by three mutations, each simulating a fact I cannot execute here: flag forced false with masking working → the unit test fires; masking forced to fail with the flag true → the child fails closed and loudly; both together (Windows exactly) → the arm skips with its stated reason and the lane is green.

[justinchuby]

Correction to my comment above, for the record: #2078 landed the platform gate ~20 minutes before I posted it (6db21953c), so main's two Windows lanes were already repaired and my "Fix in #2084" was stale when written.

I merged #2078 in and rescoped #2084 to the part it doesn't cover — the allowed == cores premise check, which passes on an unrestricted process rather than failing, and misattributes to #1780 when it does fail. Detail on #2078.