test(ep-cpu): assert placement honesty, not the spread policy, in the default sweep

[justinchuby]

Follow-up to #1805, and the change #1802 needs in order to be landable.

The defect

every_benchmarked_decode_width_realizes_the_worker_count_it_requests asserted two different things under one name:

assert_eq!(report.planned_distinct_cores, Some(true), ...);
assert_eq!(report.realized, "one-per-core", ...);

One of those claims is honest and policy-neutral — the pool places workers where it says it places them. That is the #1792 defect (the only user-facing placement control was inert), it is wrong under every policy, and it stays.

The other is #1729's spread policy stated as a law, in a test that never asked for spread. Encoding a policy as the contract inverts the argument the test exists to make.

It also blocks #1802 concretely. The standing user direction recorded in .squad/decisions/inbox/copilot-cpu-shared-host-default-2026-08-23.md (via #1729) is that a policy which wins only under exclusive quiet-host conditions is not a valid default; spread measures ~26% worse against a single ~90% co-tenant, and the ranking of the two policies inverts with load (core_topology.rs module docs: four bandwidth hogs → compact 4.54 ms/token vs spread 5.03–6.26; eight hogs → compact 15.92 vs spread 6.08). Whoever implements the co-tenancy-robust default hits this assertion, and the cheapest way past it is to weaken it rather than to fix its shape. That is worth pre-empting: @gaff-1 flagged exactly this risk on #1805 (comment 5384470084).

What this PR does

1. The default sweep asserts only what survives a policy change.

Kept, all policy-neutral: the reported placement is the placement actually in force (assert_placement_is_honest); the policy name is one this build can select; the realized-placement observation is switched on wherever the target has the query; an unreadable mask is never scored as a placed worker; and the per-width anti-vacuity guards in both directions.

Removed: both one-per-core assertions.

saw_placement_check was counting widths at which the planner's one-per-core policy was asserted. There is no such assertion left, so it now counts widths at which a policy-neutral verdict was actually produced. A counter that outlives the check it counted is worse than no counter, because it keeps reporting coverage that stopped existing.

2. Placement becomes selectable, so spread can be asserted by name.

ONNX_GENAI_CPU_DECODE_PLACEMENT=spread|compact and decode_affinity::CorePlacement. order_pin_targets keeps its signature and delegates to order_pin_targets_for(.., CorePlacement::from_env()), so all four production call sites read one policy and cannot drift apart.

• The default is unchanged (Spread). This PR does not flip anything; it makes CPU EP: #1729 made the spread placement the unconditional default, against a recorded correction that quiet-host-only policies are not valid defaults #1802's flip a one-line default change with a named alternative to flip to.
• Orthogonal to ONNX_GENAI_CPU_DECODE_AFFINITY, which chooses which CPUs the pool may use. This one only ranks a set already chosen — and both policies are permutations of their input, so no placement can widen or shrink what a cgroup or taskset allows.
• An unparseable value is reported once on stderr and treated as the default rather than being fatal, unlike DECODE_AFFINITY, because a wrong answer there is a correctness question and here it is a ranking. It is never silent.

3. Spread is asserted explicitly, where it is asked for, in an_explicit_spread_policy_places_one_worker_per_physical_core. Same claim as before, now attached to a request rather than to an assumption. #1802 cannot invalidate it: it may change what the default is, it cannot change what spread means.

Mutations

The thing being removed here is an assertion that could not be falsified by the policy it encoded, so the additions have to be shown to discriminate.

mutation	test	result
compact default	a_compact_policy_shares_cores_and_still_passes_every_policy_neutral_check	every policy-neutral claim the sweep makes, re-run against a compact pool. Measured on a 4-core/8-thread cpuset: workers=3 cores=4 realized=shared-core placement=0 honest=1 pinned=1. If any of these fails, the sweep still encodes spread and #1802 is still blocked.
explicit spread	the_spread_assertion_rejects_a_compact_placement	the one-per-core predicate is shown to fail on this host today, so the explicit-spread test is not passing because realized is a constant somewhere.
syscall-stub pin	a_pin_that_reports_success_without_the_syscall_fails_the_realized_check (pre-existing, STUB_PIN_SYSCALL)	re-run unchanged against the new assertions.
dropped/altered assignment	a_pool_that_misreports_its_placement_is_caught_end_to_end (pre-existing, ONNX_GENAI_TEST_PLACEMENT_DISHONEST)	re-run unchanged.
inert selector	compact_and_spread_are_permutations_that_differ_on_an_smt_host	fails if the knob parses and changes nothing — #1792's shape, one level down.

Plus placement_parse_accepts_exactly_the_documented_modes (every rejection must name the variable, the offending value and the accepted set), a_cpu_the_topology_does_not_know_keeps_its_place_under_either_policy, and without_a_topology_every_policy_is_the_identity.

pin_targets_are_ordered_one_per_core_then_siblings now names the policy it asserts instead of reading the ambient one, and separately asserts that the default is that policy — so a changed default fails with its own name rather than looking like an ordering bug.

Skips

Unsupported platforms still skip, but only on compile-time target properties (core_topology::DETECTION_SUPPORTED, pinning_supported(), affinity_observation_supported()) and always with a printed reason. Never on the runtime success of the call whose failure the guard exists to report — that is the fail-open shape main already fixed for Linux topology detection via require_host_for_placement.

The discrimination halves of the compact tests are gated on an exact prediction derived from the child's own cpuset, not on whole-machine SMT. Whether compact can double up is a property of the cpuset, not of the host: taskset -c 24,26,28,30 is four cores with no sibling pair, so a machine-level has_smt() gate would have demanded a shared core the cpuset cannot produce and false-failed. The prediction is exact because node_shards_with orders the whole allowed set through order_pin_targets_for and caps only the worker count, so the first workers entries of the ordering are the CPUs the workers get. It is still a real cross-check rather than a tautology: the prediction comes from the ordering policy, realized comes from kernel affinity masks. Verified on both cpusets — on 24-31 the discrimination halves run, on 24,26,28,30 both skip with the parent and child cpusets printed.

Validation

All local, bounded to 8 CPUs (taskset -c 24-31, -j4, --test-threads<=4) — no saturating runs while #1806's host lock is unlanded.

• cargo fmt --all --check ✅
• cargo clippy --locked --all-targets -p onnx-runtime-ep-cpu -- -D warnings ✅
• cargo test -p onnx-runtime-ep-cpu --lib → 1751 passed, 0 failed, 25 ignored ✅ (current with main)
• aarch64 cross-clippy, --all-targets -p onnx-runtime-ep-cpu ✅
• the four mutations above, individually, with the reports quoted ✅

Two independent reviews

Opus found one false-failure: the compact tests gated their discrimination halves on whole-machine topology.has_smt(), which would have false-failed on a one-sibling-per-core cpuset (see Skips above). Fixed by deriving the exact prediction from the child's cpuset. Also: the permutation doc now states its distinct-input precondition, and the decode_spmd call sites name Spread explicitly instead of reading the ambient policy.

Pris (tester) found five things, all fixed rather than deferred:

1. matches!(placement_policy, "spread" | "compact") could never fail — decoration, not a check. It now asserts the policy equals CorePlacement::default().as_str(), a claim that can fail.
2. parent_cpuset_matching hard-asserted, so a mid-test cgroup resize or CPU hot-unplug would red the test on an environmental event. It now returns Option and skips with both cpusets printed.
3. It compared cardinality rather than membership — two different 4-CPU sets would have matched. Now compares the set.
4. Coverage had been lost: explicit spread was checked at one width where the old assertion swept every width. It now sweeps BENCHMARKED_DECODE_WIDTHS ∩ [2, cores] plus the probe width, with a checked > 0 anti-vacuity guard.
5. predicted_placement_code's degenerate branch returned the passing value — a false oracle one level down. It now returns an UNPREDICTABLE_PLACEMENT sentinel that no comparison accepts.

Pris also confirmed the compact/spread cross-check is non-circular and that all four required mutations are wired.

Disclosed limitation

If the compact selector were ever inert, both child-side discrimination checks would skip rather than fail, because both derive from order_pin_targets_for. The backstop is the deterministic synthetic-topology unit test compact_and_spread_are_permutations_that_differ_on_an_smt_host, which runs on every host and needs no SMT of its own. Stated here rather than left for a reader to discover.

Notes for review

• Adding an env var to a test-hardening PR is scope, and deliberate: CPU EP: #1729 made the spread placement the unconditional default, against a recorded correction that quiet-host-only policies are not valid defaults #1802's own proposed resolution item 2 is "spread becomes explicit configuration", and there is no way to write an explicit-spread assertion or a compact mutation without a selector. The alternative — a test-only backdoor — would mean the mutation exercises a path production cannot take.
• CorePlacement::from_env() is OnceLock-cached process-wide, so in-process env mutation after the first read is invisible by construction. Every policy-selecting test therefore goes through a child process; direct ordering tests call order_pin_targets_for with the policy named.
• DecodeAffinity::Compact means single NUMA node — a different axis from CorePlacement::Compact (SMT siblings before next core). Same word, orthogonal knobs; that is why this is a new enum rather than a new DecodeAffinity variant.

Refs #1805, #1802, #1792, #1729

---

Also in this PR: the rest of the Windows red

I found main red on both Windows lanes and fixed it here; #2078 landed the platform half first, so what remains in this PR is the outcome half. Both are recorded because the second is only interesting once you know the first exists.

The defect. #2059 landed a default-width arm that narrows the child to one CPU per physical core, ending in an unconditional expect on set_current_thread_affinity. That function is #[cfg(target_os = "linux")]; every other target gets a documented no-op returning Err. On Windows the guards above it all pass — allowed_cpus() answers via windows_imp, require_host_for_placement() succeeds — so the child reached the expect and died. It merged with both Windows lanes already FAILURE. macOS was green only because allowed_cpus() returns None there and it returned early.

What #2078 did, and it is the right shape: skip on !cfg!(target_os = "linux") in both the child and the parent, loudly, while leaving a failure on Linux fatal. That turns the lanes green.

What this PR adds on top. The skip is decided by the platform, not by whether narrowing actually happened, and the child still has three paths on which it declines to narrow and says nothing — no readable cpuset, undetectable topology, a topology covering none of the allowed CPUs. So:

• restrict_self_to_leader_cpus returns whether it narrowed, and states a reason on every declining path;
• the child reports narrowed=1/0, and past the platform gate the parent asserts it unconditionally;
• the platform predicate is a single AFFINITY_MASKING_SUPPORTED in decode_affinity.rs rather than two copies of cfg!(target_os = "linux"), with a unit test asserting the flag never claims less than the call delivers — so implementing Windows masking later cannot leave the skip silently on forever.

narrowed is reported rather than inferred from allowed == cores because that equality is also the arm's assertion, and deriving a precondition from the conclusion lets a check confirm itself. It also does not work: allowed == cores only differs on an SMT host. The mutation below fires on a cpuset of [24, 26, 28, 30], where allowed == cores == 4 on the un-narrowed set and every assertion in the arm passes while testing nothing.

Mutations, each simulating a fact this host cannot execute

mutation	expected	observed
flag forced false, masking works	unit test fires	set_current_thread_affinity succeeded on a target where AFFINITY_MASKING_SUPPORTED is false
masking forced to fail, flag true	child fails closed	restrict the child to leader CPUs: … a failure rather than an absent capability
flag forced false — Windows exactly	arm skips loudly, green	SKIP …: process-wide CPU affinity masking is implemented only on Linux → ok
narrowing silently declines on Linux	arm fails	the child must have narrowed itself … narrowed_to_leaders: false, allowed: 4, cores: 4

The last row is the one #2078 cannot catch, and the one the extra field exists for.

[codecov]

Codecov Report

❌ Patch coverage is 63.42711% with 143 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.69%. Comparing base (bc715c3) to head (fece8d2).
⚠️ Report is 39 commits behind head on main.

Files with missing lines	Patch %	Lines
...es/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs	44.92%	136 Missing and 5 partials ⚠️
crates/onnx-runtime-ep-cpu/src/decode_affinity.rs	98.37%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2034      +/-   ##
==========================================
+ Coverage   80.32%   80.69%   +0.37%     
==========================================
  Files         426      429       +3     
  Lines      204769   215097   +10328     
  Branches   204769   215097   +10328     
==========================================
+ Hits       164483   173580    +9097     
- Misses      34665    35746    +1081     
- Partials     5621     5771     +150     

Flag	Coverage Δ
cli-ort-linux	72.51% <ø> (?)
cli-ort-windows	72.10% <ø> (ø)
mlas	85.80% <ø> (?)
offline	80.82% <63.42%> (+0.28%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-runtime-ep-cpu/src/decode_spmd.rs	90.27% <100.00%> (+0.83%)	⬆️
crates/onnx-runtime-ep-cpu/src/decode_affinity.rs	94.50% <98.37%> (-0.04%)	⬇️
...es/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs	79.80% <44.92%> (-0.47%)	⬇️

... and 68 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.