perf(ep-cpu): bound the worker spin phase's sched_yield rate

[justinchuby]

Closes item 2 of #2075.

The headline is a negative result, and it is in the constant's doc comment

This does not reduce CPU. It moves it from the sys column to the user column. A sched_yield costs ~1.05 µs of kernel time and advances the wall clock ~1.05 µs; a spin_loop advances it ~0.3 ns at ~0.3 ns of user time. Per unit of waiting, both hold the core at ~100%. Replacing one with the other therefore relabels the accounting and does not give the core back.

Measured on a 7×7 order-cancelled, per-repetition-paired matrix (bench_decode_gap, gaps 20 µs–3 ms × t=2/4/8/16):

	before	after
sys share of CPU	20–31%	3.6–5.9%
cpu-per-wall	—	−0.02% … −0.85%

Every cpu-per-wall delta is smaller than the two same-binary null arms' own drift (−0.71% and −0.43%). Only t=16 sits outside at −6.6% ± 0.4 (~2.4σ) — suggestive, not established, and not claimed.

Latency is unresolvable at this sample size: p50 medians looked 7–15% better under co-tenancy, but paired per-repetition analysis gives ratio means 1.005 / 1.016 / 0.960 with sd up to 0.515. No latency claim is made in either direction.

Co-tenant harm is ruled out precisely. A dependency-free CPU-bound competitor at 8/16/32 threads shows throughput 1.000, 1.000, 1.002 against a null of 1.000 ± 0.001.

So what is it worth?

The yield rate today is set by how fast the loop happens to run, not by policy — nothing bounds it. This makes it a bounded, tested quantity: ≤64 yields per MAX_SPIN window (YIELD_MIN_INTERVAL = 10 µs = MIN_SPIN/2), which is 5–12× less run-queue traffic imposed on the host, with proven-zero harm to co-tenants.

Whether that reduction helps other tenants via run-queue lock contention is plausible and unmeasured, so it is deliberately not claimed.

The attribution behind it (this part is solid)

Before changing anything I checked #2075's premise with a regression rather than an anecdote. Nine arms spanning a 94× range of yield counts:

sys ≈ 1.053 µs × yields + 68.3 µs × parks + ~0        R² = 0.989, n = 18 (pooled with a replicate)

The fit was given only counter totals, and it recovers both syscall costs — ~1.05 µs for an uncontended sched_yield and ~68 µs for a futex round-trip — from data that contained neither. Replication agrees to 1.6%.

The control fails: sys ~ user_s fits worse (R²=0.86) and impossibly, with a −0.43 s intercept. Sys share spans 2.4%→31.6% across arms, so kernel time is not a fixed fraction of CPU. Yields explain 77–101% of kernel time in every arm above 50 CLK_TCK ticks (arms below that are 10 ms-quantised and are not quoted).

So #2075's attribution is correct. The payoff simply is not there.

Implementation

Phase 2 of spin_for_dispatch becomes:

let elapsed = start.elapsed();
if elapsed >= spin_window { break Expired; }
if elapsed >= next_yield { spin_yield(); next_yield = elapsed + YIELD_MIN_INTERVAL; }
else { spin_loop(); }

• The start.elapsed() read is the one fix(ep-cpu): bound the decode pool's readiness barrier and release its workers on failure #1825 already mandates per phase-2 iteration; it now gates both the deadline and the yield. No added clock read.
• next_yield is computed from the pre-yield elapsed, which makes the limit self-disable under contention: an 11 ms yield returns with the next one already due, so a genuinely contended box behaves exactly as before.
• The deadline keeps strict precedence over the yield. Leaving late costs a syscall, which is the thing both this change and fix(ep-cpu): bound the decode pool's readiness barrier and release its workers on failure #1825 exist to stop.
• Caught/Shutdown latency strictly improves: the epoch is re-read every ~25 ns instead of being invisible for up to 11 ms inside a blocking yield.

Independent Opus review — one real find, taken

[RISK] I had claimed "the phase's first iteration always yields", so the tests asserted performed > 0. But the deadline check now precedes the yield, and phase 1's last stride check is at spins == 4032 — iterations 4033–4095 run with no clock read. On a loaded box the window can therefore already be expired at phase-2 entry, giving zero yields with spins >= SPIN_LOOP_BUDGET. The assertion could hard-fail on correct code.

The reviewer offered relaxing the assertion, or yielding before honouring an expired deadline. I rejected the latter as actively wrong (it spends a syscall to leave late) and rejected merely relaxing as losing the test's teeth. Instead the assertion is now exact, using a discriminator that needs no clock:

Conditional on outcome == Expired (asserted first): spins == SPIN_LOOP_BUDGET ⟺ the loop broke on the first phase-2 pass without yielding; spins > SPIN_LOOP_BUDGET ⟹ the first pass yielded, because next_yield starts at zero. So performed > 0 ⟺ spins > SPIN_LOOP_BUDGET.

Proved by construction, not by argument: with CLOCK_CHECK_STRIDE mutated to 1<<20 and the window to 1 ns, the run reaches phase 2 with the deadline already blown and a probe confirms spins=4096, performed=0 — the reviewer's exact scenario, which the old assertion would have failed and the new three-way iff handles.

Seven further items were traced and cleared: spins u32 wrap (~107 s of looping vs a 500 µs window cap, and a wrap self-corrects within ≤64 iterations), next_yield overflow, #1825's guarantee and its test's continued discriminating power, self-disable under contention, the YIELD_COUNT change, and release-build cost on the spin_window <= 130 µs path.

Tests

• the_yield_phase_rate_limits_its_yields_against_the_wall_clock — absolute ceiling of 64 yields per window, with a non-vacuity gate.
• the_spin_phase_yield_counter_counts_exactly_the_yields_the_window_performed — the three-way exact iff above.
• YIELD_COUNT's increment moved out of spin_yield's injection branch so it is unconditional ground truth (still #[cfg(test)], zero release cost).

Mutation-proved: removing the limit → 221 yields against the 64 ceiling → fails. Delaying the first yield → fails all three of the new lower bound, the counter identity, and the pre-existing #1825 deadline test.

Where the loss actually is

The pool burns ~15 cores at a 300 µs gap and the yield has nothing to do with it. That is spin-window policy — MAX_SPIN/MIN_SPIN adaptation — which I am filing separately with the cpu-per-wall framing this matrix established.

Validation

cargo test -p onnx-runtime-ep-cpu --lib 1828 passed / 0 failed; --features mlas 1854 / 0; clippy --all-targets --all-features -D warnings clean; fmt --check clean.

Scope corrections to #2075 (restated)

There is one yield site, not two — :429/:520 was my misreading. And DISPATCHER_YIELD_STRIDE did not regress anything; it landed in #1201, the same commit that introduced the pool.

[codecov]

Codecov Report

❌ Patch coverage is 97.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 80.37%. Comparing base (9c5886a) to head (a55a85d).
⚠️ Report is 19 commits behind head on main.

Files with missing lines	Patch %	Lines
...rates/onnx-runtime-ep-cpu/src/task_runtime/pool.rs	97.50%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2151       +/-   ##
===========================================
+ Coverage   72.01%   80.37%    +8.36%     
===========================================
  Files          12      430      +418     
  Lines        5227   213120   +207893     
  Branches     5227   213120   +207893     
===========================================
+ Hits         3764   171303   +167539     
- Misses       1330    36022    +34692     
- Partials      133     5795     +5662     

Flag	Coverage Δ
cli-ort-linux	72.51% <ø> (?)
cli-ort-windows	72.01% <ø> (ø)
mlas	85.90% <ø> (?)
offline	80.49% <97.50%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...rates/onnx-runtime-ep-cpu/src/task_runtime/pool.rs	95.78% <97.50%> (ø)

... and 418 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.