perf(cpu-ep): run the elementwise split on ORT's pool, not a second one

[justinchuby]

What

When our CPU EP runs inside an ORT session, our elementwise activation kernels
stop starting a second thread pool. They borrow ORT's intra-op pool through
OrtApi::KernelContext_ParallelFor instead.

Root cause

simd_activations::run_chunked splits any f32 slice of ≥ 1 Mi across a rayon
pool. That is the right thing for the native executor, which owns the machine.
Inside an ORT session it is not: ORT already has an intra-op pool of its own,
and its workers spin. Sixteen rayon workers next to sixteen spinning ORT
workers puts 32 runnable threads on 16 cores.

Measured on this machine (AMD EPYC 9V74, taskset -c 0-15, ORT 1.28,
intra_op_num_threads = 16, 1 Mi f32, p50, same .so, RAYON_NUM_THREADS
1 vs 16):

op	serial	our rayon split
Sqrt	252 µs	777 µs	3.08× slower
Relu	345 µs	607 µs	1.76× slower
Clip	399 µs	858 µs	2.15× slower
Sigmoid	521 µs	993 µs	1.91× slower
Tanh	569 µs	944 µs	1.66× slower
QuickGelu	790 µs	1320 µs	1.67× slower
Erf	1020 µs	1388 µs	1.36× slower
FastGelu	1040 µs	1479 µs	1.42× slower
Gelu	1570 µs	1808 µs	1.15× slower

Parallelising made every op slower.

Why a bigger threshold is not the fix

The obvious repair is to raise PAR_MIN_LEN until the split only fires where
it still wins. It does not work, because the same sweep at
intra_op_num_threads = 1 says the opposite — there the split is a large win
from 1 Mi upwards:

op, 4 Mi f32	serial	our rayon split
Erf	3551 µs	700 µs	5.07× faster
Gelu	5315 µs	1047 µs	5.08× faster

No constant satisfies both, because the variable that decides the answer is not
the slice length. It is how much of the machine the host is already using, and
only the host knows that.

Fix

Ask the host.

• onnx-runtime-ep-api::host_parallel — a thread-local seam. It is the only
  crate both onnx-runtime-ep-cpu (which has the kernels) and
  onnx-runtime-ep-plugin (which has the OrtKernelContext) already depend on,
  so neither has to learn about the other. HostParallel::run(total, body) is
  blocking, which is what makes borrowing a &dyn Fn from the caller's frame
  sound; bodies run with in_host_task() set so a nested split can see it is
  already on a host thread.
• onnx-runtime-ep-plugin::host_pool — the ORT-backed implementation over
  KernelContext_ParallelFor (num_batch = 0, so ORT's workers claim indices
  dynamically). install() returns an RAII guard; compute_execute holds one
  for the whole call.
• simd_activations — run_chunked and run_chunked_rows prefer the host
  pool when one is installed, and only fall back to rayon when there is none.

Because we cannot ask ORT how wide its pool is, the host split cuts by size
rather than by thread count: MAX_HOST_CHUNKS = 64 pieces, floored at the
existing PAR_MIN_CHUNK. That also makes the chunk boundaries — and therefore
the bits — independent of the machine and of the session's thread count.

Nothing is handed to ORT's CPU EP

This is a threading change, not an assignment change. Every node our EP claims
is still computed by our kernels; the only thing that comes from ORT is the
threads they run on. No op is declined, no capability filter changes.

Correctness

cargo test -p onnx-runtime-ep-cpu --features mlas --lib → 1339 passed
(1332 on main + 7), -p onnx-runtime-ep-api → 51 (+9), -p onnx-runtime-ep-plugin
→ 238 (+7).

New coverage:

• host_pool_split::{unary,bias}_kernels_match_the_unsplit_result — every unary
  and bias-fused kernel, split across a real four-thread stand-in host, is
  bit-identical to the unsplit result. A serial stand-in would prove the
  arithmetic but not the disjointness of the ranges, which is the part that
  would corrupt an output tensor.
• the_rayon_pool_is_not_used_when_a_host_is_installed — the whole point of the
  change, asserted: exactly host_chunk_len(N).1 chunks dispatched, none to rayon.
• a_nested_split_stays_serial — a kernel reached from inside a host task does
  not dispatch again.
• serial_scope_still_suppresses_the_split — the f16/bf16 sandwich stays serial
  on the host path too.
• host_chunk_policy_holds_across_lengths — whole vectors, never below
  SIMD_MIN_LEN, never a cut through a bias row, never an empty final range,
  never past the cap; swept to usize::MAX / 2.
• host_pool::{a_refused_dispatch_still_runs_every_index, a_panicking_body_does_not_unwind_into_ort}
  — a failed KernelContext_ParallelFor still runs every index (a short write
  would leave the output tensor uninitialised), and a Rust panic is caught on
  the worker and re-raised on the calling thread rather than unwinding into C++.
• host_parallel::{scope_restores_on_unwind, a_handle_is_not_visible_from_another_thread}
  — a leaked handle would be a dangling OrtKernelContext.

Benchmarks

The arm that decides this PR: the host-pool split vs staying serial. Every
number in "Root cause" compares our rayon split against serial, and serial won
every row — which says nothing about the change this PR actually makes, routing
the split onto ORT's own pool. That arm is measured here, and the
host-pool split beats serial by ~5.5–6.6×.

Measured on a 13th-gen i7-13800H (14 physical cores / 20 logical), release
build, 1 Mi f32, three arms in one process, interleaved per rep (p50 µs over
201 reps), repeated three times. intra_op is modelled at 14 rather than 16
because the stand-in's spinning workers are real threads: on a 14-core box,
15 spinners + the caller already oversubscribe and would starve the serial arm
before rayon even runs, so the width is matched to the cores. The stand-in
reproduces the one thing that matters — an intra-op pool that spins while
idle, which is exactly why a coexisting rayon pool oversubscribes — and it
latches HOST_HELPED the honest way (a worker, not the dispatcher, runs a
chunk), so prefer_host reaches steady state through the real mechanism.

The rayon-split arm is a built-in control: its result is already known from
the table above (serial wins at intra_op = 16). It reproduces here — rayon is
slower than serial in every run — which validates the harness before the
host-pool number is trusted.

op	run	serial	rayon split	host-pool split	rayon / serial	host / serial
Sqrt	1	224	297	41	1.33× (loss)	0.18× (5.5× faster)
Sqrt	2	215	490	40	2.28× (loss)	0.18× (5.4× faster)
Sqrt	3	223	302	38	1.35× (loss)	0.17× (5.9× faster)
Gelu	1	1376	1514	209	1.10× (loss)	0.15× (6.6× faster)
Gelu	2	1395	1509	209	1.08× (loss)	0.15× (6.7× faster)
Gelu	3	1413	1510	213	1.07× (loss)	0.15× (6.6× faster)

The host-pool win (~5.5–6.6×) is an order of magnitude larger than each arm's
own p10..p90 spread (e.g. Sqrt host 34..62 µs, serial 187..256 µs), so it is a
real signal, not noise. Process CPU time (contention-immune) reproduced to
16.6 / 16.2 / 12.8 s across the three runs.

Why so much larger than the rayon split's best case? Because the host cut floors
at HOST_MIN_CHUNK (4 Ki), not the rayon path's 256 Ki: 1 Mi becomes 64 tasks
the host's threads claim dynamically, where the rayon path would make four. The
warm, already-spinning pool turns that into near-linear speedup with none of the
wake-up cost that made a second pool lose.

The no-host fall-through still wins at intra_op = 1

With no host installed (native executor, ORT < 1.17, or a null context) the
kernels keep the rayon path. On a free machine that path is still the right one
(p50 µs, same box):

op	serial	rayon split
Gelu, 4 Mi	4012 µs	1687 µs	2.38× faster
Sqrt, 4 Mi	2044 µs	914 µs	2.24× faster
Gelu, 1 Mi	1012 µs	631 µs	1.60× faster
Sqrt, 1 Mi	181 µs	172 µs	1.05× (memory-bound; unchanged from main)

(The absolute multiples are smaller than the EPYC's 5× because this box has
fewer, heterogeneous P+E cores; the direction — rayon split wins on a free
machine — is what the fall-through must preserve, and it does.)

Conclusion. Dispatching the elementwise split onto the host's own pool
beats staying serial by ~5.5–6.6× at intra_op = 16, while the rayon split
loses under the same busy pool. The simpler "stay serial under a host"
alternative would leave that entire ~6× on the table — so the host-pool split,
KernelContext_ParallelFor and all, is the correct fix, not merely the more
complex one. The PR merges as written.

The harness is committed as two #[ignore]d tests in
simd_activations::three_arm_bench, run with
EP_BENCH=1 cargo test --release -p onnx-runtime-ep-cpu --lib three_arm_bench -- --ignored --nocapture --test-threads=1.

Limitations

• Sessions on an ORT older than 1.17 have a null KernelContext_ParallelFor and
  keep the rayon path.
• Only the two elementwise split points consume the seam so far. Other kernels
  that start their own rayon work under an ORT session have the same
  oversubscription and are not addressed here — filed as perf(cpu-ep): route the remaining kernels' parallelism through the host pool (MatMul, int4) — follow-up to #1143 #1145
  (follow-up to The CPU EP's fast paths depend on a host-set scope: standalone and plugin hosts get the slow path #1138), because MatMul and the int4 paths matter far more than
  elementwise and deserve their own measured change.

[codecov]

Codecov Report

❌ Patch coverage is 79.01829% with 218 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.00%. Comparing base (11043a0) to head (17ec3d5).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...nnx-runtime-ep-cpu/src/kernels/simd_activations.rs	58.96%	204 Missing and 2 partials ⚠️
crates/onnx-runtime-ep-api/src/host_parallel.rs	96.33%	7 Missing and 1 partial ⚠️
crates/onnx-runtime-ep-plugin/src/host_pool.rs	98.73%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1143      +/-   ##
==========================================
- Coverage   80.53%   80.00%   -0.54%     
==========================================
  Files         368      370       +2     
  Lines      161313   162350    +1037     
  Branches   161313   162350    +1037     
==========================================
- Hits       129921   129885      -36     
- Misses      26659    27730    +1071     
- Partials     4733     4735       +2     

Flag	Coverage Δ
cli-ort-linux	83.79% <ø> (ø)
cli-ort-windows	83.40% <ø> (ø)
mlas	85.67% <ø> (-0.14%)	⬇️
offline	79.75% <79.01%> (-0.57%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-runtime-ep-plugin/src/compute.rs	78.94% <100.00%> (+0.01%)	⬆️
crates/onnx-runtime-ep-plugin/src/lib.rs	100.00% <ø> (ø)
crates/onnx-runtime-ep-plugin/src/host_pool.rs	98.73% <98.73%> (ø)
crates/onnx-runtime-ep-api/src/host_parallel.rs	96.33% <96.33%> (ø)
...nnx-runtime-ep-cpu/src/kernels/simd_activations.rs	88.58% <58.96%> (-8.54%)	⬇️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	gather/medium_f32_threads=1-internal/32768	3.75 µs	6.39 µs	+70.2%
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	503.32 µs	744.26 µs	+47.9%
🔴	gather/small_bf16_threads=1-internal/4096	442.8 ns	654.2 ns	+47.8%
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.27 ms	1.83 ms	+44.7%
🔴	add/large_bf16_threads=1-internal/4194304	1.53 ms	2.05 ms	+33.9%
⚠️	add/large_f16_threads=1-internal/4194304	1.53 ms	1.98 ms	+29.9%
⚠️	matmul/small_generic_f16_threads=8/1x256x256	27.92 µs	34.91 µs	+25.1%
⚠️	reduce_mean/large_f32_threads=1-internal/262144	919.93 µs	1.13 ms	+22.9%
⚠️	tokenization/decode_tokens_per_second	5.68 ms	6.96 ms	+22.6%
⚠️	matmul/small_generic_bf16_threads=8/1x256x256	29.15 µs	35.44 µs	+21.6%
⚠️	reduce_mean/medium_f32_threads=1-internal/65536	231.33 µs	280.20 µs	+21.1%
⚠️	grammar_masking/llguidance_compute_mask/32	69.26 µs	83.21 µs	+20.1%
⚠️	gather/small_f16_threads=1-internal/4096	456.7 ns	547.6 ns	+19.9%
⚠️	tokenization/encode_tokens_per_second	344.02 µs	411.05 µs	+19.5%
⚠️	qwen3_sampling_processors/top_p_fast_after_top_k	482.50 µs	574.58 µs	+19.1%
⚠️	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.26 ms	3.77 ms	+15.9%
⚠️	gather/large_f32_threads=1-internal/131072	26.19 µs	30.31 µs	+15.7%
⚠️	sampling_latency/min_p_per_token	192.43 µs	221.35 µs	+15.0%
✅	matmul/small_generic_f16_threads=1/1x256x256	29.04 µs	33.37 µs	+14.9%
✅	reduce_mean/small_f32_threads=1-internal/4096	16.06 µs	18.42 µs	+14.7%
✅	matmul/small_generic_bf16_threads=1/1x256x256	27.85 µs	31.91 µs	+14.6%
✅	qwen3_sampling_processors/top_k_partial_selection	132.12 µs	150.81 µs	+14.1%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	426.78 µs	484.29 µs	+13.5%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.25 ms	5.92 ms	+12.7%
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	40.70 µs	45.70 µs	+12.3%
✅	gather/medium_bf16_threads=1-internal/32768	2.23 µs	2.49 µs	+11.4%
✅	matmul/medium_generic_f32_threads=8/32x512x512	882.79 µs	982.94 µs	+11.3%
✅	add/medium_bf16_threads=1-internal/262144	95.64 µs	106.05 µs	+10.9%
✅	sampling_latency/greedy_per_token	2.99 µs	3.30 µs	+10.2%
✅	add/large_f32_threads=1-internal/4194304	665.45 µs	732.08 µs	+10.0%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	155.31 µs	170.67 µs	+9.9%
✅	sampling_latency/top_p_per_token	354.60 µs	384.50 µs	+8.4%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.99 ms	2.14 ms	+7.9%
✅	qwen3_sampling_processors/top_k_top_p_fast	611.53 µs	658.39 µs	+7.7%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	1.99 ms	2.11 ms	+6.3%
✅	logit_processing/seven_processor_chain_per_step	299.29 µs	316.80 µs	+5.9%
✅	matmul/medium_generic_f16_threads=8/32x512x512	27.39 µs	28.89 µs	+5.5%
✅	gather/small_f32_threads=1-internal/4096	627.2 ns	661.7 ns	+5.5%
✅	kv_cache/alloc_dealloc_pages	36.63 µs	38.54 µs	+5.2%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.65 ms	3.82 ms	+4.7%
✅	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	40.42 µs	42.12 µs	+4.2%
✅	matmul/small_generic_f32_threads=8/1x256x256	48.46 µs	50.00 µs	+3.2%
✅	matmul/small_generic_f32_threads=1/1x256x256	33.73 µs	34.70 µs	+2.9%
✅	gather/large_f16_threads=1-internal/131072	13.60 µs	13.96 µs	+2.6%
✅	add/small_f16_threads=1-internal/1024	428.1 ns	439.1 ns	+2.6%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	78.74 µs	80.69 µs	+2.5%
✅	sampling_latency/top_k_per_token	48.46 µs	49.27 µs	+1.7%
✅	add/medium_f16_threads=1-internal/262144	99.69 µs	101.13 µs	+1.4%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	369.46 µs	374.28 µs	+1.3%
✅	matmul/medium_generic_f16_threads=1/32x512x512	27.70 µs	27.97 µs	+1.0%
✅	gather/medium_f16_threads=1-internal/32768	2.60 µs	2.62 µs	+0.8%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.13 ms	2.14 ms	+0.6%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	8.77 ms	8.80 ms	+0.4%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	74.19 µs	74.46 µs	+0.4%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	492.20 µs	486.83 µs	-1.1%
✅	add/small_f32_threads=1-internal/1024	206.1 ns	194.5 ns	-5.6%
✅	add/small_bf16_threads=1-internal/1024	470.9 ns	422.1 ns	-10.4%
🟢	gather/large_bf16_threads=1-internal/131072	14.80 µs	12.48 µs	-15.7%
🟢	add/medium_f32_threads=1-internal/262144	30.71 µs	23.45 µs	-23.6%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.84 3.51 5.20 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

The seam is the right design and I want to say why before the objection: the plugin installs it itself.

compute.rs takes the guard from the OrtKernelContext ORT handed the call, not from anything our engine sets. That makes it the structural opposite of IN_SPMD_SCOPE (#1138), where only onnx-genai-engine set the thread-local and every third-party host inherited the costs with none of the benefit. Here any ORT host gets it with no cooperation. That is the contract shape #1138 is asking for, and #1142 only solved half of.

Gates pass on my box: -p onnx-runtime-ep-cpu --lib 1310 passed / 0 failed, -p onnx-runtime-ep-api 48 passed. The fall-through is also right -- with no host installed the rayon path is unchanged, so the native executor keeps the 5.07x it gets at intra_op = 1.

The blocking gap: the benchmark that decides this PR has not been run

The Benchmarks section says:

Filling in: interleaved same-machine A/B against ORT at intra_op 1 and 16, before and after, for decode and prefill sizes.

Everything measured so far compares our rayon split against staying serial, and serial wins every row -- Sqrt 252 vs 777 us, and eight more. No measurement in this PR shows that dispatching to ORT's pool beats staying serial. That is the change being made.

So the evidence supports a strictly simpler alternative that the PR never tests:

when a host pool is installed, stay serial

...which needs no KernelContext_ParallelFor, no 64-chunk policy, no disjointness argument, no panic-catching across the FFI boundary, and no ORT-1.17 fallback -- roughly 700 of the 1060 added lines. If the ORT-pool split does not beat serial at intra_op = 16, that one-line version is the correct fix and this PR is mostly cost.

I am not asserting it loses. ORT's pool is the right pool, its workers are already warm, and num_batch = 0 dynamic claiming is the right call. It is entirely plausible it beats both. But the arm that would tell us apart is the one arm not in the table, and this is the fourth time this month the measured mechanism has contradicted the predicted one -- including on #1142, where the intuitive repair (parallelise the shards) measured slower than doing nothing, for the same fork-join reason that motivates this PR.

Please add one row: Sqrt and Gelu, 1 Mi, intra_op = 16, three arms in one process, interleaved -- serial / rayon split / host-pool split. Two ops is enough; the nine-row sweep already told us the direction is uniform. If host-pool wins, the PR merges as written and that row is the whole justification. If it does not, we delete most of it and keep the seam only where it pays.

Smaller points, none blocking

• MAX_HOST_CHUNKS = 64 cutting by size rather than by pool width is a good call for a reason worth keeping in the comment: it makes the bit pattern independent of the session's thread count, so results do not vary with intra_op. That is a correctness property, not a tuning one.
• a_refused_dispatch_still_runs_every_index and a_panicking_body_does_not_unwind_into_ort are the two tests I would have asked for. A short write leaving an output tensor uninitialised is exactly the failure that would surface as a wrong logit three layers later.
• the_rayon_pool_is_not_used_when_a_host_is_installed asserts the actual point of the change rather than a proxy for it. Good.
• The four-thread stand-in host rather than a serial one is the right choice -- a serial stand-in proves the arithmetic but not the disjointness, and disjointness is the part that corrupts tensors.
• Limitation 2 (other kernels still start rayon under an ORT session) is the real remaining scope. Once the seam is proven, MatMul and the int4 paths matter far more than elementwise. Please file that rather than grow this PR.

[justinchuby]

Three-arm measurement: the host-pool split beats serial ~5.5–6.6×

The one arm that decides this PR — host-pool split vs staying serial — is now measured. It wins. The "stay serial under a host" simplification would leave ~6× on the table, so the KernelContext_ParallelFor machinery is justified, not merely more complex. The PR merges as written.

i7-13800H (14 physical / 20 logical), release, 1 Mi f32, three arms in one process, interleaved per rep, p50 over 201 reps, ×3 runs. intra_op modelled at 14: the stand-in's spinning workers are real threads, so 15 spinners + caller would oversubscribe a 14-core box and starve the serial arm before rayon even runs — width matched to cores. The stand-in reproduces the one thing that matters, an intra-op pool that spins while idle, and latches HOST_HELPED honestly (a worker, not the dispatcher, runs a chunk).

The rayon-split arm is a built-in control: its loss at busy intra_op is already known from the PR's nine-row EPYC table. It reproduces here (rayon slower than serial every run), which validates the harness before the host-pool number is trusted.

op	run	serial µs	rayon split µs	host-pool split µs	rayon / serial	host / serial
Sqrt	1	224	297	41	1.33× (loss)	0.18× (5.5× faster)
Sqrt	2	215	490	40	2.28× (loss)	0.18× (5.4× faster)
Sqrt	3	223	302	38	1.35× (loss)	0.17× (5.9× faster)
Gelu	1	1376	1514	209	1.10× (loss)	0.15× (6.6× faster)
Gelu	2	1395	1509	209	1.08× (loss)	0.15× (6.7× faster)
Gelu	3	1413	1510	213	1.07× (loss)	0.15× (6.6× faster)

The win (~5.5–6.6×) is an order of magnitude larger than each arm's own p10..p90 spread (Sqrt host 34..62 µs, serial 187..256 µs) — real signal, not noise. Process CPU time (contention-immune, per the profiling SKILL) reproduced to 16.6 / 16.2 / 12.8 s across runs while another agent built on the same box.

Why so much larger than the rayon split's best case? The host cut floors at HOST_MIN_CHUNK (4 Ki), not the rayon path's 256 Ki: 1 Mi becomes 64 tasks the host's warm threads claim dynamically (num_batch = 0), where the rayon path would make four and pay wake-up cost on a second pool.

No-host fall-through still wins at intra_op = 1

With no host installed the kernels keep the rayon path, and on a free machine it is still right:

op	serial µs	rayon split µs
Gelu, 4 Mi	4012	1687	2.38× faster
Sqrt, 4 Mi	2044	914	2.24× faster
Gelu, 1 Mi	1012	631	1.60× faster
Sqrt, 1 Mi	181	172	1.05× (memory-bound; unchanged from main)

(Smaller absolute multiples than the EPYC's 5× because this box has fewer, heterogeneous P+E cores; the direction — rayon wins on a free machine — is what the fall-through must preserve, and does.)

Harness

Committed as two #[ignore]d, EP_BENCH-gated tests in simd_activations::three_arm_bench. It drives the real kernel entry points (sqrt_f32_slice, erf_gelu_f32_slice) and the real HostParallel seam (probe latch and all), not a reimplementation:

EP_BENCH=1 cargo test --release -p onnx-runtime-ep-cpu --lib three_arm_bench -- --ignored --nocapture --test-threads=1

The same commit also strengthens the MAX_HOST_CHUNKS comment to record why the cut is by size not pool width: it keeps the chunk boundaries — and thus the output bit pattern — independent of the session's thread count. That is a correctness property (a tensor produces the same bits at intra_op = 1 or 16), not a tuning one.

Gates (this branch, after rebase onto origin/deckard/host-parallel @ ee217724)

• cargo test -p onnx-runtime-ep-cpu --lib → 1316 passed, 0 failed, 16 ignored
• cargo test -p onnx-runtime-ep-api --lib → 53 passed, 0 failed
• cargo test -p onnx-runtime-ep-plugin --lib → 246 passed, 0 failed
• cargo clippy -p onnx-runtime-ep-cpu -p onnx-runtime-ep-api -p onnx-runtime-ep-plugin --all-targets -- -D warnings → clean

Follow-up

Limitation 2 (other kernels — MatMul, int4 — still start rayon work under an ORT session) is filed as #1145, referencing #1138 and this PR, rather than growing this PR's scope.

[justinchuby]

Reviewed the harness rather than the table, because the table is only as good as the arm that produced it. It holds up. Merging.

What convinces me

The control arm. The rayon-split arm reproduces its already-known loss (slower than serial in all six runs, matching the nine-row EPYC table). That is the strongest evidence shape available here: an arm the change provably cannot help, moving the way prior data says it should, before the headline number is read. Without it, "host-pool is 6x serial" would just be "14 threads beat 1 thread", which nobody disputed.

The stand-in is honest about what it models. I read three_arm_bench on 17ec3d5a. It is not a reimplementation of the kernel:

• workers claim indices dynamically off a cursor, as ParallelFor with num_batch = 0 does;
• the calling thread drains too (drain(total, is_worker)), because ORT runs tasks on the caller -- so the caller is one of the intra_op threads, not a spectator. Getting this wrong would have inflated the host arm by a thread;
• idle workers spin_loop(), which is the single property that makes a coexisting rayon pool oversubscribe -- i.e. the mechanism under test is present, not assumed;
• worker_helped is set by a worker, never the dispatcher, so the probe latches on real positive evidence.

Sizing workers + 1 == intra_op at 14 on a 14-physical-core box is the right call and the reasoning is stated: 15 spinners plus a caller would have starved the serial arm before rayon ever ran, which would have manufactured the result.

The margin dwarfs the spread. 5.5-6.6x against a p10..p90 of 34..62 us (host) and 187..256 us (serial). And CPU time was reported alongside wall clock, which is what makes a number taken while another agent was building on the same box readable at all.

What is still not measured, and why it cannot flip this

The pool is a stand-in, not ORT's KernelContext_ParallelFor. Real dispatch overhead, real batching behaviour under num_batch = 0, and real worker wake-up policy are all modelled rather than observed.

That is a genuine residual, and it should be said out loud rather than left implicit in the word "stand-in". But it cannot change the decision this PR turns on. The choice was between the ParallelFor machinery and the one-line "stay serial under a host" simplification. For serial to win, the real host pool would have to be more than 5.5x worse than the model -- not 20% worse, not 2x worse. Nothing about ORT's pool suggests that. The decision is robust to the modelling error by an order of magnitude, which is the only reason a modelled arm is admissible here at all.

Two things worth keeping for their own sake

MAX_HOST_CHUNKS cutting by size rather than pool width makes the chunk boundaries -- and therefore the output bit pattern -- independent of the session's intra_op. A tensor produces the same bits at 1 thread or 16. The comment now says so. That is a correctness property masquerading as a tuning constant, and it is exactly the kind of thing that gets "optimised" away by someone who reads it as tuning.

And scoping Limitation 2 out to #1145 instead of growing a 2150-line PR is the right instinct.

Verified read-only; I did not re-run the bench myself, because a second agent was building on this box and a contended re-run would be worse evidence than the interleaved one already taken, not better.