fix(cpu): undo the 2.3x unary regression #1130 shipped

[justinchuby]

What happened

#1130 (mine) wrapped the Clip MLAS call in run_chunked so it would use the
thread pool. run_chunked is generic, so doing that from selection.rs created a new
instantiation in a module that had never had one.

The runtime path of every other unary op was untouched — not one instruction — and all
1332 tests still passed. But the crate's codegen units repartitioned and the AVX2 unary
kernels in simd_activations.rs stopped being vectorised.

This shipped. It was found while collecting numbers for the final report: Sqrt had
gone from beating ORT by 1.9× to losing at 0.7×, and the ORT-relative position of five
ops had collapsed in a way no code change explained.

Evidence

Bisected by rebuilding main with one file at a time reverted to 1f1ce4b74
(the commit before #1130). n = 65536, 1 thread, taskset -c 8-23, 4 interleaved
rounds, µs p50. Only selection.rs matters — reverting relu.rs or conv.rs
alone restored nothing:

op	main (#1130)	revert relu.rs	revert selection.rs	revert conv.rs	pre-#1130
Sqrt	48.5	48.4	21.3	44.6	21.5
Tanh	55.2	57.0	31.6	60.0	31.8
Sigmoid	56.9	55.6	31.0	57.1	31.1
QuickGelu	64.5	64.5	42.5	64.2	42.5
FastGelu	79.6	79.3	55.5	79.3	55.8
Erf	62.2	62.4	62.2	62.2	62.5
Relu	21.3	26.5	22.6	23.0	20.4

Reproduced independently by building the same commit in a second worktree, so it is not
a build-directory artefact. Adding #[inline] to run_chunked did not help, which
rules out a plain inlining decision and points at codegen-unit partitioning.

The diagnosis, confirmed mechanically

If the cause really is codegen-unit partitioning, then forcing the crate into a single
codegen unit should erase the regression with no source change at all. It does.
CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1 on the unfixed commit 34095af0f, same
machine, 4 interleaved rounds, µs p50:

op	n	main, default CGUs	main, codegen-units=1	this PR, default CGUs
Sqrt	64 Ki	48.3	21.6	21.2
Sqrt	1 Mi	667.4	261.4	274.4
Sigmoid	1 Mi	813.3	504.8	441.6
QuickGelu	1 Mi	929.4	575.8	573.0
FastGelu	1 Mi	1171.2	788.2	808.5

So the diagnosis is not inferred from a bisect alone — the proposed mechanism, applied
directly, reproduces the cure.

codegen-units = 1 (or LTO) in the release profile would remove this whole fragility
class permanently, and is the more durable answer. It is deliberately not in this PR:
it is a workspace-wide build-policy change that affects every crate and every
contributor's build time (the plugin alone went 14 s to 63 s here), it would need its own
measurement across the whole EP rather than the activation kernels, and it does not
belong in a regression fix. Filed as the follow-up this PR's limitation section points
at. The source fix costs nothing and is independent of it.

The fix

run_chunked is private to simd_activations.rs again — the compiler now enforces the
rule, not a convention. Callers elsewhere go through one of two entry points that are
instantiated in that module:

• run_chunked_fn(input, output, body: fn(&[f32], &mut [f32])) — deliberately a fn
  pointer, not impl Fn, so every caller shares one instantiation. Used by Relu and
  SiLU.
• clip_chunked(input, output, min, max) — Clip needs captured bounds. It takes the
  serial decision itself so the short case is a direct call rather than one through a
  closure the optimiser can no longer see into; without that, Clip itself paid 12%.

Result

n = 65536 and 1 Mi, 1 thread, 5 interleaved rounds, µs p50:

op	n	main (#1130)	this PR	pre-#1130
Sqrt	64 Ki	48.5	21.4	21.7
Sqrt	1 Mi	667.4	259.2	275.1
Tanh	1 Mi	776.1	437.8	436.4
Sigmoid	1 Mi	807.6	440.1	502.2
QuickGelu	1 Mi	923.1	574.7	607.0
FastGelu	1 Mi	1166.9	785.3	788.1
Clip	1 Mi	268.7	267.5	267.4
Swish	1 Mi	728.8	720.8	1385.6

Everything is back to its pre-#1130 level and #1130's own win is kept: Swish is
still 1.92× faster than before #1130, and Clip/Relu still reach the pool at
≥ PAR_MIN_LEN.

Regression guard

chunking_instantiation_is_local::no_module_outside_this_one_instantiates_run_chunked
walks the crate source and fails if any module other than simd_activations.rs
instantiates run_chunked, naming the offending file and line.

Verified to falsify: re-widening the visibility and pointing relu.rs:144 back at
run_chunked makes it fail with
Offending call sites: ["…/kernels/relu.rs:144"].

This class of bug produces no wrong answers, no test failures and no diff in the file
that slows down, so a mechanical guard is the only thing that catches it.

Tests

• cargo test -p onnx-runtime-ep-cpu --features mlas --lib → 1332 passed, 0 failed
• cargo test -p onnx-runtime-ep-cpu --lib → 1305 passed, 0 failed
• cargo fmt --all clean.
• No numerical change: clip_chunked's serial branch calls exactly the function the
  closure called, and run_chunked_fn forwards unchanged.

Limitations

• The mechanism is codegen-unit partitioning, which the compiler makes no promises
  about. The guard encodes the rule that was measured to work on this toolchain; it
  cannot prove the next refactor is safe. That is why the guard names the symptom and
  the measured cost in its failure message.
• Measured on one machine (AMD EPYC 9V74, AVX2/FMA/F16C, taskset-pinned, 5 rounds).
  The direction is unambiguous — up to 2.3× on Sqrt and roughly 1.5–1.8× on the
  other four — but the exact figures are not portable.

[codecov]

Codecov Report

❌ Patch coverage is 90.90909% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.94%. Comparing base (34095af) to head (34a50b2).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
...nnx-runtime-ep-cpu/src/kernels/simd_activations.rs	90.90%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #1136    +/-   ##
========================================
  Coverage   79.94%   79.94%            
========================================
  Files         368      368            
  Lines      160611   160826   +215     
  Branches   160611   160826   +215     
========================================
+ Hits       128395   128579   +184     
- Misses      27499    27525    +26     
- Partials     4717     4722     +5     

Flag	Coverage Δ
cli-ort-linux	83.79% <ø> (ø)
cli-ort-windows	83.40% <ø> (+0.09%)	⬆️
mlas	85.64% <ø> (-0.10%)	⬇️
offline	79.69% <90.90%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...tes/onnx-runtime-ep-cpu/src/kernels/activations.rs	91.71% <ø> (-0.21%)	⬇️
crates/onnx-runtime-ep-cpu/src/kernels/relu.rs	90.74% <ø> (ø)
...rates/onnx-runtime-ep-cpu/src/kernels/selection.rs	82.64% <ø> (ø)
...nnx-runtime-ep-cpu/src/kernels/simd_activations.rs	97.11% <90.90%> (-0.12%)	⬇️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	58.43 µs	106.53 µs	+82.3%
🔴	grammar_masking/llguidance_compute_mask/32	83.32 µs	117.59 µs	+41.1%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	83.73 µs	117.19 µs	+40.0%
🔴	gather/small_f16_threads=1-internal/4096	518.5 ns	697.3 ns	+34.5%
🔴	matmul/medium_generic_f32_threads=8/32x512x512	1.29 ms	1.71 ms	+33.0%
⚠️	matmul/large_generic_f32_threads=8/32x1024x1024	5.05 ms	6.52 ms	+29.0%
⚠️	sampling_latency/min_p_per_token	214.97 µs	268.08 µs	+24.7%
⚠️	gather/large_f16_threads=1-internal/131072	16.49 µs	20.32 µs	+23.3%
⚠️	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	777.68 µs	956.81 µs	+23.0%
⚠️	matmul/large_generic_f32_threads=1/32x1024x1024	9.02 ms	11.02 ms	+22.1%
⚠️	sampling_latency/top_p_per_token	392.85 µs	471.22 µs	+20.0%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	585.79 µs	667.19 µs	+13.9%
✅	reduce_mean/small_f32_threads=1-internal/4096	16.06 µs	18.05 µs	+12.4%
✅	gather/large_f32_threads=1-internal/131072	39.44 µs	44.26 µs	+12.2%
✅	reduce_mean/medium_f32_threads=1-internal/65536	259.42 µs	289.21 µs	+11.5%
✅	sampling_latency/top_k_per_token	52.83 µs	58.46 µs	+10.7%
✅	matmul/medium_generic_f16_threads=8/32x512x512	49.85 µs	54.80 µs	+9.9%
✅	add/large_bf16_threads=1-internal/4194304	1.78 ms	1.95 ms	+9.8%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.82 ms	4.17 ms	+9.2%
✅	qwen3_sampling_processors/top_k_partial_selection	154.71 µs	162.11 µs	+4.8%
✅	logit_processing/seven_processor_chain_per_step	331.32 µs	344.83 µs	+4.1%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	88.22 µs	90.80 µs	+2.9%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.20 ms	2.24 ms	+2.0%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	522.81 µs	533.33 µs	+2.0%
✅	matmul/small_generic_f16_threads=1/1x256x256	36.61 µs	37.23 µs	+1.7%
✅	matmul/small_generic_f32_threads=8/1x256x256	55.59 µs	56.27 µs	+1.2%
✅	gather/small_bf16_threads=1-internal/4096	520.2 ns	517.4 ns	-0.5%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	187.20 µs	186.17 µs	-0.6%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	508.18 µs	501.66 µs	-1.3%
✅	add/medium_f16_threads=1-internal/262144	111.99 µs	109.75 µs	-2.0%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	606.50 µs	594.30 µs	-2.0%
✅	add/small_bf16_threads=1-internal/1024	479.9 ns	469.1 ns	-2.2%
✅	gather/small_f32_threads=1-internal/4096	761.4 ns	743.3 ns	-2.4%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.33 ms	6.11 ms	-3.5%
✅	qwen3_sampling_processors/top_k_top_p_fast	724.37 µs	690.59 µs	-4.7%
✅	matmul/small_generic_f16_threads=8/1x256x256	35.48 µs	33.54 µs	-5.5%
✅	gather/medium_f32_threads=1-internal/32768	4.19 µs	3.95 µs	-5.9%
✅	tokenization/encode_tokens_per_second	416.90 µs	390.31 µs	-6.4%
✅	sampling_latency/greedy_per_token	3.44 µs	3.21 µs	-6.6%
✅	matmul/medium_generic_f16_threads=1/32x512x512	39.18 µs	36.32 µs	-7.3%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.29 ms	2.12 ms	-7.7%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.13 ms	1.03 ms	-8.2%
✅	gather/medium_bf16_threads=1-internal/32768	2.91 µs	2.66 µs	-8.5%
✅	gather/large_bf16_threads=1-internal/131072	18.24 µs	16.58 µs	-9.1%
✅	add/large_f32_threads=1-internal/4194304	675.98 µs	614.22 µs	-9.1%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	106.64 µs	96.69 µs	-9.3%
✅	add/medium_f32_threads=1-internal/262144	27.68 µs	25.03 µs	-9.6%
✅	add/large_f16_threads=1-internal/4194304	1.77 ms	1.57 ms	-11.5%
✅	add/small_f16_threads=1-internal/1024	530.5 ns	468.9 ns	-11.6%
✅	matmul/small_generic_f32_threads=1/1x256x256	45.66 µs	40.25 µs	-11.9%
✅	matmul/small_generic_bf16_threads=8/1x256x256	48.20 µs	42.47 µs	-11.9%
✅	kv_cache/alloc_dealloc_pages	47.29 µs	40.82 µs	-13.7%
✅	tokenization/decode_tokens_per_second	6.72 ms	5.76 ms	-14.3%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	45.42 µs	37.56 µs	-17.3%
🟢	matmul/large_generic_bf16_threads=8/32x1024x1024	2.10 ms	1.67 ms	-20.5%
🟢	add/medium_bf16_threads=1-internal/262144	137.35 µs	108.38 µs	-21.1%
🟢	add/small_f32_threads=1-internal/1024	277.7 ns	189.4 ns	-31.8%
🟢	matmul/medium_generic_f32_threads=1/32x512x512	3.54 ms	2.37 ms	-32.9%
🟢	gather/medium_f16_threads=1-internal/32768	3.98 µs	2.47 µs	-37.9%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 8.11 4.38 5.87 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)