perf(cpu-ep): AVX2 kernels for LeakyRelu, HardSigmoid, ThresholdedRelu and Selu

[justinchuby]

AVX2 kernels for LeakyRelu, HardSigmoid, ThresholdedRelu and Selu

LeakyRelu, HardSigmoid, ThresholdedRelu and Selu had no slice kernel, so
they took ActivationKernel's generic path: widen into a Vec, map() a
per-element match, collect into a second Vec, copy that into the output
tensor. This PR gives each a slice kernel and routes them through the zero-copy
contiguous_f32 fast path (established in #1240), and adds a genuinely
vectorised exponential for Selu.

Stacked on #1240 (which is stacked on #1235).

---

Reviewer measurements (Intel Core i7-13800H) — where the win actually comes from

Measured by the reviewer on an Intel Core i7-13800H (14C/20T, AVX2+FMA, no
AVX-512), Windows 11. No ORT build is available on this box, so these are
ORT-independent, same-binary A/Bs, 1M f32 elements, single-thread, min-of-200 ×
9 rounds.

Two different comparisons, because they answer different questions:

1. Fast-path routing vs the generic path (what this PR changes at the call
   site). Emulating the old generic path (to_vec → map → copy_from_slice)
   against the new *_f32_slice fast path, for LeakyRelu:

   LeakyRelu, 1M, 1 thread	generic path	fast path	speedup
   min-of-200 × 9	1.8784 ms	0.1712 ms	~10.97×

   This — eliminating two allocations and two extra passes — is the dominant win
   for the three cheap ops.

2. The AVX2 kernel vs its scalar reference (isolating the hand-written SIMD
   arithmetic alone, both in the same release binary):

   op, 1M, 1 thread	scalar	AVX2	speedup
   LeakyRelu	0.1533 ms	0.1524 ms	~1.01×
   HardSigmoid	0.1599 ms	0.1583 ms	~1.01×
   ThresholdedRelu	0.1644 ms	0.1619 ms	~1.02×
   Selu	1.4301 ms	0.3391 ms	~4.22×

   For the three cheap ops the hand-written AVX2 kernel is no faster than the
   scalar loop, because LLVM already auto-vectorises a compare-and-blend in
   release mode. Selu is the exception: its exponential does not auto-vectorise,
   so the vector kernel is a real ~4.2× on the arithmetic itself.

Honest attribution: for LeakyRelu/HardSigmoid/ThresholdedRelu the speedup is
the fast-path plumbing, not the SIMD; for Selu it is both. That is worth stating
plainly so nobody credits the compare-and-blend intrinsics with a win the
allocator elimination actually produced.

Peer measurement (original PR, AMD EPYC 9V74)

Originally measured on an AMD EPYC 9V74, 16 physical cores, no AVX-512 as a
ratio ours / ORT in a shared process, at 1M/1-thread: ThresholdedRelu
11.30 → 0.93, HardSigmoid 8.62 → 0.71, LeakyRelu 7.58 → 0.65, Selu 2.56 → 0.27.
These are peers, not corrections to the reviewer's numbers: different
microarchitecture, core count, and reference (one is ours/ORT, the other is an
internal same-binary A/B). The AMD ours/ORT ratios and the i7 fast-path-vs-
generic ratio are consistent — both attribute the cheap-op gains to removing the
generic path's allocations and passes.

The PR's own note that the three cheap ops remain ~1.2–1.5× ORT at 4k (a
per-call plugin-dispatch floor, not the kernel) stands and is a good separate
follow-up.

---

Correctness — falsifiers verified by the reviewer

The PR ships two load-bearing invariants, both confirmed RED-on-break:

• HardSigmoid clamp operand order. minps/maxps return their second
  operand for an unordered input, so the value must be second or a NaN silently
  becomes 1 then 0. hard_sigmoid_special_values pins NaN→NaN.
• Selu signed-zero. ONNX's x > 0 sends -0 down the exp branch; ORT
  returns +0, so both scalar and vector spell it alpha*(exp(x)-1) (there is
  no AVX2 expm1). scalar_paths_agree_with_the_vector_paths pins scalar and
  vector to the same answer, and selu_zero_sign_does_not_depend_on_dtype pins
  it across dtypes.

Bit-identity where the op is exact (LeakyRelu, ThresholdedRelu asserted
bit-identical to their scalars over 20 003 points); HardSigmoid allows one FMA
rounding; Selu uses an alpha-scaled bound.

---

Verification (reviewer, Intel Core i7-13800H, Windows 11)

• cargo test -p onnx-runtime-ep-cpu --lib — 1415 passed / 0 failed / 17
  ignored for the activation family (all four ops' special-value,
  scalar-vs-vector bit-parity, dense-sweep and signed-zero tests green).
• One unrelated flake observed under full-suite load:
  task_runtime::pool::tests::slot_exhaustion_declines_instead_of_blocking
  (from perf(cpu): add a CPU task runtime with an adaptive-spin native pool #1201's CPU task runtime, untouched by this PR) failed once under
  contention and passes deterministically in isolation. Not introduced here.
• cargo clippy -p onnx-runtime-ep-cpu --all-targets -- -D warnings — clean.
• fix(cpu): restore map_ps's target_feature, which a merge moved onto map_bias_ps #1227's #[target_feature(enable = "avx2,fma")] confirmed intact on map_ps
  and map_bias_ps.
• MLAS remains non-default; all four kernels use dispatch! and touch no MLAS
  path.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	matmul/large_generic_f32_threads=8/32x1024x1024	4.54 ms	7.67 ms	+69.1%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	68.01 µs	112.66 µs	+65.7%
🔴	gather/large_f32_threads=1-internal/131072	34.41 µs	53.44 µs	+55.3%
🔴	matmul/small_generic_f32_threads=8/1x256x256	35.65 µs	55.17 µs	+54.8%
🔴	matmul/medium_generic_f32_threads=8/32x512x512	943.85 µs	1.41 ms	+49.6%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	86.75 µs	125.98 µs	+45.2%
🔴	matmul/medium_generic_f16_threads=1/32x512x512	27.55 µs	39.06 µs	+41.8%
🔴	add/large_f32_threads=1-internal/4194304	719.22 µs	988.77 µs	+37.5%
🔴	matmul/small_generic_f16_threads=8/1x256x256	32.03 µs	41.89 µs	+30.8%
⚠️	gather/large_f16_threads=1-internal/131072	12.34 µs	15.63 µs	+26.7%
⚠️	matmul/medium_generic_f16_threads=8/32x512x512	31.58 µs	39.87 µs	+26.2%
⚠️	gather/large_bf16_threads=1-internal/131072	12.49 µs	15.58 µs	+24.8%
⚠️	matmul/large_generic_f16_threads=1/32x1024x1024	85.78 µs	106.34 µs	+24.0%
⚠️	matmul/small_generic_bf16_threads=1/1x256x256	34.74 µs	42.95 µs	+23.6%
⚠️	matmul/large_generic_bf16_threads=8/32x1024x1024	1.40 ms	1.71 ms	+22.4%
⚠️	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	1.00 ms	1.22 ms	+22.1%
⚠️	matmul/medium_generic_f32_threads=1/32x512x512	2.24 ms	2.67 ms	+19.0%
⚠️	matmul/small_generic_bf16_threads=8/1x256x256	32.54 µs	37.68 µs	+15.8%
✅	add/medium_f16_threads=1-internal/262144	109.58 µs	125.70 µs	+14.7%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.12 ms	2.40 ms	+13.0%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.05 ms	6.81 ms	+12.6%
✅	add/large_f16_threads=1-internal/4194304	1.73 ms	1.94 ms	+12.4%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	653.39 µs	728.33 µs	+11.5%
✅	add/large_bf16_threads=1-internal/4194304	1.77 ms	1.94 ms	+9.9%
✅	gather/medium_f32_threads=1-internal/32768	4.16 µs	4.56 µs	+9.8%
✅	grammar_masking/llguidance_compute_mask/32	73.52 µs	80.61 µs	+9.6%
✅	sampling_latency/min_p_per_token	226.91 µs	248.72 µs	+9.6%
✅	add/medium_bf16_threads=1-internal/262144	111.63 µs	122.06 µs	+9.3%
✅	gather/small_bf16_threads=1-internal/4096	457.9 ns	495.5 ns	+8.2%
✅	reduce_mean/small_f32_threads=1-internal/4096	15.62 µs	16.67 µs	+6.7%
✅	reduce_mean/medium_f32_threads=1-internal/65536	268.41 µs	285.31 µs	+6.3%
✅	qwen3_sampling_processors/top_k_top_p_fast	675.74 µs	715.60 µs	+5.9%
✅	matmul/small_generic_f32_threads=1/1x256x256	38.48 µs	40.54 µs	+5.4%
✅	add/medium_f32_threads=1-internal/262144	25.69 µs	26.73 µs	+4.0%
✅	matmul/small_generic_f16_threads=1/1x256x256	37.21 µs	38.68 µs	+3.9%
✅	gather/medium_f16_threads=1-internal/32768	2.60 µs	2.71 µs	+3.9%
✅	tokenization/encode_tokens_per_second	382.74 µs	397.41 µs	+3.8%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	96.20 µs	99.84 µs	+3.8%
✅	logit_processing/seven_processor_chain_per_step	329.98 µs	341.23 µs	+3.4%
✅	kv_cache/alloc_dealloc_pages	39.89 µs	41.08 µs	+3.0%
✅	qwen3_sampling_processors/top_k_partial_selection	152.05 µs	153.89 µs	+1.2%
✅	add/small_f16_threads=1-internal/1024	464.3 ns	469.2 ns	+1.0%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.12 ms	1.12 ms	-0.2%
✅	sampling_latency/greedy_per_token	3.33 µs	3.29 µs	-1.3%
✅	gather/medium_bf16_threads=1-internal/32768	3.01 µs	2.93 µs	-2.8%
✅	sampling_latency/top_k_per_token	59.01 µs	55.66 µs	-5.7%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	11.27 ms	10.25 ms	-9.0%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	463.40 µs	416.63 µs	-10.1%
✅	add/small_bf16_threads=1-internal/1024	464.9 ns	416.6 ns	-10.4%
✅	tokenization/decode_tokens_per_second	6.82 ms	6.10 ms	-10.6%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	598.05 µs	530.56 µs	-11.3%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.18 ms	1.92 ms	-11.8%
✅	gather/small_f16_threads=1-internal/4096	580.0 ns	511.2 ns	-11.9%
✅	add/small_f32_threads=1-internal/1024	208.7 ns	181.1 ns	-13.2%
✅	gather/small_f32_threads=1-internal/4096	866.2 ns	745.7 ns	-13.9%
✅	sampling_latency/top_p_per_token	492.23 µs	421.86 µs	-14.3%
🟢	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	4.48 ms	3.71 ms	-17.2%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	895.85 µs	640.52 µs	-28.5%
🟢	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	111.68 µs	77.33 µs	-30.8%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.33 3.69 5.66 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

Reviewer verification — validated and merging

Reproduced on Intel Core i7-13800H (14C/20T, AVX2+FMA, no AVX-512), Windows 11; ORT-independent same-binary A/Bs (no ORT build on this box).

Where the win comes from (attributed honestly):

• Fast-path routing vs generic path (LeakyRelu, 1M, 1t, min-of-200 × 9): generic 1.8784 ms → fast path 0.1712 ms (~10.97×). This — eliminating two allocations and two passes — is the dominant win for the three cheap ops.
• AVX2 kernel vs scalar reference (isolating the SIMD arithmetic): LeakyRelu ~1.01×, HardSigmoid ~1.01×, ThresholdedRelu ~1.02× — no kernel-level speedup, because LLVM already auto-vectorizes a compare-and-blend. Selu 1.4301 → 0.3391 ms (~4.22×) is a real win because its exponential does not auto-vectorize.

So for LeakyRelu/HardSigmoid/ThresholdedRelu the speedup is the plumbing, not the intrinsics; for Selu it is both. The PR's original AMD EPYC 9V74 ours/ORT ratios (11.30→0.93, 8.62→0.71, 7.58→0.65, 2.56→0.27) are peers to these, not corrections — different microarch/cores/reference, and consistent in attributing the cheap-op gains to the fast path.

Correctness falsifiers confirmed load-bearing: HardSigmoid clamp operand order pins NaN→NaN; Selu -0 → +0 matches ORT and scalar/vector agree (selu_zero_sign_does_not_depend_on_dtype). Bit-identity held for the exact ops.

Suite: lib activation tests 1415 passed / 0 failed / 17 ignored; clippy -D warnings clean; #1227 target_feature intact; MLAS untouched. One unrelated pre-existing flake (task_runtime::pool::slot_exhaustion_declines_instead_of_blocking, from #1201, passes in isolation) — not introduced here.

Merging via squash.

[codecov]

Codecov Report

❌ Patch coverage is 98.63946% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.70%. Comparing base (cd108de) to head (2c5e84d).
⚠️ Report is 39 commits behind head on main.

Files with missing lines	Patch %	Lines
...tes/onnx-runtime-ep-cpu/src/kernels/activations.rs	95.34%	2 Missing ⚠️
...nnx-runtime-ep-cpu/src/kernels/simd_activations.rs	99.20%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           main    #1243       +/-   ##
=========================================
+ Coverage      0   80.70%   +80.70%     
=========================================
  Files         0      363      +363     
  Lines         0   160682   +160682     
  Branches      0   160682   +160682     
=========================================
+ Hits          0   129680   +129680     
- Misses        0    26345    +26345     
- Partials      0     4657     +4657     

Flag	Coverage Δ
mlas	85.09% <ø> (?)
offline	80.62% <98.63%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...tes/onnx-runtime-ep-cpu/src/kernels/activations.rs	92.87% <95.34%> (ø)
...nnx-runtime-ep-cpu/src/kernels/simd_activations.rs	90.86% <99.20%> (ø)

... and 361 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.