fix(cpu): restore map_ps's target_feature, which a merge moved onto map_bias_ps by justinchuby · Pull Request #1227 · justinchuby/onnx-genai

justinchuby · 2026-08-18T06:39:38Z

What this is

map_ps is the loop driver behind every unary activation on AVX2. A merge
(the #1037 / #1074 layering) inserted map_bias_ps between map_ps's
doc comment and map_ps itself, so the comment and both of its attributes
ended up on the wrong function:

/// Apply an 8-lane kernel across a slice. ...   <- map_ps's doc
#[inline]
#[target_feature(enable = "avx2,fma")]           <- map_ps's attributes
/// Like [`map_ps`], but adds a bias row ...
pub(super) unsafe fn map_bias_ps(...)            <- ...on map_bias_ps

pub(super) unsafe fn map_ps(...)                 <- bare

map_bias_ps needs those attributes too, so nothing was broken outright —
but map_ps was left with no #[inline] and no avx2,fma.

Why the missing `target_feature` is a hazard

LLVM will not inline a callee that requires a feature its caller lacks (a
strict superset; equal feature sets inline fine). Every kernel closure map_ps takes (erf_ps, tanh_ps,
sigmoid_ps, ...) is avx2,fma. With map_ps compiled at baseline features,
those closures are not inlinable into its loop.

It is harmless today only by luck of ordering: every caller
(tanh_avx2, erf_avx2, ...) is itself avx2,fma, so map_ps gets inlined
upwards into the caller first, and once there the closure folds in. That is
an inlining decision, not a guarantee. Grow map_ps past the inline threshold
— which is exactly what an unroll experiment does — and the closure becomes a
real call per 8 elements, and a kernel like erf_ps has to re-materialise all
of its constants on every one of those calls.

I hit this while unrolling map_ps, which is how it surfaced.

Evidence that this changes nothing today

Disassembled tanh_avx2 out of the built rlib before and after:

diff <(objdump -d ... base) <(objdump -d ... fixed)   ->  identical

Byte-identical, and erf_avx2 / sigmoid_avx2 keep their instruction counts
(216 / 117). No call to any kernel remains in the lib. So this is a
latent-hazard and documentation fix with zero codegen delta — not a perf
change, and it needs no A/B.

Also recorded here: a negative result on the divide

While looking at these loops I measured one algorithmic change and it lost, so
it is written down rather than left for someone to retry.

tanh_ps and sigmoid_ps each end in _mm256_div_ps. vdivps ymm is the
only instruction in either loop that is not fully pipelined (Zen 4 retires one
about every 4.5 cycles), so replacing it with vrcpps + one Newton-Raphson
step — y1 = y0 + y0*(1 - q*y0), four pipelined uops for one blocking one —
looked like the obvious win.

It is not:

case	ORT p50 (control)	divide	rcp + NR
`bench_tanh_f32_4k`	0.0031 ms, all 6 runs	1.518 / 1.525 / 1.526	1.550 / 1.570 / 1.570

Interleaved rebuild-and-alternate, 3 rounds, 1 thread, taskset -c 8-15.
The 4k case is the only trustworthy one here — it is L1-resident and ORT's own
p50 was identical to four decimal places across all six runs, whereas on the
1M cases ORT's p50 moved 4.6x mid-session and neither arm is quotable.

Consistent ~3% regression. The rcp + 2 FMA dependency chain is about as
long as the divide it replaces, so a latency-bound kernel gains nothing and
just pays two extra uops.

It also breaks four exactness proofs that #1121 landed: tanh(±Inf) comes back
as 0.99999994 instead of exactly ±1.0, because the input clamp makes p
and q bit-equal at |v| = 9 and only an exact divide turns that into exactly
1.0. Buying that back would mean restoring the saturation blend #1121 proved
redundant over 1.05 billion inputs — to fund a change that is already slower.
Not pursued. No fallback involved; the kernel keeps its own divide.

Validation

cargo test --release -p onnx-runtime-ep-cpu --lib — 1337 passed, 0 failed
cargo clippy --release -p onnx-runtime-ep-cpu --all-targets — clean
cargo fmt --all --check — clean
tanh_avx2 disassembly identical to main

Review

Opus review: no blockers, no should-fixes.

It independently confirmed the three things the PR rests on: every one of the
16 map_ps and 2 map_bias_ps call sites is already inside a
#[target_feature(enable = "avx2,fma")] function (so nothing becomes unsound
or fails to compile), map_bias_ps keeps exactly the attributes it had, and
there are no bare/function-pointer references to either.

It also did the scan I most wanted: all ~130 fn definitions in the file were
checked for the same orphaned-doc-comment-after-attribute signature that caused
this bug. Zero other anomalies — every *_ps helper and every *_avx2
dispatcher carries the attributes it should. So this is the only instance.

NIT (applied) — the doc comment said LLVM won't inline a callee whose
features are "a superset" of the caller's. Read non-strictly that would
forbid inlining at equal feature sets, which is the opposite of what the
fix relies on. Reworded to "requires a feature its caller lacks — a strict
superset; equal feature sets inline fine".

A merge left the 'Apply an 8-lane kernel across a slice' doc comment and both of its attributes attached to map_bias_ps, which was inserted between them and their function. map_ps was left with no #[inline] and, more importantly, no #[target_feature(enable = "avx2,fma")]. That matters because LLVM will not inline a callee whose feature set is a superset of its caller's: without avx2,fma on map_ps, the kernel closures cannot be inlined into its loop. It is harmless today only because every caller is itself avx2,fma, so map_ps inlines upwards into the caller and the closure folds in afterwards -- an inlining decision, not a guarantee. Grow map_ps past the threshold and every activation takes a real call per 8 elements. Codegen is byte-identical before and after (verified by diffing the disassembly of tanh_avx2), so this is a latent-hazard fix, not a perf change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…two docs Opus review found one real defect and three smaller ones. MUST-FIX -- `celu_scalar` destroyed NaN. Rust's `f32::max`/`min` are IEEE `maxNum`/`minNum`, so they return the *other* operand for NaN: transcribing ONNX's `max(0,x) + min(0, alpha*(exp(x/alpha)-1))` literally collapses both terms to zero and answers `0.0`. The AVX2 path was deliberately written to propagate NaN, and ORT propagates it too, so the result depended on whether the slice was long enough for the vector path and on whether the host had AVX2 -- which is the one thing `dispatch!` exists to prevent. Reachable by any tensor under `SIMD_MIN_LEN`, by every `Float64` tensor (which never touches SIMD), and by any non-AVX2 host. Guarded in `celu_scalar`, and `Activation::apply` now delegates to it instead of keeping a second copy of the formula; `apply_f64` carries the same guard. Two new tests pin it, both verified to fail without the fix: `celu_scalar_path_agrees_with_the_vector_path` runs the same specials through a slice too short for SIMD and a padded one, and `celu_and_mish_propagate_nan_on_every_path` covers f32 and f64 `apply`. The existing special-value tests could not catch this: `special_inputs()` pads to `SIMD_MIN_LEN`, so all of them take the vector path. SHOULD-FIX -- `mod log_c` landed between `mod exp_c`'s doc comment and `mod exp_c`, so `exp_c` lost its `#[cfg(target_arch = "x86_64")]` and its doc described the wrong module. Same shape as the `map_ps` defect in #1227, in the same file, found the same way. Each module now carries its own doc and its own gate. SHOULD-FIX -- added `Celu`/`Mish` to the `OWNED` list in `activation_and_norm_ops_clear_every_capability_filter`, which checks the dtype filter as well as the shape filter. Its comment said to add them the moment a kernel landed; it has. NIT -- the four new kernels had been inserted into the middle of `exp_full_ps`'s doc comment, splitting it mid-sentence. Moved below it. Also refreshed `CPU_ACTIVATION_GAPS.md`: "activations with no kernel at all" is now empty, and `Celu`/`Mish`/`Log` moved to the closed table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-08-18T13:10:20Z

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`grammar_masking/llguidance_compute_mask/32`	77.27 µs	138.36 µs	+79.1%
🔴	`logit_processing/seven_processor_chain_per_step`	324.34 µs	427.71 µs	+31.9%
⚠️	`gather/large_f16_threads=1-internal/131072`	14.73 µs	18.95 µs	+28.7%
⚠️	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	5.91 ms	7.50 ms	+27.0%
⚠️	`kv_cache/alloc_dealloc_pages`	40.98 µs	51.47 µs	+25.6%
⚠️	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	3.78 ms	4.71 ms	+24.7%
⚠️	`qwen3_sampling_processors/top_k_top_p_fast`	621.42 µs	772.22 µs	+24.3%
⚠️	`gather/large_bf16_threads=1-internal/131072`	15.78 µs	18.79 µs	+19.1%
⚠️	`qwen3_sampling_processors/top_p_fast_after_top_k`	535.10 µs	620.68 µs	+16.0%
✅	`qwen3_sampling_processors/top_k_partial_selection`	156.45 µs	175.90 µs	+12.4%
✅	`gather/medium_bf16_threads=1-internal/32768`	2.39 µs	2.68 µs	+12.1%
✅	`gather/small_f32_threads=1-internal/4096`	701.0 ns	767.1 ns	+9.4%
✅	`tokenization/encode_tokens_per_second`	387.51 µs	419.48 µs	+8.2%
✅	`gather/large_f32_threads=1-internal/131072`	42.64 µs	45.03 µs	+5.6%
✅	`add/large_f32_threads=1-internal/4194304`	822.44 µs	868.48 µs	+5.6%
✅	`add/medium_bf16_threads=1-internal/262144`	112.85 µs	118.95 µs	+5.4%
✅	`matmul/small_generic_f16_threads=8/1x256x256`	38.85 µs	39.96 µs	+2.9%
✅	`gather/medium_f16_threads=1-internal/32768`	2.52 µs	2.55 µs	+1.2%
✅	`tokenization/decode_tokens_per_second`	6.60 ms	6.67 ms	+1.1%
✅	`matmul/medium_generic_f32_threads=1/32x512x512`	2.45 ms	2.47 ms	+0.9%
✅	`matmul/small_generic_f16_threads=1/1x256x256`	37.65 µs	37.47 µs	-0.5%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	43.84 µs	43.63 µs	-0.5%
✅	`reduce_mean/medium_f32_threads=1-internal/65536`	252.70 µs	251.06 µs	-0.6%
✅	`reduce_mean/small_f32_threads=1-internal/4096`	16.18 µs	15.88 µs	-1.9%
✅	`sampling_latency/greedy_per_token`	3.27 µs	3.17 µs	-3.0%
✅	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	198.64 µs	189.53 µs	-4.6%
✅	`sampling_latency/top_k_per_token`	56.68 µs	53.18 µs	-6.2%
✅	`matmul/small_generic_f32_threads=8/1x256x256`	50.84 µs	46.98 µs	-7.6%
✅	`reduce_mean/large_f32_threads=1-internal/262144`	1.07 ms	962.07 µs	-10.5%
✅	`sampling_latency/top_p_per_token`	450.04 µs	401.88 µs	-10.7%
✅	`matmul/small_generic_bf16_threads=1/1x256x256`	36.47 µs	32.30 µs	-11.4%
✅	`matmul/medium_generic_f16_threads=1/32x512x512`	40.21 µs	34.82 µs	-13.4%
✅	`add/medium_f16_threads=1-internal/262144`	145.87 µs	126.03 µs	-13.6%
✅	`add/large_bf16_threads=1-internal/4194304`	2.07 ms	1.78 ms	-14.0%
🟢	`matmul/small_generic_bf16_threads=8/1x256x256`	41.40 µs	34.97 µs	-15.5%
🟢	`gather/small_f16_threads=1-internal/4096`	619.1 ns	511.9 ns	-17.3%
🟢	`gather/medium_f32_threads=1-internal/32768`	4.91 µs	3.97 µs	-19.1%
🟢	`gather/small_bf16_threads=1-internal/4096`	614.8 ns	492.9 ns	-19.8%
🟢	`matmul/medium_generic_f32_threads=8/32x512x512`	1.72 ms	1.35 ms	-21.3%
🟢	`sampling_latency/min_p_per_token`	260.61 µs	203.74 µs	-21.8%
🟢	`matmul/large_generic_f16_threads=1/32x1024x1024`	98.70 µs	74.69 µs	-24.3%
🟢	`add/medium_f32_threads=1-internal/262144`	36.74 µs	27.69 µs	-24.6%
🟢	`add/small_bf16_threads=1-internal/1024`	677.7 ns	503.7 ns	-25.7%
🟢	`matmul/large_generic_f32_threads=1/32x1024x1024`	11.77 ms	8.71 ms	-26.0%
🟢	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.85 ms	2.09 ms	-26.4%
🟢	`matmul/medium_generic_f16_threads=8/32x512x512`	51.56 µs	37.73 µs	-26.8%
🟢	`matmul/medium_generic_bf16_threads=1/32x512x512`	731.29 µs	524.42 µs	-28.3%
🟢	`matmul/large_generic_bf16_threads=1/32x1024x1024`	2.70 ms	1.93 ms	-28.8%
🟢	`add/small_f16_threads=1-internal/1024`	690.6 ns	488.6 ns	-29.3%
🟢	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	689.28 µs	482.95 µs	-29.9%
🟢	`matmul/large_generic_f16_threads=8/32x1024x1024`	115.38 µs	80.58 µs	-30.2%
🟢	`add/small_f32_threads=1-internal/1024`	316.7 ns	214.1 ns	-32.4%
🟢	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	107.34 µs	72.45 µs	-32.5%
🟢	`add/large_f16_threads=1-internal/4194304`	2.73 ms	1.80 ms	-34.0%
🟢	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	1.38 ms	822.17 µs	-40.3%
🟢	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	135.88 µs	80.16 µs	-41.0%
🟢	`matmul/large_generic_f32_threads=8/32x1024x1024`	6.80 ms	3.84 ms	-43.5%
🟢	`matmul/large_generic_bf16_threads=8/32x1024x1024`	2.34 ms	1.25 ms	-46.7%
🟢	`matmul/medium_generic_bf16_threads=8/32x512x512`	901.39 µs	351.58 µs	-61.0%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 7.01 4.97 6.70 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

justinchuby · 2026-08-18T14:17:25Z

Verified locally and merged.

The bug is real and I confirmed it on main rather than taking the description's word for it. At simd_activations.rs:1770-1777, map_bias_ps currently carries map_ps's doc comment and its #[inline] / #[target_feature(enable = "avx2,fma")], with its own doc comment sitting oddly after the attributes — while map_ps itself (line 1816) has no attributes at all. A merge clearly slid the attribute block down one function. This PR puts them back and gives each function its own doc.

On the impact, I want to be precise rather than generous. The PR body is right that target_feature is load-bearing for inlining — LLVM will not inline a callee requiring a feature its caller lacks — and it is also honest that the effect is currently masked, because every caller is itself avx2,fma, so map_ps is inlined upward into the caller and the closure folds in afterwards. So I did not measure a throughput delta from this change, and I am not claiming one. What it fixes is a latent cliff: grow map_ps past the inline threshold and the kernel closure becomes a real call per 8 elements, which for something like erf_ps would also re-materialise its constants every call. Restoring the attribute is cheap insurance against a regression that would be very annoying to attribute later.

Verification: cargo test -p onnx-runtime-ep-cpu --lib → 1339 passed / 0 failed / 16 ignored (61s). Clippy clean. The diff is one file, +19/-5, entirely doc comment and attribute placement — no logic change.

Rebased onto current main to preserve #1227 (map_ps's target_feature), which the original branch predated. Only the erf big-branch R polynomial and the exp polynomial are converted to Estrin's scheme; the map_ps / map_bias_ps attribute arrangement from #1227 is left intact. Co-authored-by: resch <resch@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… (1.66x -> 1.36x vs ORT) (#1218) ## What this changes `erf_ps`'s big-branch `R` polynomial and the `exp_ps` polynomial it feeds are switched from Horner's rule to **Estrin's scheme**. Horner is a six-deep dependent FMA chain; Estrin regroups it as `a^3·(c0 a^3 + c1 a^2 + c2 a + c3) + (c4 a^2 + c5 a + c6)`, three deep for two extra multiplies, whose halves the out-of-order engine runs in parallel. Only the two chains actually on `erf`'s critical path are converted; the small branch stays Horner (it runs in parallel and is off-path). Rebased onto current `main` so it **preserves #1227** (`map_ps`'s `#[target_feature(enable = "avx2,fma")]`). The original branch predated #1227 and its raw diff would have moved that attribute back onto `map_bias_ps`; that hunk is dropped. Only `simd_activations.rs`'s erf/exp polynomials and one accuracy test change here. ## Numerics — measured, not assumed (reviewer) Estrin reassociates f32 FMAs, so it is **not** bit-identical to Horner by construction. The squad reviewer measured the actual error with a two-build dump-and-diff (base Horner build vs this Estrin build, identical input grid of 404,045 points spanning `[-6,6]` densely, the `0.921875` split boundary, large `|x|`, denormals, and NaN/±inf), on **Intel Core i7-13800H, Windows 11**: - **Estrin vs Horner: max difference = exactly 1 ULP.** 4,401 of 404,043 finite points (1.09%) differ, every one of them by a single ULP; max absolute diff 5.96e-8. Nothing moves by more than one ULP anywhere in the domain. - **vs f64 `erf` truth:** max |Horner − erf| = 5.77e-8, max |Estrin − erf| = 6.55e-8 (worst at x≈0.933, just past the split). Both within ~1 ULP of the correctly rounded result; Estrin is at most ~0.8e-8 looser than Horner and never worse than 1 ULP from it. - **Special values are all correct and identical between the two:** large `|x|` (7,8,10,20,50,100,1000,1e30,f32::MAX) saturate to ±1.0; denormals produce bit-identical tiny outputs; NaN → NaN; +inf → +1.0; −inf → −1.0. This is a conscious, documented 1-ULP change, not a hand-wave. The accuracy gate `erf_reassociation_costs_no_accuracy` (worst scaled error pinned at 5.97e-8 = 2^-24 over `[-6,6]`) **passes** on this box, so a future regrouping that widens the envelope fails instead of being absorbed by the looser `ERF_BOUND`. ## Performance — measured on this box, ORT-independent The original body quoted "1.66× → 1.36× vs ORT" (≈ −17.7% kernel time). I could not reproduce that specific figure: no pinned ORT reference on this Windows box, and it is different hardware. Instead I timed `erf_avx2` directly, same machine, two builds, single-thread, min-of-200 × 9 rounds over a 1,048,576-element mixed-range buffer: | build | per-round min (ms) | best | | --- | --- | ---: | | Horner (base) | 0.6346–0.6348 (8/9 rounds) | 0.6346 | | **Estrin (this PR)** | **0.6042–0.6043 (8/9 rounds)** | **0.6042** | The ranges do not overlap, so the direction is solid: **Estrin is ~5.0% faster (1.05×)** on the erf kernel here. That is a real win but smaller than the −17.7% originally claimed — reported honestly as the measured kernel-internal delta rather than an unverified ORT ratio. The mechanism (a latency-bound chain freed by shorter dependency depth) is consistent with a positive but hardware-dependent magnitude. ## Validation (reviewer) - `cargo test -p onnx-runtime-ep-cpu --lib` — **1353 passed, 0 failed, 17 ignored** (base 1345 + #1183/#1149 + this test), including the accuracy gate. - `cargo clippy -p onnx-runtime-ep-cpu --all-targets -- -D warnings` — clean. - No MLAS feature involved; `map_ps`'s target_feature (#1227) verified intact in the merged result. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Co-authored-by: resch <resch@users.noreply.github.com>

…nt ORT activation deferrals) (#1235) ## What this changes Adds native CPU-EP kernels for **`Celu`** (opset 12) and **`Mish`** (opset 18), which had no kernel at all and were therefore silently handed to ORT's CPU EP, and vectorises **`Log`** (f32), which was still taking a scalar `libm` call per element. Celu/Mish are wired through all four capability filters (registry, `PHASE1_OPS`, dtype table, shape-inference table, plugin descriptors) and added to the two activation-coverage tests. ## Behavioural claim — verified independently (reviewer) The headline "removes the last silent ORT activation deferrals" is a behavioural claim, so I falsified it rather than trusting it, on **Intel Core i7-13800H, Windows 11**: - **The deferral was real and silent.** `GetCapability` runs three fail-closed filters; an op absent from the shape-inference table (Filter 2) or the dtype descriptor list (Filter 3) is handed to ORT with no diagnostic. Celu/Mish had no kernel, so they never reached `supports_op`. - **Falsification (RED/GREEN).** With the PR applied, `activation_and_norm_ops_clear_every_capability_filter` passes. Removing just the Celu/Mish entries from the shape-inference table (`compute.rs`, Filter 2) turns it **RED** with `::Celu: no shape rule (shape filter declines it) … would be handed to ORT's CPU EP` for both ops; restoring returns it to **GREEN**. The test genuinely guards the deferral. - **"The last" is defended structurally, not by spot-check.** The coverage test's `OWNED` list now enumerates the full activation/normalisation family and holds every member to both the shape and dtype filters; the companion `every_registered_op_has_a_shape_rule_or_is_a_known_gap` enumerates all registered ops, and its remaining shape-table gap (data-dependent, internal-fusion, and inferrable-but-unwritten ops) contains **no activations**. `Log` was never a deferral — it already had a scalar kernel — so this PR is a perf change for it, not a coverage change. ## Correctness The kernels' own tests (dense sweeps vs the scalar reference, subnormals, and the full IEEE special set) pass; the scalar reference, the f64 path, and the AVX2 path are held to agree, including NaN propagation (`f32::max`/`min` drop NaN, so each kernel guards it to match ORT). Full `cargo test -p onnx-runtime-ep-cpu --lib` = **1362 passed, 0 failed, 17 ignored** (base 1353 + 9 new). `cargo clippy --all-targets -D warnings` clean. `map_ps`'s `#[target_feature]` (#1227) verified intact. ## Performance — measured on this box, ORT-independent The gaps doc's ratios were taken on an **AMD EPYC 9V74** against ORT. I have no ORT reference on this host, so I measured the ORT-independent internal win instead: the AVX2 kernel vs the scalar-native fallback (which is what runs on non-AVX2 hosts and small tensors), on **Intel Core i7-13800H (14C/20T), Windows 11**, single thread, 1,048,576 elements, min-of-200 × 9 rounds. | kernel | scalar-native (ms) | AVX2 (ms) | speedup | | --- | ---: | ---: | ---: | | `Log` (positive domain) | 2.89 | 1.14 | **~2.54×** | | `Celu` (alpha=1, mixed) | 2.77 | 0.68 | **~4.07×** | | `Mish` (mixed) | 17.82 | 2.50 | **~7.13×** | These are hardware-dependent (see `docs/performance/CPU_ACTIVATION_GAPS.md` §32 lesson) and are **not** the same quantity as the doc's ORT-relative ratios — mine compare AVX2 against the native scalar path on this machine; the AMD EPYC figures compare native against ORT. Presented as peers, each with its hardware named. Both agree on direction: the native kernels are a clear win. --------- Co-authored-by: Resch <resch@squad.local> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…1240) # Zero-copy activation fast path, and an AVX2 Elu kernel Two changes in the same seam of `ActivationKernel::execute`: 1. **Generalise the contiguous-f32 fast path.** `silu_contiguous_f32` becomes `contiguous_f32(input, output, kernel)` — the kernel is now a parameter — so Celu, Mish and Elu write their result straight into the output buffer instead of taking the general path's two allocations and three passes (`to_dense_f32_widen` → `map` → `write_dense_f32_narrow`). 2. **An AVX2 Elu kernel** (`elu_ps` / `elu_f32_slice` / `elu_avx2`) built on the `exp_full_ps` that Celu and Mish already use, replacing a scalar `expf` loop. Base: `origin/main` with #1235 (Celu/Mish/Log) landed first. --- ## Reviewer measurements (Intel Core i7-13800H) All numbers below were taken by the reviewer on an **Intel Core i7-13800H (14C/20T, AVX2+FMA, no AVX-512), Windows 11**. No ORT reference build is available on this box, so the perf figure is an **ORT-independent, same-binary A/B**: the scalar Elu reference (`elu_scalar` loop) versus the AVX2 slice kernel (`elu_f32_slice`), 1M f32 elements, single-thread, min-of-200 iterations × 9 rounds, `black_box`ed on both arms of the same release binary. | Elu, 1M elems, 1 thread | scalar-native | AVX2 slice | speedup | |---|---|---|---| | min-of-200 × 9 rounds | 2.5842 ms | 0.5107 ms | **~5.06×** | This measures the vectorisation win itself (scalar vs AVX2) rather than a ratio against ORT, which is what the same-binary A/B can establish without an ORT build present. ### Peer measurement (original PR, AMD EPYC 9V74) The PR was originally measured on an **AMD EPYC 9V74, 16 physical cores, AVX2/FMA/F16C, no AVX-512**, as a ratio `ours / ORT` in a shared process. There, Elu went from **2.06× slower than ORT to ~4.2× faster** at 1M (4.96 ms → 0.57 ms against ORT's 2.39 ms), with Celu/Mish also improving from the fast path alone (their kernels are unchanged in this PR). These two results are **peers, not corrections**: different microarchitecture, different core count, and different reference (one is `ours/ORT`, the other is an internal scalar-vs-AVX2 A/B). Both show the AVX2 Elu kernel is a substantial win; neither refutes the other. --- ## The fast-path fallback is proven, not assumed `contiguous_f32` forms `&[f32]` and `&mut [f32]` over raw tensor pointers, which is only sound when both tensors are dense f32, agree on shape and strides, and do not overlap in memory. A fast path that silently produced wrong results on an unusual layout would be worse than a slow one, so the reviewer added a direct test — `contiguous_f32_fast_path_declines_every_unsafe_layout` — that pins the guard: - **dtype mismatch** (f64 input) → declines; - **non-dense strides** (a broadcast stride of 0) → declines; - **aliasing** (input and output over the same buffer) → declines; - **control**: two distinct dense f32 buffers of equal shape → the fast path runs *and* is the thing that produced the output. The closure passed on the decline arms panics if invoked, so the test fails if the fast path is ever wrongly taken. Verified by falsification: neutering the overlap check makes the aliasing arm run the kernel and the test goes **RED** (`fast path ran on a layout it must decline`); restoring it returns **GREEN**. The overlap check refuses even exact in-place aliasing, because forming `&[f32]` and `&mut [f32]` over the same bytes is UB in Rust regardless of what the elementwise machine code would compute. --- ## Numerics - `elu_special_values` — `-Inf → -alpha`, `-0 → -0` (bit-checked), `+0 → +0`, `+Inf`, `NaN`, `±MAX`, `±1e30`, and ordinary values against the scalar. - `elu_scalar_path_agrees_with_the_vector_path` — nine specials run once through a sub-`SIMD_MIN_LEN` slice (scalar) and once padded past it (vector), compared **bit for bit**, at alpha ∈ {0.25, 1, 2, 7.5}. - `elu_matches_scalar_across_alphas` — 20 003 points over `[-90, 90]` at four alphas with an alpha-scaled bound. `-0.0`: Elu is a select so the sign survives (`Elu(-0) = -0`), where Celu's trailing addition gives `Celu(-0) = +0`; both are pinned. The scalar reference moves from `alpha * x.exp_m1()` to `alpha * (x.exp() - 1.0)` so the scalar path, the vector path and ORT are the same function (there is no AVX2 `expm1`); the two spellings differ by at most ~6e-8 absolute, inside this module's 4e-7 bound. --- ## Verification (reviewer, Intel Core i7-13800H, Windows 11) - `cargo test -p onnx-runtime-ep-cpu --lib` — **1366 passed / 0 failed / 17 ignored** (+4 over #1235: three Elu tests plus the fallback falsifier). - `cargo clippy -p onnx-runtime-ep-cpu --all-targets -- -D warnings` — clean. - #1227's `#[target_feature(enable = "avx2,fma")]` confirmed intact on **both** `map_ps` and `map_bias_ps` after the merge. - MLAS remains non-default; `elu_f32_slice` uses `dispatch!` like its neighbours and touches no MLAS path. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

…u and Selu (#1243) # AVX2 kernels for LeakyRelu, HardSigmoid, ThresholdedRelu and Selu `LeakyRelu`, `HardSigmoid`, `ThresholdedRelu` and `Selu` had no slice kernel, so they took `ActivationKernel`'s generic path: widen into a `Vec`, `map()` a per-element `match`, collect into a second `Vec`, copy that into the output tensor. This PR gives each a slice kernel and routes them through the zero-copy `contiguous_f32` fast path (established in #1240), and adds a genuinely vectorised exponential for Selu. Stacked on #1240 (which is stacked on #1235). --- ## Reviewer measurements (Intel Core i7-13800H) — where the win actually comes from Measured by the reviewer on an **Intel Core i7-13800H (14C/20T, AVX2+FMA, no AVX-512), Windows 11**. No ORT build is available on this box, so these are ORT-independent, same-binary A/Bs, 1M f32 elements, single-thread, min-of-200 × 9 rounds. **Two different comparisons, because they answer different questions:** 1. **Fast-path routing vs the generic path** (what this PR changes at the call site). Emulating the old generic path (`to_vec` → `map` → `copy_from_slice`) against the new `*_f32_slice` fast path, for LeakyRelu: | LeakyRelu, 1M, 1 thread | generic path | fast path | speedup | |---|---|---|---| | min-of-200 × 9 | 1.8784 ms | 0.1712 ms | **~10.97×** | This — eliminating two allocations and two extra passes — is the dominant win for the three cheap ops. 2. **The AVX2 kernel vs its scalar reference** (isolating the hand-written SIMD arithmetic alone, both in the same release binary): | op, 1M, 1 thread | scalar | AVX2 | speedup | |---|---|---|---| | LeakyRelu | 0.1533 ms | 0.1524 ms | ~1.01× | | HardSigmoid | 0.1599 ms | 0.1583 ms | ~1.01× | | ThresholdedRelu | 0.1644 ms | 0.1619 ms | ~1.02× | | **Selu** | 1.4301 ms | 0.3391 ms | **~4.22×** | For the three cheap ops the hand-written AVX2 kernel is **no faster than the scalar loop**, because LLVM already auto-vectorises a compare-and-blend in release mode. Selu is the exception: its exponential does not auto-vectorise, so the vector kernel is a real ~4.2× on the arithmetic itself. **Honest attribution:** for LeakyRelu/HardSigmoid/ThresholdedRelu the speedup is the fast-path plumbing, not the SIMD; for Selu it is both. That is worth stating plainly so nobody credits the compare-and-blend intrinsics with a win the allocator elimination actually produced. ### Peer measurement (original PR, AMD EPYC 9V74) Originally measured on an **AMD EPYC 9V74, 16 physical cores, no AVX-512** as a ratio `ours / ORT` in a shared process, at 1M/1-thread: ThresholdedRelu 11.30 → 0.93, HardSigmoid 8.62 → 0.71, LeakyRelu 7.58 → 0.65, Selu 2.56 → 0.27. These are **peers, not corrections** to the reviewer's numbers: different microarchitecture, core count, and reference (one is `ours/ORT`, the other is an internal same-binary A/B). The AMD `ours/ORT` ratios and the i7 fast-path-vs- generic ratio are consistent — both attribute the cheap-op gains to removing the generic path's allocations and passes. The PR's own note that the three cheap ops remain **~1.2–1.5× ORT at 4k** (a per-call plugin-dispatch floor, not the kernel) stands and is a good separate follow-up. --- ## Correctness — falsifiers verified by the reviewer The PR ships two load-bearing invariants, both confirmed RED-on-break: - **HardSigmoid clamp operand order.** `minps`/`maxps` return their *second* operand for an unordered input, so the value must be second or a NaN silently becomes `1` then `0`. `hard_sigmoid_special_values` pins NaN→NaN. - **Selu signed-zero.** ONNX's `x > 0` sends `-0` down the exp branch; ORT returns `+0`, so both scalar and vector spell it `alpha*(exp(x)-1)` (there is no AVX2 `expm1`). `scalar_paths_agree_with_the_vector_paths` pins scalar and vector to the same answer, and `selu_zero_sign_does_not_depend_on_dtype` pins it across dtypes. Bit-identity where the op is exact (LeakyRelu, ThresholdedRelu asserted bit-identical to their scalars over 20 003 points); HardSigmoid allows one FMA rounding; Selu uses an alpha-scaled bound. --- ## Verification (reviewer, Intel Core i7-13800H, Windows 11) - `cargo test -p onnx-runtime-ep-cpu --lib` — **1415 passed / 0 failed / 17 ignored** for the activation family (all four ops' special-value, scalar-vs-vector bit-parity, dense-sweep and signed-zero tests green). - One unrelated flake observed under full-suite load: `task_runtime::pool::tests::slot_exhaustion_declines_instead_of_blocking` (from #1201's CPU task runtime, untouched by this PR) failed once under contention and passes deterministically in isolation. Not introduced here. - `cargo clippy -p onnx-runtime-ep-cpu --all-targets -- -D warnings` — clean. - #1227's `#[target_feature(enable = "avx2,fma")]` confirmed intact on `map_ps` and `map_bias_ps`. - MLAS remains non-default; all four kernels use `dispatch!` and touch no MLAS path. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

justinchuby and others added 2 commits August 18, 2026 06:39

Say 'strict superset': equal feature sets do inline

2e39321

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby marked this pull request as ready for review August 18, 2026 06:47

justinchuby enabled auto-merge (squash) August 18, 2026 06:47

justinchuby added 2 commits August 18, 2026 06:48

Merge origin/main into squad/resch-act-unroll2

c631ee2

Merge origin/main into squad/resch-act-unroll2

000c8ef

justinchuby mentioned this pull request Aug 18, 2026

CPU EP: native Celu and Mish kernels, AVX2 Log (removes the last silent ORT activation deferrals) #1235

Merged

justinchuby merged commit 8a90878 into main Aug 18, 2026
7 of 17 checks passed

justinchuby deleted the squad/resch-act-unroll2 branch August 18, 2026 14:17

justinchuby mentioned this pull request Aug 18, 2026

perf(cpu-ep): ship the decode f32 GEMV that #1091 left switched off (6.85x -> 1.15x vs ORT) #1183

Merged

justinchuby mentioned this pull request Aug 18, 2026

perf(cpu-ep): evaluate erf's on-path polynomials with Estrin's scheme (hardware-dependent: ~1.36x vs ORT on many-core AMD, ~5% on i7-13800H) #1218

Merged

justinchuby mentioned this pull request Aug 18, 2026

perf(cpu-ep): zero-copy activation fast path and an AVX2 Elu kernel #1240

Merged

justinchuby mentioned this pull request Aug 18, 2026

perf(cpu-ep): AVX2 kernels for LeakyRelu, HardSigmoid, ThresholdedRelu and Selu #1243

Merged

justinchuby mentioned this pull request Aug 19, 2026

perf(cpu-ep): give x86 Relu and Clip the vector kernel they never had #1355

Closed

Navigation Menu

Repository navigation

fix(cpu): restore map_ps's target_feature, which a merge moved onto map_bias_ps - #1227

fix(cpu): restore map_ps's target_feature, which a merge moved onto map_bias_ps#1227
justinchuby merged 4 commits into
mainfrom
squad/resch-act-unroll2

justinchuby commented Aug 18, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 18, 2026

Uh oh!

justinchuby commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justinchuby commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this is

Why the missing target_feature is a hazard

Evidence that this changes nothing today

Also recorded here: a negative result on the divide

Validation

Review

Uh oh!

github-actions Bot commented Aug 18, 2026

🔴 Benchmark Regression Detected

Uh oh!

justinchuby commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

justinchuby commented Aug 18, 2026 •

edited

Loading

Why the missing `target_feature` is a hazard