[justinchuby]

Records Phase 11 and corrects the ratios that measured the wrong build.

Why

bench_generic declared required-features = ["mlas"] while containing no MLAS references at all. mlas is not a default feature of onnx-runtime-ep-cpu, so every ratio in this document measured a configuration production never builds. §29.4 spotted the harness problem and flagged it as open work; it underestimated the consequence.

What it hid

Two kernels had no vectorized production path at all, both masked for months by the MLAS arm covering for them:

• Softmax called scalar f32::exp() once per element - 9.3-10.9x ORT. scale_mask_softmax_serial shares the helper, so fused attention paid it too. Fixed in perf(cpu-ep): vectorise the production softmax exp on AVX2+FMA #1234.
• MoE allocated and filled a full transposed copy of an expert's weight matrix on every call, for every routed expert group - 29 MiB for a Mixtral fc1 with swiglu_fusion=1, to multiply as little as one token row. 39-72x ORT at t=1. Fixed in perf(cpu-ep): layout-aware gemm_bt for MoE, killing the per-call expert transpose #1241.

§18.3 had recorded every MoE t=1 cell at 1.006-1.052 and concluded the only remaining MoE problem was thread scheduling at t=8/16. On the arm that ships, t=1 was the worst cell in the entire benchmark suite.

What this PR does

Adds §31 with:

• the linker-level falsifier a feature flag cannot provide - nm -C | grep -ci mlas gives 0 on the shipping binary against 805 on the reference one;
• the corrected production-native matrix for transforms and MoE (thread-matched, interleaved, min/median/max of per-rep ratios, parity PASS throughout);
• §31.4, recording that the §19/§20 conclusion was wrong. Those sections closed the transpose question on the grounds that a layout-aware GEMM consumer was a cost "these graphs simply do not exercise" and would need a synthetic fixture built. The most expensive transpose in the runtime was never a Transpose node at all - it was inside the MoE kernel, feeding straight into a GEMM, on three of the models already in this document.
• §31.7, two methodology rules: benchmark the artifact that ships and prove it with the linker rather than the flag; and treat an untested arm as unmeasured rather than "probably similar" - here it was 10x and 70x.
• §31.8, where the CPU EP stands now, including what still loses: softmax 1.29-1.63x, MoE prefill 1.22-1.56x, phi35moe decode 1.10x, and the small-graph per-call overhead cells.

Correction banners are added to §18.3, §27 and §29.4 rather than editing their tables, so the original numbers and the reason they misled both stay readable.

Documentation only - no code changes.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	43.57 µs	106.65 µs	+144.8%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	39.78 µs	93.54 µs	+135.1%
🔴	matmul/medium_generic_f32_threads=8/32x512x512	930.47 µs	2.02 ms	+116.6%
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	555.39 µs	915.24 µs	+64.8%
🔴	matmul/small_generic_f32_threads=8/1x256x256	36.30 µs	55.34 µs	+52.5%
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.34 ms	1.86 ms	+39.5%
🔴	gather/medium_bf16_threads=1-internal/32768	2.46 µs	3.36 µs	+36.6%
🔴	matmul/large_generic_bf16_threads=1/32x1024x1024	2.01 ms	2.74 ms	+36.0%
🔴	matmul/medium_generic_f32_threads=1/32x512x512	2.33 ms	3.05 ms	+31.2%
⚠️	matmul/small_generic_f32_threads=1/1x256x256	36.58 µs	47.26 µs	+29.2%
⚠️	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	377.06 µs	478.43 µs	+26.9%
⚠️	matmul/small_generic_bf16_threads=1/1x256x256	32.11 µs	40.36 µs	+25.7%
⚠️	matmul/large_generic_f16_threads=8/32x1024x1024	85.51 µs	107.11 µs	+25.3%
⚠️	gather/large_f32_threads=1-internal/131072	28.72 µs	35.92 µs	+25.1%
⚠️	matmul/small_generic_bf16_threads=8/1x256x256	32.43 µs	39.81 µs	+22.7%
⚠️	add/large_f32_threads=1-internal/4194304	740.69 µs	858.38 µs	+15.9%
✅	gather/large_bf16_threads=1-internal/131072	13.17 µs	14.92 µs	+13.3%
✅	matmul/medium_generic_f16_threads=1/32x512x512	35.46 µs	40.10 µs	+13.1%
✅	logit_processing/seven_processor_chain_per_step	386.51 µs	434.14 µs	+12.3%
✅	matmul/small_generic_f16_threads=1/1x256x256	35.29 µs	39.03 µs	+10.6%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.94 ms	4.31 ms	+9.4%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	80.62 µs	87.72 µs	+8.8%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	10.57 ms	11.49 ms	+8.8%
✅	gather/small_bf16_threads=1-internal/4096	501.5 ns	544.0 ns	+8.5%
✅	grammar_masking/llguidance_compute_mask/32	89.48 µs	97.06 µs	+8.5%
✅	matmul/small_generic_f16_threads=8/1x256x256	42.12 µs	45.00 µs	+6.8%
✅	add/large_bf16_threads=1-internal/4194304	1.79 ms	1.89 ms	+5.5%
✅	add/large_f16_threads=1-internal/4194304	1.91 ms	1.97 ms	+3.2%
✅	matmul/medium_generic_f16_threads=8/32x512x512	42.86 µs	44.06 µs	+2.8%
✅	gather/large_f16_threads=1-internal/131072	14.80 µs	15.18 µs	+2.5%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	566.78 µs	574.80 µs	+1.4%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	581.34 µs	587.98 µs	+1.1%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	4.28 ms	4.25 ms	-0.6%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	692.05 µs	687.86 µs	-0.6%
✅	sampling_latency/top_p_per_token	445.55 µs	441.50 µs	-0.9%
✅	qwen3_sampling_processors/top_k_partial_selection	168.18 µs	165.89 µs	-1.4%
✅	gather/medium_f32_threads=1-internal/32768	4.48 µs	4.39 µs	-1.8%
✅	sampling_latency/top_k_per_token	58.79 µs	57.01 µs	-3.0%
✅	tokenization/decode_tokens_per_second	7.11 ms	6.89 ms	-3.1%
✅	sampling_latency/greedy_per_token	3.51 µs	3.40 µs	-3.1%
✅	tokenization/encode_tokens_per_second	440.61 µs	419.89 µs	-4.7%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.52 ms	2.34 ms	-7.1%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.84 ms	6.34 ms	-7.2%
✅	qwen3_sampling_processors/top_k_top_p_fast	804.25 µs	724.31 µs	-9.9%
✅	reduce_mean/small_f32_threads=1-internal/4096	18.53 µs	16.65 µs	-10.1%
✅	reduce_mean/medium_f32_threads=1-internal/65536	358.83 µs	322.11 µs	-10.2%
✅	gather/small_f16_threads=1-internal/4096	633.2 ns	556.3 ns	-12.1%
✅	kv_cache/alloc_dealloc_pages	60.86 µs	53.01 µs	-12.9%
✅	add/small_bf16_threads=1-internal/1024	628.0 ns	540.7 ns	-13.9%
🟢	add/medium_f16_threads=1-internal/262144	173.85 µs	140.83 µs	-19.0%
🟢	sampling_latency/min_p_per_token	291.72 µs	236.00 µs	-19.1%
🟢	reduce_mean/large_f32_threads=1-internal/262144	1.29 ms	1.04 ms	-19.5%
🟢	gather/medium_f16_threads=1-internal/32768	3.56 µs	2.75 µs	-22.6%
🟢	add/medium_f32_threads=1-internal/262144	39.84 µs	30.82 µs	-22.6%
🟢	gather/small_f32_threads=1-internal/4096	1.02 µs	770.5 ns	-24.7%
🟢	add/small_f32_threads=1-internal/1024	292.0 ns	218.6 ns	-25.1%
🟢	add/small_f16_threads=1-internal/1024	755.5 ns	564.1 ns	-25.3%
🟢	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	114.99 µs	84.80 µs	-26.3%
🟢	add/medium_bf16_threads=1-internal/262144	176.14 µs	113.13 µs	-35.8%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 8.26 4.78 7.50 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)