diff --git a/docs/benchmarks/2026-08-15-cpu-ep-vs-ort-attention-moe.md b/docs/benchmarks/2026-08-15-cpu-ep-vs-ort-attention-moe.md index 2b32363f5b..7c102c357b 100644 --- a/docs/benchmarks/2026-08-15-cpu-ep-vs-ort-attention-moe.md +++ b/docs/benchmarks/2026-08-15-cpu-ep-vs-ort-attention-moe.md @@ -1081,6 +1081,12 @@ per-expert loop rather than merely close - the tests assert on raw `Vec` eq ### 18.3 Full matrix, 27 cells, real ORT CPU session +> **Correction (§31).** These are **MLAS-arm** ratios. The claim below that every `t=1` +> cell sits at 1.006-1.052 is not true of the build that ships: measured on the +> production arm, the same `t=1` cells were **39-72x** ORT, because the shipping path +> materialized a transposed copy of each expert's weights on every call. Fixed in #1241. +> Current production-native numbers are in §31.5. + `ratio = native p50 / ORT p50`, **lower is better**. Interleaved arms, same host, same session options, same thread count on both sides, 5 warmups, 5 trials x 15 runs. Synthetic weights from `scripts/ort_ab/gen_moe.py` (public architecture dimensions, @@ -2173,6 +2179,10 @@ above. ## 27. Where the CPU EP actually stands +> **Correction (§31).** The 66-cell grid in this section was produced by the MLAS-linked +> `bench_generic` and does not describe the shipping build. See §31.5 for the corrected +> production-native matrix. + Full transform + KV-concat grid on `main` at `f2e6ad97d`, paired interleaved against a real ORT 1.27 CPU session, one driver invocation, 5 trials × 60 runs × 15 warmups. Ratio is native/ORT p50, **lower is better**, and `< 1.000` is a win. @@ -2436,6 +2446,11 @@ shapes is unattributed and is the next thing to chase on this axis. ### 29.4 A measurement caveat that applies to this whole document +> **Superseded by §31.** The caveat below was correct and its consequence was far larger +> than it estimated. The harness has since been fixed (#1231) and the production arm +> measured. The main table in §29.3, and every ratio in §18 and §27, is an **MLAS-arm** +> number. See §31.6 for exactly which tables are affected. + `onnx-genai-bench` declares `required-features = ["mlas"]`, so `bench_generic` can only be built with MLAS linked in. Every ratio published in this document - including §27's 66-cell grid - was therefore produced by the **MLAS-linked build**, and where a kernel @@ -2556,3 +2571,202 @@ effort. Every one of them passed when written. The only thing that separated the tests from the decorative ones was running the falsifier - and in §29.5 even the falsifier misfired, which is its own lesson: a falsifier that does not fire is not evidence the test is sound until you know *why* it did not fire. + + +## 31. Phase 11: the benchmark was measuring an arm that does not ship (#1231, #1234, #1241) + +### 31.1 The defect + +`crates/onnx-genai-bench/Cargo.toml` declared: + +```toml +[[bin]] +name = "bench_generic" +required-features = ["mlas"] +``` + +`bench_generic` contains **no MLAS references at all**. The requirement was gratuitous. +But `mlas` is not a default feature of `onnx-runtime-ep-cpu`, so the effect was that the +only binary anyone benchmarked with was the one configuration production never builds. + +Every ratio in this document up to §30 was therefore an MLAS-arm measurement, read and +acted on as though it described the shipping runtime. §29.4 spotted the harness problem +and correctly flagged it as open work; it badly underestimated the consequence. + +#1231 changed the requirement to a `bench-native` feature, added an `arm=native` / +`arm=mlas-reference` label to every result line, and added `mlas_is_not_a_default_feature` +so the crate fails loudly if MLAS ever becomes default. + +A feature flag is not proof. The falsifier that is proof: + +``` +$ nm -C target/release/bench_generic | grep -ci mlas +0 # --features bench-native (the arm that ships) +805 # --features mlas (the research/reference arm) +``` + +Every performance claim from here on is taken with that `0` verified on the same binary. + +### 31.2 What it hid + +Two kernels turned out to have no vectorized production path at all. Both had looked +fine for months because the MLAS arm was covering for them. + +**Softmax.** The `#[cfg(not(feature = "mlas"))]` arm called scalar `f32::exp()` once per +element. `scale_mask_softmax_serial` routes through the same helper, so fused attention +paid it too. The doc comment claiming an "8-lane polynomial" described only the MLAS arm. + +| model | production native | mlas reference | +|---|--:|--:| +| sm_prefill_h32_s512 | **10.012** | 0.908 | +| sm_bert_b8_s128 | **9.621** | 1.032 | +| sm_decode_h32_kv1024 | **9.302** | 1.011 | +| sm_decode_h32_kv2048 | **9.944** | 1.013 | +| sm_decode_h32_kv4096 | **10.535** | 1.028 | +| sm_decode_h32_kv8192 | **10.861** | 1.000 | + +**MoE.** Worse. The shipping path allocated and filled a full transposed copy of an +expert's weight matrix *on every call, for every routed expert group*, because expert +weights are stored `[out_features][in_features]` while `gemm` wants `[K][N]`. For a +Mixtral-shaped expert with `swiglu_fusion=1` that is a 29 MiB allocate-and-transpose for +fc1 plus 14.7 MB for fc2 - to multiply a **single token row**. + +| model | production native, before | +|---|--:| +| moe_mixtral_h1024_i3584_e8_t1 | **71.4** | +| moe_qwen3moe_h2048_i768_e16_t1 | **42.0** | +| moe_phi35moe_h2048_i6400_e4_t1 | **48.4** | + +§18.3 had recorded every `t=1` MoE cell at 1.006-1.052 and concluded the remaining MoE +problem was purely thread scheduling at `t=8/16`. On the arm that ships, `t=1` was the +worst cell in the entire benchmark suite. + +### 31.3 The fixes + +**#1234, softmax.** Both non-MLAS forms now route through one +`softmax_row_core(src, dst, d)` with an AVX2+FMA `exp`: Cody-Waite split of `ln2`, a +degree-6 Cephes minimax polynomial in FMA form, `2^k` built as `(k + 127) << 23`, runtime +`is_x86_feature_detected!` dispatch, scalar fallback for non-AVX2 and for sub-8 tails. +Non-finite lanes are handled by explicit masks: NaN by unordered-compare plus `blendv`, +`-inf` and deep underflow forced to exactly `0` by ordered-compare plus `andnot`. Review +independently measured the result at **max 1 ULP** against an f64 reference across the +whole domain. + +**#1241, MoE.** `gemm_bt(a, bt, c, m, k, n)` computes `C = A * Bt^T` directly on the +`[out][in]` expert slice, so the transposed copy is never materialized. That layout is +also the better one: for `C[i][j] = sum_k A[i][k] * Bt[j][k]` both operands are +contiguous along `k`, making the inner loop a pure contiguous dot product instead of a +strided gather. Interior tiles are swept column-panel-first so a ~256 KiB `bt` panel is +reused across row bands - without that, an expert bank larger than the 32 MiB L3 (the +Phi experts are 52 MiB) is re-streamed from DRAM once per row band, which regressed the +largest prefill shape until it was fixed. + +### 31.4 A note on what this says about the transpose question + +§19 and §20 closed the Transpose investigation on the grounds that the Transpose *kernel* +moves zero bytes in-graph, and that a layout-aware consumer was "a real cost these graphs +simply do not exercise" - so a synthetic fixture would have to be built to demonstrate it. + +That conclusion was wrong, and it was wrong because it only looked at `Transpose` nodes. +The most expensive transpose in the runtime was not a node at all: it was a materialized +transpose *inside* the MoE kernel, feeding straight into a GEMM. The fixture did not need +building. It was already the single largest term in the operator, on three of the models +in this document, and it went unseen because the MLAS arm did not take that path. + +### 31.5 Corrected production-native matrix + +All cells: production build (`bench-native`, 0 MLAS symbols), thread-matched at +`--native-threads 1 --ort-intra-threads 1`, `native_min`/`ort_min`, models interleaved +and order-alternated across reps, min/median/max of per-rep ratios. Ratio is native/ORT, +so **below 1.000 beats ORT**. Parity PASS in every cell shown. + +**Transforms, after #1234:** + +| group | model | min | med | max | +|---|---|--:|--:|--:| +| softmax | sm_prefill_h32_s512 | 1.455 | 1.460 | 1.470 | +| softmax | sm_bert_b8_s128 | 1.472 | 1.477 | 1.485 | +| softmax | sm_decode_h32_kv1024 | 1.529 | 1.529 | 1.625 | +| softmax | sm_decode_h32_kv2048 | 1.613 | 1.633 | 1.805 | +| softmax | sm_decode_h32_kv4096 | 1.574 | 1.574 | 1.574 | +| softmax | sm_decode_h32_kv8192 | 1.603 | 1.622 | 1.650 | +| softmax | sm_whisper_cross | 1.289 | 1.295 | 1.300 | +| rope | rope_llama3_s1 | 1.250 | 1.250 | 1.250 | +| rope | rope_llama3_s128 | **0.759** | **0.766** | 0.802 | +| rope | rope_llama3_s512 | **0.833** | **0.861** | 0.956 | +| rope | rope_llama3_b8_s1 | **0.900** | 1.000 | 1.000 | +| rope | rope_gptj_il_s1 | 1.250 | 1.250 | 1.500 | +| rope | rope_gptj_il_s128 | **0.821** | **0.855** | 0.861 | +| rope | rope_gptj_il_s512 | **0.941** | 1.002 | 1.017 | +| transpose | tr_bert_b8_s128 | **0.569** | **0.582** | 0.616 | +| transpose | tr_llama3_s512 | 1.023 | 1.046 | 1.647 | +| transpose | tr_whisper_s1500 | **0.679** | **0.846** | 0.994 | +| kvcat | kvcat_llama3_p1023 | 1.115 | 1.179 | 1.218 | +| kvcat | kvcat_llama3_p2047 | **0.981** | 1.015 | 1.273 | +| kvcat | kvcat_llama3_p4095 | **0.986** | 1.070 | 1.161 | +| kvcat | kvcat_llama3_p8191 | **0.999** | 1.052 | 1.066 | +| kvcat | kvcat_llama3_b8_p2047 | **0.947** | **0.975** | 0.984 | + +**MoE, before and after #1241** (before column is the production arm, freshly measured, +not the MLAS numbers from §18.3): + +| model | before (med) | after: min | med | max | +|---|--:|--:|--:|--:| +| moe_mixtral_h1024_i3584_e8_t1 | 71.366 | **0.897** | **0.926** | 0.952 | +| moe_mixtral_h1024_i3584_e8_t32 | 17.852 | **0.752** | **0.805** | 0.806 | +| moe_mixtral_h1024_i3584_e8_t512 | 3.670 | 1.217 | 1.232 | 1.240 | +| moe_qwen3moe_h2048_i768_e16_t1 | 41.958 | **0.912** | **0.918** | 0.927 | +| moe_qwen3moe_h2048_i768_e16_t32 | 8.241 | **0.975** | 1.007 | 1.057 | +| moe_qwen3moe_h2048_i768_e16_t512 | 1.877 | 1.375 | 1.383 | 1.388 | +| moe_phi35moe_h2048_i6400_e4_t1 | 48.365 | 1.094 | 1.102 | 1.136 | +| moe_phi35moe_h2048_i6400_e4_t32 | 9.008 | 1.213 | 1.225 | 1.238 | +| moe_phi35moe_h2048_i6400_e4_t512 | 1.921 | 1.553 | 1.557 | 1.571 | + +### 31.6 Which earlier tables are now known to be wrong + +Any ratio in this document produced before #1231 came from the MLAS-linked binary. +Concretely: + +- **§18.3's 27-cell MoE matrix** - MLAS arm. Its central conclusion, that `t=1` MoE is at + parity and only `t=8/16` scheduling remains, was false for the shipping build. Banner added. +- **§27's 66-cell grid** - MLAS arm. Banner added. +- **§29.3's softmax table** - MLAS arm. The *direction* of that change (removing a copy at + the shared, arm-agnostic call site) still holds, and §31.5 shows the production arm + post-fix, but the magnitudes in §29.3 are not production magnitudes. Banner added at §29.4. +- **§21's boundary instrument** - taken under the coupled profiler gate later fixed in + #1226, *and* on the MLAS arm. Both reasons to re-read it before relying on it. + +Sections that do not depend on a ratio - the structural analysis in §13, the aliasing and +capacity work in §28, the methodology in §30 - are unaffected. + +### 31.7 What this adds to the methodology rules + +§30.3 established: interleave innermost, prefer `native_min` on small graphs, distrust an +effect an order of magnitude larger than its mechanism. Phase 11 adds two more. + +1. **Benchmark the artifact that ships, and prove it with the linker, not the flag.** + `required-features` silently redefined what "the CPU EP" meant for every measurement in + this document. `nm -C | grep -ci mlas` is cheap, and it is the only statement about + linkage that a feature flag cannot make for you. + +2. **A fallback arm hides the absence of the primary one.** Neither the scalar `exp` nor + the materialized MoE transpose was subtle. Both survived because the configuration + under measurement never executed them. When a kernel has two arms, the untested arm is + not "probably similar" - it is unmeasured, and here it was 10x and 70x. + +### 31.8 Where the CPU EP now stands, honestly + +Won at `t=1`: RoPE at s128/s512, both Transpose shapes that matter, KV concat at every +context length from 2047 up, MoE decode on mixtral and qwen3moe. + +Still losing at `t=1`: **softmax, 1.29-1.63x** across the whole family - a 6-7x +improvement on #1234 but not parity, and the remaining cost is the three-pass +memory-bound structure, not `exp`. **MoE prefill, 1.22-1.56x** - microkernel efficiency +against MLAS's packed panels and wider register tiles. **phi35moe decode, 1.10x** - +bandwidth-bound, the most weight bytes per token of the three. `rope_*_s1` and +`kvcat_p1023` at 1.12-1.25x are per-call overhead on graphs too small to amortize it. + +Not started: paged/appendable KV behind an attention-owned handle (§13's boundary +stands), and beam reorder, which still does not exist on any EP. + +Nothing here defers to ORT and nothing links MLAS.