perf(cpu): retire the m == 1 half-decode handover to the fused GEBP (#1381)

[justinchuby]

The finding: the in-tree gate was the bug, not the missing one

MatMul sent every m == 1 f16/bf16 decode with k * n >= HALF_PREFILL_GEBP_MIN_WEIGHT
(1 048 576) to the fused widen-pack GEBP. Gemm never did — its m == 1 f16 path stayed on the
decode GEMV at every weight. That divergence is #1381, and the obvious reading was that Gemm was
missing a gate. I implemented that reading first, benchmarked it, and it reversed: at the model
shapes the GEBP was 19–26% slower. Escalating to the authoritative production harness showed
why — it is MatMul that was carrying a bad gate.

This PR retires the handover. #1381 closes with MatMul adopting Gemm's route rather than
the reverse.

Evidence

All numbers through crates/onnx-runtime-ep-cpu/benches/half_decode_gemv_ab.rs, which drives
ExecutionProvider::get_kernel + Kernel::execute (dispatch and output narrowing included). Arms
are selected by ONNX_GENAI_CPU_MM_HALF_GEMV/_GEBP, which are process-wide OnceLocks, so one
arm per process. Every ratio is divided by an f32 control measured in the same two processes,
because this host is heavily shared.

The original gate's evidence was 32 threads and k <= 2048 only.

• 8 threads — a loss at every full-set shape at or above the gate, /ctl 0.26–0.76
  (1.3x–3.8x), and in 20 of the 22 f16 cells of the threshold-neighbourhood band set over
  two independent repetitions, /ctl 0.20–1.14 (up to 5.0x). The GEBP's weight bandwidth is
  pinned at 20–34 GB/s independent of shape; the GEMV reaches 24–155 GB/s.
  • Reported against myself: the band re-run does not reproduce an earlier "no row above 1.00"
    claim. w1.05M_k1024 — exactly at the retired threshold, at k = 1024 — measured 1.14 then
    0.89, and five bf16 cells reach 1.00+ with their f32 controls drifted to 0.55–0.94, which
    makes those cells unusable rather than favourable. Both are tabulated in the doc rather than
    dropped. The earlier 6.7x figure is likewise not reproducible and has been walked back to the
    supported 5.0x everywhere it was cited.
• 32 threads — still a loss at every shape a 7B-class decode issues. k = 4096 qkv/mlp /ctl
  0.51–0.89; a 136M lm_head 0.86. (The original table reported 1.18/1.37 for the last two; re-run
  they are 0.88/0.86.) bf16 tracks f16 at 0.41–0.88.
• The one corner it wins — k = 1024, 1.05M–4.2M, 32 threads, /ctl 0.95–1.49 — is not a shape
  any such model emits, and its k = 2048 neighbours are not reproducible run to run (6.3M measured
  1.31 then 0.70 from the same binary).

Why there was nothing to retune

The GEBP earns its packing by reusing a KC x NR panel of B across the rows of A
(MR = 6, NR = 16, KC = 256). At m == 1, m_panels = 1 — every panel is consumed by exactly one
microkernel pass, so none of the packing is repaid.

Where the cost actually lands: the widen-pack writes k*n f32 and reads it back, but per strip
that is a bpack of at most KC * 16 * NR * 4 B = 256 KiB, i.e. L2-resident on this host — so
that traffic is not a DRAM story. What does reach DRAM is line amplification: pack_b_half
walks B column-strip-major and at the usual panels_per_strip = 1 touches NR * 2 = 32 bytes of
every 64-byte line, ~2x the GEMV's read traffic. The rest is L2 bandwidth, the widening compute, and
a fork/join over strips that one row of A cannot amortise. Both axes follow: pack work per unit of
reuse rises with k, and it can only be overlapped if there are workers to overlap it with — which
is why the arm collapses at 8 threads and merely loses at 32.

A decode-specialised GEBP that skips packing B is the GEMV. The finding is structural.

Changes

• matmul.rs: dropped && !half_decode_prefers_gebp(..) from the decode arm; deleted
  half_decode_prefers_gebp and half_decode_prefers_gebp_when (94 lines, incl. the superseded
  evidence table).
• gemm.rs: behaviourally unchanged — it was right. Adds a test-only route counter and the
  comment recording perf(cpu): give bf16 decode a GEMV, and split decode at the measured weight #1381's resolution.
• Deliberately untouched: HALF_PREFILL_GEBP_MIN_WEIGHT / half_prefill_gebp_selected. They are
  the prefill gate and they still serve a batched m == 1 half MatMul, which the non-batched
  GEMV declines and whose only alternative is the row-blocked half GEMM at 16x–21x. Pinned by
  a_batched_single_row_half_matmul_still_reaches_the_gebp.
• Coverage repaired in the same change: new PROBE_SHAPE=band puts 11 rows immediately below, at
  and above the retired threshold at three different k, so a k-dependent effect cannot hide
  inside a k * n gate again. Pins are now by execution rather than by asking a predicate — the
  predicate is what was deleted.
• Docs: new docs/benchmarks/2026-08-21-half-decode-gebp-retired.md, new §14 in
  docs/performance/CPU_MATMUL_ASSIGNMENT.md, and a superseded banner on the 2026-08-19 bf16 record
  whose second finding this reverses.

Review

Reviewed twice by claude-opus-4.8, both times with an explicit instruction to argue the opposing
case (that retiring the divert is wrong, or that a narrower gate should have been kept). It did, and
concluded the case does not survive: a k * n-only gate — the only kind the retired predicate could
express — cannot see thread count, and reinstating it to capture the k = 1024 corner net-loses on
the k = 4096 and lm_head shapes that models actually issue. All five defects from the first pass
are fixed in this head; the second pass found one further Low (a loss range that no longer matched
its own table after rescoping), which is fixed, and prompted the band re-run above.

Mutation test

Re-inserting the divert (&& geom.k * geom.n < 1_048_576 on the decode arm) fails exactly three
tests — no_half_decode_is_diverted_off_the_gemv_on_weight,
f16_decode_at_the_retired_weight_gate_keeps_the_gemv,
gemm_and_matmul_take_the_same_decode_route — so the pins are not vacuous.

Validation (local, on this head merged with origin/main 78058a4)

PASS=19 FAIL=1 over the full matrix: fmt --all, offline build, -p onnx-runtime-ep-cpu --lib
(1552 passed / 0 failed / 21 ignored), clippy -D warnings, --features mlas, native-be
clippy, aarch64-unknown-linux-gnu clippy, --no-default-features, --all-features, no-MLAS
default-artifact symbol scan, cross-compile.sh, and all eight scripts/ conformance checks
(dispatch reachability, dispatch manifest, feature-gate coverage, documented env vars, publish
order, profile table, platform naming, workspace test packages).

The one failure is step H aarch64-pc-windows-msvc, which is environmental and pre-existing:
ort-sys' bindgen cannot find stdlib.h without the Windows SDK headers. No code path in this diff
is Windows-specific.

Additionally, because the repo's quality lane is wider than that step's package list, I ran it
separately: clippy -D warnings --all-targets over the full 31-package quality set — green — and
-p onnx-genai-cli — green.

Out-of-scope observation: -p onnx-runtime-ep-cuda --features cuda clippy is red on main
today (unusual_byte_groupings in device_argmax.rs/matmul_nbits.rs, type_complexity in
gqa_shared_prefix_parity_gpu.rs, a manual div_ceil). Pre-existing 1.98.0 fallout in the lane
#1609 did not cover; this diff touches no CUDA file.

Miri is not meaningful coverage for this EP's SIMD and this diff adds no unsafe — reporting it
as no coverage, not as a pass.

Remaining losses

• The k = 1024 / 32-thread corner gives up 1.1x–1.5x. Recovering it needs a thread-count- and
  k-dependent predicate; not attempted, because the measurement that would justify it is the least
  reproducible one on this host — and §12 already records a per-op result that inverted under the
  real pool.
• The GEMV itself is not at the roofline: 24–155 GB/s against a 75.8 GB/s DRAM ceiling puts the
  large shapes at ~77% and leaves the small ones latency-bound. Separate line of work.
• Gemm's half fast path is still f16-only: bf16 operands return None at the dtype check
  and fall into the portable blocked half GEMM, where MatMul serves them from the same GEMV. A
  distinct, separately mergeable gap — documented as open, not addressed here.

Closes #1381.

[codecov]

Codecov Report

❌ Patch coverage is 68.01802% with 71 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.88%. Comparing base (78058a4) to head (0a38d2b).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/onnx-runtime-ep-cpu/src/kernels/gemm.rs	59.65%	71 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1613      +/-   ##
==========================================
- Coverage   80.94%   80.88%   -0.07%     
==========================================
  Files         382      381       -1     
  Lines      178395   175984    -2411     
  Branches   178395   175984    -2411     
==========================================
- Hits       144401   142343    -2058     
+ Misses      29061    28714     -347     
+ Partials     4933     4927       -6     

Flag	Coverage Δ
cli-ort-linux	82.60% <ø> (ø)
cli-ort-windows	82.19% <ø> (+0.09%)	⬆️
mlas	?
offline	80.82% <68.01%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-runtime-ep-cpu/src/kernels/matmul.rs	82.04% <100.00%> (-0.05%)	⬇️
crates/onnx-runtime-ep-cpu/src/kernels/gemm.rs	83.52% <59.65%> (-5.92%)	⬇️

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justinchuby]

Local validation on this head (0a38d2bbc, merged with origin/main 78058a4):

PASS: A fmt-all              PASS: I ep-cpu-no-default
PASS: B build-offline        PASS: J ep-cpu-all-features
PASS: C test-ep-cpu          PASS: K no-mlas-artifacts
PASS: D clippy-offline       PASS: L cross-compile-sh
PASS: E test-mlas-feature    PASS: S x8 (dispatch reachability/manifest,
PASS: F clippy-native-be           feature-gate coverage, documented env vars,
PASS: G clippy-aarch64-linux       publish order, profile table, platform
FAIL: H check-win-arm64            naming, workspace test packages)
PASS=19 FAIL=1

C is 1552 passed / 0 failed / 21 ignored. H (aarch64-pc-windows-msvc) is the standing environmental failure — ort-sys' bindgen cannot find stdlib.h without the Windows SDK; nothing in this diff is Windows-specific. Additionally green outside that script: clippy -D warnings --all-targets over the full 31-package quality lane, and -p onnx-genai-cli.

Mutation check re-run on this head: re-inserting && geom.k * geom.n < 1_048_576 on the decode arm fails exactly no_half_decode_is_diverted_off_the_gemv_on_weight, f16_decode_at_the_retired_weight_gate_keeps_the_gemv and gemm_and_matmul_take_the_same_decode_route.

Not covered: Miri is a no-op gate for this EP's SIMD and this diff adds no unsafe — reporting that as no coverage, not as a pass. Out of scope but observed: -p onnx-runtime-ep-cuda --features cuda clippy is red on main today (1.98 fallout #1609 did not cover); this diff touches no CUDA file.