perf(cpu): serve f16 Gemm decode with an [N, K] GEMV instead of the blocked half GEMM (22-65x loss to a win)

[justinchuby]

What

An f16 Gemm at M = 1 with transB = 1 measured 32-48 ms against ONNX Runtime's 0.16-1.5 ms at K = N = 3584 — 22x to 65x slower — and did not improve with thread count at all. It now runs in 0.3-1.5 ms: a 38x to 90x absolute speedup, up to 148x on per-run minima, and a win against ORT at 1 through 8 threads.

transB = 1 is not a corner case. It is the layout every nn.Linear export produces, so this is what a QKV, an output projection and an MLP gate look like whenever a model is exported through Gemm rather than MatMul.

Why it was slow

GemmKernel::execute disqualified both f16 fast paths if either transpose flag was set:

let half_fast_path = if self.trans_a || self.trans_b { None } else { ... };

with the reasoning, in a comment directly above it, that both "read B in its stored [K, N] order, and materialising a transpose first would give back what they save".

The premise is right and the conclusion does not follow. You only need a transpose if you insist on reusing the [K, N] kernel. A [N, K] weight does not need one — it is the better GEMV layout:

	[K, N] (transB = 0)	[N, K] (transB = 1)
one output element is	a strided gather over k	one contiguous k-run
parallel granularity	a stripe of columns, min 32 (a cache line)	one row, any width
accumulator working set	W f32 live across the whole k sweep	one f32

So the fix is a second kernel, not a transpose. The path it replaced was the portable blocked half GEMM, which packs both operands into MR x NR panels — at M = 1 there is no reuse to amortise that against. The same file already recorded exactly this failure for the untransposed case ("the worst dense region measured anywhere in this EP ... 10.07 ms at 1 thread, 10.26 ms at 8"). That case was fixed; the transposed one kept the note and the behaviour.

Measurements

K = N = 3584 (Qwen3-8B hidden), ORT and native alternating in one process so every ratio is paired. Host AMD EPYC 9V74, 32 vCPU / 16 cores, AVX2+FMA+F16C only, ORT 1.27.0, native build (no mlas).

Tabulated run is the quietest and largest of three sweeps: 9 trials, load 9-11.

threads	before ms	after ms	speedup	before ours/ORT	after p50	after p90
1	36.606	0.968	37.8x	22.46	0.655 win	0.674
2	36.383	0.633	57.5x	29.75	0.634 win	0.733
4	32.587	0.468	69.6x	36.86	0.840 win	1.021
8	36.234	0.532	68.1x	49.50	0.757 win	0.929
16	48.096	0.534	90.1x	65.26	1.706	1.224

On per-run minima: 32.203 → 0.814, 31.465 → 0.461, 31.628 → 0.294, 36.076 → 0.243, 35.394 → 0.302 ms (40x to 148x).

The before column does not move with thread count (36.6 at 1, 36.2 at 8, 48.1 at 16). The blocked path never scaled on a single-row problem; the ratio degraded purely because ORT did scale.

The ratio is now the noisy part, and the claim is narrowed accordingly

Post-fix these cells run in 0.3-1.5 ms — short enough that this shared host's other tenants move the ratio more than the kernel does. Three independent sweeps of the same five cells:

threads	1	2	4	8	16
sweep A (7 trials, load 6-13)	0.588	0.592	1.224	1.833	1.455
sweep B (7 trials, load 12-18)	0.884	1.513	1.234	1.040	2.680
sweep C (9 trials, load 9-11, tabulated)	0.655	0.634	0.840	0.757	1.706

t = 16 swings 1.46-2.68 and t = 8 swings 0.76-1.83, so neither is worth two digits. The defensible statement is a consistent win at 1-8 threads and a t = 16 ratio unresolvable on this host. A p90 taken at load 12-18 read 8.8 at two threads and 19.0 at sixteen — evidence that the tail statistic measures the other tenants at these durations, not this kernel.

What is not in doubt is the absolute change, three to four orders of magnitude above that noise: 32-48 ms → 0.3-1.5 ms in every sweep.

Controls — every other M = 1 cell, same binary, same session, unchanged within noise:

model	before (t=1/4/16)	after (t=1/4/16)
MatMul f16	1.79 / 1.49 / 2.24	1.42 / 1.59 / 2.79
MatMul f32	1.01 / 3.13 / 2.56	1.04 / 2.38 / 2.71
MatMulNBits 4-bit	2.20 / 3.84 / 7.06	1.90 / 3.79 / 6.78
MatMulNBits 8-bit	0.19 / 0.22 / 0.19	0.16 / 0.16 / 0.17

End-to-end parity against ORT: max_rel = 6.6e-4, PASS.

Numerics — the one deliberate departure

gemv_f16_kn is bit-identical to a naive sequential loop. gemv_f16_nk is not, and I would rather say so than bury it: it contracts along the lanes rather than across them, so the sum splits into 32 partial sums. Slightly more accurate, but different.

The order is fully specified in the function's doc comment and pinned two ways:

• dot_row_scalar is an executable statement of that order, and nk_simd_and_scalar_rows_agree_bit_for_bit asserts the vector path matches it exactly at every k around the 32- and 8-element loop boundaries (1, 7, 8, 9, 31, 32, 33, 39, 40, 64, 65, 96, 127, 128, 3584).
• nk_is_independent_of_the_thread_count asserts the answer does not move across pools of 1, 2, 3 and 8 workers — ROW_STRIPE partitions the output, never the contraction.

Routing is asserted rather than inferred, via a thread-local NK_GEMV_CALLS counter: values alone cannot tell which route ran, because the blocked fallback computes the same thing, only ~40x slower. Thread-local rather than global on purpose — the count is taken at entry, before any rayon dispatch, so it is exact without every caller in the crate having to agree on a lock.

What this does not fix (recorded in the work list, not left implied)

• transB f16 prefill is still badly broken. At M = 128 it measures 4.04x at 1 thread and 16.95x at 8 (156 ms vs 39 ms). Those cells run 100-160 ms and are reproducible, unlike the decode ratios. The GEMV is a decode kernel and correctly declines M > 1 — half_prefill_gemm_does_not_take_the_nk_gemv asserts exactly that — so prefill still takes the blocked path. Closing it needs a packed NT half GEMM, the f16 analogue of Native transposed-B SGEMM: eliminate the Kn dequant from default-build MatMulNBits prefill (#959, #1091) #1176's transposed-B SGEMM. Separate work, not a comment change.
• The residual loss at high thread counts is a pre-existing ceiling, not this kernel. Every one of our M = 1 GEMVs flattens at ~0.7-1.1 ms past 4-8 threads while ORT keeps scaling (4-bit 3.60 → 0.79 ms over 1..16 threads vs ORT 1.59 → 0.13; f32 1.73 → 0.94 vs 1.76 → 0.17). Worth being precise about what that rules out: we do not get slower as threads are added, so it is not fork/join overhead — we stop getting faster. It is also not task granularity — I built that fix and measured it away. gemv_f16_kn's fixed STRIPE = 512 does yield only 7 tasks at n = 3584, so it was the obvious suspect. Making the width adaptive (n / (2 * threads), rounded up to a multiple of 32) raises that to 8/16/28 tasks at 4/8/16 threads and moves nothing. Two interleaved A/B runs of the same binary pair, each with a null control, disagree in sign:

run	host load	t=4	t=8	t=16
1 (5 trials)	5.0	—	+49.1% (noise 51.9%)	+41.0% (noise 0.5%)
2 (9 trials)	9.9	-15.1% (noise 8.9%)	+1.1% (noise -1.3%)	+9.2% (noise 33.5%)

The effect is smaller than this host's between-run variability, so the change is not in this PR. The absolute numbers settle it rather than the ratios: native p50 stayed at 1.15–1.64 ms in both arms at every thread count — 4x the tasks, same time. Commit 667e4c0a6 records the rejection in the work list so the lead is not retried.

The suspect it leaves standing is the loop nest: gemv_f16_kn holds a w-wide f32 accumulator in memory with p outermost, so every FMA is a load-modify-store against L1 — w overflows the 16 available ymm registers at 512 lanes and at 128 alike, which is exactly why narrowing the stripe changed nothing. Inverting the nest onto a register-resident accumulator tile (what gemv_f16_nk already does for [N, K]) is the next thing I intend to take. It is a kernel rewrite and it is unproven.

• The 8-bit MatMulNBits "win" in the control table is mostly ORT being slow (30.6 ms at 1 thread), not us being fast.
• AVX-512 / VNNI untested — this host has neither.
• trans_a still declines. At M = 1 transposing a single-row A is meaningless.

Gates

cargo fmt --all, 1455 ep-cpu lib tests, clippy --all-targets -D warnings on x86_64, verify_documented_env_vars.py, workspace_test_packages.py verify.

aarch64 clippy fails with three "never used" errors — I reproduced them on a pristine detached worktree of origin/main before assuming they were mine, and filed #1415. This branch adds no new ones.

Files

• crates/onnx-runtime-ep-cpu/src/kernels/half_gemv.rs — gemv_f16_nk, dot_row_simd, dot_row_scalar, ROW_STRIPE, the test counter, 6 new tests.
• crates/onnx-runtime-ep-cpu/src/kernels/gemm.rs — try_half_fast_path takes trans_b and picks a kernel; execute no longer disqualifies on it; 2 new routing tests.
• docs/benchmarks/2026-08-19-f16-gemm-transb-decode.md — full record.
• docs/performance/CPU_MATMUL_ASSIGNMENT.md — 7 new transB rows (they were absent entirely) + root-cause section 5.
• scripts/ort_ab/gen_decode.py, scripts/ort_ab/sweep_decode.py — the harness; gen_gemm.py had no f16 MatMul or Gemm at all.

[justinchuby]

Nits addressed in c619459d0. Per-nit disposition:

A. gemm_f16_transposed_b_keeps_the_blocked_path doc comment stated the rationale this PR refutes — fixed.
Good catch; that comment was the thing I set out to disprove and I left a copy of it standing. It now reads "A transposed B at M > 1 keeps the blocked half GEMM: the decode GEMV declines anything but a single row, and everything past it reads B as [K, N]" and points at half_decode_gemm_takes_the_nk_gemv_when_b_is_transposed, which is why the test pins m = 4.

B. "section 2 already calls" was a false attribution — fixed.
You are right, that phrase is not in section 2; it is the dispatch comment in gemm.rs. Attributed there.

C. The p90-headed column held a min-of-minima. Fixed properly — by measuring a p90.
I had reached for the footnote because sweep_decode.py was not capturing a p90. bench_generic prints one; the regex just did not pick it up. It does now, both documents tabulate the real statistic, and the ‡ footnote is deleted rather than reworded. Marking a mismatched statistic was the wrong fix when the right one was to go and measure it.

Re-measuring changed the answer, so the tables and the claim changed with it.

This is the part worth your attention. Your reproduction (load 12–18) read 0.652 at one thread against my 0.588, which I would have written off as ordinary noise. Chasing the p90 instead showed the tail is not usable at these durations at all: at load 12–18 the p90 ratio read 8.8 at two threads and 19.0 at sixteen. Post-fix these cells run in 0.3–1.5 ms, short enough that the other tenants on this host move the ratio more than the kernel does.

Three independent sweeps of the same five cells:

threads	1	2	4	8	16
sweep A (7 trials, load 6–13)	0.588	0.592	1.224	1.833	1.455
sweep B (7 trials, load 12–18)	0.884	1.513	1.234	1.040	2.680
sweep C (9 trials, load 9–11, now tabulated)	0.655	0.634	0.840	0.757	1.706

t = 16 swings 1.46–2.68 and t = 8 swings 0.76–1.83, so neither deserved the two digits I gave it. Both documents now tabulate the quietest and largest sample, print all three sweeps beside it, and narrow the claim to what survives: a consistent win at 1–8 threads, and a t = 16 ratio not worth quoting on this host. I removed "1.2–1.8x loss at 4–16 threads" from the PR bodys premise everywhere it appeared — on the better sample 4 and 8 are wins, so that sentence was wrong in both directions.

The absolute change is untouched by any of this and is what the PR rests on: 32–48 ms before, 0.3–1.5 ms after, in every sweep — 38x to 90x, up to 148x on minima. Three to four orders of magnitude above the noise band.

Prefill re-measured the same way: 4.04x at one thread, 16.95x at eight (156 ms vs 39 ms), replacing the earlier 3.7x/10.7x. Those cells run 100–160 ms and are reproducible.

On your Finding D (dot_row_scalar unreachable in the shipped x86 path): agreed, and deliberate. It is the executable statement of the documented reduction order — nk_simd_and_scalar_rows_agree_bit_for_bit is only meaningful because the scalar path is written independently of the intrinsics — and it is the live path on any host without f16c. I would rather keep a spec that runs than a comment that cannot.

On your caveat that you did not rebuild main for the "before" column: noted and fair. The before numbers come from a sweep on this branch with the fix reverted at the dispatch site, not from a main build. Your mechanistic explanation matches what I saw: the pre-PR path is try_half_gemm with MatrixLayout::transposed(k), i.e. strided B reads through the blocked kernel, which is why transB (32–48 ms) was worse than the untransposed blocked path (~10 ms).

Gates re-run: cargo fmt --all, 1455 ep-cpu lib tests, clippy --all-targets -D warnings on x86_64, verify_documented_env_vars.py, workspace_test_packages.py verify. aarch64 clippy is red on pristine main for three unrelated symbols — #1415.

[justinchuby]

Follow-up measurement: the "next step" in this PR was wrong, and I removed it

While CI was queued I built the follow-up this PR named — adaptive stripe width in gemv_f16_kn — and it does not work. Recording it here rather than leaving a bad lead in a doc that is about to merge.

What I built. kn_stripe_width(n, threads) = clamp(next_multiple_of_32(n / (2*threads)), 32, 512). At n = 3584 that lifts the task count from a fixed 7 to 8 / 16 / 28 at 4 / 8 / 16 threads.

How I measured it. Two binaries (fixed vs adaptive), interleaved per trial through scripts/ort_ab/ab.py --null-control, so each run carries its own noise floor from the same binary under a second name. No env probe — this file's own convention is to pass the route explicitly, not to read env in production.

run	host load	t=4	t=8	t=16
1 (5 trials, 15 runs)	5.0	—	+49.1% (noise 51.9%)	+41.0% (noise 0.5%)
2 (9 trials, 25 runs)	9.9	-15.1% (noise 8.9%)	+1.1% (noise -1.3%)	+9.2% (noise 33.5%)

The two runs disagree in sign. The effect is smaller than this host's between-run variability, so it is not a result and it is not shipped.

The absolute numbers are what actually settle it, and they refute the hypothesis rather than merely failing to confirm it: native p50 stayed at 1.15–1.64 ms in both arms at every thread count. Four times the tasks, same time. Controls behaved (MatMul f16 M=128, which does not use this path, and f32 M=1, which does not touch half_gemv at all, both within noise).

Why, mechanistically. gemv_f16_kn loops p outermost over a w-wide f32 accumulator held in memory:

for (p, &av) in a.iter().enumerate().take(k) {
    let row = &b[p * n + j0..p * n + j0 + w];
    for (slot, &bits) in acc.iter_mut().zip(row.iter()) {
        *slot = half::f16::from_bits(bits).to_f32().mul_add(av, *slot);
    }
}

Every FMA is a load-modify-store against L1, because w overflows the 16 available ymm registers at 512 lanes and at 128 alike. Narrowing the stripe keeps the accumulator in L1 either way — which is precisely why it changed nothing. The fix is to invert the nest onto a register-resident accumulator tile across the whole k contraction, which is what gemv_f16_nk in this PR already does for [N, K]. That is a kernel rewrite, not a constant tweak, and I am claiming it as unproven.

Pushed 667e4c0a6, docs-only: the rejection table and the corrected mechanism now sit in CPU_MATMUL_ASSIGNMENT.md and in the benchmark record, and the PR body no longer points at the dead end. No code changed — half_gemv.rs and gemm.rs are byte-identical to the reviewed c619459d0, so the approval still covers what it reviewed.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	79.82 µs	132.75 µs	+66.3%
🔴	gather/large_f16_threads=1-internal/131072	13.83 µs	20.06 µs	+45.1%
🔴	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	98.30 µs	141.51 µs	+44.0%
🔴	gather/large_bf16_threads=1-internal/131072	12.63 µs	17.84 µs	+41.3%
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	802.45 µs	1.13 ms	+41.1%
🔴	qwen3_sampling_processors/top_k_partial_selection	168.82 µs	232.43 µs	+37.7%
🔴	grammar_masking/llguidance_compute_mask/32	89.42 µs	121.23 µs	+35.6%
⚠️	matmul/small_generic_bf16_threads=8/1x256x256	31.13 µs	40.38 µs	+29.7%
⚠️	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.85 ms	4.90 ms	+27.3%
⚠️	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	66.28 µs	83.41 µs	+25.8%
⚠️	matmul/large_generic_bf16_threads=8/32x1024x1024	1.81 ms	2.24 ms	+23.9%
⚠️	matmul/small_generic_bf16_threads=1/1x256x256	33.37 µs	41.32 µs	+23.8%
⚠️	matmul/small_generic_f16_threads=1/1x256x256	33.22 µs	39.74 µs	+19.6%
⚠️	matmul/large_generic_f16_threads=8/32x1024x1024	99.58 µs	118.97 µs	+19.5%
⚠️	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	520.08 µs	610.21 µs	+17.3%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	551.36 µs	632.40 µs	+14.7%
✅	logit_processing/seven_processor_chain_per_step	341.95 µs	390.74 µs	+14.3%
✅	matmul/small_generic_f16_threads=8/1x256x256	34.45 µs	39.23 µs	+13.9%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.19 ms	2.45 ms	+11.7%
✅	gather/medium_bf16_threads=1-internal/32768	2.55 µs	2.82 µs	+10.9%
✅	matmul/small_generic_f32_threads=1/1x256x256	44.76 µs	49.61 µs	+10.8%
✅	sampling_latency/top_p_per_token	393.63 µs	434.29 µs	+10.3%
✅	tokenization/decode_tokens_per_second	6.79 ms	7.46 ms	+10.0%
✅	gather/large_f32_threads=1-internal/131072	39.07 µs	42.95 µs	+9.9%
✅	gather/medium_f16_threads=1-internal/32768	2.50 µs	2.73 µs	+9.3%
✅	sampling_latency/min_p_per_token	215.39 µs	235.30 µs	+9.2%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	10.37 ms	11.14 ms	+7.5%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.33 ms	2.50 ms	+7.4%
✅	gather/small_bf16_threads=1-internal/4096	555.6 ns	583.0 ns	+4.9%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	616.46 µs	643.05 µs	+4.3%
✅	tokenization/encode_tokens_per_second	422.98 µs	438.94 µs	+3.8%
✅	gather/small_f16_threads=1-internal/4096	519.0 ns	538.5 ns	+3.8%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	6.29 ms	6.50 ms	+3.5%
✅	gather/medium_f32_threads=1-internal/32768	4.51 µs	4.64 µs	+2.9%
✅	gather/small_f32_threads=1-internal/4096	731.1 ns	729.4 ns	-0.2%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.83 ms	2.80 ms	-0.8%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	7.32 ms	7.12 ms	-2.7%
✅	add/small_f16_threads=1-internal/1024	528.9 ns	514.6 ns	-2.7%
✅	sampling_latency/top_k_per_token	63.20 µs	60.94 µs	-3.6%
✅	qwen3_sampling_processors/top_k_top_p_fast	819.78 µs	782.17 µs	-4.6%
✅	kv_cache/alloc_dealloc_pages	52.40 µs	49.85 µs	-4.9%
✅	add/small_f32_threads=1-internal/1024	259.2 ns	246.4 ns	-4.9%
✅	add/medium_f16_threads=1-internal/262144	111.06 µs	104.44 µs	-6.0%
✅	add/large_f16_threads=1-internal/4194304	1.82 ms	1.66 ms	-9.1%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.21 ms	1.09 ms	-9.6%
✅	add/small_bf16_threads=1-internal/1024	513.6 ns	444.3 ns	-13.5%
🟢	sampling_latency/greedy_per_token	4.39 µs	3.71 µs	-15.4%
🟢	add/large_f32_threads=1-internal/4194304	785.31 µs	662.46 µs	-15.6%
🟢	matmul/small_generic_f32_threads=8/1x256x256	51.50 µs	43.35 µs	-15.8%
🟢	add/medium_f32_threads=1-internal/262144	29.60 µs	24.85 µs	-16.1%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	59.08 µs	49.15 µs	-16.8%
🟢	matmul/large_generic_f16_threads=1/32x1024x1024	113.87 µs	92.85 µs	-18.5%
🟢	matmul/medium_generic_bf16_threads=1/32x512x512	783.38 µs	626.44 µs	-20.0%
🟢	reduce_mean/small_f32_threads=1-internal/4096	19.07 µs	15.13 µs	-20.6%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.85 ms	1.46 ms	-21.2%
🟢	add/medium_bf16_threads=1-internal/262144	130.80 µs	102.06 µs	-22.0%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	332.15 µs	253.90 µs	-23.6%
🟢	matmul/medium_generic_f16_threads=1/32x512x512	48.72 µs	36.26 µs	-25.6%
🟢	add/large_bf16_threads=1-internal/4194304	2.26 ms	1.67 ms	-26.4%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 6.76 5.02 6.33 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[codecov]

Codecov Report

❌ Patch coverage is 94.69027% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.50%. Comparing base (06b372c) to head (287f0fe).
⚠️ Report is 41 commits behind head on main.

Files with missing lines	Patch %	Lines
...rates/onnx-runtime-ep-cpu/src/kernels/half_gemv.rs	94.25%	9 Missing and 1 partial ⚠️
crates/onnx-runtime-ep-cpu/src/kernels/gemm.rs	96.15%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1417      +/-   ##
==========================================
+ Coverage   80.36%   80.50%   +0.14%     
==========================================
  Files         378      379       +1     
  Lines      167774   169645    +1871     
  Branches   167774   169645    +1871     
==========================================
+ Hits       134836   136578    +1742     
- Misses      28088    28213     +125     
- Partials     4850     4854       +4     

Flag	Coverage Δ
cli-ort-linux	82.60% <ø> (?)
cli-ort-windows	82.19% <ø> (ø)
offline	80.43% <94.69%> (+0.13%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-runtime-ep-cpu/src/kernels/gemm.rs	89.43% <96.15%> (+0.63%)	⬆️
...rates/onnx-runtime-ep-cpu/src/kernels/half_gemv.rs	96.82% <94.25%> (-1.61%)	⬇️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justinchuby]

Re-validated on latest main, pure-native — and a 12% prefill regression found and fixed

Brought up to main (which meanwhile landed #1381, renaming gemv_f16_kn → gemv_half_kn(format, ..), simd_available() → simd_available(format) and the test helper f16v → halfv(format, ..)) and re-measured from scratch.

The A/B is unusually clean

Only two .rs files differ from main. So both binaries are built in the same tree from the same command, and the "before" one is produced by checking out only gemm.rs and half_gemv.rs from origin/main. Everything else in the two artifacts is identical, and neither runs ONNX Runtime in-process (--native-only) — this is a native-vs-native comparison, not a paired ORT ratio. nm -C on both: 0 MLAS symbols.

8 interleaved reps, --runs 9 --warmups 3, medians. decode_matmul_f32_m1 is the untouched control.

model	t	main	PR	gain	med-of-min gain	parity
gemm_f16_m1	1	31.912	0.823	38.8x	40.1x	PASS
gemm_f16_m1	4	31.905	0.213	149.8x	152.8x	PASS
gemm_f16_m1	8	31.812	0.139	229.7x	242.5x	PASS
gemm_f16_m1	16	31.748	0.122	260.2x	277.9x	PASS
gemm_f16_m128	1	155.284	154.805	1.00x	1.01x	PASS
gemm_f16_m128	4	81.683	81.841	1.00x	1.00x	PASS
gemm_f16_m128	8	70.275	69.892	1.01x	1.00x	PASS
gemm_f16_m128	16	76.284	81.686	0.93x	1.03x	PASS
matmul_f32_m1	1	1.390	1.364	1.02x	1.01x	PASS
matmul_f32_m1	4	1.349	1.358	0.99x	0.96x	PASS
matmul_f32_m1	8	0.917	0.906	1.01x	1.02x	PASS
matmul_f32_m1	16	0.630	0.641	0.98x	0.89x	PASS

These are larger than the 37.8x–90.1x in the PR body, which was measured with ORT alternating in the same process. Removing that contention is what moved 90x to 260x. The original figures were conservative, not wrong.

The regression, and why it was not what it looked like

An earlier revision of this sweep showed gemm_f16_m128 — the transB prefill control, a path this PR declines outright — consistently at 0.87–0.90x. It reproduced across four independent sweeps, so it was not host noise.

try_half_fast_path returns Ok(None) for trans_b before touching anything, so there is no semantic difference at m = 128. Three extra binaries settled what it actually was:

binary	what it is	m = 128 vs main
layout	the PR's 300 lines of new kernel, main's dispatch condition	1.00x
v2	PR, condition written trans_a || (trans_b && m != 1) — at m = 128 this is literally None without calling the function	0.89x
noinline	PR + #[inline(never)] on try_half_fast_path	1.00x

layout rules out the new code's mere presence. v2 rules out the call, because at m = 128 it is runtime-identical to main and still regressed. What is left is the compiler laying execute out around a larger inlined body — carrying a second kernel pushed try_half_fast_path past the point where inlining it was free.

So the fix is #[inline(never)] on a once-per-execute, Option-returning dispatch probe. It has one call site and is not in any loop; the per-row work is in gemv_f16_nk, a separate #[target_feature] function that would not be inlined regardless. 10 reps at --runs 9, median of per-run minima: main 69.7 ms, PR + inline(never) 69.7 ms, PR 78.9 ms.

The measurement is in the doc comment, so the attribute cannot be deleted as decoration.

Local gate matrix — 19/20

fmt · offline build · ep-cpu tests · clippy -D warnings · --features mlas tests · clippy for a native big-endian target · clippy aarch64-unknown-linux-gnu · --no-default-features · --all-features · no-MLAS-symbol artifact scan · cross-compile script · 8 repo guard scripts.

cargo test --release -p onnx-runtime-ep-cpu --lib → 1543 passed, 0 failed.

The one failure is aarch64-pc-windows-msvc, where onnx-genai-ort-sys bindgen needs the Windows SDK ('stdlib.h' file not found). Reproduced on unmodified main — pre-existing. Windows ARM64 is therefore checked only as far as this Linux host can check it; that is the exact scope, not a pass.

This validation pass also turned up two gates already red on main, both now fixed and merged: verify_documented_env_vars (#1536, a knob left in the docs after #1527 deleted the kernel that read it) and cargo fmt (#1546, four files across two crates). Neither was mine. strict_required_status_checks_policy is false here, so a check that passed against an older base stays green against a newer one — which is how both survived.

Review

Opus review: APPROVE_WITH_COMMENTS, and its one BLOCKING finding was that the #[inline(never)] fix was still uncommitted — correct, and now committed in 287f0feea. It independently confirmed there is no semantic difference at m = 128 (the m == 1 short-circuit means b.is_contiguous(), b.validate() and self.prepack.dense() are never evaluated, so nothing is validated, allocated, or cached on a path main never took), that the v2 experiment is decisive, and that #[inline(never)] costs nothing at M = 1. It also traced gemv_f16_nk's k % 32 / k % 8 boundaries at k=33/39/40/48/64 against the five documented summation steps, confirmed every SIMD load is in bounds, and confirmed the order is genuinely independent of thread count and ROW_STRIPE. Non-blocking findings — two docs still naming the renamed gemv_f16_kn, and a "two things" introducing four bullets — are fixed in the same commit.

Still open, unchanged by this PR

transB prefill is still broken and this does not touch it: M = 128 measures 4.0x at 1 thread and 17.0x at 8 against ORT. The GEMV correctly declines M > 1, so prefill still takes the blocked path; closing it needs a packed NT half GEMM, the f16 analogue of #1176's transposed-B SGEMM. The residual loss at high thread counts is the section 1 ceiling, not this kernel — and task granularity was measured and rejected as the cause, which is recorded rather than quietly dropped.