[justinchuby]

Closes the scheduler's llama3_8b_qkv_t8 report.

The report

gemm_nbits_llama3_8b_qkv_t8 was ~10x behind ORT measured native-alone (not the 41x paired artifact), flat across 8/16/32 threads. It reproduces on f8f3878ba at 6.837 ms against the note's 6.90 ms — three significant figures — and we actually regress from 16 to 32 threads while ORT keeps scaling.

Shape is k = 4096, n = 6144, m = 8: 201 M MACs against 12.58 MiB of packed int4.

Two defects

1. The fixed path was switched off. borrowed_int4_prefill_block_enabled() defaulted to false, so every build anyone ran took the row-serial route. This is #1091's default-off SGEMV and #1080's mlas-gated f16 fix a third time: a toggle added "until the win is measured", and then nothing measured it. §36.2 of the phase-18 document had already written down "it is dead in the default build".

2. The kernel re-decoded the weight for every activation row. Per 32-lane chunk of one column the int4 inner loop spends ~18 instructions unpacking nibbles — load, two masks, a shift, two unpacks, eight widen/converts — to feed four FMAs. >80% of the instruction stream is decode, and the row loop sat outside all of it.

Two independent measurements agree:

• Time was exactly linear in m: 0.890 / 0.859 / 0.851 ms per row at m = 1 / 8 / 128. That is what zero row reuse looks like.
• Roofline: 402 MFLOP in 6.87 ms is 58.6 GFLOP/s = 0.61 FLOP/cycle/thread, against ORT's 6.07.

What it was not — built and rejected, not assumed

• Not bandwidth. 100.7 MiB in 6.87 ms is only 14.6 GB/s, and the whole 15.7 MiB working set fits in one 32 MiB L3 slice. I built the fix anyway (64-column tile, rows inner, so weight and activations sit in L2) and it moved nothing at 16 or 32 threads. Removed.
• Not fork/join. ONNX_GENAI_CPU_MM_INT4_PREFILL=1 selects the path whose entire purpose is replacing m fork-joins with one: 6.874 ms vs the default's 6.897 ms.

The fix

borrowed_int4_rowblock_avx2 decodes each K block's nibbles once and drives PREFILL_ROW_BLOCK = 4 rows through the decoded vectors — ~22 instructions per 32 MACs down to ~8.5. Within any one row it is instruction-for-instruction the single-column case of the existing NCols4 kernel. Default flipped on; the variable stays as a kill switch.

ROWS = 4 is chosen by a sweep, not by assertion — I had written "ROWS = 8 measured slower" into the constant's docs before measuring, and 8 is in fact faster on the target cell. 4 takes eight of the twelve cells. 8 takes three t8 cells and one wide one, llama3_8b_qkv_t128, by 1.15x — which the register-pressure argument does not predict and I cannot explain; it is one cell against the seven other wide prefills, which 4 takes by up to 1.31x. Full table in the constant's docs.

Result — 32 threads, native-alone

Separate --native-only / --ort-only arms per sebastian-paired-harness-coresidency. "before" is ONNX_GENAI_CPU_MM_INT4_PREFILL=0, exactly the path that shipped.

cell	before	after	speedup	ours/ORT was	now
llama3_8b_qkv_t8	7.087 ms	2.339 ms	3.03x	9.35	3.09
llama3_8b_qkv_t128	103.250 ms	28.862 ms	3.58x	12.49	3.49
llama3_8b_qkv_t512	509.199 ms	97.591 ms	5.22x	23.97	4.59
llama3_8b_mlp_t8	15.350 ms	5.327 ms	2.88x	10.47	3.63
llama3_8b_mlp_t128	195.611 ms	56.196 ms	3.48x	13.10	3.76
llama3_8b_mlp_t512	810.630 ms	227.322 ms	3.57x	15.49	4.34
qwen3_0p6b_qkv_t8	0.895 ms	0.261 ms	3.43x	9.32	2.72
qwen3_0p6b_qkv_t128	13.645 ms	4.981 ms	2.74x	19.98	7.29
qwen3_0p6b_qkv_t512	51.084 ms	15.337 ms	3.33x	8.50	2.55
qwen3_0p6b_mlp_t8	1.718 ms	0.678 ms	2.53x	10.10	3.99
qwen3_0p6b_mlp_t128	26.891 ms	6.981 ms	3.85x	18.83	4.89
qwen3_0p6b_mlp_t512	104.909 ms	30.250 ms	3.47x	17.16	4.95

Twelve cells, twelve wins, 2.53x–5.22x, nothing regressed. It also scales again — target cell 12.182 → 4.714 (t=8), 5.646 → 1.893 (t=16), 6.596 → 2.332 (t=32).

Row reuse now shows up where its absence used to: per-row cost falls with m — 0.292 / 0.226 / 0.191 ms at m = 8 / 128 / 512, against a flat ~0.9 before.

Numerics

Row blocking ends byte-identity with the per-element path (~2 ULP): that path reduces every 32-lane block to a scalar, this one keeps a lanewise accumulator and reduces once — the trade borrowed_int4_nblock4_avx2 already documents. Nibble decode is untouched.

The test that asserted byte-identity is rewritten, not deleted: it now checks both paths against an f64 oracle over the dequantized weights and requires the blocked path to be no worse, so "it reassociates" cannot quietly become "it lost accuracy". Errors normalise by sum |a| * |w| rather than by the result — these dot products cancel hard (a k = 256 cell sums terms of order 0.25 down to a result of order 0.005), so normalising by the result would measure the conditioning of the test data instead of the kernel.

Byte-identity is still asserted everywhere it is load-bearing: across fan-out partitions, across row-tile boundaries (prefill_is_independent_of_the_row_tiling), and against NCols4 for a single row (rowblock_matches_nblock_for_a_single_row). the_blocked_int4_prefill_is_on_by_default pins the default — the part a future "add a toggle, default off" would silently undo.

Constraints

Pure native throughout: no ORT fallback, no runtime MLAS, no admin CI bypass. Based on merged main (f8f3878ba).

Gates

cargo fmt --check; 1450 onnx-runtime-ep-cpu lib tests (4 new, 1 rewritten); clippy x86_64 --all-targets -D warnings; verify_documented_env_vars.py. aarch64 clippy fails on three pre-existing dead-code errors (WIDE_PREFILL_MACS, prefill_fan_out, prefill_column_grain) that reproduce on a pristine origin/main worktree — filed as #1415, not from this diff.

What this does not fix

Still 2.5x–7.3x behind ORT. The remaining gap is the decode itself: MLAS SQNBit's CompInt8 path quantizes activations to int8 and uses integer dot products, avoiding f32 dequantization entirely, where this kernel still widens every nibble to f32. That is structural, not tuning, and VNNI — which this host lacks — is what makes it pay. qwen3_0p6b_qkv_t128 at 7.29x is now the outlier. The m == 1 decode route is untouched.

Full record: docs/benchmarks/2026-08-19-int4-prefill-row-blocking.md.

[justinchuby]

Opus adversarial review: APPROVE WITH NITS — all three fixed in a2efe2af7.

The review verified the new kernel line-by-line against borrowed_int4_nblock4_avx2 (pointer arithmetic and slice indexing in bounds, affine/zero-point/scale handling and the ragged-tail scalar fallback matching the pre-existing kernels expression-for-expression, row-tile remainder correct), confirmed the default flip is guarded on all three axes (#[cfg(target_arch = "x86_64")] + runtime non-Scalar AVX2 dispatch + m >= 2 + block_size % 32 == 0), confirmed the debug_assert! is not load-bearing in release because the real guard is the runtime branch at the call site, and ran the tests and clippy clean. No blocking findings.

1. The ROWS sweep summary was wrong — again. I wrote that ROWS = 8 "leads only on the t8 cells and only just". My own table has 8 taking llama3_8b_qkv_t128 — a wide cell — by 1.15x (23.201 vs 26.697 ms). Since that paragraph exists specifically to retract an earlier unmeasured ROWS claim, shipping a second inaccurate one about the same sweep is not acceptable. It now states what the table says: 4 takes eight of twelve; 8 takes three t8 cells (up to 1.18x) and that one wide cell; and the wide exception is flagged as unexplained by the register-pressure argument rather than smoothed over. The decision is unchanged — 4's wins are the larger ones (up to 1.31x) and cover the other seven wide prefills. Corrected in the constant's docs, the benchmark doc, and this PR body.

2. Dangling reference to the renamed test. prefill_column_fan_out_matches_its_serial_self's doc comment still pointed at ..._bit_identical. Now ..._within_reassociation.

3. Hardened the dispatch lock (raised as a latent opportunity, taken). every_probe_perturbing_test_takes_the_dispatch_lock now also requires the lock from any test naming borrowed_int4_prefill_block_enabled. It moves no probe counter, but this PR makes it read process-global env on the default path, so an unlocked reader races any test flipping ONNX_GENAI_CPU_MM_INT4_PREFILL. Scoped to this gate deliberately: adding a blanket set_var( pattern would sweep in ~10 unrelated NXRT_CPU_GEMM_BACKEND tests, which is not this PR's business.

Two review observations I am recording rather than acting on: the kernel is target_feature(enable = "avx2,fma") while dispatch detects only AVX2 — that is the pre-existing convention shared by borrowed_int4_nblock4_avx2 and borrowed_int4_block_dot_x86, so not a regression here; and the ~2 ULP reassociation on the now-default path is disclosed, oracle-checked, and passes the SPMD/parity subprocess tests.

Gates after the fixes: cargo fmt --check clean, 1450 lib tests pass / 0 fail, clippy x86_64 --all-targets -D warnings clean. Still blocked on CI — 39 runs queued repo-wide with none executing, main itself included. No admin bypass.

[justinchuby]

Reconciliation required: #1356 and #1431 collide on int4 prefill

I dry-ran the full merge order in a scratch worktree. #1356 and #1431 both make
the #1117 int4 prefill path default-on, by different routes, and they conflict
semantically, not just textually.

• perf(cpu-ep): fuse the int4 dequant into the prefill GEMM pack step (23.7x) #1356 adds int4_prefill_gebp and dispatches it first, with an early
  return, at m >= INT4_PREFILL_GEBP_MIN_ROWS = 4.
• Decode int4 nibbles once per row tile in MatMulNBits prefill #1431 improves borrowed_affine_int4_matmul_prefill, which sits after that
  early return.

So a naive merge makes #1431's kernel unreachable for every m >= 4 — including
the m = 8 cell its own benchmark doc is built on. The measured win would ship
as dead code.

A/B of the two kernels

Both paths exist in one binary and are selectable by their kill switches, so I
measured them against each other with the build, data and harness held fixed and
dispatch as the only variable. Medians of three 20-run native-alone
measurements, ms (qkv = k 4096/n 6144, mlp = k 1024/n 6144):

m	qkv blocked (#1431)	qkv GEBP (#1356)	mlp blocked	mlp GEBP
4	1.31	3.13	0.36	0.70
6	2.24	3.10	0.60	0.70
8	2.56	3.25	0.71	1.03
10	3.47	3.05	0.98	1.11
12	3.78	3.18	1.05	0.96
32	10.00	4.37	2.68	1.31
128	40.47	8.56	10.56	2.44
512	129.07	26.45	44.65	6.93

Neither kernel dominates, and the reason is structural: the blocked kernel
decodes each nibble once per row-block and never materializes a panel, so it is
O(m); GEBP dequantizes B into f32 panels once and reuses them, so it is
nearly flat in m but pays the panel up front. The blocked kernel wins while
m is too small to repay the panel; GEBP wins once it is not.

A useful internal check: at m = 2 and m = 3 the two arms agree to within
noise (1.002 vs 1.015), because GEBP is below its threshold and both arms are
running the same code. That confirms the switches select what I think they do.

Consequence for #1356's threshold

INT4_PREFILL_GEBP_MIN_ROWS = 4 was chosen against the row-serial kernel,
before the blocked path existed. Against the blocked path, GEBP at m = 4 is a
2.4x loss on the qkv shape. Once both land, the correct crossover is 12 —
the first row count where both shapes prefer GEBP. (At 10 they disagree and the
margin is under 15% either way; that is disclosed in the constant's docs.)

Resolution

I will carry this in the merge commit when the second of the two lands: keep
both kernels, raise INT4_PREFILL_GEBP_MIN_ROWS to 12, and rewrite the
constant's doc table with the head-to-head data above. Verified on the merged
tree — 1457 tests pass, and the reconciled default is the lower envelope of
the two columns at every m measured.

Reconciled default vs current main (row-serial), native-alone medians:

cell	main	reconciled	gain	vs ORT after
llama3_8b_qkv m=4	5.05	1.29	3.91x	0.39x
llama3_8b_qkv m=8	9.52	2.55	3.73x	0.26x
llama3_8b_qkv m=32	38.26	4.23	9.04x	0.37x
llama3_8b_qkv m=128	149.97	8.46	17.72x	0.63x
llama3_8b_qkv m=512	638.48	26.00	24.56x	0.82x
qwen3_0p6b_mlp m=8	2.48	0.70	3.54x	0.25x
qwen3_0p6b_mlp m=512	154.08	6.93	22.24x	0.77x

Neither PR needs a code change on its own branch; the interaction only exists in
the merged tree, which is where I will fix it.

[justinchuby]

Latest-main root cause for the *_qkv/mlp_t8 gap: zero row amortization

Re-measured on latest main (a7bd79f72/dbade34c1), pure-native (0 MLAS symbols), native-alone vs ORT-alone. This identifies why these cells lose, and it is not what issue #1453 assumed.

1. The kernel is compute/issue-bound, not bandwidth-bound

Interleaved thread scaling, llama3_8b_qkv_t8, native-alone:

threads	1	2	4	8	16
qkv (ms)	56.2	28.1	14.2	10.7	6.43
mlp (ms)	185.3	102.8	46.4	27.7	14.29

That is 8.75x / 12.97x on 16 threads. Sustained DRAM on this host is 75.8 GB/s, which a bandwidth-bound kernel would saturate by ~4–6 threads. It scales, so it is issue-bound — consistent with borrowed_int4_nblock4_avx2 spending ~16 decode instructions per 4 FMAs (80% of the instruction stream is nibble unpacking).

2. The decisive measurement: cost per row is flat

Interleaved m-sweep on llama3_8b_qkv, latest main:

m	1	2	4	8	16	32	64	128
native / m (ms/row)	1.209	1.446	1.325	1.348	1.276	1.307	1.256	1.284
ORT / m (ms/row)	0.252	0.243	0.135	0.094	0.069	0.132	0.094	0.050

Native per-row cost is constant at ~1.28 ms/row from m=1 to m=128 — B is fully re-decoded for every single row, zero amortization. ORT's per-row cost falls 5x (0.252 → 0.050): it dequantizes once and runs SGEMM over all rows.

That is the entire gap, and it is why the deficit grows with m (4.7x at m=1 → 20.8x at m=128). The fix is amortizing the decode across rows — precisely what #1356 (GEBP panels) and #1431 (column-blocked row reuse) do. A better inner loop cannot close this; the f32 widen+convert has a hard floor of ~8 instructions per 32 values.

3. The reconciled stack, validated on latest main

origin/main + #1356 + #1431 with the measured INT4_PREFILL_GEBP_MIN_ROWS = 12 crossover. Three arms interleaved per iteration; statistic is the minimum of 4 (see note 4).

m	main (ms)	reconciled (ms)	speedup	gap vs ORT (main)	gap vs ORT (reconciled)
1	1.202	1.199	1.00x	4.73x	4.72x
2	2.373	1.005	2.36x	5.08x	2.15x
4	4.808	1.329	3.62x	8.99x	2.48x
8	10.027	2.551	3.93x	12.27x	3.12x
16	20.722	4.066	5.10x	5.22x	1.03x
32	40.145	4.942	8.12x	10.16x	1.25x
64	82.024	6.079	13.49x	15.91x	1.18x
128	159.562	9.234	17.28x	20.84x	1.21x

native/m for the reconciled stack now falls 1.20 → 0.094 ms/row, the same shape as ORT's curve. m=1 is exactly neutral (1.00x) — as it must be, since both prefill paths are gated (m >= 12 for GEBP, m >= 2 for blocked), so decode executes identical code.

Remaining loss is concentrated at m=4–8 (2.5–3.1x); from m=16 up we are within 1.03–1.25x of ORT.

4. Two measurement artifacts I found and am retracting

Both were mine, and both would have become false claims:

• "16 threads beats 32 by 1.69x." A sequential thread sweep showed llama3_8b_qkv_t8 at 9.975 ms (t32) vs 5.892 ms (t16). Re-run interleaved, the difference vanished: 1.00x on three of four cells. It was host drift, not SMT oversubscription.
• "The reconciled stack regresses m=1 by 47%." Medians said 1.209 → 1.777 ms. But intra-arm spread at m=1 is 2.14–2.57x, so a median of 9 is meaningless there; the minima agreed to 1.2% (1.184 vs 1.170), and static analysis confirms no dispatch change is even reachable at m=1.

Methodology this establishes for these cells: interleave the arms every iteration, and use the minimum, not the median — this host is shared, so contention can only ever add time.

Consequence for issue #1453

#1453 proposed an int8-activation inner product (vpmaddubsw/vpmaddwd) on the theory that nibble-decode ALU cost was the binding constraint. The ALU-bound diagnosis is confirmed, but the premise that it needs a numerics-changing fix is not: amortization recovers the gap to 1.03–1.25x for m>=16 while staying bit-identical and fp32-exact. I am deprioritizing #1453 accordingly and retargeting the residual at m=4–8.

[justinchuby]

Where the residual m=4–8 gap lives: GEBP materializes a panel 6.4x larger than its own input

Follow-up to the amortization result. Measured on latest main + #1356 + #1431 (MIN_ROWS = 12), pure-native, 0 MLAS symbols, minimum of 4 interleaved runs.

Fitting the reconciled GEBP curve t = fixed + marginal * m (least squares over m = 16/32/64/128):

cell	f32 panel	measured m=16/32/64/128 (ms)	fixed	marginal
llama3_8b_qkv (k=4096, n=6144)	100.7 MB	3.849 / 4.731 / 6.066 / 9.165	3.135 ms	0.0470 ms/row
qwen3_0p6b_qkv (k=1024, n=3072)	12.6 MB	0.714 / 0.880 / 1.160 / 1.710	0.587 ms	0.0088 ms/row

The fixed term is the panel, and it is running at DRAM speed:

• llama3_8b_qkv: panel write + read-back = 2 x 100.66 MB = 201.3 MB in 3.135 ms = 64.2 GB/s, which is 84.7% of this host's 75.8 GB/s sustained DRAM read.
• The int4 source is only 15.73 MB (12.58 MB packed + 3.15 MB scales). So we move 201 MB of traffic to consume 15.7 MB of weights — 12.8x amplification.
• Shrinking the panel 8x shrinks the fixed cost 5.3x (3.135 -> 0.587 ms), confirming the term tracks panel size rather than being fixed overhead.

Why this is the whole remaining story

Nibble decode itself is cheap when done once: 25.2M weights at ~12 instructions per 32 values is ~9.4M instructions, which at 16 cores is well under 0.1 ms. The earlier finding that the kernel is ALU-bound was about the row-serial path, which paid that cost m times. Having fixed that, the binding constraint moved — GEBP no longer re-decodes, but it pays a 6.4x memory blow-up (int4 -> f32) and streams the result through DRAM.

This also explains the shape of the residual precisely: the fixed term is amortized by large m (which is why m>=16 is already within 1.03–1.25x of ORT) and is not amortized at m=4–8, which is exactly where the remaining 2.5–3.1x sits.

The fix

ORT's CompFp32 never materializes a full panel: it dequantizes a B tile sized for L2, consumes it for all m rows while resident, and discards it. We should do the same — fuse the dequant into the blocked GEMM at L2-tile granularity instead of expanding all of B up front.

Expected effect on llama3_8b_qkv: remove ~3.1 ms of the fixed term, taking m=16 from 3.85 ms to roughly 0.8 ms, and collapsing the m=4–8 residual since the panel cost that dominates there disappears. It is also bit-identical — the same values are computed in the same order, only their lifetime changes.

Filed as a separate issue; not in scope for #1356/#1431, which should land as measured (2.36x–17.28x over main, m=1 exactly neutral).

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	40.94 µs	112.89 µs	+175.7%
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	498.14 µs	1.24 ms	+148.0%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	38.90 µs	83.54 µs	+114.8%
🔴	matmul/medium_generic_f32_threads=8/32x512x512	885.92 µs	1.86 ms	+109.8%
🔴	matmul/medium_generic_bf16_threads=8/32x512x512	357.64 µs	699.50 µs	+95.6%
🔴	matmul/large_generic_f32_threads=8/32x1024x1024	3.59 ms	6.70 ms	+86.4%
🔴	matmul/small_generic_f32_threads=8/1x256x256	31.42 µs	57.38 µs	+82.6%
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.27 ms	2.25 ms	+77.6%
🔴	gather/large_bf16_threads=1-internal/131072	9.00 µs	15.66 µs	+74.1%
🔴	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	49.36 µs	84.50 µs	+71.2%
🔴	gather/large_f16_threads=1-internal/131072	8.89 µs	14.81 µs	+66.6%
🔴	matmul/medium_generic_f16_threads=8/32x512x512	27.67 µs	44.10 µs	+59.4%
🔴	matmul/medium_generic_f16_threads=1/32x512x512	27.42 µs	40.09 µs	+46.2%
🔴	matmul/small_generic_bf16_threads=1/1x256x256	28.37 µs	40.81 µs	+43.9%
🔴	gather/large_f32_threads=1-internal/131072	23.23 µs	32.45 µs	+39.7%
🔴	matmul/large_generic_f16_threads=1/32x1024x1024	73.25 µs	98.69 µs	+34.7%
🔴	matmul/small_generic_f16_threads=1/1x256x256	27.71 µs	36.97 µs	+33.4%
🔴	matmul/small_generic_f16_threads=8/1x256x256	27.95 µs	36.99 µs	+32.3%
⚠️	matmul/large_generic_f16_threads=8/32x1024x1024	77.89 µs	101.06 µs	+29.7%
⚠️	matmul/small_generic_f32_threads=1/1x256x256	33.57 µs	43.45 µs	+29.4%
⚠️	matmul/medium_generic_bf16_threads=1/32x512x512	492.47 µs	621.50 µs	+26.2%
⚠️	matmul/small_generic_bf16_threads=8/1x256x256	30.95 µs	38.40 µs	+24.1%
⚠️	matmul/large_generic_f32_threads=1/32x1024x1024	8.53 ms	10.32 ms	+21.0%
⚠️	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	343.55 µs	415.38 µs	+20.9%
⚠️	gather/medium_f16_threads=1-internal/32768	2.24 µs	2.64 µs	+18.0%
⚠️	matmul/large_generic_bf16_threads=1/32x1024x1024	1.85 ms	2.18 ms	+17.3%
⚠️	matmul/medium_generic_f32_threads=1/32x512x512	2.14 ms	2.51 ms	+17.2%
⚠️	sampling_latency/top_k_per_token	47.88 µs	56.06 µs	+17.1%
⚠️	sampling_latency/min_p_per_token	191.21 µs	223.58 µs	+16.9%
⚠️	tokenization/decode_tokens_per_second	5.47 ms	6.30 ms	+15.2%
✅	gather/medium_bf16_threads=1-internal/32768	2.29 µs	2.63 µs	+14.6%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	1.95 ms	2.20 ms	+12.6%
✅	gather/small_f16_threads=1-internal/4096	447.8 ns	503.0 ns	+12.3%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.27 ms	5.88 ms	+11.6%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	480.65 µs	531.69 µs	+10.6%
✅	sampling_latency/top_p_per_token	352.96 µs	384.81 µs	+9.0%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.26 ms	3.54 ms	+8.5%
✅	sampling_latency/greedy_per_token	2.96 µs	3.21 µs	+8.4%
✅	tokenization/encode_tokens_per_second	357.11 µs	380.44 µs	+6.5%
✅	gather/small_bf16_threads=1-internal/4096	443.0 ns	463.3 ns	+4.6%
✅	gather/medium_f32_threads=1-internal/32768	3.45 µs	3.61 µs	+4.5%
✅	qwen3_sampling_processors/top_k_partial_selection	137.86 µs	138.05 µs	+0.1%
✅	gather/small_f32_threads=1-internal/4096	686.4 ns	684.0 ns	-0.3%
✅	grammar_masking/llguidance_compute_mask/32	82.84 µs	78.09 µs	-5.7%
✅	logit_processing/seven_processor_chain_per_step	366.18 µs	332.03 µs	-9.3%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.09 ms	986.32 µs	-9.8%
✅	add/medium_f16_threads=1-internal/262144	132.28 µs	118.43 µs	-10.5%
✅	reduce_mean/small_f32_threads=1-internal/4096	20.58 µs	18.28 µs	-11.2%
✅	qwen3_sampling_processors/top_k_top_p_fast	749.03 µs	654.61 µs	-12.6%
✅	add/large_f16_threads=1-internal/4194304	2.23 ms	1.94 ms	-13.0%
🟢	kv_cache/alloc_dealloc_pages	49.13 µs	40.60 µs	-17.4%
🟢	add/medium_f32_threads=1-internal/262144	30.73 µs	25.29 µs	-17.7%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	306.93 µs	249.12 µs	-18.8%
🟢	add/large_bf16_threads=1-internal/4194304	2.51 ms	2.03 ms	-19.1%
🟢	add/small_bf16_threads=1-internal/1024	575.4 ns	439.6 ns	-23.6%
🟢	add/small_f16_threads=1-internal/1024	734.5 ns	462.9 ns	-37.0%
🟢	add/medium_bf16_threads=1-internal/262144	171.51 µs	108.06 µs	-37.0%
🟢	add/small_f32_threads=1-internal/1024	332.1 ns	194.0 ns	-41.6%
🟢	add/large_f32_threads=1-internal/4194304	1.19 ms	621.92 µs	-47.8%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.06 3.50 6.01 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[codecov]

Codecov Report

❌ Patch coverage is 98.03922% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.30%. Comparing base (4a9f4ec) to head (ec1869d).
⚠️ Report is 80 commits behind head on main.

Files with missing lines	Patch %	Lines
...es/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs	98.03%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1431       +/-   ##
===========================================
- Coverage   82.10%   80.30%    -1.80%     
===========================================
  Files          12      377      +365     
  Lines        5471   166791   +161320     
  Branches     5471   166791   +161320     
===========================================
+ Hits         4492   133947   +129455     
- Misses        780    28006    +27226     
- Partials      199     4838     +4639     

Flag	Coverage Δ
cli-ort-linux	82.60% <ø> (?)
cli-ort-windows	82.10% <ø> (ø)
offline	80.22% <98.03%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...es/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs	78.52% <98.03%> (ø)

... and 364 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justinchuby]

Converged on latest main (000694cc7), Opus-reviewed, full local gate green

What changed since the last push

Two commits on top of the row-blocked kernel:

1. The GEBP crossover is now size-aware. This branch was cut before perf(cpu-ep): fuse the int4 dequant into the prefill GEMM pack step (23.7x) #1356 landed, and perf(cpu-ep): fuse the int4 dequant into the prefill GEMM pack step (23.7x) #1356 changed what this kernel competes against. INT4_PREFILL_GEBP_MIN_ROWS was tuned at 4 against the row-serial borrowed kernel, which re-decoded every packed byte once per row. The column-blocked kernel here amortises that decode over four rows, which moves the crossover out by 3-6x. Left alone the two changes compose into a regression: every prefill with 4 <= m < 12 would take a GEBP path this kernel beats.

   Same binary, ONNX_GENAI_CPU_MM_INT4_GEBP the only variable, native-alone, min of 5, ratio = column-blocked / GEBP (>1.00x = GEBP wins):

   m	qwen3-0.6B QKV (1.0 MB)	llama3-8B QKV (12.6 MB)	llama3-8B MLP (29.4 MB)
   2	0.50x	0.33x	0.29x
   4	0.32x	0.36x	0.44x
   8	0.50x	0.73x	0.81x
   12	0.69x	1.01x	1.01x
   16	0.80x	1.35x	1.26x
   24	1.14x	1.80x	1.98x
   32	1.68x	2.24x	2.35x

   It is not one row count — it tracks whether the packed weight stays cache-resident across the row loop. So: 12 above 4 MB of packed weight, 24 at or below. A flat 12 would hand the small shape a 1.45x regression at m = 12..23; a flat 24 would give up 1.35x-1.80x on the large shapes at m = 16..23. The 4 MB split is fitted to exactly two weight sizes. It is a threshold, not a model.

2. Review fix (blocking + substantive). Opus review caught two things I missed:

   • clippy::assertions_on_constants in the new test — a hard error under -D warnings, would have failed CI.
   • Raising the crossover strands block_size = 16. Both column-blocked kernels require 32-element blocks, so for the ONNX-minimum block size GEBP is the only vectorised prefill that exists; raising its threshold drops m ∈ [4, 12) to a scalar per-block dot on the decode pool (the comment right above measures that pool at ~2.4x slower for prefill) rather than to a better kernel. Those weights now hold at the pre-retune threshold of 4. Pinned by int4_prefill_gebp_crossover_holds_for_blocks_the_row_kernels_reject.

   Non-blocking and accepted as-is: self.k * self.n / 2 under-counts the true packed size for ragged K (real count is n * ceil(k/block_size) * block_size/2). Immaterial against a 4 MB threshold; usize is 64-bit on every target that compiles this path.

#1431 vs origin/main 000694cc7

Separate binaries built from each tree, native-alone, min of 7 interleaved reps, parity=PASS on all 18:

cell (int4)	main	#1431	speedup
qwen3_0p6b_qkv_t1	0.278	0.274	1.01x
qwen3_0p6b_qkv_t8	0.401	0.347	1.16x
qwen3_0p6b_qkv_t128	1.464	1.485	0.99x
qwen3_0p6b_mlp_t1	0.435	0.416	1.05x
qwen3_0p6b_mlp_t8	0.771	0.565	1.36x
qwen3_0p6b_mlp_t128	2.357	2.343	1.01x
llama3_8b_qkv_t1	1.284	1.242	1.03x
llama3_8b_qkv_t4	2.725	1.012	2.69x
llama3_8b_qkv_t8	2.961	1.933	1.53x
llama3_8b_qkv_t16	3.241	3.259	0.99x
llama3_8b_qkv_t32	4.023	4.039	1.00x
llama3_8b_qkv_t128	8.280	8.301	1.00x
llama3_8b_mlp_t1	2.822	2.683	1.05x
llama3_8b_mlp_t4	6.024	2.944	2.05x
llama3_8b_mlp_t8	6.481	4.424	1.46x
llama3_8b_mlp_t16	7.061	7.004	1.01x
llama3_8b_mlp_t32	8.589	8.595	1.00x
llama3_8b_mlp_t128	17.387	17.483	0.99x

18 cells, worst 0.99x. The wins land exactly where predicted — m = 4..8, the band the retune reclaims — and m = 1, 16, 32, 128 are neutral because both trees dispatch identically there. t1 is untouched by design.

Local gate matrix

Run on the merged head ec1869da8, latest main merged in normally:

step	result
A cargo fmt --all --check	PASS
B build offline (default features)	PASS
C cargo test -p onnx-runtime-ep-cpu	PASS
D cargo clippy --all-targets -- -D warnings	PASS
E test with mlas feature	PASS
F clippy native backend	PASS
G clippy --target aarch64-unknown-linux-gnu -- -D warnings	PASS
H check --target aarch64-pc-windows-msvc	FAIL — environmental
I ep-cpu --no-default-features	PASS
J ep-cpu --all-features	PASS
K no-MLAS-symbols in default artifact	PASS (nm -D = 0)
L scripts/check_cross_compile.sh	PASS
S × 8 guard scripts	PASS
Miri (strided, provider, dtype, task_runtime)	PASS, 4/4 suites, 0 failures

19/20. Step H fails because onnx-genai-ort-sys bindgen needs the Windows SDK (fatal error: 'stdlib.h' file not found) — reproduced on plain origin/main in a clean worktree, so it is pre-existing and not caused by this branch. scripts/check_cross_compile.sh is a no-op for cross-arch on Linux (it resolves TARGET to the native triple), which is why step G runs the aarch64 clippy explicitly rather than trusting L.

Every item touched here is #[cfg(target_arch = "x86_64")] with no non-x86 caller, so aarch64 keeps the pre-existing route unchanged — step G is the evidence for that.

Remaining losses

Unchanged by this PR and stated in the doc: int4 prefill is still 2.5x-7.3x behind ORT, and the gap is now structural rather than a tuning one — MLAS SQNBit's CompInt8 path quantises activations to int8 and uses integer dot products, where this kernel still widens every nibble to f32. AVX-512/VNNI, which is exactly what would make an int8 decode path pay off, is untestable on this host.