[justinchuby]

What

New reproducible perf/capture harness marlin_bench (crates/onnx-genai-bench/src/bin/marlin_bench.rs, --features bench-native,cuda) that gates the full Marlin int4 GEMM build (Lever A, greenlit in #957 / docs/research/decode-remaining-levers-feasibility.md), plus the recorded BEFORE baseline on today's code so the moment Marlin lands the same commands produce a decisive delta.

New file only — zero overlap with Deckard's kernel branch (squad/marlin-kernel). Uses only the public NativeDecodeSession::load_with_resolved_io API.

The four gate axes (BEFORE baseline, H200 SM90)

Full table, per-shape ncu rows, and exact reproduce commands are in the decision drop .squad/decisions/inbox/sebastian-marlin-bench.md.

Axis 1 — M=1 int4 GEMV DRAM% (ncu --graph-profiling node): qwen MLP gate_up ~29.5% (reproduces the doc §3 "~29% DRAM on an MLP shape" precondition), down_proj ~40%, attn ~24%; glm untuned gemv_f16_general_bs 11–17%. All far below the ≥55% Marlin M=1 pass line → GEMV is latency/dispatch-bound, as diagnosed.

Axis 2 — M-scaling wall (the cliff) (marlin_bench, M∈{1,2,4,8,16}, min ms): glm past=32 cliff 67.7 ms, M=8 91 ms, tail 1.65 ms/row — reproduces Deckard's Increment-0 (~67 ms / ~87 ms / ~1.85). glm past=512 cliff 88.7 ms; qwen 120.9 ms.

Axis 3 — e2e tok/s native vs ORT (profile_native --steady): qwen native 127.8 vs ORT 89.2 tok/s (1.43×); glm native 98.6 tok/s (matches the 96.4 doc reference). ORT can't load the native-authored glm ONNX (GroupQueryAttention rotary_embedding_dim) — foundry-local ORT-genai head-to-head is a follow-up.

Axis 4 — capture-safety re-probe (existing leverb_phase0 test, re-run): segments=41, captured M=8/M=1 = 8.77×, NO-GO for Lever B today. Wired to re-run against Marlin (target: segments→1, ratio→~1). Its Part-C eager sweep independently reproduces Axis 2's cliff (67.2 ms) / tail (1.88 ms/row) — two harnesses agree.

Measurement discipline / portability

• Shared 8× H200 box: GPU pinned to a verified-idle high index, nvidia-smi re-checked before runs; harness reports median + p10/p90/max so contention is visible; min is the contention-invariant estimator used in the table.
• Rule 11: every number is the current split-K/general_bs fallback layout (pre-Marlin), so re-running the identical harness proves <SM80 does not regress. marlin_bench prints device compute_cap.

Validation

cargo fmt --all -- --check clean · cargo clippy clean · marlin_bench + profile_native build under bench-native,(bench-ort,)cuda.

Refs #957 · docs/research/decode-remaining-levers-feasibility.md

⚠️ Draft: this is the BEFORE baseline + harness. Multi-turn — AFTER numbers land once Marlin (Deckard) merges.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.77%. Comparing base (401af46) to head (fa32d1b).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #962      +/-   ##
==========================================
+ Coverage   78.98%   79.77%   +0.79%     
==========================================
  Files         354      356       +2     
  Lines      147988   150211    +2223     
  Branches   147988   150211    +2223     
==========================================
+ Hits       116883   119837    +2954     
+ Misses      26465    25718     -747     
- Partials     4640     4656      +16     

Flag	Coverage Δ
cli-ort-linux	83.52% <ø> (ø)
cli-ort-windows	83.02% <ø> (-0.10%)	⬇️
mlas	81.33% <ø> (?)
offline	79.61% <ø> (+0.80%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	40.66 µs	77.82 µs	+91.4%
🔴	gather/small_f32_threads=1-internal/4096	659.1 ns	1.01 µs	+53.4%
🔴	gather/small_f16_threads=1-internal/4096	505.0 ns	722.8 ns	+43.1%
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	497.36 µs	695.27 µs	+39.8%
🔴	matmul/large_generic_f16_threads=1/32x1024x1024	81.47 µs	109.97 µs	+35.0%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	42.23 µs	56.11 µs	+32.9%
🔴	matmul/large_generic_f16_threads=8/32x1024x1024	90.13 µs	119.53 µs	+32.6%
⚠️	matmul/large_generic_bf16_threads=1/32x1024x1024	1.95 ms	2.51 ms	+28.6%
⚠️	matmul/large_generic_f32_threads=8/32x1024x1024	4.49 ms	5.77 ms	+28.4%
⚠️	reduce_mean/large_f32_threads=1-internal/262144	909.52 µs	1.12 ms	+22.9%
⚠️	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.31 ms	4.03 ms	+21.8%
⚠️	sampling_latency/top_p_per_token	352.06 µs	415.94 µs	+18.1%
⚠️	add/large_f16_threads=1-internal/4194304	42.00 ms	49.15 ms	+17.0%
⚠️	qwen3_sampling_processors/top_k_full_sort_baseline	1.96 ms	2.28 ms	+16.8%
⚠️	sampling_latency/top_k_per_token	50.26 µs	58.58 µs	+16.5%
⚠️	tokenization/encode_tokens_per_second	351.11 µs	406.00 µs	+15.6%
⚠️	matmul/medium_generic_f16_threads=8/32x512x512	31.37 µs	36.24 µs	+15.5%
⚠️	sampling_latency/min_p_per_token	205.87 µs	237.53 µs	+15.4%
✅	reduce_mean/small_f32_threads=1-internal/4096	14.36 µs	16.01 µs	+11.5%
✅	qwen3_sampling_processors/top_k_partial_selection	129.45 µs	143.82 µs	+11.1%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	347.09 µs	383.31 µs	+10.4%
✅	tokenization/decode_tokens_per_second	5.85 ms	6.44 ms	+10.1%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.34 ms	1.45 ms	+8.3%
✅	reduce_mean/medium_f32_threads=1-internal/65536	236.79 µs	255.93 µs	+8.1%
✅	add/small_f16_threads=1-internal/1024	13.50 µs	14.39 µs	+6.6%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	524.77 µs	552.61 µs	+5.3%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	71.99 µs	75.77 µs	+5.3%
✅	matmul/small_generic_f32_threads=8/1x256x256	41.09 µs	42.76 µs	+4.1%
✅	add/large_bf16_threads=1-internal/4194304	43.21 ms	44.78 ms	+3.6%
✅	sampling_latency/greedy_per_token	3.06 µs	3.16 µs	+3.3%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	529.72 µs	539.67 µs	+1.9%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.48 ms	9.49 ms	+0.1%
✅	add/small_bf16_threads=1-internal/1024	14.19 µs	14.11 µs	-0.6%
✅	grammar_masking/llguidance_compute_mask/32	82.10 µs	80.79 µs	-1.6%
✅	gather/medium_f32_threads=1-internal/32768	4.54 µs	4.47 µs	-1.7%
✅	add/small_f32_threads=1-internal/1024	213.0 ns	207.9 ns	-2.4%
✅	add/large_f32_threads=1-internal/4194304	44.22 ms	42.35 ms	-4.2%
✅	matmul/medium_generic_f16_threads=1/32x512x512	34.97 µs	33.05 µs	-5.5%
✅	add/medium_bf16_threads=1-internal/262144	3.10 ms	2.82 ms	-8.9%
✅	gather/medium_bf16_threads=1-internal/32768	2.78 µs	2.53 µs	-9.0%
✅	logit_processing/seven_processor_chain_per_step	362.95 µs	329.85 µs	-9.1%
✅	add/medium_f32_threads=1-internal/262144	3.02 ms	2.73 ms	-9.6%
✅	gather/small_bf16_threads=1-internal/4096	572.4 ns	517.1 ns	-9.7%
✅	matmul/medium_generic_f32_threads=8/32x512x512	1.19 ms	1.07 ms	-10.3%
✅	qwen3_sampling_processors/top_k_top_p_fast	753.58 µs	661.14 µs	-12.3%
✅	gather/medium_f16_threads=1-internal/32768	2.77 µs	2.39 µs	-13.5%
✅	add/medium_f16_threads=1-internal/262144	3.21 ms	2.77 ms	-13.7%
✅	kv_cache/alloc_dealloc_pages	46.39 µs	39.68 µs	-14.5%
🟢	matmul/medium_generic_f32_threads=1/32x512x512	2.52 ms	2.13 ms	-15.3%
🟢	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.87 ms	5.78 ms	-15.9%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	38.82 µs	31.25 µs	-19.5%
🟢	matmul/small_generic_f32_threads=1/1x256x256	48.95 µs	39.16 µs	-20.0%
🟢	matmul/small_generic_f16_threads=1/1x256x256	41.42 µs	32.73 µs	-21.0%
🟢	gather/large_f16_threads=1-internal/131072	14.82 µs	11.35 µs	-23.4%
🟢	gather/large_bf16_threads=1-internal/131072	16.90 µs	12.35 µs	-26.9%
🟢	matmul/small_generic_f16_threads=8/1x256x256	49.02 µs	35.46 µs	-27.7%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	534.65 µs	386.41 µs	-27.7%
🟢	gather/large_f32_threads=1-internal/131072	41.91 µs	29.44 µs	-29.8%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	51.15 µs	31.55 µs	-38.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 1.81 2.83 5.76 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

Follow-up: glm-vs-ORT resolved + long-prompt prefill sweep (head 1297ab5d)

glm ORT head-to-head (was blocked). No stock ORT loads the native-authored glm ONNX — ORT 1.27 EP, pip ORT 1.28, and onnxruntime-genai 0.14.1's bundled ORT all reject its GroupQueryAttention (Unrecognized attribute: rotary_embedding_dim); no foundry-local glm package exists. Doc-specified path = re-export: ran the onnxruntime-genai model builder on glm-4-9b-gptq-source (gptq→int4, -p int4 -e cuda) → clean stock ORT-genai chatglm int4 package that loads + runs.

Decisive BEFORE numbers (H200, idle GPU, contention-invariant min):

Prompt len	glm native prefill	glm ORT-genai prefill	gap
128	199.7 tok/s	8 002 tok/s	40×
512	214.2 tok/s	21 044 tok/s	98×
1024	217.1 tok/s	26 331 tok/s	121×

Decode: glm native ~95 vs ORT-genai 233 tok/s. qwen native prefill flat 147→154 tok/s (128→1024).

Interpretation: native prefill is flat and scales linearly with wall → the M=len forward gets zero batching efficiency because MatMulNBits has no efficient M>1 int4 GEMM. ORT-genai's prefill rises with length as a real GEMM amortizes. The ~100–120× prefill gap is exactly the Marlin M>1 gate — after Marlin the same --prefill-lens 128,512,1024 command must show native rising toward the ORT curve.

Caveats: ORT side is builder-requantized (gptq→int4), accuracy parity not verified; methodology differs from qwen (same-ONNX ORT-EP). Prefill M>1 gap is architectural regardless of quant. Full table + reproduce commands in .squad/decisions/inbox/sebastian-marlin-bench.md.

Truthful negative: 'native beats ORT' holds for qwen (1.43×) but not glm today — Marlin is needed even more urgently for glm prefill.

[justinchuby]

AFTER measurements — Marlin (Lever A) landed (snapshot c692bffe, H200 GPU7)

Merged Deckard's P1 (full M>1 Marlin, zero tiled fallbacks, byte-identical) and ran every gate MARLIN-OFF vs ON (ONNX_GENAI_MARLIN_M_GT_1=1) on one binary. Contention-invariant min.

🎯 HEADLINE — capture re-probe (leverb-phase0 Part D INC0, glm)

Metric	BEFORE (OFF)	AFTER (ON)	Δ	target
captured M=8/M=1 B*	8.76×	4.99×	−43%	≤~2
M=8 segments	41	120¹	—	~1
M=8 seams	GQA×40, MatMulNBits×240, MatMul×1, LN×80	GQA×40, MatMul×1, LN×80 (NBits×240 gone)	−240	—
eager tail slope	1.814 ms/row	0.744	−59%	—
verdict	NO-GO	NO-GO (much closer)		GO@B*≤2

¹ segments rose because Marlin made all 240 MatMulNBits nodes capture-safe — the graph's now-capturable runs surface as distinct segments bounded by the residual GroupQueryAttention×40 + SkipSimplifiedLayerNorm×80 seams. Those are the capture-safety lever (not Marlin's scope). Marlin removed the GEMM barrier; the residual barrier to segments→1/B*≤2 is now cleanly isolated to attention+layernorm.

Prefill sweep (native OFF→ON) + ORT gap

L	glm OFF	glm ON	×	ORT-genai	gap OFF→ON
128	209.8	415.4	1.98×	8 002	38→19×
512	216.4	417.3	1.93×	21 044	97→50×
1024	218.4	425.6	1.95×	26 331	121→62×

qwen prefill 147→291 / 152→295 / 154→300 tok/s (~1.97×). Marlin ~doubles native prefill, halving the glm-vs-ORT gap.

M-scaling wall (past=32, min ms)

Model	cliff (M1→M2)	tail ms/row	M16/M1
glm OFF→ON	67.3→37.7	2.21→0.93 (−58%)	8.37→4.77×
qwen OFF→ON	94.2→78.1	3.26→1.10 (−66%)	13.45→9.24×

Tail-slope collapse (the pure GEMM signal) confirms Deckard's kernel 6.9×→~2× at the forward level. The cliff (per-op dispatch + attn/LN, not GEMM) drops less → again the capture-safety lever.

Portability: Marlin-OFF rows == portable tiled fallback == unchanged BEFORE baseline → <SM80 provably does not regress.

Verdict: Lever A is a decisive GEMM-level win (B* −43%, prefill ~2×, all NBits seams cleared) and moved speculative-capture 43% toward the GO line. Full B*≤2 GO now requires the complementary GQA+LayerNorm capture-safety lever. Full tables + reproduce commands in .squad/decisions/inbox/sebastian-marlin-bench.md.

[justinchuby]

Capture re-probe @ c842b759 — segments 120 → 1 (Deckard's GQA + LayerNorm M>1 capture landed)

Merged c842b759 (18d00f90: GroupQueryAttention×40 + SkipSimplifiedLayerNorm×80 now advertise M>1 capture). leverb-phase0 Part D INC0, glm-4-9b, M=8, pre-warmed, GPU7.

Config	segments	residual seams	captured M=8	B*
Marlin OFF (BEFORE)	41	GQA×40, NBits×240, MatMul×1, LN×80	87.3 ms	8.76×
Marlin M>1 (c692bffe)	120	GQA×40, MatMul×1, LN×80	50.7 ms	4.99×
+ GQA/LN capture-safe (canonical)	1 ✅	MatMul×1	51.8 ms	5.10×
+ split-K (exploratory)	1	MatMul×1	27.3 ms	2.69×

segments → 1 achieved — the whole M=8 verify captures as a single replayable graph; only a lone MatMul×1 seam remains. But capture-safety alone did NOT move B* (5.10× vs 4.99×): collapsing 120→1 segments moved the wall ~0 (50.7→51.8ms) → the captured M=8 wall is compute-bound, not segmentation-bound.

Split-K is the lever that moves B*: exploratory ONNX_GENAI_MARLIN_SPLITK=1 (also capture-safe, also byte-identical) drops B* 5.10×→2.69× (wall 51.8→27.3ms) — right at the ≤2 GO line. Path to speculative-capture GO = capture-safety (done) + split-K on the M=K verify.

Numerics: greedy tokens byte-identical to tiled with AND without split-K (24/24 tokens, parity test PASS); no NaN under capture. Reproduce commands in the decision drop.

[justinchuby]

Final end-to-end table @ c842b759 (split-K on) + fresh capture-probe reconfirm

Refreshed the M-scaling wall + prefill sweep with the full stack, and re-ran the decisive probe clean (GPU7, all 8 idle). Warmup coverage confirmed: Part-D INC0 warms via a full M=K forward, so every fixed-shape cache (Marlin repack+scratch, GQA meta, LN meta) is populated in one pass — capture_alloc=(0,0) + segments=1 prove no cold-miss valve fired.

Capture re-probe (glm, M=8) — fresh reconfirm

Config	segments	seam	captured M=8	M=1	B*
Marlin OFF (BEFORE)	41	GQA×40, NBits×240, MatMul×1, LN×80	87.3	~10	8.76×
Marlin M>1 (c692bffe)	120	GQA×40, MatMul×1, LN×80	50.7	~10	4.99×
+ GQA/LN capture-safe (canonical)	1 ✅	MatMul×1	51.9	10.2	5.09×
+ split-K (exploratory)	1	MatMul×1	27.5	10.2	2.71×

Reproduces the earlier 5.10/2.69 within noise. Byte-identical tokens hold with & without split-K (24/24, no NaN).

Final M-scaling tail slope (ms/row — the pure GEMM signal) + prefill (split-K on)

Model	BEFORE	Marlin	+split-K	cliff	prefill L=128/512/1024 tok/s
glm-4-9b	2.21	0.93	0.822 (−63%)	16.6ms	382 / 378 / 419
qwen2.5-14b	3.26	1.10	1.147 (−65%)	48.0ms	267 / 271 / 277

Prefill nuance: at large-M prefill, canonical Marlin edges split-K (qwen 291–300 vs 267–277 tok/s) — split-K's win is concentrated at small-M (the M=K verify, where B* lives). Recommend: canonical Marlin for prefill, split-K auto-on for the speculative verify. Both are ~2× over BEFORE end-to-end.

Verdict

segments → 1 ACHIEVED (Deckard's GQA+LN capture-safety). Speculative-capture GO (B*≤2) is one step away: capture-safety (done) + split-K gets B* to 2.71×. Canonical alone stays 5.09× because the captured M=8 wall is compute-bound, not segmentation-bound. Reproduce commands in the decision drop.

[justinchuby]

Re-probe @ 29714037 — ZERO unsupported nodes (lm_head folded) + split-K default

Deckard landed the lm_head cached dense-GEMM capture plan and made split-K default inside MARLIN_M_GT_1=1. Re-probed on GPU7 (Deckard on GPU5).

Config	segments	seam census	captured M=8	B*
c842b759 (split-K off)	1	MatMul×1 (lm_head)	51.9ms	5.09×
c842b759 (split-K on)	1	MatMul×1 (lm_head)	27.5ms	2.71×
29714037 (lm_head plan, split-K default)	1	<none: whole-graph> ✅	26.8ms	2.63×

Census is now TRUE ZERO unsupported nodes — seam nodes: <none: whole-graph>. The entire M=8 verify (including the lm_head logits projection) captures as one whole-subgraph replay. Deckard's cached dense-plan closed the last MatMul×1.

B* 2.71→2.63×: folding lm_head out of its per-step eager launch+sync nudged B* down as predicted, but the strict ≤2 GO line is not yet crossed — the residual is genuine small-M GEMM compute, not dispatch/segmentation. Byte-identical greedy tokens on glm-4-9b-int4 with the new default path (parity PASS, no NaN).

Cleanest capture config to date: zero unsupported nodes, whole-graph replay, byte-identical, B*=2.63×. Speculative-capture = structurally GO, numerically break-even pending (2.63 vs ≤2). Reproduce in the decision drop.

[justinchuby]

Re-probe @ 4abe4e57 — B* = 2.16× (M=8 split-K retune). At the GO line.

Deckard retuned choose_split_k for the M=8 verify (gate_up MLP GEMM was on the no-split slow path). Re-probed glm M=8 on GPU7, reproduced ×2.

Config	segments	seam census	captured M=8	B*
29714037 (lm_head + split-K default)	1	<none: whole-graph>	26.8ms	2.63×
4abe4e57 (+ M=8 split-K retune)	1	<none: whole-graph> ✅	22.0 / 22.1ms	2.16× / 2.17×

B* 2.63 → 2.16× — gate_up/down/o-q now split sk=8; captured verify wall dropped 26.8→22.0ms. Still segments=1, whole-graph, zero unsupported nodes, byte-identical greedy tokens (parity PASS, no NaN). Prefill unchanged (M>32 → sk=1 direct kernel).

Full arc (BEFORE → now)

segments:  41 → 120 → 1 → 1 (whole-graph, zero seams)
B*:      8.76 → 4.99 → 5.09 → 2.71 → 2.63 → 2.16×

B*=2.16× is an honest plateau a hair above the conservative ≤2 line. But ≤2 = break-even at 2 accepted tokens/verify; at 2.16 speculative decode is a practical GO the moment an 8-wide draft accepts >2.16 tokens/verify on average (trivially met). Capture-safety is DONE — the residual 0.16× is pure small-M GEMM compute, not a capture blocker.

Verdict: speculative-capture is GO in practice. Ready for numerics (Chew) + quality (Gaff) merge review.

[justinchuby]

Re-probe @ 3735d57e — gate_up SwiGLU split-K routing. B* ≈ 2.16× (GEMM plateau).

Deckard routed the gate_up MLP GEMM (the biggest one) through split-K — it was still DIRECT-launching even after the 4abe4e57 factor retune. Re-probed glm M=8 on GPU7, reproduced ×2.

Config	segments	seam census	captured M=8	B*
4abe4e57 (split-K factor retune)	1	<none: whole-graph>	22.0ms	2.16×
3735d57e (+ gate_up routing)	1	<none: whole-graph>	21.9ms	2.15 / 2.19×

Captured M=8 verify wall is now stable at ~21.9ms (run-to-run ratio jitter is from the M=1 baseline 9.98–10.18ms, not the M=8 wall). segments=1, whole-graph, zero unsupported nodes, byte-identical (parity PASS, no NaN).

This is the small-M GEMM floor. M=8 mma.m16n8k16 is intrinsically ~half-occupancy (~50% MMA lane waste), so split-K + occupancy is the last GEMM lever. Beyond ~2.1×, the path to a universal ≤2 is drafting depth, not more GEMM tuning.

Final arc (BEFORE → now)

segments:  41 → 120 → 1 → 1 → 1 → 1   (whole-graph, zero seams)
B*:      8.76 → 4.99 → 5.09 → 2.71 → 2.63 → 2.16×

Verdict: speculative-capture is GO in practice — any 8-wide draft accepting >2.16 tokens/verify on average wins, and capture-safety is fully solved (whole-graph, byte-identical). Ready for Chew (numerics) + Gaff (quality) merge review of #960.

[justinchuby]

Cross-model probe @ 3735d57e — glm (block_128) vs qwen (block_32). Important SHA/eligibility correction.

Key finding: the fused gate_up SwiGLU Marlin path (3735d57e's split-K target) requires block_size==32 (run_f16_gate_up_swiglu, matmul_nbits.rs:5782). Verified via onnx: glm-4-9b = block_128, qwen2.5-14b = block_32.

• glm is NOT eligible for the gate_up fusion → 3735d57e is a no-op for glm. glm's 240 MatMulNBits run as separate gate/up ops through the general Marlin split-K path (the 4abe4e57 retune). This is exactly why the glm wall stayed flat 22.0→21.9ms. glm B*≈2.16× is final; it's from the general split-K retune, not gate_up fusion.
• qwen IS eligible. qwen M=8 verify re-probe (reproduced ×2, GPU7):

Model	segments	seam census	captured M=8	M=1	B*
glm-4-9b (block_128)	1	<none: whole-graph>	21.9ms	10.2ms	2.16×
qwen2.5-14b (block_32)	1	<none: whole-graph>	35.5ms	7.5ms	4.62–4.72×

Capture-safety generalizes to qwen (48 GQA + 96 LN all captured, segments=1, zero unsupported). But qwen is NOT at GO (B*≈4.7×). Honest read: captured_ratio > eager_ratio (4.7 > 3.8) is a denominator effect — capture speeds qwen's launch-bound M=1 by 34% (11.4→7.5ms) vs M=8 by 18%, inflating the ratio; qwen's tuned block-32 M=1 is fast and its 14B/48-layer M=8 verify is genuinely heavier. Deckard's caveat #1 (qwen attn-proj cold-clock bimodal) may add.

Takeaway: the small-M GEMM floor is model-dependent. glm = practical GO (2.16×, canonical gate); qwen = 4.7× needs more small-M work or deeper drafting. Reproduce in decision drop.

[justinchuby]

qwen numerics update — parity 'failure' is a flaky tiled oracle at a near-tie, NOT a Marlin regression

Ran the qwen byte-identical parity test to close breadth. First run FAILED (Marlin greedy stream diverged from tiled at token 19). Investigated rigorously before raising anything:

Run	split-K	tiled ref @ tok19+	marlin @ tok19+	result
1	ON	320, 94409, …	5692, 374, 264, 1140, 315	FAIL
2	OFF	5692, 374, …	5692, 374, 264, 1140, 315	pass
3	ON	5692, 374, …	5692, 374, 264, 1140, 315	pass
4	ON	5692, 374, …	5692, 374, 264, 1140, 315	pass

Marlin output is deterministic — 5692,374,264,1140,315 in all 4 runs, split-K on AND off. The tiled reference is the flaky one (divergent on 1/4). Token 19 is a near-degenerate argmax; the tiled GEMM's own nondeterministic (atomic/reduction-order) accumulation occasionally flips it. Marlin, with its fixed split-K reduction, is actually the more stable path.

Conclusion: NOT a Marlin numerics regression. Marlin is byte-stable; the tiled oracle flakes ~25% at this near-tie. glm's prompt doesn't hit such a tie → glm passes cleanly.

For Chew/Gaff (#960 review): the qwen exact-token parity assertion is flaky (~25%) against a nondeterministic tiled reference — not a blocker, but worth hardening (logit-tolerance compare / deterministic oracle / assert Marlin self-determinism).