[justinchuby]

Design-only feasibility scoping (mirrors the TP feasibility doc #933) of the two remaining multi-week decode levers, after the three cheap ones were closed this week (#928 kernel +2.7% capped, #933 TP NO-GO for speed, #932/#935/spec-14b eager-M=K speculative KILL).

Recommendation: build Lever B (capture-stable padded M=K verify graph) first, gated on a cheap Phase-0 capture-stability probe. B attacks the actual dispatch binding and — because a captured M=K replay reads weights once (weight-DRAM is per-token, not per-M) — amortizes both dispatch and weight-read over K tokens (floor ~1.0x, ceiling ~2-3x), reuses existing frozen-bucket capture machinery, and unlocks prompt-lookup + EAGLE-3/MTP from one build. Lever A (Marlin int4 relayout) is the unconditional ~1.3-1.6x Amdahl-capped fallback.

Docs + decision drop only; no code.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.81%. Comparing base (843bbb0) to head (49cb60a).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #938      +/-   ##
==========================================
- Coverage   78.56%   77.81%   -0.76%     
==========================================
  Files         352      354       +2     
  Lines      145983   148153    +2170     
  Branches   145983   148153    +2170     
==========================================
+ Hits       114698   115282     +584     
- Misses      26799    28377    +1578     
- Partials     4486     4494       +8     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	82.92% <ø> (-0.10%)	⬇️
mlas	80.73% <ø> (?)
offline	77.55% <ø> (-0.83%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	655.71 µs	1.48 ms	+125.4%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	59.00 µs	129.28 µs	+119.1%
🔴	matmul/large_generic_f32_threads=8/32x1024x1024	3.72 ms	6.08 ms	+63.4%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	64.37 µs	97.38 µs	+51.3%
🔴	matmul/large_generic_f16_threads=1/32x1024x1024	73.15 µs	107.18 µs	+46.5%
🔴	matmul/large_generic_f16_threads=8/32x1024x1024	78.65 µs	108.18 µs	+37.6%
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.64 ms	2.14 ms	+30.6%
⚠️	grammar_masking/llguidance_compute_mask/32	79.72 µs	99.36 µs	+24.6%
⚠️	add/large_f32_threads=1-internal/4194304	42.57 ms	52.12 ms	+22.5%
⚠️	gather/large_f32_threads=1-internal/131072	33.68 µs	40.18 µs	+19.3%
⚠️	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.99 ms	6.89 ms	+15.1%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.84 ms	2.09 ms	+13.3%
✅	add/medium_f16_threads=1-internal/262144	3.25 ms	3.61 ms	+10.9%
✅	gather/medium_f16_threads=1-internal/32768	2.31 µs	2.54 µs	+10.3%
✅	logit_processing/seven_processor_chain_per_step	348.39 µs	378.19 µs	+8.6%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.05 ms	1.12 ms	+6.4%
✅	matmul/small_generic_f32_threads=8/1x256x256	32.15 µs	33.97 µs	+5.7%
✅	gather/small_f32_threads=1-internal/4096	672.8 ns	708.6 ns	+5.3%
✅	sampling_latency/top_k_per_token	55.00 µs	57.35 µs	+4.3%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	406.02 µs	423.11 µs	+4.2%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	10.28 ms	10.68 ms	+3.9%
✅	matmul/small_generic_bf16_threads=8/1x256x256	28.83 µs	29.78 µs	+3.3%
✅	tokenization/encode_tokens_per_second	399.06 µs	411.89 µs	+3.2%
✅	sampling_latency/greedy_per_token	3.38 µs	3.49 µs	+3.2%
✅	reduce_mean/medium_f32_threads=1-internal/65536	249.94 µs	257.23 µs	+2.9%
✅	add/large_f16_threads=1-internal/4194304	43.73 ms	44.83 ms	+2.5%
✅	matmul/medium_generic_f16_threads=8/32x512x512	27.79 µs	28.21 µs	+1.5%
✅	add/large_bf16_threads=1-internal/4194304	45.56 ms	46.05 ms	+1.1%
✅	add/medium_bf16_threads=1-internal/262144	3.20 ms	3.23 ms	+1.0%
✅	matmul/small_generic_f32_threads=1/1x256x256	34.49 µs	34.60 µs	+0.3%
✅	matmul/medium_generic_f16_threads=1/32x512x512	27.65 µs	27.67 µs	+0.1%
✅	qwen3_sampling_processors/top_k_top_p_fast	696.73 µs	694.58 µs	-0.3%
✅	reduce_mean/small_f32_threads=1-internal/4096	14.94 µs	14.81 µs	-0.9%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.22 ms	2.19 ms	-1.3%
✅	matmul/small_generic_bf16_threads=1/1x256x256	31.23 µs	30.76 µs	-1.5%
✅	sampling_latency/top_p_per_token	394.10 µs	385.93 µs	-2.1%
✅	matmul/small_generic_f16_threads=1/1x256x256	28.39 µs	27.76 µs	-2.2%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.78 ms	3.66 ms	-3.2%
✅	gather/medium_bf16_threads=1-internal/32768	2.46 µs	2.38 µs	-3.4%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	505.19 µs	487.96 µs	-3.4%
✅	add/small_bf16_threads=1-internal/1024	14.86 µs	14.35 µs	-3.4%
✅	add/small_f16_threads=1-internal/1024	14.22 µs	13.57 µs	-4.5%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	569.22 µs	543.37 µs	-4.5%
✅	matmul/medium_generic_f32_threads=8/32x512x512	960.83 µs	915.91 µs	-4.7%
✅	gather/small_bf16_threads=1-internal/4096	550.3 ns	523.5 ns	-4.9%
✅	tokenization/decode_tokens_per_second	6.71 ms	6.37 ms	-5.0%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	556.24 µs	527.04 µs	-5.3%
✅	matmul/small_generic_f16_threads=8/1x256x256	29.90 µs	28.17 µs	-5.8%
✅	qwen3_sampling_processors/top_k_partial_selection	150.50 µs	140.94 µs	-6.4%
✅	sampling_latency/min_p_per_token	230.22 µs	211.05 µs	-8.3%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.33 ms	2.14 ms	-8.4%
✅	kv_cache/alloc_dealloc_pages	43.86 µs	39.90 µs	-9.0%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	86.84 µs	78.98 µs	-9.0%
✅	gather/large_bf16_threads=1-internal/131072	11.03 µs	9.93 µs	-10.0%
✅	gather/small_f16_threads=1-internal/4096	518.7 ns	461.8 ns	-11.0%
🟢	gather/medium_f32_threads=1-internal/32768	4.73 µs	4.02 µs	-15.1%
🟢	gather/large_f16_threads=1-internal/131072	11.41 µs	9.49 µs	-16.8%
🟢	add/small_f32_threads=1-internal/1024	249.0 ns	206.0 ns	-17.3%
🟢	add/medium_f32_threads=1-internal/262144	4.34 ms	2.58 ms	-40.5%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 5.68 4.17 6.76 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)