[justinchuby]

TP feasibility scoping (Roper, design-only). Verdict: 🟥 NO-GO for tok/s as the next lever — decode is latency-bound (~15% of HBM roofline; byte-fold −75% bytes → +2.8%), so TP shards the non-binding axis and adds 104 all-reduces/token (net −3% to −7%). Precondition to flip: single-GPU decode must be made bandwidth-bound (>~55% peak) first. 🟢 GO for fit/capacity (split weights+KV → bigger models/longer context that don't fit one H200). Includes full cost model, Megatron 1-D sharding scheme (kv_heads=2 → clean only at N=2), NCCL×graph-capture assessment, and a Phase-0 microbench design. Docs-only.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.81%. Comparing base (1a6310a) to head (1614e9a).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #933      +/-   ##
==========================================
+ Coverage   77.74%   77.81%   +0.07%     
==========================================
  Files         352      354       +2     
  Lines      145872   148153    +2281     
  Branches   145872   148153    +2281     
==========================================
+ Hits       113407   115290    +1883     
- Misses      27983    28370     +387     
- Partials     4482     4493      +11     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	83.01% <ø> (+0.09%)	⬆️
mlas	81.10% <ø> (?)
offline	77.55% <ø> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	809.35 µs	1.70 ms	+110.2%
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.62 ms	2.78 ms	+72.0%
🔴	sampling_latency/top_k_per_token	51.38 µs	80.99 µs	+57.6%
🔴	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.22 ms	8.09 ms	+54.9%
🔴	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	80.26 µs	105.50 µs	+31.5%
⚠️	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.23 ms	4.16 ms	+28.7%
⚠️	logit_processing/seven_processor_chain_per_step	307.22 µs	392.30 µs	+27.7%
⚠️	grammar_masking/llguidance_compute_mask/32	75.72 µs	96.61 µs	+27.6%
⚠️	qwen3_sampling_processors/top_k_full_sort_baseline	1.98 ms	2.49 ms	+25.5%
⚠️	qwen3_sampling_processors/top_p_fast_after_top_k	492.92 µs	603.13 µs	+22.4%
⚠️	qwen3_sampling_processors/top_k_top_p_fast	635.79 µs	764.57 µs	+20.3%
⚠️	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	54.07 µs	64.75 µs	+19.7%
⚠️	sampling_latency/greedy_per_token	3.53 µs	4.13 µs	+17.0%
⚠️	matmul/large_generic_bf16_threads=1/32x1024x1024	2.04 ms	2.38 ms	+16.8%
⚠️	matmul/medium_generic_bf16_threads=8/32x512x512	368.61 µs	427.54 µs	+16.0%
✅	sampling_latency/min_p_per_token	194.48 µs	218.50 µs	+12.4%
✅	sampling_latency/top_p_per_token	350.80 µs	393.96 µs	+12.3%
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	51.90 µs	57.92 µs	+11.6%
✅	kv_cache/alloc_dealloc_pages	37.44 µs	41.04 µs	+9.6%
✅	tokenization/encode_tokens_per_second	370.67 µs	406.00 µs	+9.5%
✅	add/small_bf16_threads=1-internal/1024	12.31 µs	13.48 µs	+9.5%
✅	qwen3_sampling_processors/top_k_partial_selection	130.90 µs	142.85 µs	+9.1%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	106.46 µs	114.11 µs	+7.2%
✅	add/medium_f32_threads=1-internal/262144	2.38 ms	2.53 ms	+6.6%
✅	gather/large_f32_threads=1-internal/131072	26.12 µs	27.66 µs	+5.9%
✅	gather/large_f16_threads=1-internal/131072	12.14 µs	12.14 µs	-0.0%
✅	add/large_f32_threads=1-internal/4194304	37.79 ms	37.70 ms	-0.2%
✅	add/medium_bf16_threads=1-internal/262144	2.48 ms	2.47 ms	-0.5%
✅	add/large_f16_threads=1-internal/4194304	38.24 ms	37.91 ms	-0.9%
✅	add/medium_f16_threads=1-internal/262144	2.44 ms	2.40 ms	-1.6%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	510.43 µs	486.68 µs	-4.7%
✅	tokenization/decode_tokens_per_second	6.78 ms	6.40 ms	-5.6%
✅	add/large_bf16_threads=1-internal/4194304	41.92 ms	38.74 ms	-7.6%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	11.02 ms	10.13 ms	-8.1%
✅	gather/medium_bf16_threads=1-internal/32768	2.49 µs	2.27 µs	-8.8%
✅	gather/small_f16_threads=1-internal/4096	498.6 ns	448.2 ns	-10.1%
✅	add/small_f16_threads=1-internal/1024	13.79 µs	12.25 µs	-11.2%
✅	gather/medium_f32_threads=1-internal/32768	4.21 µs	3.70 µs	-12.2%
✅	reduce_mean/medium_f32_threads=1-internal/65536	275.22 µs	241.38 µs	-12.3%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	570.78 µs	496.84 µs	-13.0%
✅	matmul/small_generic_f32_threads=1/1x256x256	38.98 µs	33.83 µs	-13.2%
✅	matmul/small_generic_bf16_threads=8/1x256x256	33.87 µs	29.23 µs	-13.7%
✅	gather/small_bf16_threads=1-internal/4096	514.0 ns	442.1 ns	-14.0%
✅	add/small_f32_threads=1-internal/1024	223.5 ns	190.8 ns	-14.6%
🟢	gather/small_f32_threads=1-internal/4096	730.3 ns	616.7 ns	-15.5%
🟢	reduce_mean/large_f32_threads=1-internal/262144	1.08 ms	908.63 µs	-15.6%
🟢	matmul/medium_generic_f32_threads=1/32x512x512	2.68 ms	2.13 ms	-20.5%
🟢	reduce_mean/small_f32_threads=1-internal/4096	17.57 µs	13.79 µs	-21.6%
🟢	gather/medium_f16_threads=1-internal/32768	2.93 µs	2.26 µs	-23.0%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	39.48 µs	29.95 µs	-24.1%
🟢	gather/large_bf16_threads=1-internal/131072	14.14 µs	10.51 µs	-25.6%
🟢	matmul/small_generic_f32_threads=8/1x256x256	45.57 µs	32.88 µs	-27.8%
🟢	matmul/small_generic_f16_threads=1/1x256x256	39.62 µs	28.06 µs	-29.2%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	6.19 ms	4.29 ms	-30.7%
🟢	matmul/medium_generic_f16_threads=1/32x512x512	40.33 µs	27.65 µs	-31.4%
🟢	matmul/small_generic_f16_threads=8/1x256x256	42.59 µs	28.01 µs	-34.2%
🟢	matmul/large_generic_f16_threads=1/32x1024x1024	115.65 µs	76.00 µs	-34.3%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.75 ms	877.60 µs	-49.8%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	100.05 µs	27.52 µs	-72.5%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 9.64 6.00 6.57 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)