[justinchuby]

A durable record of the load-bearing results from this push, with every claim tied to a merged executable test — and, deliberately, the claims that were refuted.

Three of the most useful outcomes were corrections to things I had stated as achieved:

• seq-major had not realised the 8x floor. Enable stride-described seq-major flash prefill #794 measured head-major and seq-major committing identical physical bytes, because the bindings did not consume the layout descriptor. Fixed in Make native CUDA KV binding commit geometry layout-aware #797; I corrected the floor table before merging.
• The decoder was not declining capture. \captures=0\ came from a cached \ONNX_GENAI_CUDA_GRAPH=0\ in a long-lived test process (captures=0 was cached ONNX_GENAI_CUDA_GRAPH=0, not a model decline #804) — two PRs had misattributed a failed measurement to a model gate.
• A fixed KV stride does not remove growth-triggered re-capture. True in mechanism (seq-major moves 0 bytes vs 688,576), irrelevant in practice, because the engine invalidates unconditionally on growth (Seq-major 4 invalidations = head-major 4: growth policy still re-captures #805).

Also records the #736 audit's recurring finding — four of five slices found over-reservation rather than an ungoverned allocation — and the guidance that follows: start from use, not from allocation, because governing a bypass without sizing it to use converts invisible waste into charged waste, which tightens admission and costs concurrency.

Docs only, no code paths touched.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.55%. Comparing base (fe3114f) to head (d63a86c).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #808       +/-   ##
===========================================
+ Coverage   79.43%   83.55%    +4.12%     
===========================================
  Files         331       12      -319     
  Lines      137783     5192   -132591     
  Branches   137783     5192   -132591     
===========================================
- Hits       109442     4338   -105104     
+ Misses      23994      677    -23317     
+ Partials     4347      177     -4170     

Flag	Coverage Δ
cli-ort-linux	83.50% <ø> (ø)
cli-ort-windows	82.99% <ø> (-0.10%)	⬇️
mlas	?
offline	?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 319 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	71.81 µs	125.08 µs	+74.2%
🔴	gather/large_bf16_threads=1-internal/131072	13.69 µs	19.20 µs	+40.2%
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.71 ms	2.36 ms	+38.2%
🔴	matmul/medium_generic_f16_threads=8/32x512x512	35.98 µs	47.12 µs	+31.0%
🔴	gather/medium_f16_threads=1-internal/32768	2.74 µs	3.57 µs	+30.0%
⚠️	gather/medium_bf16_threads=1-internal/32768	2.57 µs	3.30 µs	+28.4%
⚠️	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	458.16 µs	565.08 µs	+23.3%
⚠️	logit_processing/seven_processor_chain_per_step	357.64 µs	436.25 µs	+22.0%
⚠️	matmul/large_generic_f32_threads=8/32x1024x1024	4.38 ms	5.32 ms	+21.7%
⚠️	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	96.57 µs	112.32 µs	+16.3%
⚠️	kv_cache/alloc_dealloc_pages	42.70 µs	49.61 µs	+16.2%
✅	add/large_bf16_threads=1-internal/4194304	47.32 ms	52.71 ms	+11.4%
✅	gather/small_bf16_threads=1-internal/4096	511.7 ns	564.6 ns	+10.3%
✅	tokenization/encode_tokens_per_second	404.94 µs	438.52 µs	+8.3%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.21 ms	2.40 ms	+8.2%
✅	sampling_latency/top_p_per_token	424.58 µs	459.31 µs	+8.2%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	99.88 µs	107.90 µs	+8.0%
✅	gather/small_f16_threads=1-internal/4096	534.6 ns	577.3 ns	+8.0%
✅	qwen3_sampling_processors/top_k_top_p_fast	724.43 µs	781.63 µs	+7.9%
✅	add/large_f16_threads=1-internal/4194304	46.40 ms	49.93 ms	+7.6%
✅	grammar_masking/llguidance_compute_mask/32	81.98 µs	88.05 µs	+7.4%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	10.71 ms	11.43 ms	+6.7%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	89.21 µs	95.04 µs	+6.5%
✅	reduce_mean/small_f32_threads=1-internal/4096	19.24 µs	20.24 µs	+5.2%
✅	add/medium_f16_threads=1-internal/262144	2.93 ms	3.07 ms	+5.1%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	88.95 µs	93.44 µs	+5.0%
✅	sampling_latency/greedy_per_token	3.40 µs	3.56 µs	+4.6%
✅	matmul/small_generic_f32_threads=1/1x256x256	42.25 µs	43.64 µs	+3.3%
✅	gather/large_f16_threads=1-internal/131072	14.17 µs	14.52 µs	+2.5%
✅	gather/medium_f32_threads=1-internal/32768	4.71 µs	4.82 µs	+2.2%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.52 ms	6.64 ms	+1.9%
✅	sampling_latency/top_k_per_token	56.58 µs	57.55 µs	+1.7%
✅	matmul/medium_generic_f16_threads=1/32x512x512	38.56 µs	39.13 µs	+1.5%
✅	sampling_latency/min_p_per_token	246.68 µs	250.31 µs	+1.5%
✅	gather/large_f32_threads=1-internal/131072	36.74 µs	37.00 µs	+0.7%
✅	add/medium_bf16_threads=1-internal/262144	2.99 ms	3.01 ms	+0.6%
✅	tokenization/decode_tokens_per_second	6.91 ms	6.94 ms	+0.5%
✅	add/small_bf16_threads=1-internal/1024	14.87 µs	14.86 µs	-0.0%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	616.16 µs	611.49 µs	-0.8%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	632.75 µs	627.27 µs	-0.9%
✅	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	1.26 ms	1.24 ms	-1.0%
✅	add/medium_f32_threads=1-internal/262144	2.87 ms	2.83 ms	-1.3%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	589.73 µs	574.12 µs	-2.6%
✅	add/large_f32_threads=1-internal/4194304	45.34 ms	42.96 ms	-5.2%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.69 ms	2.43 ms	-9.5%
✅	add/small_f16_threads=1-internal/1024	14.56 µs	13.16 µs	-9.6%
✅	matmul/small_generic_f32_threads=8/1x256x256	60.42 µs	54.27 µs	-10.2%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.82 ms	2.49 ms	-11.6%
✅	matmul/small_generic_f16_threads=1/1x256x256	41.86 µs	36.51 µs	-12.8%
✅	matmul/small_generic_f16_threads=8/1x256x256	46.75 µs	40.64 µs	-13.1%
✅	matmul/small_generic_bf16_threads=8/1x256x256	45.29 µs	39.26 µs	-13.3%
✅	qwen3_sampling_processors/top_k_partial_selection	186.65 µs	159.99 µs	-14.3%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	4.81 ms	4.11 ms	-14.5%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.33 ms	1.13 ms	-14.7%
✅	reduce_mean/medium_f32_threads=1-internal/65536	326.10 µs	277.36 µs	-14.9%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.60 ms	1.33 ms	-17.0%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	52.54 µs	40.17 µs	-23.5%
🟢	add/small_f32_threads=1-internal/1024	270.3 ns	203.8 ns	-24.6%
🟢	gather/small_f32_threads=1-internal/4096	1.10 µs	774.9 ns	-29.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.24 3.76 4.86 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)