[justinchuby]

Records two owner decisions that redirect in-flight work, each with the measured counter-evidence it has to beat.

Contiguous-VA KV for batching (#750). Each sequence gets its own contiguous device virtual address with physical granules mapped underneath on demand; the attention kernel keeps seeing a flat buffer and never learns page tables. This supersedes #721 stage 3 unless the route fails. The record carries the result it must beat: #721 stage 4 measured a stable full-context VA committing 1.5 GB where bucket growth commits 48 MB, because the decode kernel reads the full padded shape and masks, forbidding tail decommit.

Managed no-spill VMM as default (#755), with automatic weight streaming instead of hard OOM. Sequenced explicitly behind #716, because weight offload and CUDA graph capture are mutually exclusive today — flipping the default first would silently disable capture for exactly the models that need offload, giving back the 154 -> 34 segment collapse from #708/#728.

Docs only, no code paths touched.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.36%. Comparing base (f861905) to head (9a52e1c).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #756      +/-   ##
==========================================
- Coverage   80.10%   79.36%   -0.74%     
==========================================
  Files         331      331              
  Lines      135684   137462    +1778     
  Branches   135684   137462    +1778     
==========================================
+ Hits       108686   109098     +412     
- Misses      22704    24024    +1320     
- Partials     4294     4340      +46     

Flag	Coverage Δ
cli-ort-linux	83.41% <ø> (ø)
cli-ort-windows	82.90% <ø> (-0.10%)	⬇️
mlas	80.92% <ø> (ø)
offline	79.17% <ø> (-0.78%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 22 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	matmul/medium_generic_bf16_threads=8/32x512x512	365.37 µs	721.91 µs	+97.6%
🔴	matmul/medium_generic_f16_threads=8/32x512x512	30.15 µs	53.73 µs	+78.2%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	40.40 µs	69.24 µs	+71.4%
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	536.87 µs	904.60 µs	+68.5%
🔴	matmul/small_generic_f16_threads=8/1x256x256	36.98 µs	54.94 µs	+48.6%
🔴	matmul/large_generic_f32_threads=8/32x1024x1024	3.71 ms	5.37 ms	+44.7%
🔴	matmul/medium_generic_f16_threads=1/32x512x512	30.18 µs	41.72 µs	+38.2%
🔴	matmul/medium_generic_bf16_threads=1/32x512x512	520.36 µs	709.49 µs	+36.3%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	57.64 µs	77.78 µs	+34.9%
🔴	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	398.63 µs	524.61 µs	+31.6%
⚠️	add/large_f32_threads=1-internal/4194304	44.62 ms	57.31 ms	+28.4%
⚠️	matmul/medium_generic_f32_threads=8/32x512x512	934.14 µs	1.18 ms	+25.9%
⚠️	gather/large_f32_threads=1-internal/131072	27.37 µs	34.29 µs	+25.3%
⚠️	sampling_latency/top_k_per_token	58.13 µs	72.79 µs	+25.2%
⚠️	add/large_f16_threads=1-internal/4194304	41.92 ms	52.49 ms	+25.2%
⚠️	add/medium_bf16_threads=1-internal/262144	2.76 ms	3.44 ms	+24.6%
⚠️	gather/large_bf16_threads=1-internal/131072	12.92 µs	15.89 µs	+23.0%
⚠️	matmul/small_generic_bf16_threads=8/1x256x256	31.52 µs	38.48 µs	+22.1%
⚠️	matmul/large_generic_f16_threads=8/32x1024x1024	85.23 µs	103.71 µs	+21.7%
⚠️	matmul/small_generic_f32_threads=8/1x256x256	35.45 µs	43.04 µs	+21.4%
⚠️	matmul/large_generic_bf16_threads=8/32x1024x1024	1.31 ms	1.59 ms	+20.8%
⚠️	gather/small_f16_threads=1-internal/4096	477.6 ns	574.7 ns	+20.3%
⚠️	gather/large_f16_threads=1-internal/131072	12.98 µs	15.53 µs	+19.7%
⚠️	reduce_mean/medium_f32_threads=1-internal/65536	242.45 µs	289.80 µs	+19.5%
⚠️	add/large_bf16_threads=1-internal/4194304	41.71 ms	48.89 ms	+17.2%
⚠️	reduce_mean/small_f32_threads=1-internal/4096	15.14 µs	17.59 µs	+16.2%
✅	matmul/small_generic_f16_threads=1/1x256x256	31.19 µs	35.13 µs	+12.6%
✅	gather/medium_f32_threads=1-internal/32768	3.72 µs	4.19 µs	+12.6%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.02 ms	1.14 ms	+12.2%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.28 ms	10.39 ms	+11.9%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.33 ms	2.61 ms	+11.8%
✅	add/medium_f32_threads=1-internal/262144	2.63 ms	2.92 ms	+10.7%
✅	add/small_bf16_threads=1-internal/1024	13.83 µs	15.24 µs	+10.2%
✅	sampling_latency/greedy_per_token	3.89 µs	4.25 µs	+9.3%
✅	gather/small_bf16_threads=1-internal/4096	481.1 ns	520.8 ns	+8.3%
✅	tokenization/decode_tokens_per_second	7.85 ms	8.50 ms	+8.3%
✅	matmul/small_generic_f32_threads=1/1x256x256	36.61 µs	39.61 µs	+8.2%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	81.45 µs	88.06 µs	+8.1%
✅	gather/small_f32_threads=1-internal/4096	686.7 ns	741.5 ns	+8.0%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.94 ms	2.04 ms	+5.5%
✅	qwen3_sampling_processors/top_k_top_p_fast	687.71 µs	721.57 µs	+4.9%
✅	grammar_masking/llguidance_compute_mask/32	88.57 µs	92.72 µs	+4.7%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.37 ms	6.64 ms	+4.2%
✅	gather/medium_f16_threads=1-internal/32768	2.66 µs	2.73 µs	+2.5%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	78.27 µs	79.81 µs	+2.0%
✅	qwen3_sampling_processors/top_k_partial_selection	155.95 µs	158.24 µs	+1.5%
✅	matmul/small_generic_bf16_threads=1/1x256x256	35.07 µs	35.15 µs	+0.2%
✅	sampling_latency/top_p_per_token	416.27 µs	410.94 µs	-1.3%
✅	sampling_latency/min_p_per_token	240.75 µs	236.77 µs	-1.7%
✅	gather/medium_bf16_threads=1-internal/32768	2.60 µs	2.53 µs	-2.6%
✅	add/small_f32_threads=1-internal/1024	237.2 ns	229.1 ns	-3.4%
✅	logit_processing/seven_processor_chain_per_step	357.30 µs	344.51 µs	-3.6%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.44 ms	2.35 ms	-3.7%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	583.25 µs	556.87 µs	-4.5%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.91 ms	3.64 ms	-6.9%
✅	tokenization/encode_tokens_per_second	485.00 µs	447.88 µs	-7.7%
✅	kv_cache/alloc_dealloc_pages	41.77 µs	38.49 µs	-7.9%
✅	add/medium_f16_threads=1-internal/262144	3.56 ms	3.26 ms	-8.5%
✅	add/small_f16_threads=1-internal/1024	16.38 µs	14.61 µs	-10.8%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 2.78 3.64 5.69 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)