[justinchuby]

Replaces the stale 27x slower than doing nothing row with the measured result from #874 and, more importantly, the mechanism behind it.

Measured on qwen14b-zp, solo with nvidia-smi verified empty before every run, byte-identical output: WDDM 8.09/7.78 tok/s with htod_bytes_per_token = 0 (true zero-copy) against managed streaming 0.11 tok/s. main measured 0.05-0.08 tok/s on the same model minutes earlier, so the end-to-end effect on this box is ~100x.

The row now carries the structural cause rather than a bare ratio: each weight is read exactly once per decode step, so both paths move the same bytes over the same link, but ours adds a CPU memcpy into pinned staging, a VRAM allocation, a cuMemMap, an eviction and a synchronize -- to buy VRAM residency discarded before it is ever re-read. Copying to VRAM only pays off when the data is re-read from VRAM before eviction, and there is no intra-step reuse.

Also records three things a future reader needs and would otherwise have to rediscover:

• Keeping the no-spill arena while merely disabling offload is not an option. The physical-handle pool is a hard cap and cuMemCreate cannot spill, so with nothing paging the over-budget remainder the load fails outright.
• The Compatibility arm has no device budget at all, so on a host short of RAM an over-budget model can thrash and the governor cannot see system-wide pressure (WDDM pages out our VMM (cuMemCreate/cuMemMap) granules to host RAM under over-commit — #798 no-spill is an accounting guarantee, not physical residency, on WDDM #863).
• Linux is unchanged and must stay so -- with no shared-memory fallback an over-budget model simply fails there, so managed streaming competes with 'does not run', not with something faster (KV layout should be a stride descriptor negotiated per EP, not a growing enum — and the choice is not memory-neutral #783's lesson about not inheriting a platform-specific conclusion).

Supersedes the historical figures the row carried (6.01 tok/s WDDM, 27x behind, the h2d/staging/vram_free/vram_alloc split, capture disabled under offload): capture now runs under offload (#796) and the eviction-policy defect was fixed (#723).

Docs only.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.23%. Comparing base (b702e45) to head (d35260f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #876      +/-   ##
==========================================
+ Coverage   77.78%   78.23%   +0.45%     
==========================================
  Files         354      354              
  Lines      147834   148003     +169     
  Branches   147834   148003     +169     
==========================================
+ Hits       114987   115785     +798     
+ Misses      28358    27720     -638     
- Partials     4489     4498       +9     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	82.92% <ø> (ø)
mlas	81.33% <ø> (+0.41%)	⬆️
offline	77.98% <ø> (+0.46%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	gather/small_f16_threads=1-internal/4096	500.2 ns	770.5 ns	+54.1%
⚠️	gather/small_f32_threads=1-internal/4096	659.6 ns	825.2 ns	+25.1%
⚠️	reduce_mean/large_f32_threads=1-internal/262144	1.01 ms	1.22 ms	+20.6%
⚠️	matmul/large_generic_bf16_threads=1/32x1024x1024	2.05 ms	2.43 ms	+18.3%
⚠️	qwen3_sampling_processors/top_k_top_p_fast	665.37 µs	777.58 µs	+16.9%
⚠️	tokenization/decode_tokens_per_second	6.64 ms	7.67 ms	+15.6%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	2.73 ms	3.06 ms	+12.1%
✅	add/medium_bf16_threads=1-internal/262144	2.74 ms	3.02 ms	+10.5%
✅	gather/small_bf16_threads=1-internal/4096	532.2 ns	582.5 ns	+9.4%
✅	sampling_latency/top_k_per_token	53.82 µs	58.00 µs	+7.8%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.75 ms	6.04 ms	+4.9%
✅	kv_cache/alloc_dealloc_pages	40.01 µs	41.72 µs	+4.3%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	104.23 µs	107.97 µs	+3.6%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	80.49 µs	82.57 µs	+2.6%
✅	grammar_masking/llguidance_compute_mask/32	78.88 µs	80.20 µs	+1.7%
✅	sampling_latency/greedy_per_token	3.27 µs	3.32 µs	+1.6%
✅	gather/large_bf16_threads=1-internal/131072	14.72 µs	14.91 µs	+1.3%
✅	gather/medium_bf16_threads=1-internal/32768	2.48 µs	2.50 µs	+1.1%
✅	gather/large_f32_threads=1-internal/131072	46.47 µs	46.85 µs	+0.8%
✅	gather/medium_f16_threads=1-internal/32768	2.60 µs	2.62 µs	+0.6%
✅	add/small_bf16_threads=1-internal/1024	12.88 µs	12.79 µs	-0.7%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.86 ms	9.78 ms	-0.8%
✅	reduce_mean/small_f32_threads=1-internal/4096	15.43 µs	15.18 µs	-1.6%
✅	logit_processing/seven_processor_chain_per_step	369.46 µs	357.58 µs	-3.2%
✅	sampling_latency/top_p_per_token	400.49 µs	386.39 µs	-3.5%
✅	add/large_f32_threads=1-internal/4194304	42.49 ms	40.73 ms	-4.2%
✅	add/medium_f32_threads=1-internal/262144	2.63 ms	2.51 ms	-4.6%
✅	tokenization/encode_tokens_per_second	392.07 µs	371.97 µs	-5.1%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	528.40 µs	490.72 µs	-7.1%
✅	add/large_f16_threads=1-internal/4194304	45.11 ms	41.66 ms	-7.7%
✅	qwen3_sampling_processors/top_k_partial_selection	150.87 µs	138.94 µs	-7.9%
✅	add/large_bf16_threads=1-internal/4194304	47.36 ms	43.32 ms	-8.5%
✅	sampling_latency/min_p_per_token	231.43 µs	211.23 µs	-8.7%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.72 ms	3.35 ms	-9.9%
✅	matmul/small_generic_f32_threads=1/1x256x256	42.76 µs	38.42 µs	-10.2%
✅	matmul/small_generic_f16_threads=8/1x256x256	38.81 µs	34.67 µs	-10.7%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.46 ms	2.20 ms	-10.7%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	91.80 µs	80.99 µs	-11.8%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	625.76 µs	545.83 µs	-12.8%
✅	gather/medium_f32_threads=1-internal/32768	4.68 µs	4.07 µs	-13.1%
🟢	add/small_f16_threads=1-internal/1024	15.74 µs	12.98 µs	-17.6%
🟢	add/medium_f16_threads=1-internal/262144	3.20 ms	2.59 ms	-19.0%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	291.07 µs	234.11 µs	-19.6%
🟢	matmul/small_generic_f16_threads=1/1x256x256	38.33 µs	30.74 µs	-19.8%
🟢	matmul/small_generic_f32_threads=8/1x256x256	50.36 µs	39.76 µs	-21.0%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	43.05 µs	33.68 µs	-21.8%
🟢	gather/large_f16_threads=1-internal/131072	15.32 µs	11.96 µs	-21.9%
🟢	qwen3_sampling_processors/top_k_full_sort_baseline	2.65 ms	1.96 ms	-26.1%
🟢	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	573.18 µs	410.81 µs	-28.3%
🟢	matmul/medium_generic_f16_threads=1/32x512x512	40.74 µs	29.03 µs	-28.7%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	41.66 µs	29.49 µs	-29.2%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	676.15 µs	467.21 µs	-30.9%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	49.64 µs	34.16 µs	-31.2%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	130.19 µs	88.42 µs	-32.1%
🟢	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	102.68 µs	68.21 µs	-33.6%
🟢	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	1.27 ms	780.91 µs	-38.5%
🟢	add/small_f32_threads=1-internal/1024	333.1 ns	203.3 ns	-39.0%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	7.31 ms	4.08 ms	-44.1%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.97 ms	893.63 µs	-54.5%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.55 3.77 5.48 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)