[justinchuby]

I published 11.41 GB/s for host-mapped weight reads on #877 and labelled it an upper bound. #880 measured the real thing and it was a meaningfully loose bound — correcting it here rather than leaving the optimistic number in the single-source doc.

What changed. #877 used a sequential cuMemcpyDtoD from mapped host memory. #880 ran the real strided int4 GEMV — the exact kernel and launch config the engine selects for qwen14b-zp — with only the weight pointer differing, activation/scales/zero-points/output resident in both arms. Result: ~5.6 GB/s steady-state host-mapped against ~100–133 GB/s resident, outputs bit-identical at every size. I reproduced it independently before merging #880.

Size the hybrid's cold path at 5.6 GB/s, not 11.4.

Why the correction is trustworthy. It cross-checks against a measurement taken for an entirely different purpose: WDDM reaches ~8 tok/s on qwen14b-zp by keeping ~7.7 GB resident and reading the ~0.6 GB over-budget remainder from host each step — ~4.8 GB/s of PCIe traffic, the same order. If zero-copy were really ~1 GB/s, #874's result would be impossible. Two independent paths landing on the same order is much stronger than either alone.

Also recorded:

• Sweeping the packed size mattered. The realistic ~12 MiB per-tensor read fits this GPU's L2, so a single-point measurement there compares a cached read against a PCIe read and yields a badly wrong ratio. The real 8.33 GB layer walk reads each weight once and vastly exceeds L2, so the ≥64 MiB points are the honest ones.
• CUDA graph capture with a host-mapped weight pointer is SUPPORTED and replays bit-identically to eager — measured, not assumed. The cold path therefore does not forfeit the captures > 0 / fallbacks == 0 gates (Page offloaded weights under a stable VA so offload keeps CUDA graph capture (#716) #796), which I had flagged as an open design risk.

The design survives. A ~2× penalty is not the ~10× that would have killed it. Residency remains the dominant lever (~9.7× at the realistic per-op shape, ~15–24× at large N), so the priority order from #877 stands: maximise the resident fraction first, de-copy the remainder second.

Docs only.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.79%. Comparing base (11558e3) to head (28c90c5).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #881      +/-   ##
==========================================
- Coverage   78.39%   77.79%   -0.61%     
==========================================
  Files         354      354              
  Lines      148003   148003              
  Branches   148003   148003              
==========================================
- Hits       116027   115133     -894     
- Misses      27480    28378     +898     
+ Partials     4496     4492       -4     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	82.92% <ø> (-0.10%)	⬇️
mlas	80.92% <ø> (-0.19%)	⬇️
offline	77.53% <ø> (-0.64%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	509.02 µs	1.10 ms	+115.4%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	44.97 µs	92.36 µs	+105.4%
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.26 ms	2.51 ms	+98.8%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	45.87 µs	70.41 µs	+53.5%
🔴	matmul/large_generic_bf16_threads=1/32x1024x1024	1.82 ms	2.75 ms	+51.0%
🔴	matmul/large_generic_f16_threads=8/32x1024x1024	78.17 µs	112.91 µs	+44.4%
🔴	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	349.23 µs	491.87 µs	+40.8%
🔴	add/large_f16_threads=1-internal/4194304	43.51 ms	57.71 ms	+32.6%
🔴	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.66 ms	4.86 ms	+32.6%
🔴	add/small_f32_threads=1-internal/1024	204.6 ns	266.4 ns	+30.2%
⚠️	add/large_bf16_threads=1-internal/4194304	41.34 ms	51.97 ms	+25.7%
⚠️	tokenization/decode_tokens_per_second	5.78 ms	7.24 ms	+25.1%
⚠️	sampling_latency/greedy_per_token	2.98 µs	3.66 µs	+22.9%
⚠️	tokenization/encode_tokens_per_second	360.59 µs	437.06 µs	+21.2%
⚠️	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	72.63 µs	86.29 µs	+18.8%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.20 ms	2.48 ms	+12.8%
✅	sampling_latency/top_k_per_token	51.55 µs	57.71 µs	+11.9%
✅	gather/medium_f16_threads=1-internal/32768	2.51 µs	2.77 µs	+10.4%
✅	qwen3_sampling_processors/top_k_partial_selection	147.25 µs	161.81 µs	+9.9%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	73.86 µs	80.59 µs	+9.1%
✅	add/medium_bf16_threads=1-internal/262144	2.64 ms	2.88 ms	+9.1%
✅	gather/large_f32_threads=1-internal/131072	31.71 µs	34.48 µs	+8.7%
✅	add/large_f32_threads=1-internal/4194304	41.99 ms	45.53 ms	+8.4%
✅	add/small_bf16_threads=1-internal/1024	12.47 µs	13.15 µs	+5.5%
✅	reduce_mean/small_f32_threads=1-internal/4096	15.53 µs	16.38 µs	+5.5%
✅	add/small_f16_threads=1-internal/1024	13.88 µs	14.52 µs	+4.6%
✅	reduce_mean/medium_f32_threads=1-internal/65536	255.10 µs	265.95 µs	+4.3%
✅	gather/small_f16_threads=1-internal/4096	476.2 ns	495.8 ns	+4.1%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	537.89 µs	555.88 µs	+3.3%
✅	matmul/small_generic_bf16_threads=8/1x256x256	29.62 µs	30.24 µs	+2.1%
✅	gather/medium_f32_threads=1-internal/32768	4.32 µs	4.38 µs	+1.2%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.69 ms	3.71 ms	+0.6%
✅	matmul/medium_generic_f32_threads=8/32x512x512	905.94 µs	909.86 µs	+0.4%
✅	kv_cache/alloc_dealloc_pages	42.33 µs	42.51 µs	+0.4%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	486.59 µs	486.65 µs	+0.0%
✅	matmul/small_generic_f32_threads=1/1x256x256	34.15 µs	34.12 µs	-0.1%
✅	add/medium_f32_threads=1-internal/262144	2.58 ms	2.58 ms	-0.1%
✅	matmul/medium_generic_f16_threads=1/32x512x512	27.72 µs	27.63 µs	-0.3%
✅	matmul/small_generic_bf16_threads=1/1x256x256	28.84 µs	28.68 µs	-0.6%
✅	matmul/small_generic_f16_threads=8/1x256x256	28.44 µs	28.18 µs	-0.9%
✅	matmul/small_generic_f32_threads=8/1x256x256	32.30 µs	31.85 µs	-1.4%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	8.77 ms	8.59 ms	-2.0%
✅	matmul/medium_generic_f16_threads=8/32x512x512	28.72 µs	28.07 µs	-2.2%
✅	matmul/small_generic_f16_threads=1/1x256x256	28.57 µs	27.88 µs	-2.4%
✅	add/medium_f16_threads=1-internal/262144	2.76 ms	2.67 ms	-3.2%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.02 ms	978.13 µs	-4.4%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.22 ms	5.82 ms	-6.5%
✅	sampling_latency/min_p_per_token	221.40 µs	205.85 µs	-7.0%
✅	sampling_latency/top_p_per_token	458.36 µs	425.34 µs	-7.2%
✅	qwen3_sampling_processors/top_k_top_p_fast	706.99 µs	654.62 µs	-7.4%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.33 ms	2.16 ms	-7.5%
✅	gather/small_f32_threads=1-internal/4096	761.9 ns	676.0 ns	-11.3%
✅	gather/medium_bf16_threads=1-internal/32768	2.64 µs	2.31 µs	-12.7%
✅	gather/small_bf16_threads=1-internal/4096	555.9 ns	478.2 ns	-14.0%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	419.08 µs	355.68 µs	-15.1%
🟢	logit_processing/seven_processor_chain_per_step	382.80 µs	323.46 µs	-15.5%
🟢	grammar_masking/llguidance_compute_mask/32	108.92 µs	81.69 µs	-25.0%
🟢	gather/large_bf16_threads=1-internal/131072	15.61 µs	11.20 µs	-28.2%
🟢	gather/large_f16_threads=1-internal/131072	13.99 µs	10.04 µs	-28.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 2.79 3.16 5.27 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)