[justinchuby]

Two updates so PROGRESS carries the measurements rather than just the verdicts.

Batching stage 2a (#884) is measured. A stateless batch-N fused forward over length-1 rows with an empty past gives batch-invariant totals — htod_bytes 3,575,949,312 and page_ins 301 identical at N = 1/2/4/8, evictions 0 — with per-token figures falling exactly 1/N. That is the signature of amortization, and the record now says why the knob cannot have produced it: ONNX_GENAI_WEIGHT_OFFLOAD controls whether weights stream, not the ratio, so the 1/N comes solely from dividing a batch-invariant stream by the row count.

Also records the staging correction: assumptions #1 (input binding) and #2 (KV batch) are inseparable for an attention decoder — persistent_state_shapes pins KV batch axis 0 = 1, the mask is [1, max_len], and ONNX attention couples the batch dim across QKV/mask/past/present — so stage 2a used the stateless escape hatch #844 had anticipated and left cuda.rs untouched, which is what keeps the #796 capture gates provably unaffected.

A rejected approach recorded under item 3, so nobody repeats it. Byte-aware residency corrupts decode output — token identity violated, generation collapsing to 3 tokens — whenever offload actually engages on the managed streaming path, with and without CUDA graph; the default-off path is unaffected. The byte-identical token constraint is what caught it. A rejected design with its measurements attached is far more useful than one carrying only a verdict, and this is the first time today that constraint has fired on a real proposal.

Docs only.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.39%. Comparing base (209f7ed) to head (e5a9f76).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #887      +/-   ##
==========================================
+ Coverage   77.79%   78.39%   +0.60%     
==========================================
  Files         354      354              
  Lines      148003   148003              
  Branches   148003   148003              
==========================================
+ Hits       115134   116031     +897     
+ Misses      28377    27476     -901     
- Partials     4492     4496       +4     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	83.01% <ø> (+0.09%)	⬆️
mlas	81.29% <ø> (+0.32%)	⬆️
offline	78.16% <ø> (+0.63%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

⚠️ Benchmark Change Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
⚠️	gather/medium_f16_threads=1-internal/32768	2.56 µs	3.29 µs	+28.3%
⚠️	gather/medium_f32_threads=1-internal/32768	4.14 µs	4.88 µs	+17.8%
⚠️	gather/medium_bf16_threads=1-internal/32768	2.77 µs	3.21 µs	+15.9%
✅	tokenization/decode_tokens_per_second	6.52 ms	6.74 ms	+3.3%
✅	matmul/small_generic_f16_threads=8/1x256x256	37.68 µs	38.49 µs	+2.1%
✅	gather/large_f32_threads=1-internal/131072	36.21 µs	36.64 µs	+1.2%
✅	matmul/small_generic_f32_threads=1/1x256x256	39.11 µs	39.29 µs	+0.5%
✅	sampling_latency/greedy_per_token	3.38 µs	3.38 µs	-0.0%
✅	tokenization/encode_tokens_per_second	393.27 µs	386.62 µs	-1.7%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.07 ms	1.04 ms	-2.4%
✅	gather/small_bf16_threads=1-internal/4096	496.6 ns	477.4 ns	-3.9%
✅	gather/small_f32_threads=1-internal/4096	727.5 ns	697.3 ns	-4.2%
✅	gather/small_f16_threads=1-internal/4096	519.1 ns	492.7 ns	-5.1%
✅	add/small_f16_threads=1-internal/1024	14.99 µs	14.15 µs	-5.7%
✅	matmul/small_generic_f32_threads=8/1x256x256	47.13 µs	43.75 µs	-7.2%
✅	add/medium_f16_threads=1-internal/262144	3.06 ms	2.84 ms	-7.3%
✅	reduce_mean/medium_f32_threads=1-internal/65536	296.58 µs	273.26 µs	-7.9%
✅	matmul/small_generic_f16_threads=1/1x256x256	33.94 µs	30.69 µs	-9.6%
✅	matmul/medium_generic_f32_threads=8/32x512x512	1.43 ms	1.29 ms	-9.7%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	80.74 µs	72.86 µs	-9.8%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.74 ms	8.76 ms	-10.0%
✅	sampling_latency/top_k_per_token	54.81 µs	49.12 µs	-10.4%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	550.83 µs	488.34 µs	-11.3%
✅	kv_cache/alloc_dealloc_pages	42.42 µs	37.29 µs	-12.1%
✅	logit_processing/seven_processor_chain_per_step	341.47 µs	299.04 µs	-12.4%
✅	add/medium_f32_threads=1-internal/262144	3.08 ms	2.66 ms	-13.4%
✅	sampling_latency/min_p_per_token	224.23 µs	194.09 µs	-13.4%
✅	matmul/small_generic_bf16_threads=1/1x256x256	37.37 µs	32.34 µs	-13.5%
✅	matmul/medium_generic_f16_threads=8/32x512x512	38.90 µs	33.62 µs	-13.6%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	86.14 µs	73.86 µs	-14.3%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	563.00 µs	479.93 µs	-14.8%
✅	add/small_f32_threads=1-internal/1024	258.6 ns	220.4 ns	-14.8%
🟢	sampling_latency/top_p_per_token	418.88 µs	355.28 µs	-15.2%
🟢	qwen3_sampling_processors/top_k_partial_selection	155.14 µs	131.47 µs	-15.3%
🟢	qwen3_sampling_processors/top_k_top_p_fast	718.94 µs	608.64 µs	-15.3%
🟢	add/medium_bf16_threads=1-internal/262144	3.40 ms	2.87 ms	-15.5%
🟢	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.24 ms	5.25 ms	-15.8%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	40.34 µs	33.89 µs	-16.0%
🟢	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.94 ms	3.27 ms	-17.0%
🟢	add/small_bf16_threads=1-internal/1024	17.19 µs	14.21 µs	-17.3%
🟢	matmul/medium_generic_f32_threads=1/32x512x512	2.85 ms	2.36 ms	-17.4%
🟢	add/large_f16_threads=1-internal/4194304	52.04 ms	42.88 ms	-17.6%
🟢	add/large_f32_threads=1-internal/4194304	52.81 ms	43.43 ms	-17.8%
🟢	matmul/large_generic_f16_threads=8/32x1024x1024	96.53 µs	79.08 µs	-18.1%
🟢	grammar_masking/llguidance_compute_mask/32	85.75 µs	69.61 µs	-18.8%
🟢	matmul/medium_generic_f16_threads=1/32x512x512	35.08 µs	27.94 µs	-20.4%
🟢	qwen3_sampling_processors/top_k_full_sort_baseline	2.56 ms	2.00 ms	-21.8%
🟢	gather/large_bf16_threads=1-internal/131072	17.35 µs	13.34 µs	-23.1%
🟢	matmul/large_generic_bf16_threads=1/32x1024x1024	2.38 ms	1.83 ms	-23.4%
🟢	matmul/large_generic_bf16_threads=8/32x1024x1024	1.72 ms	1.28 ms	-25.8%
🟢	gather/large_f16_threads=1-internal/131072	17.55 µs	12.99 µs	-26.0%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	516.16 µs	362.31 µs	-29.8%
🟢	add/large_bf16_threads=1-internal/4194304	60.71 ms	42.29 ms	-30.3%
🟢	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	516.97 µs	349.33 µs	-32.4%
🟢	reduce_mean/small_f32_threads=1-internal/4096	24.75 µs	16.06 µs	-35.1%
🟢	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	66.66 µs	42.14 µs	-36.8%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	5.88 ms	3.66 ms	-37.7%
🟢	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	914.80 µs	520.41 µs	-43.1%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	83.17 µs	41.20 µs	-50.5%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 2.40 3.32 6.09 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)