[justinchuby]

Summary

A truth pass over the memory / offload / KV / capture documentation. Over the
past days a series of hardware measurements (RTX 4060 Laptop GPU, Windows/WDDM)
overturned several claims still asserted as fact in docs/. This PR finds those
claims, corrects them against what was measured, and — where a document described
an intended design in the present tense as though it were implemented — marks it
as a design proposal with its current status. No wrong claim is silently deleted:
each is replaced with what was measured and the PR/issue that superseded it, so a
reader who remembers the old number can see it was tested and retired.

Docs-only. No code or tests changed.

Corrected claims

File	What it said	What is true	Superseded by
ORT2.md §30.4–30.8	"Offloading doesn't mean stalling — prefetch overlaps with compute"; "If compute time > transfer time: zero visible overhead"; "Overlap compute + transfer: Always"; // Async prefetch next layer's weights (overlaps with current compute)	Sequential weight prefetch produced no usable overlap and was deleted; the bottleneck is disk/transfer bandwidth, not overlap. On a model that doesn't fit there isn't enough independent compute to hide the transfer behind.	PR #715; AirLLM analysis #718 (rates prefetch ~10%)
ORT2.md §30.6 / §30.7	"30% weights on CPU → prefetch mostly hidden by compute"; memory.prefetch_layers presented as an active knob	Estimate, not measured; prefetch removed. Managed offload was ~27× slower than doing nothing (WDDM) on a non-fitting model.	PR #715
DESIGN.md §39.5	KV table "tokens per granule, 8B GQA (2048 B/token): 2 MiB → 1024"; floor num_layers × 2 × 2 MiB	KV is per-binding/head-major: a fixed full-context stride commits objects × granule, not tokens/granule. qwen2.5-0.5b commits 96 head-stripes × 2 MiB = 192 MiB for ~12 KiB live. Crossover is granule/(head_dim × sizeof(dtype)), model-size independent (~8,192 tok @ head_dim 128/fp16; ~16,384 @ 64/fp16); quantized KV doubles it. 2 MiB is min and recommended — no finer granule on this device.	PR #776
MEMORY_ARCHITECTURE.md	h2d attribution 1.7% (already partly recorded); KV geometry, crossover, read-bounded commit, offload/capture exclusion not consolidated	Re-measured h2d transfer 18.8%. Added the central KV reservation-geometry section: per-object granule floor, the crossover formula, 2 MiB min==recommended, the read-bounded-commit result (read—not reservation—forces the commit; #721 stage-4 32× regression = 1.5 GB vs 48 MB), offload/capture mutual exclusion, and prefetch deletion.	PR #715, #716, #772, #776
MEMORY_ARCHITECTURE.md (KV page storage row)	"still host-only" without VMM ownership	Device KV paging is owned by the CUDA VMM layer (CudaVmmAllocator, #740 pool), with committed-granule admission (#745) and growth grants (#748); native_decode/cuda.rs has no PageTable consumer and paged_gqa.rs is a batch-1 CPU primitive.	#740, #745, #748
WEIGHT_OFFLOAD.md §7.3, Phase 4	"Prefetch the next exact/predicted expert wave … while the current wave computes"; Phase 4 predictive prefetch as the plan	Marked as a design proposal whose central premise (prefetch overlap) is measured false so far; plumbing (stream-ordered H2D, fences, double-buffer scheduler) is distinct from prefetch producing a throughput gain.	PR #715, #718
WEIGHT_OFFLOAD.md §0.4	Did not mention offload/capture exclusion, the residency-cache defect, or committed-physical budgeting	Added: offload disables capture today (#716 is the fix; #708/#728 = 154→34 segments); residency cache had a 0% hit rate for its whole life, fixed to 74.18% with evictions 6,286→0 (#723); a nominal 6 GB budget physically consumes ~6.51 GB.	#716, #723
CUDA_GRAPH_CAPTURE.md	No mention of offload interaction or VMM remap survival	Added: offload/capture mutual exclusion (#716); a captured graph survives a VMM remap at the same VA, writing into the new physical pages — sentinel-proven; unmapping during in-flight replay and cuMemMap-during-capture are not proven.	#716, #727

Deliberately left unchanged

GQA_KV_MATERIALIZATION_DESIGN.md, KV_INSERTION_DESIGN.md,
DECODE_POOL_RESIDENCY_DESIGN.md, NATIVE_CUDA_DECODE.md, CUDA_STRATEGY.md,
native-ort-kv-capacity.md — checked against the relevant facts and found
accurate/self-consistent for their scope (CPU-kernel data movement, CPU
decode-pool threading, the native-decode milestone plan, the KV-capacity policy).
Correct docs were not restructured, and subject areas outside this pass (CPU EP,
quantization formats, CLI, diffusion) were not touched.

Preventing recurrence

Added a documentation-hygiene note to MEMORY_ARCHITECTURE.md: a performance
claim in docs must cite the measurement that produced it; when measurement
overturns a claim, correct it in place and name what superseded it rather than
deleting it; and keep measured fact, design intent, and not-yet-built visibly
distinct.

Refs #715, #716, #718, #721, #723, #727, #740, #745, #748, #772, #776.

[justinchuby]

Coordinator review — approving and merging.

The find that justifies this pass on its own is \DESIGN.md\ §39.5: it modelled KV virtual-memory cost as tokens per granule, with a table claiming a 2 MiB granule holds ~1024 tokens and a sequence floors at
um_layers x 2 x 2 MiB. That is the same error I made publicly and had to retract. KV is stored per-binding/head-major, so a fixed full-context stride commits \objects x granule, not \content / granule. Measured, qwen2.5-0.5b commits 96 head-stripes x 2 MiB = 192 MiB to hold ~12 KiB of live content — the old model presented a ~192 MiB commit as if it were ~12 KiB worth of tokens, hiding a three-order-of-magnitude over-commit and erasing the crossover \granule / (head_dim x sizeof(dtype))\ (~8,192 tokens at head_dim 128/fp16) below which a stable full-context VA is strictly worse than bucket growth. That crossover is exactly the mechanism behind the #721 stage-4 32x regression.

The correction style is what I asked for and is the reason this is safe to merge: the falsified prefetch design in \ORT2.md\ §30.5 and the \prefetch_weights_async\ sketch are retained and labelled as discarded design with the superseding PR cited, rather than silently deleted. A reader who remembers the old claim can see it was tested and overturned. KV geometry is consolidated into \MEMORY_ARCHITECTURE.md\ as the single source with the other documents linking to it, which is what should stop this from drifting again.

Verified docs-only: all 5 changed files are markdown, no code or tests touched.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.40%. Comparing base (d2db1f5) to head (f330ecd).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #781   +/-   ##
=======================================
  Coverage   79.40%   79.40%           
=======================================
  Files         331      331           
  Lines      137528   137528           
  Branches   137528   137528           
=======================================
  Hits       109199   109199           
  Misses      23985    23985           
  Partials     4344     4344           

Flag	Coverage Δ
cli-ort-linux	83.41% <ø> (ø)
cli-ort-windows	82.90% <ø> (-0.10%)	⬇️
mlas	80.96% <ø> (ø)
offline	79.21% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	1.12 ms	1.52 ms	+35.4%
🔴	matmul/small_generic_f32_threads=8/1x256x256	39.17 µs	51.16 µs	+30.6%
⚠️	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	95.92 µs	113.45 µs	+18.3%
⚠️	sampling_latency/top_p_per_token	393.26 µs	461.75 µs	+17.4%
⚠️	sampling_latency/top_k_per_token	53.04 µs	62.10 µs	+17.1%
⚠️	sampling_latency/min_p_per_token	228.25 µs	262.57 µs	+15.0%
✅	tokenization/decode_tokens_per_second	7.15 ms	8.21 ms	+14.8%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	617.99 µs	704.27 µs	+14.0%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	552.78 µs	629.66 µs	+13.9%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.29 ms	2.60 ms	+13.5%
✅	sampling_latency/greedy_per_token	3.69 µs	4.18 µs	+13.4%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	554.70 µs	626.90 µs	+13.0%
✅	matmul/small_generic_f32_threads=1/1x256x256	38.87 µs	42.75 µs	+10.0%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	83.07 µs	90.09 µs	+8.4%
✅	matmul/small_generic_bf16_threads=8/1x256x256	38.27 µs	41.45 µs	+8.3%
✅	matmul/medium_generic_f16_threads=8/32x512x512	42.74 µs	45.31 µs	+6.0%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.82 ms	4.00 ms	+4.7%
✅	qwen3_sampling_processors/top_k_top_p_fast	685.98 µs	714.19 µs	+4.1%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.06 ms	6.29 ms	+3.7%
✅	logit_processing/seven_processor_chain_per_step	338.70 µs	345.74 µs	+2.1%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	563.88 µs	569.78 µs	+1.0%
✅	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	130.73 µs	130.56 µs	-0.1%
✅	matmul/small_generic_f16_threads=1/1x256x256	36.43 µs	36.35 µs	-0.2%
✅	matmul/medium_generic_f16_threads=1/32x512x512	39.36 µs	39.14 µs	-0.6%
✅	gather/large_f16_threads=1-internal/131072	18.97 µs	18.86 µs	-0.6%
✅	matmul/small_generic_f16_threads=8/1x256x256	39.34 µs	38.56 µs	-2.0%
✅	matmul/medium_generic_f32_threads=8/32x512x512	1.61 ms	1.58 ms	-2.0%
✅	kv_cache/alloc_dealloc_pages	42.49 µs	41.32 µs	-2.8%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	5.86 ms	5.63 ms	-3.8%
✅	gather/large_bf16_threads=1-internal/131072	18.89 µs	18.08 µs	-4.3%
✅	matmul/small_generic_bf16_threads=1/1x256x256	38.55 µs	36.65 µs	-4.9%
✅	reduce_mean/small_f32_threads=1-internal/4096	19.89 µs	18.61 µs	-6.4%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	92.64 µs	86.08 µs	-7.1%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.24 ms	2.05 ms	-8.7%
✅	matmul/medium_generic_f32_threads=1/32x512x512	3.00 ms	2.69 ms	-10.4%
✅	qwen3_sampling_processors/top_k_partial_selection	166.46 µs	148.82 µs	-10.6%
✅	grammar_masking/llguidance_compute_mask/32	96.33 µs	85.69 µs	-11.1%
✅	tokenization/encode_tokens_per_second	507.12 µs	441.71 µs	-12.9%
✅	gather/small_f32_threads=1-internal/4096	937.1 ns	814.8 ns	-13.1%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	103.44 µs	89.69 µs	-13.3%
✅	gather/large_f32_threads=1-internal/131072	45.63 µs	39.27 µs	-13.9%
🟢	gather/medium_bf16_threads=1-internal/32768	3.16 µs	2.68 µs	-15.2%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	332.32 µs	280.36 µs	-15.6%
🟢	gather/small_bf16_threads=1-internal/4096	599.5 ns	505.4 ns	-15.7%
🟢	gather/medium_f16_threads=1-internal/32768	3.31 µs	2.67 µs	-19.5%
🟢	matmul/large_generic_bf16_threads=8/32x1024x1024	2.33 ms	1.87 ms	-19.5%
🟢	add/large_f32_threads=1-internal/4194304	56.94 ms	45.20 ms	-20.6%
🟢	matmul/large_generic_f32_threads=1/32x1024x1024	12.42 ms	9.70 ms	-21.9%
🟢	add/large_bf16_threads=1-internal/4194304	62.92 ms	48.62 ms	-22.7%
🟢	add/large_f16_threads=1-internal/4194304	60.34 ms	46.60 ms	-22.8%
🟢	add/medium_f32_threads=1-internal/262144	3.70 ms	2.70 ms	-26.9%
🟢	add/small_f16_threads=1-internal/1024	19.80 µs	14.43 µs	-27.1%
🟢	add/medium_f16_threads=1-internal/262144	3.79 ms	2.73 ms	-28.0%
🟢	reduce_mean/large_f32_threads=1-internal/262144	1.60 ms	1.14 ms	-28.9%
🟢	gather/small_f16_threads=1-internal/4096	729.4 ns	517.2 ns	-29.1%
🟢	gather/medium_f32_threads=1-internal/32768	6.58 µs	4.46 µs	-32.2%
🟢	add/medium_bf16_threads=1-internal/262144	4.11 ms	2.76 ms	-32.8%
🟢	add/small_f32_threads=1-internal/1024	326.5 ns	215.8 ns	-33.9%
🟢	add/small_bf16_threads=1-internal/1024	21.25 µs	14.05 µs	-33.9%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 5.05 4.05 6.99 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)