[justinchuby]

#837 item 3 — residency policy gap: attribution (the deliverable) + a rejected design, measured

Refs #837 (item 3 only; items 1 & 2 landed in #842/#843). Do not merge / do not close #837 — this is a report PR. A rejected design with its measurements attached is more useful than a verdict alone, so both are here.

---

1. The deliverable: the 1.158 GB/step, attributed

bypassed_page_ins was 704 in the last measured run and had never been explained. It now is. A "bypass" is a page-in that admit_committed_span maps, H2D-fills, hands to the kernel transiently, then frees on Drop — it is counted in htod_bytes but never joins the resident set, so its bytes re-stream every decode step. I added byte-weighted accounting for it (GlobalOffloadStats::bypassed_page_in_bytes, bypassed_byte_share(), printed by profile_native).

Measured on qwen14b-zp, managed path, solo (nvidia-smi --query-compute-apps verified empty immediately before the run), 16 tokens:

metric	value
byte_hit_rate	70.16 %
htod_bytes_per_token	2.349 GB
bypassed_page_ins	336 (11 % of page-in events)
bypassed_page_in_bytes	16.77 GB / 16 tok
bypassed_byte_share	44.63 % of streamed bytes ≈ 1.05 GB/step
avg bypass size	49.9 MB (large projections)
avg retained page-in	7.8 MB (norms / biases)

Bypass traffic is 11 % of page-in events but 44.63 % of streamed bytes ≈ ~1.05 GB/step ≈ ~90 % of the 1.158 GB/step recoverable gap. This converts an unexplained counter into the explanation for the whole 1.97×-above-floor figure. It is exactly the count-vs-byte trap #837 warned about: size-blind StableResident retains the small tensors (norms/biases) that win the first-fit race for leftover budget and re-streams the large projections transiently, every step. This attribution stands on its own even with the policy change below abandoned, and is what redirects whoever picks item 3 up next.

---

2. The obvious fix, implemented and rejected: byte-aware residency

Env knob ONNX_GENAI_WEIGHT_OFFLOAD_BYTE_AWARE (default-OFF, opt-in). When enabled, admission keeps the largest tensors resident: retain an incoming tensor iff it is strictly larger than the smallest evictable resident, evicting smallest-first (strictly-smaller only, to avoid thrash when the large set alone exceeds budget).

Before / after (managed arm, each run solo, nvidia-smi clear beforehand)

config	tokens	token IDs	offload engaged	byte_hit_rate	htod/token
byte-aware OFF (baseline), steady, graph ON	16 ✓	[96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]	yes (page_ins=3008)	70.16 %	2.349 GB
byte-aware ON, steady, graph ON	3 ✗	— early EOS	yes	— (aborted)	—
byte-aware ON, steady, graph OFF	3 ✗	— early EOS	yes	— (aborted)	—
byte-aware ON, non-steady	16 ✓	identical to baseline	no (page_ins=0)	n/a	0

The "after" arm is being abandoned: the policy cannot be safely enabled, so no bytes were moved. The counters are attached because a measured rejection is the useful artifact.

What went wrong: silent numeric corruption, not an error

Byte-aware residency corrupts decode output — token-identity violated, generation collapses to 3 tokens — whenever offload actually engages on the managed streaming path, both with and without CUDA graph. The default-off path is unaffected.

This is silent numeric corruption, not an admission error, and that is proven, not inferred, from the profiler's error-propagation structure: generate_with_callback is ?-propagated with .context("steady measured generation") before the bail!("generation emitted N tokens") check. The message we observed is the bail!, which means generation returned Ok with only 3 tokens (an early EOS from corrupted logits) — a surfaced WeightHandleError would have produced the .context(...) error instead. The runs that stayed byte-identical are exactly those where offload never engaged (page_ins=0), i.e. the branch never ran.

The token-identity hard constraint is what caught this. That check exists precisely so a plausible optimisation cannot ship while silently producing wrong output — and this is the first time it has fired on a real proposal in this line of work. Without it, byte-aware would have "improved" the count-based hit-rate report while emitting garbage.

Mechanism, as far as I got (and a defect worth flagging)

A residency policy decides what is resident, not what is computed. Both the resident-hit and bypass paths fill the correct bytes at a stable per-tensor VA; every eviction here only targets strong_count == 1 pages, and the admit loop drains both the compute and copy streams before any eviction. The only behavioural change vs baseline is the eviction target: smallest_evictable picks the smallest-by-bytes page, whereas next_evictable_index (used by both Lru and StableResident) picks the front-of-order oldest evictable page.

Under those guards, a change of eviction target should be value-neutral. That it corrupts values implicates state that depends on eviction order — most plausibly the physical granule / retained-handle pool accounting — rather than the policy decision itself. Reproduction with CUDA graph OFF rules out captured-VA baking as the sole cause. I could not isolate, within this task, whether this is:

• (a) a bug in the byte-aware admission loop, or
• (b) a latent defect in the existing offload path, merely exposed (not caused) by reordering evictions — i.e. something downstream assumes the front-of-order eviction discipline and reads/reuses a physical granule it should not when that order changes.

Possibility (b) is the more valuable one and I cannot exclude it. If it holds, it is a latent correctness defect independent of this knob, and should be filed and investigated separately (an instrumented run dumping the resident set + per-tensor checksums across steps under both eviction orders would decide it). Flagging rather than asserting.

---

3. Conclusion and where the effort should go

The count-vs-byte residency gap cannot be closed by an eviction-policy reorder. Closing it needs the structural lever #837 deferred: a dedicated transient staging zone so a large tensor can be handed to the kernel without displacing a resident page (today even a bypass must evict a resident for physical room). That redirects effort toward #864's hybrid and #750 batching — as the issue anticipated, and as batch-stage2a independently corroborated (batch-N amortizes the weight stream exactly 1/N: 301→37.6 pages/token for N=1→8, and the byte counters are contention-invariant process-local EP accounting).

#864 raises the stakes for item 3, it does not lower them

Because the managed residency cache only governs Linux / forced-managed Windows / the #864 hybrid (see scope), one might think item 3 is now niche. The opposite: #880 measured the resident hot set at ~9.7× realistic / ~15–24× at large N against ~5.6 GB/s host-mapped reads. Which weights are resident is worth more than it was when this issue was filed — a well-chosen resident hot set is exactly the edge the hybrid claims over the driver's blind WDDM choice. The attribution in §1 is the input to choosing that hot set well.

---

Scope

Since #874 the Windows default is the WDDM zero-copy path, which has no residency cache (the driver chooses). So item 3 governs behaviour only on the managed path, which is the arm measured here (ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1): Linux (managed streaming is the only way to run over-budget), Windows with ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1 / explicit --vram-limit, and prospectively the #864 hybrid.

Hard constraints / gates

• Token identity: default (byte-aware OFF) path byte-identical — [96347, …, 752], matches prior baselines. (Byte-aware ON deliberately fails this; that is the finding, and it is default-OFF.)
• Baseline: captures=2 (>0), fallbacks=0, oversubscribed_bytes=0, peak_committed_physical_bytes=7,725,907,968 < managed_limit=7,726,753,178, ref_underflows=0, byte_underflows=0, unaccounted_committed_bytes=0.
• No assertions added in Drop.
• Mandatory mobius_seqmajor_growth_parity_native_cuda (--ignored --test-threads=1): GREEN, run solo with nvidia-smi verified clear (treated as binding per Latent gap: seq-major capture-retention proof covers KV binding pointers only, not weight buffers (original flaky-gate claim RETRACTED — contended measurement) #851).
• cargo fmt (per-package), cargo clippy -p onnx-runtime-ep-cuda --features cuda -D warnings, cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -D warnings, cargo test -p onnx-runtime-session, cargo test -p onnx-runtime-ep-cuda --lib weight_paging::tests (18 pass): all green.
• CPU unit tests exercise the size-greedy admission decision function on an abstract model and are explicitly documented not to validate the GPU path.

Measurement hygiene

Wall-clock is deliberately not headlined (WDDM paging our own memory ranges 3.9–28 tok/s here, #863). Each GPU run verified nvidia-smi --query-compute-apps empty immediately beforehand; GPU serialized with siblings batch-stage2a / zerocopy-kernel. One transient contender appeared during a short non-steady probe and is noted; it did not touch the steady A/B arms, which were clean.

[justinchuby]

Validated and merging. Gates on a worktree with main merged in: cargo fmt -p onnx-runtime-ep-cuda -p onnx-genai-bench -- --check pass, cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings pass (the exact CI gate), byte-aware unit tests 3 passed.

The attribution is the deliverable, and it is a good one. 11% of page-in events carrying 44.6% of streamed bytes is the count-vs-byte trap in its purest form: size-blind StableResident retains 7.8 MB norms and biases that win the first-fit race for leftover budget, while 49.9 MB projections re-stream transiently every step. That converts a counter nobody had explained into ~90% of the 1.158 GB/step gap. Adding bypassed_page_in_bytes / bypassed_byte_share so it stays visible is the right move — the next person does not have to rediscover it.

The negative result is stronger than a null result, because you isolated why. This table is what makes it conclusive:

The only runs where byte-aware ON stays byte-identical are those where offload never engaged (page_ins=0) — i.e. the branch never ran. Every run where it did engage collapsed generation to 3 tokens.

That rules out flakiness and rules out capture (it reproduces graph-on and graph-off), and identifying the failure as a normal early-EOS from profile_native::main rather than an admission error is what pins it on decode numerics rather than on the allocator. A weaker report would have said "byte-aware didn't help".

And the mechanism is the part with lasting value: the scan-resistant recency discipline is load-bearing for correctness on the managed path, not merely for hit rate — reordering evictions by size corrupts output. I would not have predicted that, and it is exactly the kind of thing that would have been shipped by someone optimizing the count-based hit rate in good faith.

Two judgements I want to endorse explicitly:

• Reporting no byte movement rather than a fabricated hit-rate delta. The policy cannot be safely enabled, so there is no honest "after" number, and you said so. That is the anti-artifact discipline this issue asked for, and it is the same failure mode that produced qwen14b KV committed-bytes floor: head-major vs seq-major (honest negative) + #384 metadata generator #834's false conclusion and my own retracted claim on Latent gap: seq-major capture-retention proof covers KV binding pointers only, not weight buffers (original flaky-gate claim RETRACTED — contended measurement) #851.
• Shipping it default-OFF and documented as unsafe rather than deleting it. A rejected design with its evidence attached is reviewable; a deleted branch is a story. Given the mechanism is surprising, keeping the reproducer is worth the small surface.

The conclusion redirects effort correctly. "The count-vs-byte gap cannot be closed by an eviction-order reorder alone — it needs a transient staging zone so a large tensor can be handed to the kernel without displacing a resident page" is a real finding about the shape of the remaining work, not a shrug. It also raises the value of #864's hybrid, where a zero-copy cold read is precisely a transient hand-off that displaces nothing — measured at ~5.6 GB/s against ~100–133 GB/s resident in #880.

Scope caveat correctly stated: since #874 the Windows default is WDDM zero-copy with no residency cache, so this governs Linux, forced-managed Windows, and the hybrid. All measurements on the managed arm, which is the honest arm to measure.

Merging as a report. #837 stays open; item 3's remaining lever is now named (transient staging), not vague.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.79%. Comparing base (ab75a12) to head (2198ad4).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #886      +/-   ##
==========================================
- Coverage   77.79%   77.79%   -0.01%     
==========================================
  Files         354      354              
  Lines      148003   148003              
  Branches   148003   148003              
==========================================
- Hits       115137   115134       -3     
- Misses      28375    28377       +2     
- Partials     4491     4492       +1     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	83.01% <ø> (ø)
mlas	80.96% <ø> (-0.14%)	⬇️
offline	77.53% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	matmul/medium_generic_f16_threads=8/32x512x512	37.12 µs	52.89 µs	+42.5%
🔴	sampling_latency/min_p_per_token	235.70 µs	324.76 µs	+37.8%
🔴	matmul/medium_generic_bf16_threads=1/32x512x512	539.89 µs	734.71 µs	+36.1%
⚠️	gather/medium_f32_threads=1-internal/32768	4.48 µs	5.37 µs	+19.9%
⚠️	sampling_latency/top_k_per_token	59.76 µs	70.01 µs	+17.1%
⚠️	sampling_latency/top_p_per_token	463.78 µs	539.39 µs	+16.3%
⚠️	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.80 ms	4.41 ms	+16.0%
✅	gather/small_bf16_threads=1-internal/4096	516.7 ns	583.9 ns	+13.0%
✅	qwen3_sampling_processors/top_k_partial_selection	152.34 µs	170.51 µs	+11.9%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.15 ms	2.35 ms	+9.6%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.41 ms	2.54 ms	+5.2%
✅	sampling_latency/greedy_per_token	3.60 µs	3.75 µs	+4.0%
✅	matmul/medium_generic_f16_threads=1/32x512x512	36.06 µs	36.46 µs	+1.1%
✅	kv_cache/alloc_dealloc_pages	41.32 µs	41.31 µs	-0.0%
✅	tokenization/decode_tokens_per_second	7.32 ms	7.26 ms	-0.7%
✅	gather/small_f16_threads=1-internal/4096	565.7 ns	554.2 ns	-2.0%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	85.32 µs	83.24 µs	-2.4%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	576.31 µs	561.60 µs	-2.6%
✅	tokenization/encode_tokens_per_second	459.35 µs	446.24 µs	-2.9%
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	66.79 µs	64.57 µs	-3.3%
✅	grammar_masking/llguidance_compute_mask/32	84.22 µs	79.37 µs	-5.8%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.41 ms	6.04 ms	-5.9%
✅	logit_processing/seven_processor_chain_per_step	351.74 µs	327.84 µs	-6.8%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	10.57 ms	9.78 ms	-7.5%
✅	add/medium_bf16_threads=1-internal/262144	3.26 ms	2.99 ms	-8.4%
✅	add/large_f32_threads=1-internal/4194304	48.83 ms	44.72 ms	-8.4%
✅	add/medium_f16_threads=1-internal/262144	3.04 ms	2.78 ms	-8.5%
✅	matmul/small_generic_f32_threads=8/1x256x256	50.86 µs	46.45 µs	-8.7%
✅	add/large_f16_threads=1-internal/4194304	49.48 ms	44.97 ms	-9.1%
✅	matmul/small_generic_f16_threads=1/1x256x256	35.86 µs	32.11 µs	-10.5%
✅	add/small_f16_threads=1-internal/1024	15.50 µs	13.85 µs	-10.7%
✅	reduce_mean/medium_f32_threads=1-internal/65536	289.45 µs	256.08 µs	-11.5%
✅	matmul/small_generic_f32_threads=1/1x256x256	43.74 µs	38.65 µs	-11.6%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	104.15 µs	91.36 µs	-12.3%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.21 ms	1.06 ms	-12.7%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.98 ms	1.73 ms	-12.9%
✅	gather/small_f32_threads=1-internal/4096	805.9 ns	698.7 ns	-13.3%
✅	matmul/small_generic_bf16_threads=1/1x256x256	38.43 µs	33.31 µs	-13.3%
✅	add/medium_f32_threads=1-internal/262144	3.26 ms	2.81 ms	-13.7%
✅	qwen3_sampling_processors/top_k_top_p_fast	732.50 µs	631.99 µs	-13.7%
✅	matmul/small_generic_bf16_threads=8/1x256x256	42.90 µs	36.91 µs	-14.0%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.73 ms	2.33 ms	-14.5%
✅	matmul/small_generic_f16_threads=8/1x256x256	39.58 µs	33.78 µs	-14.7%
🟢	gather/medium_f16_threads=1-internal/32768	2.78 µs	2.35 µs	-15.3%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	6.01 ms	5.06 ms	-15.8%
🟢	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	513.28 µs	429.89 µs	-16.2%
🟢	add/small_bf16_threads=1-internal/1024	16.76 µs	14.02 µs	-16.4%
🟢	matmul/large_generic_f16_threads=1/32x1024x1024	106.72 µs	86.49 µs	-19.0%
🟢	gather/medium_bf16_threads=1-internal/32768	2.88 µs	2.32 µs	-19.4%
🟢	add/large_bf16_threads=1-internal/4194304	54.82 ms	44.12 ms	-19.5%
🟢	gather/large_bf16_threads=1-internal/131072	18.55 µs	14.87 µs	-19.9%
🟢	reduce_mean/small_f32_threads=1-internal/4096	20.16 µs	16.06 µs	-20.3%
🟢	add/small_f32_threads=1-internal/1024	330.1 ns	240.4 ns	-27.2%
🟢	gather/large_f32_threads=1-internal/131072	46.05 µs	32.99 µs	-28.4%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	613.67 µs	439.42 µs	-28.4%
🟢	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	936.44 µs	656.57 µs	-29.9%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.31 ms	915.82 µs	-30.3%
🟢	gather/large_f16_threads=1-internal/131072	21.27 µs	14.75 µs	-30.7%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	92.95 µs	60.00 µs	-35.5%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.30 4.13 5.86 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)