[justinchuby]

#837 item 3 — and the isolation of #886 to a single tensor

Closes #837 (item 3). Do not merge without your review.

Headline update from round 2

Your control killed my conservation/churn explanation, and following the trigger
you predicted led exactly where you hoped: the #886 corruption is one tensor.

• Over-pinning 67 tensors / 2.371 GB (threshold 30 MB, budget 2.4 GB) forces
  more churn than the 21-key arm (htod 2.349 → 2.626 GB/step, +11.8%;
  page_ins_per_token 321 → 375) and stays byte-identical. Mass-forced
  evict-and-re-admit does not corrupt. The trigger is which tensor, not churn.

• Bisecting the 21 bypasser keys via the PIN_KEYS allow-list:

  Pinned keys	Count	Result
  919,920,287,247,912,914,855,874,232,304	10	FAIL (3 tokens)
  919,920,287,247,912	5	FAIL
  919,920	2	FAIL
  919	1	FAIL — [96347, 3375, 724]
  920 (its int4 scales)	1	PASS (16 IDs)
  287 (a 35 MiB block weight)	1	PASS (16 IDs)

Pinning only key 919 corrupts. 919 = 389,283,840 B = 152 064 (vocab) × 5 120 (hidden) × 0.5 — the int4-quantised lm_head / vocab-projection weight,
the one tensor that directly produces the logits deciding token identity. Its
scale tensor (920) alone is safe, so it is the weight matrix itself, not the
quant metadata. This turns MEMORY_ARCHITECTURE.md §3.5's "unisolated
value-corruption" into a single-tensor bug.

Reconciliation — why threshold-pin is safe and bypasser-pin is not

Reproducing your 2.4 GB threshold arm with the key trace on shows key 919
retained_page_ins=0 bypass_page_ins=16 — the lm_head is never pinned, even
at 2.4 GB / 67 tensors. It is the last large tensor touched each step (logits
run after every block), so the size+arrival budget is spent before it is reached.
The threshold rule is safe because it structurally cannot pin the poisonous
tensor; the explicit rule corrupts because its allow-list contains it. The
difference between the two selection rules is the finding — exactly as you
asked me to state it.

Mechanism: narrowed, and a hypothesis already refuted by a control

The stable-slot machinery (#716) exists for CUDA-graph pointer baking, so the
obvious cause is a graph hazard when the lm_head is promoted from bypass
(throwaway VA, never baked) into the stable-slot population. Ruled out:
pinning 919 with ONNX_GENAI_CUDA_GRAPH=0 (captures=0 replays=0) still
corrupts. The first three tokens match the reference before the collapse, so the
retained lm_head reads correctly for the first steps and then goes wrong —
consistent with retained-page staleness (a pinned page is filled once and
served as a hit thereafter, never re-filled), not a first-read address error.
Stated as a leading hypothesis, not a proven cause. Honest state: isolated to
one tensor and one transition (retain-across-steps vs stream-fresh-each-step);
precise mechanism not yet root-caused.

This also means #892's "evict-and-re-admit" localisation is at best incomplete:
here the page is pinned and never re-admitted, yet it still corrupts.

Round-1 evidence (unchanged, verified)

• 867/867 keys reads_per_step = 1.000 — no high-reuse tensor.
• Gap = 23 single-read large tensors, bypassed_page_in_bytes/step = 1.048 GB
  to the byte (~90% of the 1.158 GB recoverable gap).
• Fresh #837 item 3: attribute residency streaming gap; byte-aware policy rejected (negative result) #886 reproduction on 29d1515a; softened reuse-vs-retention wording per
  your note (Managed offload is 30x SLOWER than WDDM demand paging on Windows — and it is the default: the copy-to-VRAM buys nothing for a single-touch access pattern #864).
• Arm 4a (threshold 1.2 GB, 33 tensors): byte-identical PASS, htod unchanged —
  you reproduced this to the byte.

On item 3 itself

The practical answer is unchanged: the gap is only safely recoverable by the
funded #864 zero-copy hybrid (never evicts a retained page for a cold weight;
~8× on Linux #925/#936; WDDM-blocked by the aperture ceiling, not admission), so
no managed-path admission policy should ship. The value here is the isolation,
not a policy.

Gates (run locally; no source changed since they last passed)

• cargo fmt -p onnx-runtime-ep-cuda -p onnx-genai-engine -- --check ✅
• cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings ✅
• cargo test -p onnx-runtime-ep-cuda --features cuda --lib — 374 passed ✅
• cargo test -p onnx-genai-engine --features native-backend,cuda --lib native_decode — 96 passed ✅
• mobius parity SOLO (--ignored --test-threads=1) — 2 passed ✅

All bisection runs were env-only (the PIN_KEYS/threshold knobs, off by default);
the shipped size-blind path is untouched and byte-identical.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.81%. Comparing base (29d1515) to head (b27e933).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #944      +/-   ##
==========================================
- Coverage   78.44%   77.81%   -0.64%     
==========================================
  Files         354      354              
  Lines      148153   148153              
  Branches   148153   148153              
==========================================
- Hits       116226   115287     -939     
- Misses      27429    28372     +943     
+ Partials     4498     4494       -4     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	82.92% <ø> (-0.10%)	⬇️
mlas	80.96% <ø> (+0.23%)	⬆️
offline	77.55% <ø> (-0.68%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	43.30 µs	124.35 µs	+187.2%
🔴	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	375.81 µs	627.87 µs	+67.1%
🔴	matmul/large_generic_f16_threads=8/32x1024x1024	109.53 µs	157.18 µs	+43.5%
⚠️	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	77.96 µs	98.94 µs	+26.9%
⚠️	matmul/large_generic_bf16_threads=8/32x1024x1024	1.87 ms	2.17 ms	+16.1%
⚠️	matmul/medium_generic_bf16_threads=8/32x512x512	607.53 µs	699.65 µs	+15.2%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.02 ms	2.31 ms	+14.0%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	10.37 ms	11.80 ms	+13.8%
✅	gather/medium_f16_threads=1-internal/32768	2.56 µs	2.67 µs	+4.3%
✅	gather/medium_f32_threads=1-internal/32768	4.39 µs	4.51 µs	+2.8%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.17 ms	2.22 ms	+2.6%
✅	qwen3_sampling_processors/top_k_top_p_fast	665.85 µs	678.99 µs	+2.0%
✅	reduce_mean/large_f32_threads=1-internal/262144	987.40 µs	1.01 ms	+1.9%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	7.23 ms	7.32 ms	+1.3%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	530.44 µs	535.09 µs	+0.9%
✅	gather/medium_bf16_threads=1-internal/32768	2.46 µs	2.47 µs	+0.7%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.87 ms	5.84 ms	-0.6%
✅	qwen3_sampling_processors/top_k_partial_selection	143.62 µs	142.38 µs	-0.9%
✅	grammar_masking/llguidance_compute_mask/32	79.17 µs	78.43 µs	-0.9%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.65 ms	3.61 ms	-1.2%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	112.68 µs	109.90 µs	-2.5%
✅	logit_processing/seven_processor_chain_per_step	338.01 µs	328.60 µs	-2.8%
✅	kv_cache/alloc_dealloc_pages	41.40 µs	40.25 µs	-2.8%
✅	matmul/medium_generic_f16_threads=8/32x512x512	48.03 µs	46.40 µs	-3.4%
✅	sampling_latency/min_p_per_token	223.48 µs	215.62 µs	-3.5%
✅	gather/small_f16_threads=1-internal/4096	565.5 ns	540.7 ns	-4.4%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.49 ms	2.37 ms	-4.7%
✅	matmul/medium_generic_f32_threads=8/32x512x512	1.90 ms	1.79 ms	-6.2%
✅	matmul/small_generic_f16_threads=8/1x256x256	41.95 µs	38.33 µs	-8.6%
✅	sampling_latency/greedy_per_token	3.74 µs	3.34 µs	-10.5%
✅	sampling_latency/top_k_per_token	58.49 µs	52.32 µs	-10.6%
✅	tokenization/encode_tokens_per_second	460.75 µs	410.46 µs	-10.9%
✅	matmul/small_generic_f32_threads=8/1x256x256	57.11 µs	49.92 µs	-12.6%
✅	add/small_f16_threads=1-internal/1024	15.48 µs	13.39 µs	-13.5%
✅	add/medium_f32_threads=1-internal/262144	3.12 ms	2.67 ms	-14.3%
✅	gather/small_bf16_threads=1-internal/4096	601.5 ns	514.6 ns	-14.4%
✅	add/medium_f16_threads=1-internal/262144	3.15 ms	2.69 ms	-14.7%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	649.24 µs	553.62 µs	-14.7%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	46.06 µs	39.07 µs	-15.2%
🟢	gather/large_f16_threads=1-internal/131072	16.52 µs	14.01 µs	-15.2%
🟢	matmul/small_generic_f16_threads=1/1x256x256	41.69 µs	35.20 µs	-15.6%
🟢	gather/large_f32_threads=1-internal/131072	38.69 µs	32.66 µs	-15.6%
🟢	add/small_bf16_threads=1-internal/1024	16.73 µs	13.89 µs	-17.0%
🟢	sampling_latency/top_p_per_token	481.44 µs	393.26 µs	-18.3%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	44.32 µs	36.00 µs	-18.8%
🟢	matmul/medium_generic_f16_threads=1/32x512x512	44.77 µs	36.19 µs	-19.2%
🟢	gather/small_f32_threads=1-internal/4096	912.4 ns	730.5 ns	-19.9%
🟢	add/large_f32_threads=1-internal/4194304	53.32 ms	42.06 ms	-21.1%
🟢	tokenization/decode_tokens_per_second	7.99 ms	6.21 ms	-22.3%
🟢	add/medium_bf16_threads=1-internal/262144	3.56 ms	2.75 ms	-22.9%
🟢	add/large_bf16_threads=1-internal/4194304	57.39 ms	43.34 ms	-24.5%
🟢	matmul/small_generic_f32_threads=1/1x256x256	56.70 µs	42.35 µs	-25.3%
🟢	add/large_f16_threads=1-internal/4194304	58.98 ms	43.25 ms	-26.7%
🟢	add/small_f32_threads=1-internal/1024	281.2 ns	201.2 ns	-28.4%
🟢	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	983.74 µs	693.45 µs	-29.5%
🟢	reduce_mean/small_f32_threads=1-internal/4096	23.94 µs	15.92 µs	-33.5%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	388.09 µs	253.56 µs	-34.7%
🟢	gather/large_bf16_threads=1-internal/131072	24.93 µs	13.83 µs	-44.5%
🟢	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	79.28 µs	41.68 µs	-47.4%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 2.98 3.37 5.72 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

Validated. Merging — and this is the most valuable thing to come off this line.

Reproduced, including the control

Solo, managed streaming forced, qwen14b-zp, --tokens 16 --steady:

Pinned	Result
919 alone (389,283,840 B = 152064 × 5120 × 0.5 — int4 lm_head)	corrupts — generation emitted 3 tokens
920 alone (its own int4 scales, 48,660,480 B)	passes — token IDs byte-identical
67 keys / 2.371 GB by size threshold	passes (my earlier counter-arm)

The scales control is what makes this conclusive. Same layer, same quantization group, adjacent key, ~1/8 the size — and it is safe. So it is the weight matrix itself, not "something about that layer" and not "something about large tensors".

Why this matters more than the item it closes

#886 has been open since it was filed with the exact value-corruption line unisolated; MEMORY_ARCHITECTURE.md §3.5 records it as needing granule-level checksums and compute-sanitizer, neither of which is available on this box. It is now isolated to one tensor without either.

And the tensor is the one that would produce exactly this symptom: lm_head is the vocabulary projection — the only weight that computes token identity. Everything upstream can be perfectly correct and you would still get wrong tokens. That also explains the shape of the failure: three correct tokens then generation stops rather than degenerating into garbage, which is what a stale logits projection selecting EOS looks like.

Two corrections this forces, both worth more than the closure

1. My conservation explanation was wrong, and your control killed it rather than mine — I only showed churn mass was insufficient. You then showed why the threshold rule is safe: at 2.4 GB it still never pins 919 (retained=0 bypass=16), because 919 is the last large tensor touched each step and the budget is spent before reaching it. The selection-rule difference was the finding, exactly as it needed to be.

2. #892's evict-and-re-admit localisation is incomplete, and this PR proves it. Here the page is pinned and never re-admitted, and it still corrupts. That has been load-bearing in the design docs — MEMORY_ARCHITECTURE.md §3.5 states a static hot set "provably does not exercise the corrupting path", and #912's hybrid safety argument rests on it. It is now known to be false in at least one case. I will correct those docs.

Ruling out CUDA graph by re-running with ONNX_GENAI_CUDA_GRAPH=0 was the right next control, and stating retained-page staleness as a hypothesis rather than a cause is the right level of confidence for what the evidence supports.

Gates

fmt clean · exact CI CUDA clippy gate exit 0 · onnx-runtime-ep-cuda --lib 374 passed · native_decode 96 passed · mobius 2 passed · origin/main merges clean · the 919/920 arms reproduced above.

All knobs default off and the shipped path is unchanged, so the instrumentation merges on its own merits regardless of where the mechanism hunt lands.

What I am doing next

Filing the mechanism as its own issue rather than leaving it in #837's tail: why does a pinned, never-evicted, never-re-admitted page of the int4 lm_head weight go stale? That is now a bounded question with a one-tensor reproduction, which is a completely different proposition from where #886 has been sitting.