Make CUDA weight-residency budget elastic against KV occupancy (#857) by justinchuby · Pull Request #866 · justinchuby/onnx-genai

justinchuby · 2026-08-13T04:18:45Z

Closes #857 — do not merge yet (opened for review per the issue).

TL;DR (read this first — the framing changed after #863/#864)

This change is correct, byte-identical, and safe, and within the managed weight-offload path it recovers the idle full-context KV reservation and cuts weight streaming 1.68× (htod_bytes_per_token 3.94 GB → 2.35 GB, page_ins_per_token 372 → 188, hit_rate 57% → 78%). But two hardware findings landed while this was in flight and they bound how much it is worth:

WDDM pages out our VMM (cuMemCreate/cuMemMap) granules to host RAM under over-commit — #798 no-spill is an accounting guarantee, not physical residency, on WDDM #863: WDDM pages out our own VMM-mapped granules under system-wide pressure, not just cuMemAlloc. So oversubscribed_bytes == 0 / peak_committed < managed_limit are ledger guarantees, not proof of physical residency. I now measure nvidia-smi device-resident alongside the ledger (below) and treat residency as real only where the two agree.
Managed offload is 30x SLOWER than WDDM demand paging on Windows — and it is the default: the copy-to-VRAM buys nothing for a single-touch access pattern #864: the managed copying offload path is ~30× slower than the legacy WDDM zero-copy path (5.53 vs 0.18 tok/s, n=5/4), because each weight is read exactly once per decode step, so copying it into VRAM pays alloc/map/evict/sync overhead that WDDM avoids. This PR makes a path that trails WDDM trail it by less; it does not close the path-level gap.

So I am not manufacturing a win. I present this as a correct, well-evidenced within-path improvement plus an explicit statement (bottom) of where the real leverage now sits. If the reviewer prefers to hold it until the hybrid direction is decided, that is reasonable.

What this does

At load the weight-residency budget was derived once as resolved_device_budget − kv_bytes_per_token × max_context and never revisited, statically withholding the full declared-max-context KV reservation (1.611 GB on qwen14b-zp, ctx 8192) from weights even for short requests. This lends that reservation to weights and lets KV growth reclaim it on demand, still guaranteeing a sequence reaches its declared max_context, and leaves a tunable device headroom (load-bearing per #863).

Design (points 1–4)

1. Reclaim path. Budget/accounting change that unlocks already-built machinery. reconcile_cuda_offload_budget_after_native_load withholds only an initial KV floor (kv_capacity_bucket(1, max), ~50 MB) instead of the full reservation. Bytes above the floor (less headroom) are lent to the weight cache. As KV grows, ensure_capacity_seq_major_fixed → prepare_mapped_growth(KvCache) drives MappedGrowthAuthority: it consumes KV's allowance, then unassigned capacity, then transfers allowance from the weight holder (reclaim_priority()=0) and calls reclaim_mapped() (LRU eviction of unreferenced pages). Transactional with rollback, serialized (no deadlock), requester filtered from its own victim list. Weights shed in step with KV growth; no refusal path.

2. Hysteresis. The existing monotonic KV bucket growth (power-of-two) gives O(log max_context) reclaim events, each permanently lowering the weight ceiling. The lending high-water mark is a fixed budget set once at load, not re-lent per event. No thrash.

3. Capture. Budget/limit changes move no VA and no physical mapping. Stable-VA weight paging (#716/#796) keeps captured graphs valid. Measured: captures=2, fallbacks=0.

4. Scope. Single-sequence only. Serving/admission (#750) out of scope.

Gating. Only when managed_no_spill && commits_on_demand && dynamic_kv_weight_lending_enabled() — the exact condition registering the weight cache as a reclaimable holder. Otherwise the static reservation is kept. Operator override unaffected. The non-override auto budget is recomputed from the elastic available (the plan-time device_budget_bytes carried the full reservation and would otherwise clamp the budget straight back — the one real trap, fixed).

Safety: WDDM headroom (question RESOLVED by #863)

#863 proved single-process and deterministically that this card pages out our own cuMemCreate/cuMemMap granules: an arena-mirroring allocator committed and touched all of 9,984 MiB on an 8,188 MiB card, but device-resident capped at ~7,942 MiB while host working set hit 9.49 GB — ~2 GB of our granules spilled to host RAM. cuMemCreate confers no pin. So this PR does not lend to the last byte: it keeps a tunable device headroom unlent below the managed limit (ONNX_GENAI_ELASTIC_LENDING_HEADROOM_BYTES, default 512 MiB), never dropping below the static-reservation baseline (never a regression). The headroom is now load-bearing, not precautionary.

Two nuances (from #863). (a) The spill is a latency effect, transparent to kernel correctness — the driver makes a granule resident before a kernel uses it, so it costs tok/s, not correctness; it is not the #851 illegal address. (b) It is very likely WDDM-specific: on TCC / datacenter GPUs cuMemCreate should fail at the physical limit rather than spill, so the headroom is a WDDM-deployment safeguard and the right default may differ (smaller/zero on TCC, larger under concurrent GPU use). The tunable exists precisely so the default can be set per target.

Results — qwen14b-zp, ctx 8192, 16 tokens, greedy (byte-identical)

All runs solo, nvidia-smi verified clear before each (card coordinated with the #851 agent). nvidia-smi peak = max device-resident memory.used sampled at 300 ms during the run (includes ~229 MiB OS/display baseline).

metric	baseline (main, managed)	elastic 512 MiB (default)	elastic headroom=0	legacy WDDM (`LEGACY_ALLOCATOR=1`)
htod_bytes_per_token	3,943,690,240	2,349,010,944	2,349,010,944	0 (zero-copy)
page_ins_per_token	372	188	188	0
hit_rate	57.09%	78.32%	78.32%	n/a
evictions	5,248	2,672	2,672	0
weight budget_bytes	6,116,140,442	7,139,550,618	7,676,421,530	— (offload off)
weight mapped_physical	5,662,310,400	6,406,799,360	6,406,799,360	—
peak_committed (ledger)	6,983,516,160	7,725,907,968	7,725,907,968	—
nvidia-smi peak resident MiB	—	7,479	—	7,951
oversubscribed / underflows	0	0	0	0
throughput tok/s (single, noisy)	—	0.72–1.42	—	7.42

Reading the table honestly

The byte win is real and byte-identical. Token IDs identical across every column: [96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]. Elastic lending cuts managed-path streaming 1.68×.
Solo, the lent residency is physically real. nvidia-smi peaked 7,479 MiB while the ledger peak_committed was 7,368 MiB (7,725,907,968 B) and physical_owned 7,194 MiB — device-resident tracked our commitment rather than capping below it. No spill solo (7,368 MiB fits under the 7,959 MiB free with ~590 MiB to spare). The WDDM pages out our VMM (cuMemCreate/cuMemMap) granules to host RAM under over-commit — #798 no-spill is an accounting guarantee, not physical residency, on WDDM #863 spill is a contention effect our ledger cannot see; on this shared box that is the realistic exposure, hence the headroom and this caveat. Precise statement (refined jointly with WDDM pages out our VMM (cuMemCreate/cuMemMap) granules to host RAM under over-commit — #798 no-spill is an accounting guarantee, not physical residency, on WDDM #863): Make managed no-spill VMM the default with automatic weight streaming (#755) #798 no-spill holds physically solo under managed_limit — nvidia-smi tracks our commitment 1:1; spill is specifically the system-wide over-commit case (a 2nd process, or engine-at-limit + anything else), which is invisible to our ledger because our own committed bytes do not change.
The right baseline is WDDM, and it wins decisively. WDDM moves zero explicit HtoD bytes (it reads cold weights zero-copy from shared host memory, nvidia-smi sitting at the 7,951 MiB physical ceiling) and still runs 7.42 tok/s here vs the managed path's 0.72–1.42 — corroborating Managed offload is 30x SLOWER than WDDM demand paging on Windows — and it is the default: the copy-to-VRAM buys nothing for a single-touch access pattern #864's 30.7×. Wall-clock is not a headline on this box (documented 3.9–28 tok/s spread), but the direction is unambiguous and structural, not noise.
The 512 MiB headroom costs nothing at 16 tokens (resident working set 6.4 GB sits below both ceilings); it caps the ceiling for high-residency runs, bounding our footprint under the contention where WDDM pages out our VMM (cuMemCreate/cuMemMap) granules to host RAM under over-commit — #798 no-spill is an accounting guarantee, not physical residency, on WDDM #863 spill occurs.

max_context reachability (the property under risk)

New governor test kv_reclaims_the_full_lent_budget_from_weights_across_bucketed_growth drives the production reclaim code (prepare_mapped_growth, allowance transfer, reclaim_mapped): weights hold the full budget, KV grows bucket-by-bucket and reclaims its entire reservation, reaching full reservation with oversubscribed_bytes == 0 throughout, weights shedding exactly the reclaimed bytes. The mandatory GPU gate mobius_seqmajor_growth_parity_native_cuda passes solo with dynamic KV/weight lending active, head-major == seq-major byte-identical, captures>0, fallbacks=0.

Where the real leverage sits (per #864)

Weights are read once per decode step, so within a step there is no reuse to amortize a copy against; the only reuse is across steps, which is exactly what a resident cache captures — and what this PR enlarges. But the copying cache pays alloc/map/evict/sync per miss that WDDM's zero-copy reads avoid, so at the whole-path level WDDM dominates for the streamed (cold) fraction. The durable direction is a hybrid: keep a resident hot set (where a larger budget — this PR — genuinely helps) and serve the cold fraction by zero-copy direct reads rather than copy-then-discard. Lending more budget to a copying cache has real but bounded value: it helps the resident fraction, not the streamed one. This PR is a correct, safe increment for the resident fraction and an honest data point that the streamed fraction wants a different mechanism, not more copy budget.

Validation (all local, per issue)

cargo fmt -p {onnx-genai-engine, onnx-runtime-ep-cuda, onnx-runtime-memory-governor} -- --check ✓
cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings ✓
cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend and --features cuda,native-backend -- -D warnings ✓
cargo test -p onnx-runtime-session ✓; cargo test -p onnx-runtime-memory-governor ✓ (new reclaim test); cargo test -p onnx-genai-engine --features cuda,native-backend --lib elastic ✓ (4 tests)
MANDATORY mobius_seqmajor_growth_parity_native_cuda --ignored --test-threads=1, solo, GPU clear ✓

Files

crates/onnx-runtime-ep-cuda/src/{provider,lib}.rs: public dynamic_kv_weight_lending_enabled().
crates/onnx-genai-engine/src/engine/load.rs: elastic KV floor, tunable headroom high-water mark, non-regression clamp, gating; unit tests.
crates/onnx-runtime-memory-governor/src/lib.rs: reclaim-to-full-reservation test.

At load, the weight-residency budget was derived once as esolved_device_budget - kv_bytes_per_token * max_context and never revisited, statically withholding the full declared-max-context KV reservation (1.611 GB on qwen14b-zp) from weights even for short requests that commit a fraction of it. Weights then stream bytes that would have fit in the idle reservation. When the dynamic KV/weight reclaim path is guaranteed active (managed no-spill + on-demand-committing arena + lending enabled -- the exact condition that registers the weight cache as a reclaimable mapped holder in adopt_memory_governor), withhold only an initial KV floor (the first KV bucket the engine commits at load) instead of the full-context reservation. The bytes above the floor are lent to the weight cache and reclaimed by KV growth on demand, one bucket at a time, through the existing transactional MappedGrowthAuthority reclaim. A sequence can therefore still reach its declared max context: KV reclaims the lent budget rather than finding it statically reserved. The auto (non-override) budget must be recomputed from the elastic �vailable; the plan-time device_budget_bytes was derived with the full reservation and would otherwise clamp the budget straight back via equested.min(available). Scope: single-sequence. No kernel or capture changes -- budget/limit changes move no VA and no physical mapping, so captured decode graphs stay valid. - Add public dynamic_kv_weight_lending_enabled() to the CUDA EP. - Elastic KV floor sizing + gating in reconcile_cuda_offload_budget_after_native_load. - Unit tests: engine (floor/gate), governor (KV reclaims full lent budget from weights across bucketed growth with oversubscribed==0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov · 2026-08-13T04:26:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.20%. Comparing base (de5d38c) to head (0a5290d).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #866      +/-   ##
==========================================
- Coverage   78.46%   78.20%   -0.26%     
==========================================
  Files         351      353       +2     
  Lines      145472   147671    +2199     
  Branches   145472   147671    +2199     
==========================================
+ Hits       114138   115480    +1342     
- Misses      26850    27695     +845     
- Partials     4484     4496      +12

Flag	Coverage Δ
cli-ort-linux	`83.42% <ø> (ø)`
cli-ort-windows	`82.92% <ø> (-0.10%)`	⬇️
mlas	`81.33% <ø> (?)`
offline	`77.95% <100.00%> (-0.32%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-runtime-memory-governor/src/lib.rs	`83.36% <100.00%> (+0.24%)`	⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The first cut lent every byte above the KV floor to weights, pushing the device to within ~845 KB of the managed no-spill limit. On WDDM a device that near full can have physical granules paged to host RAM by the OS under demand -- a spill invisible to the governor ledger, so oversubscribed_bytes would read 0 while the device is in fact spilling. It is not yet established whether our VMM-mapped granules (cuMemCreate/cuMemMap) are subject to that eviction or only ordinary cuMemAlloc. Until that is settled, keep a headroom margin unlent as a high-water mark below the managed limit rather than lending to the last byte. The margin is a tunable (ONNX_GENAI_ELASTIC_LENDING_HEADROOM_BYTES, default 512 MiB); a cheap follow-up can lower it or set it to 0 once the WDDM question is answered in our favour. The reverse mistake -- a hidden host-spill regression that only shows up as wall-clock variance -- is not cheap. The headroom never drops the budget below what the static full-context reservation would have granted, so elastic lending is never a regression versus baseline. Add elastic_weight_budget_bytes as the pure, testable core of that arithmetic plus unit tests for the headroom and non-regression clamp. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby · 2026-08-13T04:53:49Z

Validated and merging. Gates I ran myself on a worktree with main merged in: per-package cargo fmt --check pass; cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings pass; cargo test -p onnx-runtime-memory-governor 48 passed including the new reclaim test; and the mandatory mobius_seqmajor_growth_parity_native_cuda -- --ignored --test-threads=1 gate solo with nvidia-smi verified empty beforehand — 2 passed in 58.14s.

The result is real and the counters are the right ones. htod_bytes_per_token 3,943,690,240 → 2,349,010,944 (1.68×), page_ins_per_token 372 → 188, hit_rate +21.2 points, captures/fallbacks unchanged at 2/0, ledger counters all 0, token IDs byte-identical. And the baseline arm reproduced the issue's numbers to the byte, which is the check that says the comparison is measuring what it claims.

Three things you did that I want to name, because they are the difference between a number and a result.

You tested the property the change puts at risk, where it actually lives. kv_reclaims_the_full_lent_budget_from_weights_across_bucketed_growth drives the production reclaim path — prepare_mapped_growth, allowance transfer, reclaim_mapped — rather than asserting on a simulated budget. The max-context guarantee is exactly what elastic lending endangers, and testing it against the real machinery is what makes this mergeable.

You turned #863 into a design constraint instead of ignoring it. The headroom is not caution, it is evidence-backed, and the non-regression clamp — the margin never drops the budget below what the static reservation would have granted — means elastic lending cannot be worse than baseline. That is the right shape for a change that trades a guarantee for performance.

You reported that the default headroom costs nothing at 16 tokens, and explained why. The resident working set (6.4 GB mapped_physical) sits below both ceilings, so the win is identical at headroom 0 — the margin only bounds the ceiling for high-residency runs, which is precisely where #863's spill occurs. Reporting "this safety margin is free in the measured case, and here is the case where it is not" is far more useful than a headline that quietly benefits from a knob.

Also good catch on the plan-time device_budget_bytes clamping the elastic budget straight back via requested.min(available). That is the kind of trap that would have produced a change with no measurable effect and a plausible explanation for why.

On the reframing against the WDDM baseline (#864): correct, and I want it on the record that you did it unprompted after I told you your work might be worth less than it looked. The durable leverage is the hybrid — resident hot set plus zero-copy cold reads — not lending more budget to a copying cache, because for a single-touch access pattern the copy has no reuse to amortize. This PR still stands on its own: it makes the resident fraction larger and cheaper, and that remains true under the hybrid. It is a real improvement to the arm we keep.

One refinement now settled (from the #851/#863 agent, using your measurement): #798 no-spill holds physically when solo and under managed_limit — nvidia-smi tracked your commitment 1:1 (resident 7,479 MiB ≥ committed 7,368 MiB). Spill is specifically the system-wide over-commit case, invisible to our ledger because our own committed bytes do not change. That is sharper than either of our earlier statements and is the version that should propagate.

Merging. #857 stays open — this is single-sequence; multi-request lending interacts with admission (#750).

github-actions · 2026-08-13T05:04:44Z

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	44.31 µs	118.95 µs	+168.5%
🔴	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	55.21 µs	107.30 µs	+94.3%
🔴	`matmul/large_generic_f32_threads=8/32x1024x1024`	3.98 ms	7.44 ms	+87.1%
🔴	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	382.30 µs	562.32 µs	+47.1%
🔴	`matmul/large_generic_bf16_threads=8/32x1024x1024`	1.68 ms	2.41 ms	+43.5%
🔴	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	77.81 µs	103.68 µs	+33.2%
⚠️	`matmul/large_generic_f32_threads=1/32x1024x1024`	9.76 ms	12.60 ms	+29.1%
⚠️	`sampling_latency/min_p_per_token`	236.84 µs	303.72 µs	+28.2%
⚠️	`matmul/medium_generic_f16_threads=1/32x512x512`	35.00 µs	44.09 µs	+26.0%
⚠️	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.34 ms	2.94 ms	+25.3%
⚠️	`matmul/medium_generic_bf16_threads=8/32x512x512`	551.81 µs	675.14 µs	+22.3%
⚠️	`matmul/medium_generic_f16_threads=8/32x512x512`	40.94 µs	49.40 µs	+20.7%
⚠️	`matmul/medium_generic_bf16_threads=1/32x512x512`	573.55 µs	680.98 µs	+18.7%
⚠️	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	954.47 µs	1.12 ms	+17.4%
⚠️	`matmul/large_generic_f16_threads=8/32x1024x1024`	103.15 µs	120.88 µs	+17.2%
⚠️	`matmul/large_generic_bf16_threads=1/32x1024x1024`	2.20 ms	2.55 ms	+15.9%
⚠️	`gather/large_f32_threads=1-internal/131072`	46.16 µs	53.22 µs	+15.3%
✅	`matmul/small_generic_f16_threads=8/1x256x256`	36.32 µs	41.34 µs	+13.8%
✅	`matmul/small_generic_bf16_threads=1/1x256x256`	37.67 µs	41.34 µs	+9.7%
✅	`sampling_latency/top_p_per_token`	463.56 µs	493.72 µs	+6.5%
✅	`matmul/large_generic_f16_threads=1/32x1024x1024`	98.13 µs	102.53 µs	+4.5%
✅	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	4.07 ms	4.25 ms	+4.2%
✅	`grammar_masking/llguidance_compute_mask/32`	83.48 µs	86.86 µs	+4.0%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	46.41 µs	47.55 µs	+2.5%
✅	`qwen3_sampling_processors/top_k_partial_selection`	159.05 µs	162.63 µs	+2.2%
✅	`tokenization/encode_tokens_per_second`	433.95 µs	434.93 µs	+0.2%
✅	`tokenization/decode_tokens_per_second`	7.39 ms	7.12 ms	-3.7%
✅	`qwen3_sampling_processors/top_p_fast_after_top_k`	608.82 µs	578.99 µs	-4.9%
✅	`reduce_mean/medium_f32_threads=1-internal/65536`	289.54 µs	270.96 µs	-6.4%
✅	`kv_cache/alloc_dealloc_pages`	48.28 µs	45.04 µs	-6.7%
✅	`add/large_f32_threads=1-internal/4194304`	50.30 ms	46.71 ms	-7.1%
✅	`gather/large_f16_threads=1-internal/131072`	18.53 µs	17.20 µs	-7.2%
✅	`logit_processing/seven_processor_chain_per_step`	398.78 µs	368.49 µs	-7.6%
✅	`matmul/medium_generic_f32_threads=1/32x512x512`	3.07 ms	2.84 ms	-7.6%
✅	`sampling_latency/greedy_per_token`	3.65 µs	3.36 µs	-8.0%
✅	`sampling_latency/top_k_per_token`	67.93 µs	62.38 µs	-8.2%
✅	`reduce_mean/large_f32_threads=1-internal/262144`	1.17 ms	1.07 ms	-8.7%
✅	`gather/large_bf16_threads=1-internal/131072`	22.27 µs	20.33 µs	-8.7%
✅	`reduce_mean/small_f32_threads=1-internal/4096`	18.89 µs	17.08 µs	-9.6%
✅	`gather/small_f32_threads=1-internal/4096`	771.9 ns	683.4 ns	-11.5%
✅	`matmul/small_generic_bf16_threads=8/1x256x256`	47.06 µs	41.25 µs	-12.4%
✅	`gather/small_f16_threads=1-internal/4096`	546.0 ns	468.8 ns	-14.1%
✅	`matmul/medium_generic_f32_threads=8/32x512x512`	2.02 ms	1.73 ms	-14.4%
🟢	`gather/medium_bf16_threads=1-internal/32768`	3.70 µs	3.12 µs	-15.7%
🟢	`add/medium_f32_threads=1-internal/262144`	3.41 ms	2.87 ms	-16.0%
🟢	`add/large_f16_threads=1-internal/4194304`	53.22 ms	44.50 ms	-16.4%
🟢	`add/large_bf16_threads=1-internal/4194304`	55.33 ms	45.79 ms	-17.2%
🟢	`matmul/small_generic_f32_threads=8/1x256x256`	69.01 µs	55.70 µs	-19.3%
🟢	`add/small_f32_threads=1-internal/1024`	308.3 ns	243.3 ns	-21.1%
🟢	`add/medium_f16_threads=1-internal/262144`	3.60 ms	2.83 ms	-21.3%
🟢	`qwen3_sampling_processors/top_k_top_p_fast`	896.48 µs	694.61 µs	-22.5%
🟢	`matmul/small_generic_f16_threads=1/1x256x256`	49.79 µs	38.47 µs	-22.7%
🟢	`add/medium_bf16_threads=1-internal/262144`	3.63 ms	2.72 ms	-25.0%
🟢	`add/small_bf16_threads=1-internal/1024`	18.95 µs	14.19 µs	-25.1%
🟢	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	8.31 ms	5.89 ms	-29.2%
🟢	`gather/medium_f32_threads=1-internal/32768`	5.50 µs	3.88 µs	-29.5%
🟢	`add/small_f16_threads=1-internal/1024`	20.17 µs	13.65 µs	-32.3%
🟢	`gather/small_bf16_threads=1-internal/4096`	713.3 ns	479.3 ns	-32.8%
🟢	`gather/medium_f16_threads=1-internal/32768`	3.94 µs	2.54 µs	-35.5%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 5.82 4.10 6.99 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

…877) Adds `scripts/zero_copy_host_map_probe.py`, which measures the two capabilities the #864 hybrid depends on — against the real weight blob, with no engine involvement. The hybrid is: keep the hot weight set resident in VRAM, and for the cold remainder do **not** copy — let the device read it in place from host memory, since each weight is read exactly once per decode step and a copy has no reuse to amortize. That rests on `cuMemHostRegister` over an existing read-only file mapping plus `cuMemHostGetDevicePointer`, which the tree does not currently use anywhere. **Measured** (RTX 4060 Laptop, 1 GiB slice of `qwen14b-zp`, best-of-3, CUDA-event timed, `can_map_host_memory = 1`): | path | bandwidth | |---|---:| | device reading mapped host memory (zero-copy) | **11.41 GB/s** | | explicit HtoD copy — what the managed path does today | 11.28 GB/s | | already-resident VRAM read — the ceiling | **110.46 GB/s** | `cuMemHostRegister` on a 1 GiB read-only mapping **succeeds in 3.1 ms**, so registration is cheap enough to do once at load over the whole blob rather than per page-in. **Two conclusions, and they reorder the hybrid's priorities — in the opposite direction to my assumption when I filed #864.** **Zero-copy is not a bandwidth win.** 11.41 vs 11.28 GB/s is 1.01×; PCIe limits both. What zero-copy removes is the copy's *second* pass and everything the copy drags along — the CPU memcpy into pinned staging, the VRAM allocation, the `cuMemMap`, the eviction, the synchronize. Those are exactly what #837 items 1–2 were separately trying to shave, so de-copying subsumes rather than competes with them. **The resident hot set is worth ~9.7×, and is the larger prize.** Every byte moved from the streamed fraction into the resident fraction gets ~10× faster; every byte moved from copied-stream to zero-copy-stream gets ~1× faster on bandwidth and only saves overhead. So the ordering should be: maximise residency first (#866 already moved the weight budget 6.12 → 7.14 GB and `htod_bytes_per_token` 3.94 → 2.35 GB), de-copy the cold remainder second. It also explains #874's result exactly: WDDM reaches 8.09 tok/s doing precisely this — zero-copy at PCIe rate, no copy machinery, `htod_bytes_per_token = 0`. We were paying the same PCIe pass *plus* the machinery. The hybrid beats WDDM only by holding a **larger and better-chosen** resident set than the driver picks blind, which is the one edge we have — and per #863 it is advisory rather than enforceable on WDDM. **Caveat, recorded in the docstring:** a `cuMemcpyDtoD` *from* mapped host memory measures the rate at which the device pulls those bytes across PCIe — the right proxy for a kernel reading in place, but not literally a strided GEMV. Treat the zero-copy figure as an upper bound on that arm. Also noted: registering all 8.33 GB of referenced weights page-locks that much host RAM (63.8 GB total / 34.4 GB free here), which is fine on this box but is a real claim on smaller hosts and needs a policy. Probe-only; no engine or runtime code touched. Verified it runs clean end to end (exit 0) after fixing the buffer-release ordering that `mmap.close()` requires. Refs #864 #837 #866 #874 #863. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

…ems (#879) `PROGRESS.md` tracked the dispatch-bound work (models that **fit** device memory) but carried nothing about the other regime — models whose weights **do not** fit — despite that line producing the largest single change today. A reader looking for "what happened to large-model support" would have found nothing, while the answers were scattered across eight issues. ## Added milestone - **#874** — stop auto-enabling managed weight streaming on Windows/WDDM. `8.09 tok/s` with `htod_bytes_per_token = 0` against `0.11` managed; `main` immediately prior measured 0.05–0.08, so **~100× end to end**, byte-identical output. Records the *structural* cause rather than the ratio: each weight is read **exactly once per decode step**, so both paths move the same bytes over the same link, but ours added a CPU memcpy, a VRAM allocation, a `cuMemMap`, an eviction and a synchronize — to buy residency discarded before it is ever re-read. Notes that explicit requests are still honoured, that the force knob is parsed so a *typo* keeps the fast default, and that **Linux is unchanged and must stay so**. - **#866** — elastic weight budget (1.68× fewer bytes, +20.3 points byte-weighted hit rate, max-context reachability tested through the production reclaim path). - **#853/#856** — `total_weight_bytes` was **2.00× too large** (50% orphaned external-data prefix). Records *how it was found*: arithmetic that could not close, with measured traffic below its own theoretical floor. - **#863** — `no-spill` is an accounting guarantee, with the later refinement that it **does** hold physically when solo and under `managed_limit`; spill is specifically the system-wide over-commit case. - **#869** — byte-weighted hit rate, and why the count-based one misleads. - **#877** — hybrid feasibility, including that it **reversed the assumed priority order**: residency is worth ~9.7×, zero-copy is not a bandwidth win. ## Added framing **Two regimes, and wins do not transfer.** Weights fit ⇒ node-dispatch bound (#870). Weights do not fit ⇒ PCIe-streaming bound. Node-count reduction is invisible in the streaming regime; the streaming work is invisible on a model that fits. This is worth stating explicitly because both lines are active and a throughput number from one says nothing about the other. ## Added open items, with measured sizes - **The #864 hybrid**, with its unresolved risk stated plainly: #877's figure is a *sequential* `cuMemcpyDtoD`, not a strided GEMV, and a real kernel could be far worse — which would kill the zero-copy half. Capture compatibility with host-mapped pointers is untested. - **#837 item 3** — 1.97× above the streaming floor, **1.158 GB/step recoverable**, ~91 ms/step at measured bandwidths; `bypassed_page_ins` still unattributed. - **#750 batching** — with the condition stage 1 established: it amortizes **only** as one fused forward with `M = N`. - **#851** — parked with the three eliminated hypotheses and what unblocks it, so nobody repeats the search. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

…ication trade, measured (#750) (#891) ## Stage 2b of #750 — batched-KV fused forward + the KV-multiplication trade, measured (scoped by #844, continuing #884) Refs #750. Continues the staging plan in #844 / #884. **Do not merge; #750 stays open.** Stage 2a (#884) proved weight-streaming amortization with a **stateless, empty-past** batch-N fused forward. It explicitly owed stage 2b two things: (1) the **batched KV layout / N-reservation decision**, and (2) **recapture accounting — quantified, not assumed**. This PR delivers the decision and the measurement, lands the first batched forward that carries a **real (non-empty) batched KV past**, and reports an honest crossover analysis. It deliberately does **not** rewrite `cuda.rs` — see "Why this is where 2b splits." ### TL;DR - **Design decision (firm, evidenced): batched *dense* layout `[N, kv_heads, cap, head_dim]`, not N separate VMM reservations** — because the native decode binds past-KV to `session.run` and the ONNX attention op couples the batch dim across QKV/mask/past-KV. N disjoint VA reservations cannot be presented as one dense ONNX tensor; that requires a custom paged kernel, which the owner explicitly deferred to the paged/#721-stage-3 path. - **Amortization reconfirmed on hardware, and shown KV-content-invariant**: `htod_bytes_per_token` falls exactly `1/N` and is **identical at past_len 0, 512, 2048**. - **Recapture is *not* the net-loss risk for uniform fixed-N 2b** (it's ~1 recapture, amortized); it is the central risk for the ragged 2c path. - **The real cost is KV multiplication**, and it does **not** amortize: batched decode adds a per-token floor `L × kv_bytes_per_token` that is independent of N, plus a hard batch ceiling `N_max ≈ budget / (L × kv_bytes_per_token)`. Batching still wins per-token up to that ceiling; the win *saturates* at the KV floor. --- ### 1. Measurement — led by the deterministic per-token counters Model `qwen14b-zp` (8.33 GB referenced weights, genuinely over-budget), streaming arm `ONNX_GENAI_WEIGHT_OFFLOAD=1` (the arm where `htod_bytes` is nonzero and its `1/N` collapse is directly observable; the post-#874 WDDM default reports `htod_bytes = 0`), `ONNX_GENAI_CUDA_GRAPH=1`, RTX 4060 Laptop (8188 MiB). Each measurement window verified `gpu_contention_check: clear` **per run** (#851). **1a. Stage-2a amortization, reconfirmed byte-for-byte** (`--fused-forward-amortization 1,2,4,8`): | batch N | htod_bytes (total) | page_ins (total) | **htod_bytes_per_token** | **page_ins_per_token** | hit_rate | evictions | |--------:|-------------------:|-----------------:|-------------------------:|-----------------------:|---------:|----------:| | 1 | 3,575,949,312 | 301 | **3,575,949,312** | **301.0** | 65.28% | 0 | | 2 | 3,575,949,312 | 301 | **1,787,974,656** | **150.5** | 65.28% | 0 | | 4 | 3,575,949,312 | 301 | **893,987,328** | **75.2** | 65.28% | 0 | | 8 | 3,575,949,312 | 301 | **446,993,664** | **37.6** | 65.28% | 0 | **1b. NEW — the batched forward with a non-empty length-L KV past** (`--fused-forward-kv-sweep 1@0,2@0,4@0,8@0,1@512,…,8@2048`). The counters are **identical across every past_len** at each N: | batch N | past_len L | htod_bytes (total) | page_ins | **htod_bytes_per_token** | hit_rate | evictions | |--------:|-----------:|-------------------:|---------:|-------------------------:|---------:|----------:| | 1 | 0 / 512 / 2048 | 3,575,949,312 | 301 | **3,575,949,312** | 65.28% | 0 | | 2 | 0 / 512 / 2048 | 3,575,949,312 | 301 | **1,787,974,656** | 65.28% | 0 | | 4 | 0 / 512 / 2048 | 3,575,949,312 | 301 | **893,987,328** | 65.28% | 0 | | 8 | 0 / 512 / 2048 | 3,575,949,312 | 301 | **446,993,664** | 65.28% | 0 | **Reading this honestly.** The `1/N` amortization is real and robust, and it does **not depend on KV content** — a genuinely useful negative control (rules out any artifact where KV pressure was secretly producing the number). But it also means **this stateless probe cannot measure the weight-reclaim trade**: a transient host-supplied KV input does not compete with the *persistent* weight-residency budget the way persistent batched-KV bindings would. So the reclaim cost is real but is a property of the stage-2b-*impl* persistent bindings, and I quantify it analytically below rather than pretend the probe measured it. **1c. Batch-1 real *decode* baseline** (`--tokens 48 --steady`), the anchor for the gates and the recapture count: | metric | value | gate | |---|---:|---| | htod_bytes_per_token | 2,349,010,944 (~2.35 GB) | — | | cuda_graph captures / fallbacks / invalidations | **2 / 0 / 1** | `captures>0, fallbacks==0` ✅ | | peak_committed_physical_bytes | 7,725,907,968 | `< limit 7,726,753,178` ✅ (**99.99% full**) | | oversubscribed_bytes | 0 | ✅ | | ref_underflows / byte_underflows / unaccounted_committed_bytes | 0 / 0 / 0 | ✅ | --- ### 2. Design decision: batched dense layout, not N reservations The native decode path binds `past_key_values.*` as ONNX tensors into `session.run` and relies on the graph's attention op, under CUDA-graph capture. A single **fused** forward with `M = N` (the discipline #844 proved is the *only* implementation that amortizes) therefore needs past-KV as **one dense tensor `[N, kv_heads, cap, head_dim]`**. N separate VMM reservations (the owner's per-sequence contiguous-VA route recorded on #750) cannot be handed to a stock attention op as one tensor — that route needs a custom flat/paged kernel that reads each sequence from its own VA, which is exactly the paged path (#721 stage 3 / 2c) the owner deferred. **For stage 2b (uniform length, stock ORT attention decode), the dense `[N,…]` layout is the correct and only viable choice.** The per-sequence-VA route remains correct for the ragged/paged future; it is not this stage. ### 3. Recapture accounting — quantified From the capture state machine plus the batch-1 baseline (`captures=2, invalidations=1`): - **Initial capture:** 1, at the first decode step of a given batch shape. - **KV-growth recaptures:** identical to batch-1 and **independent of N** — head-major realloc invalidates+recaptures per bucket growth (the `invalidations=1` above); seq-major fixed-stride keeps device pointers stable and recaptures 0×. Batching does not change this axis. - **Batch-shape recaptures:** exactly **1 per distinct N**. Uniform fixed-N decode has one shape ⇒ folded into the initial capture ⇒ **~1 total, amortized over the whole run ⇒ negligible.** **Conclusion: recapture is not a net-loss risk for uniform fixed-N (2b).** The recapture *thrash* risk lives in **2c** (ragged/continuous batching), where N changes per step and the captured graph can be rebuilt up to `max_batch` times — potentially every step if N oscillates. That is the number to gate 2c on, and it is out of scope here. ### 4. The real trade: KV multiplication (honest crossover) Model per-step streaming under the elastic budget (#866), with `C` = weight bytes that miss the resident budget at batch-1, `kvpt` = kv_bytes_per_token: ``` htod_per_token(N, L) ≈ C/N + L·kvpt + activation_overhead(N)/N ``` - `C/N` — the weight base **amortizes** with N (the win). - `L·kvpt` — the KV you reclaim is **per-token constant** (KV scales with N, so KV-per-token does not), so it **never amortizes**. It is the floor the win saturates to. - **Hard ceiling:** KV must fit — `N·L·kvpt < budget − min_activations`. On this box (`budget ≈ 7.73 GB`, qwen14b `kvpt = 196,608 B/token`): `N_max ≈ 7.73 GB / (L·196,608)` → ~**19 at L=2048**, fewer at longer context. So batching **always improves per-token streaming up to `N_max`** (monotonic: `C/N + L·kvpt < C + L·kvpt`), but the benefit **saturates at `L·kvpt`** and is bounded by the KV-fit ceiling. At the measured 99.99%-full batch-1 commit, every KV byte reclaims a weight byte 1:1, so `htod_bytes_per_token` **rises toward that floor as N (and L) grow** — exactly the trade #866 predicts. **This is not a net loss at any reachable N** for uniform decode on this model; it is a *saturating* win with a firm N ceiling. It would only invert into a loss if recapture thrash were in play, which (per §3) it is not for uniform fixed-N — it is for 2c, informing the #864 hybrid direction there. ### 5. Why this is where 2b splits (evidence, per #844's invitation) A *correct* persistent batched-KV decode requires coordinated changes across ≥6 coupled sites in the 3,900-line `cuda.rs` (`persistent_state_shapes` axis0 `1→N`; all `kv_commit` geometry; mask `[1,max]→[N,max]`; `write_decode_inputs`; symbolic-axis pinning; capture signature) **plus** the engine loop carrying N sequences — landing all of it at once, against the 99.99%-full memory limit and the mandatory Mobius CUDA parity gate, is large and risky. Per the explicit guidance ("prefer a smaller increment that lands"), 2b lands the **decision + the on-hardware measurement + the batched-forward-with-real-past mechanism**, and the persistent-binding rewrite becomes **stage 2b-impl** (dense `[N,…]` bindings, guarded by the byte-identical batch-1 baseline in §1c and the recapture count in §3). ### What landed - `NativeDecodeSession::run_fused_batch_forward(token_ids, past_len)` — the stage 2a fused forward generalized to a non-empty length-L batched KV past; `run_fused_batch_prefill` now delegates with `past_len=0` (**byte-identical**). Still **stateless**: never touches `self.past`, `current_len`, or the CUDA persistent bindings. - `make_past_input_tensor_batched` (existing `make_empty_input_tensor_batched` delegates with `past_len=0`, byte-identical). - `profile_native --fused-forward-kv-sweep N@L,…` — resets the offload counters around each `N@L`, per-run contention check (#851), leads with the deterministic per-token counters. ### Batch-1 byte-identical — confirmed - **Structural:** `cuda.rs` is **0 lines changed** (`git diff --stat`: only `mod.rs`, `tensor.rs`, `profile_native.rs`, `fused_batch_prefill.rs`), so the batch-1 decode path is untouched. - **Test:** `native_fused_batch_forward_with_past_row_identical_to_batch_one` — row `i` of a batch-N forward at past_len L is **bit-for-bit** (`f32::to_bits`) equal to a batch-1 forward of the same token at the same L. `native_fused_batch_forward_past_zero_matches_prefill` — `run_fused_batch_forward(_, 0)` byte-identical to `run_fused_batch_prefill`. Plus the existing stage-2a guards. - **Empirical:** the batch-1 fused-forward counters (§1a/1b) and decode baseline (§1c) reproduce #884/#837 exactly. ### Gates / validation - `cargo fmt -p onnx-genai-engine -p onnx-genai-bench -- --check` ✅ - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✅ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✅ - `cargo clippy -p onnx-genai-bench --features bench-native,cuda --bin profile_native -- -D warnings` ✅ - `cargo test -p onnx-genai-engine --features native-backend` ✅ (472 + suites, 0 failed) - `cargo test -p onnx-runtime-session` ✅ (0 failed) - `cargo test -p onnx-genai-bench --features bench-native --test fused_batch_prefill` ✅ (6 guards incl. non-empty-past byte-identical) - **Mobius CUDA parity gate: N/A.** It gates memory/workspace-governance changes; `cuda.rs`, `kv_commit.rs`, and the memory governor are **0 lines changed** here (same reasoning accepted for #884). The stateful persistent batched-KV bindings of stage 2b-impl **will** require it. Do not wait for CI; validated locally. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

## Investigation of #888 — which explanation, with evidence **Verdict: explanation (1).** Decode correctness does **not** depend on eviction order. The #886 byte-aware corruption is caused by its **retain-vs-bypass flip**, not by reordering evictions. #886's speculation that "the scan-resistant recency discipline is load-bearing for correctness" is **refuted** — recency is load-bearing for *hit rate* only. The defect is in the *rejected* change, not the shipped path. This is an investigation PR. It changes **no shipped/default behaviour**; every knob added is default-OFF and byte-identical on the default size-blind path (mirroring how #886 kept byte-aware wired for reviewability). **Do not close #888.** Do not merge without a squad review. ### The discriminating experiment A residency policy changes two independent things: *which* weights are resident, and *in what order they evict*. Byte-aware changes **both** at once — it starts **retaining** large tensors the shipped path streams transiently, **and** it evicts smallest-first. To separate them, `ONNX_GENAI_WEIGHT_OFFLOAD_EVICT_ORDER` changes only the eviction **victim** while keeping the shipped always-bypass decision. Runs: qwen14b-zp, `ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`, `--tokens 16 --steady`, each arm solo with `nvidia-smi` verified clear beforehand. Baseline tokens `[96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]`. | arm | tokens | byte_hit_rate | evictions | page_ins | verdict | |---|---|---|---|---|---| | baseline (LRU, byte-aware OFF) | **16 ✓** | 70.16% | 2672 | 3008 | reference | | evict order = **MRU** (reverse recency) | **16 ✓ identical** | 79.15% | 1952 | 2160 | clean | | evict order = **Smallest** (byte-aware's exact victim) | **16 ✓ identical** | 71.77% | **10192** | — | clean | | byte-aware ON (graph ON) | **3 ✗** early EOS | — | — | >0 | corrupt | | byte-aware ON (graph OFF) | **3 ✗** early EOS | — | — | >0 | corrupt | Two independent, still-correct eviction *orders* — including byte-aware's **exact** smallest-first victim under extreme churn (10,192 evictions, hit rate driven down to ~22% in a harsher variant) — stay **byte-identical** with clean ledgers. **If any non-recency order corrupted, that would be evidence for (2); none does.** Changing eviction order alone is value-neutral. The corruption tracks *only* the retain flip. ### Mechanism — what it is NOT - **Not captured-VA baking** — graph-OFF corrupts too. - **Not a copy/compute fence / WAR hazard** — `ONNX_GENAI_WEIGHT_OFFLOAD_SYNC_BEFORE_FILL=1` (full drain of both compute and copy streams before every page-in fill) does **not** fix it. (The `htod_async_elapsed_ms` fill already host-syncs every page-in — the class of confusion flagged in #837/#851 — so fences are not the issue.) - **One concrete secondary bug found and confirmed to occur:** a *slotted* key that later bypasses gets `stable_slot = true` (Drop decommits-but-keeps-VA) yet is never re-inserted into `pages` (observed: key=20, 4.4 MB). `ONNX_GENAI_WEIGHT_OFFLOAD_RETAIN_SLOTTED=1` closes it — but closing it does **not** stop the corruption. So it is real-but-**secondary**; unreachable on the shipped size-blind path. ### Honest boundary The exact value-corruption *line* is not isolated. The primary path is byte-aware retaining/re-admitting **large stable-slot tensors served as hits** (no re-fill) across steps — most plausibly granule-ownership under multi-granule decommit + reused-slot recommit. Settling it needs **granule-level checksums across steps**; `compute-sanitizer` is unavailable here (CUDA runtime via pip wheels, no toolkit), which is the honest limit of what counters + targeted sync + code reading can prove. What *is* proven is the discrimination above: the shipped path's correctness is order-independent. ### Impact on #864 (and #866 / #750) The shipped size-blind path is **safe** — it never retains large tensors, so the buggy path is unreachable, and the probe confirms it stays byte-identical under reorder + heavy churn. Therefore **#864 is not blocked by an eviction-order invariant.** A hybrid that pins a **static** hot set (retain once, never evict/re-admit; zero-copy the cold remainder — exactly #864's stated shape) does not exercise the corrupting retain-then-churn path. The hazard is specifically **evicting and re-admitting large stable-slot residents**; any dynamic scheme that churns large weight pages (byte-aware; potentially #866 elastic reclaim, #750 admission) must validate token identity and should prefer a pinned non-churning hot set. ### What changed (all default-OFF, byte-identical on default path) - `weight_paging.rs`: `EvictOrderProbe` (lru|mru|smallest|largest) + env parse + victim selection wiring; `SYNC_BEFORE_FILL` and `RETAIN_SLOTTED` diagnostic knobs; one-shot slotted-bypass diagnostic; updated `WEIGHT_OFFLOAD_BYTE_AWARE_ENV` doc with the #888 resolution; 1 unit test. - `lib.rs`/`provider.rs`/`memory_strategy.rs`: export + wire the probe from env. - `profile_native.rs`: env-echo at start of `main`. - `.squad/decisions/inbox/copilot-eviction-order-correctness.md`: decision note for #864/#866/#750 owners. ### Gates fmt (per-package `--check`), clippy `-p onnx-runtime-ep-cuda --features cuda -D warnings`, clippy `--locked --all-targets -p onnx-genai-engine --features native-backend -D warnings`, `weight_paging::tests` (all green incl. new test), and **`mobius_seqmajor_growth_parity_native_cuda --ignored --test-threads=1` run SOLO with GPU clear → GREEN**. Final baseline re-run with the final binary is byte-identical with `captures>0`, `fallbacks==0`, `oversubscribed_bytes==0`, `peak_committed_physical_bytes < managed_limit_bytes`, and `ref_underflows/byte_underflows/unaccounted_committed_bytes == 0`. Refs #888. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

…ne changes it (#896) Four measured findings about the CUDA weight-residency policy lived only in issue threads and PR bodies. Two of them cost a full investigation each, and all four are the kind a future change would otherwise rediscover the hard way. Adding them to §3.5, where someone editing admission or eviction will actually see them. **1. Decode correctness does *not* depend on eviction order (#888/#892).** Default LRU, MRU (reverse recency) and smallest-first all produce **byte-identical tokens** — the last driven to 10,816 page-ins with the count hit rate at 22%. The earlier speculation that *"the scan-resistant recency discipline is load-bearing for correctness"* is **refuted**; recency is load-bearing for hit rate only. **2. But evicting and *re-admitting* large stable-slot residents does corrupt output.** A byte-aware policy (#886) silently collapsed greedy generation 16 → 3 tokens whenever offload actually engaged, with **and without** graph capture. The corruption tracks the **retain flip**, not the reorder — isolated by a probe that changes only the eviction victim while keeping the shipped always-bypass decision. Not captured-VA baking (graph-OFF corrupts too), not a fence (a full drain of both streams before every fill does not fix it). The exact value-corruption line is **not** isolated, and the doc says so along with what would settle it. The design consequence is the part worth having written down: a **static** hot set — retain once, never evict, never re-admit — provably does not exercise the corrupting path, which is why #864's hybrid is safe *in that shape*. Any **dynamic** scheme that churns large weight pages (#866's elastic reclaim, #750 admission, any future byte-aware attempt) must **assert token identity rather than assume it**. **3. The residency gap is an *admission* problem, not an eviction problem (#837/#886).** ~90% of the recoverable 1.158 GB/step is bypass traffic — 11% of page-in events carrying 44.6% of streamed bytes — because `StableResident` bypasses on `!can_fit(bytes)`, which is arrival-order first-fit. Small tensors (avg 7.8 MB) win the free budget; large projections (avg 49.9 MB) then fail `can_fit` on every subsequent step and re-stream forever, **never entering the eviction population at all**. That is the mechanical reason a reorder cannot close the gap. **4. Judge policy on `byte_hit_rate`.** The count-based rate once moved 57.09% → 81.31% while the gap to the streaming floor **widened** 1.78× → 2.30×. Also records **#893** (MRU measured better than the shipped LRU: page-ins 3,008 → 2,160, `byte_hit_rate` 70.16% → 79.15%, byte-identical) as **open and explicitly not acted on** — one model, one budget, and favourable, which is the shape of the results this project has had to retract twice. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

…ch-1 baseline (#905) ## #750 stage 2b-impl — scope + on-hardware batch-1 baseline **This is a well-evidenced scoping result, not the full rewrite.** After reading the persistent decode path end-to-end and measuring the batch-1 baseline on hardware, the honest conclusion is: **the smallest *correct* (token-producing) batch-N persistent-binding decode step does not fit in one reviewable PR.** This PR lands the split and the regression anchor that makes each following increment safe to attempt — the same deliverable shape you accepted for stage 1 (#844) and stage 2b (#891), and explicitly invited ("bindings first, mask second… take it"; "a truthful negative result is acceptable and valuable"). New doc: `docs/NATIVE_BATCH_DECODE_2B_IMPL_SCOPING.md` (+ a pointer from `docs/NATIVE_CUDA_DECODE.md`). **No runtime code changed** — so batch-1 is structurally byte-identical and the #796 capture gates are untouched. ### Batch-1 baseline — measured solo (`nvidia-smi` verified empty before the run) `qwen14b-zp`, RTX 4060, `ONNX_GENAI_CUDA_GRAPH=1`, `ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`, 16 tokens, current `main`: | metric | value | |---|---:| | `generated_token_ids` | `[96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]` — **exact** reference match ✓ | | `htod_bytes_per_token` | 2,349,010,944 | | `page_ins_per_token` | 188.0 | | `hit_rate` | 78.32% | | `cuda_graph_measured` | `captures=1 replays=14 fallbacks=0 invalidations=1` → **recapture ≈ 1** (confirms #891) | | `peak_committed_physical_bytes` | 7,725,907,968 `< managed_limit 7,726,753,178` ✓ | | `oversubscribed_bytes` / `ref_underflows` / `byte_underflows` / `unaccounted_committed_bytes` | 0 / 0 / 0 / 0 ✓ | Not headlining wall-clock (`1.72 GB/s`, `23.8 s/token`) — WDDM-paging noise (#863). Per-token bytes differ from #844's 4.11 GB because current `main` runs the elastic-budget (#866) managed path at the 7.73 GB auto budget, not the earlier fixed 4 GiB synthetic; this is the correct current-main anchor. ### Why one PR isn't achievable - The stage 2a/2b guards exercise the **stateless** `session.run` fused-forward (`run_fused_batch_forward`), which already takes `[N,…]`. The path 2b-impl must change is `DecodeCudaState` — **GPU-only, no CPU test seam**. - **No partial guard exists.** A token-producing decode needs *all* of: batched input binding, KV allocation, KV growth/commit, mask, logits read, greedy argmax, and capture-symbol pinning. Drop one → hard-fail or silent row corruption (#892). So increments are validated by batch-1 non-regression until the final one flips batch-N on with a row-identity guard. - The batch-aware **caller** (present N sequences, consume N rows) is the generation driver / `continuous_batch_manager` = **2c, out of scope**. ### Key finding The KV growth **stride** math (`kv_growth_byte_layout`, `copy_kv_prefix_device_to_device*`) is **already batch-general** (block-count based). The real blockers are shape allocation (`persistent_state_shapes` pins axis 0→1), the mask, the input/logits bindings, the **device argmax** (in `onnx-runtime-ep-cuda`, separate clippy gate), the seq-major commit scatter (`live_prefix_ranges` in `onnx-genai-kv`), and **capture-symbol pinning** (`collect_unit_symbols` collapses the batch symbol to 1; it must instead pin to N). Full file/line inventory in the doc. ### Proposed split (each holds batch-1 byte-identical + passes mobius solo) 1. **2b-impl-1** shapes & IO bindings (thread `batch`, constructor-fixed to 1). 2. **2b-impl-2** batched KV growth/commit (+ pure range-math unit tests). 3. **2b-impl-3** batched device argmax (`onnx-runtime-ep-cuda`). 4. **2b-impl-4** batch symbol pinning + entry points + batch-N row-identity guard + measured recapture-per-shape. ### KV-multiplication trade (restated vs #891) `htod/token ∝ 1/N` but with a never-amortizing floor `L × kv_bytes_per_token` and `N_max ≈ 19 @ 2048 ctx`. Under #866's elastic budget, batched KV growth reclaims weight residency, so 2b-impl-4 must report both the 1/N fall **and** the `hit_rate`-driven offset, with a crossover if batched decode ever loses at a reachable N. ### Validation `cargo fmt`/`clippy`/tests are unaffected (docs-only). The baseline was produced by the protocol's own `profile_native` build (`--features bench-native,cuda`), run solo. Not merging; #750 stays open. Refs #750 #844 #884 #891 #796 #798 #866 #892. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

…he host-mapped cold recommendation (#915) Review pass on #907 before it goes to the wider ORT stack, with the disagreements applied directly rather than left as comments. The structure holds — contracts/invariants/flows is the right shape, the responsibility boundaries are clean, and the appendix's evidence table is the single best thing in the document because it lets a reader tell measurement from intent. My changes are where the document is either **missing a decision** or **recommending something we have since measured to fail**. ## The one I disagree with most The appendix told readers to prefer *"a resident hot set plus host-mapped cold, single-touch weights over copy-map-evict churn."* We built exactly that this week, and it lost twice (#912): - **Correctness.** Above ~0.44–0.65 GB of *distinct* host-mapped bytes read per decode step, the device silently returns stale data — generation collapsed 16 tokens to 3 with no error raised. A single host-mapped read is bit-identical at 1/8/16/32 cold weights, and a copy-instead A/B isolates the fault to the read, so it is an aggregate aperture ceiling, not a per-read bug. - **Throughput.** Even capped at a provably safe budget, 0.73 tok/s against WDDM demand paging's 7.84 on the same model in the same session. The safe aperture (0.26 GB/step) is smaller than the traffic it must displace (~0.6 GB/step) — too short by construction, not by tuning. Shipping a design doc that recommends this to other teams would have cost each of them the same week. Replaced with the narrower claim the evidence supports, and I kept what survived: the mechanism works and capture replays bit-identically, so this is a capacity verdict about one class of GPU, not a dead technique. Datacenter parts should be re-measured, not assumed to inherit it in either direction. ## The question the document never asked Every contract describes *how* to manage memory. None asks **whether managing a given class of bytes beats leaving it to the OS** — and the honest answer is sometimes it does not. That is not a caveat; it decides when the rest of the design pays for itself. The rule: **copying into device memory only pays when the data is re-read from device memory before eviction.** A weight read exactly once per decode step has no reuse to amortize, so we pay the same PCIe bytes as the OS *plus* a host copy, an allocation, a mapping, an eviction and a sync. Measured 30× slower than WDDM demand paging (#864); deferring recovered ~100× end-to-end (#874). We were not losing to a better algorithm — we were losing to not being in the path. Added as a decision table plus two obligations that are contracts, not advice: **a governor must be able to decline and report it** (that is what I3 exists for), and **the unmanaged path must stay runnable as a control**, since an A/B arm is the only thing that can tell you the control plane costs more than it saves. It also connects batching to memory management properly — batch-N is what *creates* the reuse that makes managing pay, which is why they are one lever and not two features (#884/#891, ceiling `N_max ≈ 19 @ 2048 ctx`). ## Corrections from measurements this design would otherwise repeat - **C7** must report host-mapped read capacity and resident-vs-host bandwidth — a tiering decision needs both and can infer neither. A platform that cannot report the first is treated as **zero**, not unbounded. - **I8b (new): an authority accounts charge, not residency.** On WDDM the two diverge with no notification (#863). A design that treats the ledger as physical truth is correct on TCC and quietly wrong on every consumer Windows machine. C8's numbers are now labelled accordingly, and residency claims must name their measurement source. - **I8c (new): reservations are elastic against occupancy.** A load-time `bytes_per_token × max_context` split is charged in full from the first token; making it elastic cut streamed bytes 1.68× with the max-context guarantee tested through the production reclaim path (#857/#866). - **I8**: bytes are *referenced extents*, not file or blob size — that error overstated one model by exactly 2.00× and made every derived budget wrong (#853/#856). - **C8**: policy rates must be byte-weighted. A count-based hit rate rose 57.09% → 81.31% while the byte gap *widened* 1.78× → 2.30× (#869). - **Risk table**: do not spend the dynamic-lending fix on eviction policy. Order is ruled out as the cause (#892) and bounded at ~10% of the recoverable gap (#901); ~90% is **admission** — 11% of events but 44.6% of streamed bytes. Without this a reader would reasonably spend a week on the wrong lever. ## What I deliberately did not change The Foundry/cluster layering, the transaction ordering rules, and the PagedAttention-as-a-view position all read correctly to me and I have no measurement that contradicts them. The `ClusterCoordinator` → `ServingMemoryCoordinator` split in "Relationship to existing design" is the right call. Every issue and PR number cited was verified to exist **and** to match the figure attributed to it. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

…missing edge in the diagram (#917) Two problems with taking #907 to the ORT team as written. ## The motivation was written for us, not for them It named **no ORT mechanism**, so an ORT reader could reasonably conclude this is a GenAI-side concern that does not touch their code. And it never answered the first objection anyone who knows ORT's memory model will raise: *these are configuration knobs — set `gpu_mem_limit`, choose an arena extend strategy, size `OrtArenaCfg`, share allocators through `OrtEnv`.* Rewritten around that objection, because it is a fair one and the answer is the whole point of the design: **ORT gives each pool a bound; it gives no holder a way to lend to another, and no way to reclaim from a holder that is over-provisioned right now.** It now names the concrete mechanisms — BFC high-water behaviour and precisely where end-of-`Run` shrinkage does and does not help, custom device allocators not being BFC-wrapped (so the governed path and the arena path are two implementations today, not one policy), and `RegisteredAllocator` leaking because ORT does not expose the last session user. Also puts **three measurements up front**. An architecture proposal with no evidence of harm reads as speculation, and we now have the harm: - a load-time `device_budget − kv_bytes_per_token × max_context` split that stranded **1.611 GB** from the first token; making it elastic cut streamed bytes **1.68×** (#857/#866) - weight bytes measured from the external-data file rather than referenced extents — wrong by exactly **2.00×**, so every derived budget was wrong (#853/#856) - managing single-touch weights measured **30× slower** than letting the OS demand-page them; deferring recovered ~**100×** (#864/#874) The first two are accounting failures that a single authority prevents *by construction*, which is the argument for the design. The third is why it also has to say when **not** to manage. ## The diagram had a real omission The appendix assigns **transient-peak reporting** to `InferenceSession`/EP, and I8 requires admission to derive exact bytes *including transient peaks* — but there was **no edge from `Session` to `ProcessMemoryManager` at all**. That is the principal ORT-side integration point, so leaving it out made the ORT-side work look smaller than it is and hid the one interface ORT actually has to add. Added the edge, added the matching entry to *Required architecture changes*, and stated explicitly that the session is a **reporter, not a holder**, so the new edge is not misread as an ownership change. Also split control-plane (solid) from data-plane (dotted) edges, since only the control plane is new work and a reviewer should be able to see that without reading the prose. ## One overclaim of my own, corrected I first wrote that the BFC arena "does not return memory". That is **wrong** — ORT supports arena shrinkage at a `Run` boundary, and an ORT reviewer would have caught it immediately and discounted everything around it. The accurate criticism is narrower and still holds: shrinkage is **caller-scheduled at a `Run` boundary, not demand-driven**, so it cannot answer pressure that arrives mid-generation. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

… so capture and streaming coexist (2b-impl-4, #750) Correction to the 2b-impl-4 measurement harness. The first sweep loaded the native decode session via `load_with_resolved_io`, which passes `cuda_offload_policy: None`; `load_with_cuda_options_and_io` then leaves `weight_offload_stable_va` at its conservative `false` default (`load.rs:793`), so the CUDA-graph capture predicate `weight_offload_enabled && !weight_offload_stable_va` declined capture by construction whenever offload was on. The reported "capture and weight streaming are mutually exclusive" was therefore a harness artifact of the entry point, not a runtime property. Fix: the sweep now builds the fully governed `Engine::from_dir` and drives `decode_greedy_batch` on `Engine::native_decode_session_mut()`. That path resolves the effective offload policy from the memory-strategy plan (managed no-spill + stable-VA authority), so a captured graph that baked a weight pointer stays valid across evict->repage on the reserved-once VMM arena (#716/#796/#836). Measured on qwen14b-zp with managed streaming forced: capture stays enabled (captures=1, fallbacks=0, invalidations=0) while weights stream (htod ~= 3.0 GB/token at N=1), and the 1/N amortization and the within-sweep #866 byte-hit-rate offset now come from the real captured decode path. - Add `Engine::native_decode_session_mut()` (native-backend) so a harness can drive the governed session's batch-N entry points. - Rewire `run_native_decode_batch_sweep` to construct the governed engine per batch shape instead of the bare loader; drop the ad-hoc `set_weight_residency_budget` override (the governor now owns the budget). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

…n batch-N on (2b-impl-4, #750) (#930) ## What this is Increment **4 of 4** of native batch-N decode (issue #750), stage 2b-impl-4: pin the batch symbol, wire the token writer, turn batch-N on. Increments 1-3 (#913, #920, #924) were byte-identical by construction with `batch` constructor-fixed to `1`. **This is the increment that changes behaviour**, so it is measured rather than asserted. **#750 stays open** for the 2c caller. > **Correction (2nd commit).** The first version of this PR reported that > "capture and weight-streaming are mutually exclusive on this build." **That > was a harness artifact, not a runtime property**, and it has been corrected > in place. The sweep loaded the native session via `load_with_resolved_io`, > which passes `cuda_offload_policy: None`; `load_with_cuda_options_and_io` > then leaves `weight_offload_stable_va` at its conservative `false` default > (`native_decode/load.rs:793`), so the capture predicate > `weight_offload_enabled && !weight_offload_stable_va` declined capture *by > construction* whenever offload was on. The knob I set (the entry point) set > the flag that produced the decline that I read as a finding. The harness now > drives the **fully governed** `Engine` session, whose effective offload > policy carries the managed no-spill stable-VA authority — and **capture and > streaming coexist**, exactly as #796/#836 built and verified. All the > capture/streaming numbers below are re-measured on that governed path. ## What is on and under what flag - **`ONNX_GENAI_NATIVE_DECODE_BATCH`** (default `1`). At the default the decode path is **byte-identical to `main`**. Setting `N>1` binds a stable batch extent at session construction and drives N rows through the persistent captured decode path. - Batch symbol pinned for aux outputs via `persistent_output_shape` (axis-0 → `batch`); KV seq-axis pinning unchanged. - Token writer wired: `write_decode_inputs_batch` (N token IDs → N slots), `write_position_binding_batch` (rank-R mrope per sequence), `extend_mask` (N mask rows). Single-token entry points delegate to the batch writers (byte-identical at batch 1). - `decode_cuda_greedy_batch` / `NativeDecodeSession::decode_greedy_batch` is the batch-N greedy entry point; `native_decode_batch()` exposes the extent. - `Engine::native_decode_session_mut()` (new, native-backend) lets the measurement harness drive batch-N decode against the governed session. - `profile_native --native-decode-batch-sweep <list> [--native-decode-batch-context L]` builds the governed `Engine` per batch shape and exercises the captured decode path. **The batch-N generation driver (real per-sequence prompt seeding) is stage 2c and out of scope here** — the harness fans one identical token to all N rows, which is a strict correctness probe and, because the weight/page/capture counters are content-invariant (#884), the trustworthy measurement vehicle. ## Mandatory gates — all re-run locally after the correction | Gate | Result | |------|--------| | `cargo fmt -p onnx-genai-engine -- --check` (+ `-p onnx-genai-bench`) | **pass** | | `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` | **exit 0** | | `cargo test -p onnx-genai-engine --features native-backend,cuda --lib native_decode` | **96 passed, 0 failed** | | **mobius parity** `--test mobius_seqmajor_growth_parity_native_cuda -- --ignored --nocapture --test-threads=1` | **2 passed** (SOLO) | | **batch-1 byte-identity on hardware** | **exact match** (below) | ### Batch-1 byte-identity with counters Reference token IDs matched **exactly** with the feature compiled in and batch configured to 1 (`qwen14b-zp`, `--steady --tokens 16`): ``` [96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752] ``` Counters matched the documented baseline: `captures=2`, `replays=28`, `fallbacks=0`, `htod_bytes_per_token=2349010944`, `page_ins_per_token=188`, `hit_rate=78.32%`, `byte_hit_rate=70.16%`. ## Correctness of batch-N (row identity) `qwen14b-zp`, all N, every step, on the governed captured+streaming arm: **all N rows == row 0** (the #892 cross-row-corruption detector) and **row 0 == the batch-1 stream** at N=2,4,8 (the scoping-doc §3.8 guard). Token *values* at N>1 are content-arbitrary (real per-sequence seeding is 2c); the counters are not. ## Recapture per shape (#891) — **measured, capture survives batch-N WHILE streaming** `qwen14b-zp`, governed engine, managed streaming forced (`ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`, `ONNX_GENAI_CUDA_GRAPH=1`), empty past, measured window, per batch shape. **Non-zero htod confirms weights are actually streaming during these captured steps:** | N | graph enabled | captures | replays | fallbacks | invalidations | htod_bytes_per_token | |---|---------------|----------|---------|-----------|---------------|----------------------| | 1 | true | 1 | 6 | 0 | 0 | 2,994,222,848 | | 2 | true | 1 | 6 | 0 | 0 | 1,502,098,688 | | 4 | true | 1 | 6 | 0 | 0 | 736,831,040 | | 8 | true | 1 | 6 | 0 | 0 | 424,642,624 | **Capture survives batch-N as implemented, on the real streaming decode path** — one capture per shape, zero invalidations, zero fallbacks at every N, with ~3 GB/token of weights streaming underneath. This is the positive resolution of #891's open risk the line has been waiting for, and it is the part the original PR got wrong by measuring the wrong arm. ## 1/N weight-stream amortization on the captured decode path — **measured** Same governed captured+streaming arm as above (managed path, #874 note: the WDDM default is the OS shared-memory path with htod=0, so managed streaming is forced). Model `qwen14b-zp` (8.3 GB weights), empty past, `--tokens 8`. Emitted tokens = steps × N. | N | htod_bytes_per_token | page_ins_per_token | abs page_ins/forward | evictions | byte_hit_rate | |---|----------------------|--------------------|----------------------|-----------|---------------| | 1 | 2,994,222,848 | 276.8 | 2214 | 1273 | 61.96% | | 2 | 1,502,098,688 | 139.9 | 2239 | 1293 | 61.83% | | 4 | 736,831,040 | 66.6 | 2132 | 1185 | 62.55% | | 8 | 424,642,624 | 41.6 | 2662 | 1678 | 56.84% | `htod_bytes_per_token` falls **~1/N** (2.99 GB → 1.50 → 0.74 → 0.42; a 7.05× reduction N=1→8 against the ideal 8×). Amortization vs the N=1 ideal extrapolation: N=2 **+0.3%**, N=4 **−1.6%**, N=8 **+13.5%**. The N=8 shortfall is not noise — it is the #866 offset (next section). All four shapes ran SOLO with `nvidia-smi` verified empty; no contention flags. **The amortization is approximate here, not byte-exact.** On this governed managed arm the evictor's working set and page order vary per session load, so the per-forward `abs page_ins` drifts (2214 / 2239 / 2132 / 2662) rather than staying identical. That is the honest behaviour of the *captured streaming* path. For contrast, the earlier *eager* offload arm (offload on, capture declined, fixed `DEVICE_BYTES=6e9`) gave **byte-exact** 1/N with an identical `abs page_ins=1791` at every N — because a fixed budget with no capture is fully deterministic. Both arms agree that the weight stream is paged once per forward and the per-produced-token cost falls ~1/N; the captured arm is the one that matters for #750 and it now measures cleanly. Per the task instruction, **no throughput delta is reported as a finding** (this box ranges 3.9–28 tok/s across identical configs). The deterministic counters above are the evidence. ## The #866 offset — **measured within-sweep on the governed arm** #866 makes the weight budget elastic against occupancy, so batch-N (N× the working set) should shrink the weight residency and push `byte_hit_rate` down. The governed captured arm shows exactly this **within a single sweep**: - N = 1, 2, 4: `byte_hit_rate` ≈ **62%** (61.96 / 61.83 / 62.55, flat within run-to-run noise), evictions ≈ 1200–1300. - N = 8: `byte_hit_rate` drops to **56.84% — a −5.12-point offset**, evictions rise to 1678, and `abs page_ins/forward` rise 2214 → 2662. That offset is precisely why the N=8 per-token htod runs **+13.5% above** the ideal 1/N line: the N× working set ate into the elastic weight budget, so each forward had to re-stream more. This is the #866 offset the line predicted, measured on the real decode path rather than inferred. (The earlier PR could only show it as an *arm-to-arm* budget difference because the un-governed sweep never let the elastic budget respond to batch; the governed engine does.) ## Honest summary of what is and isn't proven - **Proven:** batch symbol pinned; token/mask/position writer wired; batch-N runs end-to-end N=1,2,4,8; batch-1 byte-identity; row identity and cross-batch identity; **capture survives batch-N *while weights stream*** (captures=1, invalidations=0, htod≈3 GB/token) on the governed engine path; ~1/N weight-stream amortization on the captured decode path; the within- sweep #866 byte-hit-rate offset (−5.12 pts at N=8). - **Corrected:** the original "capture and streaming are mutually exclusive" claim was a harness artifact of `load_with_resolved_io`'s conservative `weight_offload_stable_va=false` default; the governed path proves they coexist. - **Named limitation (in the spirit of #920's seq-major `bail!`):** on the governed managed-streaming arm the 1/N amortization is *approximate*, not byte-exact, because the evictor's per-load working set varies and the #866 offset grows the stream at N=8. This is a property of managed streaming, not a batch-N defect. - **Out of scope (2c):** real per-sequence prompt seeding / the batch-N generation driver. The harness fans one identical token to all rows, so token *values* at N>1 are content-arbitrary; the counters are content- invariant (#884) and are the measurement. **Do not merge** — leaving #750 open for the 2c caller. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

…mpl-4 (#939) docs(progress): record the #750 batch-N line as complete through 2b-impl-4 The four-way split (#905) is done: #913, #920, #924, #930 all landed. The entry still described increments 2-4 as remaining and carried none of what they found. The headline is the result the line was started for, reproduced independently rather than taken from the PR: capture survives batch-N while weights stream. Solo, nvidia-smi verified empty, managed streaming forced, qwen14b-zp at context 64 -- N=2 and N=4 both show enabled=true captures=1 invalidations=0 fallbacks=0 with htod_bytes_per_token 1,234,271,232 and 547,528,704, byte_hit_rate 68.64% -> 72.17%. captures=0 replays=8 inside the measured window is capture-once/replay-many working as intended. Also records what each increment actually contributed, because three of them returned a correction to my scoping rather than the scoping: - #920 drew the distinction the scoping had missed -- a fixed full-context stride commit is batch-general in both layouts, a growing bucket only in head-major -- and refuses seq-major growing-bucket growth at batch>1 with a named bail! rather than computing a capacity-dependent stride that would have passed every batch-1 test and corrupted the first batch-N run. - #924 proved batch-1 tie-breaking unchanged two ways, including that argmax_update is a total order so the reduction cannot depend on tree shape. And it records the near-miss, because that is the part worth remembering. #930's first version reported capture and weight streaming as mutually exclusive, from a real decline message naming a real predicate. The sweep harness called load_with_resolved_io, which passes cuda_offload_policy: None, so stable_va fell to its conservative unwrap_or(false) and capture declined by construction. The harness asserted pointer-instability; the runtime was never asked. That would have closed the most valuable open question on this line with a convincing wrong answer. Finally, the #866 offset is now measured within-sweep rather than inferred: byte_hit_rate flat ~62% at N=1,2,4 then -5.12 points at N=8 as evictions go 1273 -> 1678, which is why N=8 sits +13.5% above the ideal 1/N line. Batch-N multiplies KV occupancy by N and eats the elastic weight budget -- the mechanism #866 predicted, now observed rather than argued. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

…truction Answer the follow-up: is the shipped size-blind path safe by construction, or by an accident of arrival order and this card's VRAM ceiling? Measured on qwen14b-zp (RTX 4060 Laptop, 8 GB): - Raising ONNX_GENAI_WEIGHT_OFFLOAD_DEVICE_BYTES to 6.5 GB leaves 919 bypassing (retained_page_ins=0), baseline byte-identical. Raising the physical ceiling toward holding the full 8.33 GB weight set OOMs at load, so natural clean-fit retention of 919 is physically unreachable on this card. The default's safety here rests on the VRAM ceiling, not the policy. - A non-pin residency policy (ONNX_GENAI_WEIGHT_OFFLOAD_BYTE_AWARE=1) that retains 919 by evicting smaller residents reproduces the identical cross-generation collapse (run 1 byte-identical, run 2 diverges). So the corruption is not a PIN_KEYS instrument artifact: a policy that retains 919 corrupts too. On a bigger card / smaller model / with #866 elastic budget, the shipped default could retain 919 with no knob set and corrupt silently. Also eliminate two fix hypotheses. Both the #888 fill-time drain (ONNX_GENAI_WEIGHT_OFFLOAD_SYNC_BEFORE_FILL=1) and a new reused-slot-bypass retain control (ONNX_GENAI_WEIGHT_SLOT_BYPASS_RETAIN=1, default OFF, added here) leave the corruption intact under both PIN_KEYS=919 and byte-aware. The #888 stable_slot/residency disagreement fires on every corrupting run but is a symptom, not the fix site: the damaging event is the earlier eviction of a slotted key's physical granules back to the shared pool, not the re-admission bookkeeping or the fill fence. Exact line still not isolated (no compute-sanitizer/nvcc on this box; NVRTC only). New knob is env-gated and default OFF; shipped path byte-identical ([96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]). Gates: fmt, clippy -D warnings, ep-cuda lib 374 passed, engine native_decode 96 passed, mobius parity 2 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

justinchuby mentioned this pull request Aug 13, 2026

WDDM pages out our VMM (cuMemCreate/cuMemMap) granules to host RAM under over-commit — #798 no-spill is an accounting guarantee, not physical residency, on WDDM #863

Open

justinchuby merged commit b3d6958 into main Aug 13, 2026
13 of 15 checks passed

justinchuby deleted the squad/elastic-kv-weight-budget branch August 13, 2026 04:53

This was referenced Aug 13, 2026

Large-model decode is streaming-bound: 0.24 tok/s on qwen14b, with 57k VMM map/unmap per 16 tokens and ~4.9 GB/s effective H2D #837

Closed

docs(progress): record the over-budget/streaming line and its open items #879

Merged

justinchuby mentioned this pull request Aug 13, 2026

Investigate #888: eviction-order correctness of weight offload #892

Merged

justinchuby mentioned this pull request Aug 13, 2026

docs(memory): record the measured residency-policy facts before someone changes it #896

Merged

This was referenced Aug 13, 2026

docs(#750): scope native batch-N decode step (2b-impl) + hardware batch-1 baseline #905

Merged

Native backend cannot batch: continuous batching is ORT-only, so --max-batch is silently ignored and every request decodes alone #750

Closed

justinchuby mentioned this pull request Aug 14, 2026

docs(memory-design): add the decide-whether-to-manage rule; retract the host-mapped cold recommendation #915

Merged

justinchuby mentioned this pull request Aug 14, 2026

docs(memory-design): write the motivation for an ORT reviewer; fix a missing edge in the diagram #917

Merged

justinchuby mentioned this pull request Aug 14, 2026

feat(native-decode): make the device argmax / greedy token selection batch-aware (2b-impl-3, #750) #924

Merged

justinchuby mentioned this pull request Aug 14, 2026

feat(native-decode): pin the batch symbol, wire the token writer, turn batch-N on (2b-impl-4, #750) #930

Merged

justinchuby mentioned this pull request Aug 14, 2026

docs(progress): record the #750 batch-N line as complete through 2b-impl-4 #939

Merged

Navigation Menu

Repository navigation

Make CUDA weight-residency budget elastic against KV occupancy (#857) - #866

Make CUDA weight-residency budget elastic against KV occupancy (#857)#866
justinchuby merged 2 commits into
mainfrom
squad/elastic-kv-weight-budget

justinchuby commented Aug 13, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

justinchuby commented Aug 13, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

justinchuby commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TL;DR (read this first — the framing changed after #863/#864)

What this does

Design (points 1–4)

Safety: WDDM headroom (question RESOLVED by #863)

Results — qwen14b-zp, ctx 8192, 16 tokens, greedy (byte-identical)

Reading the table honestly

max_context reachability (the property under risk)

Where the real leverage sits (per #864)

Validation (all local, per issue)

Files

Uh oh!

codecov Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

justinchuby commented Aug 13, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 13, 2026

🔴 Benchmark Regression Detected

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

justinchuby commented Aug 13, 2026 •

edited

Loading

codecov Bot commented Aug 13, 2026 •

edited

Loading