Repository navigation
feat(memory): prefer WDDM shared-memory over managed weight streaming by default (#864) - #874
Merged
Merged
Conversation
… by default (#864) On Windows/WDDM, auto-enabling managed weight streaming for an inferred over-budget model is ~30x slower than letting the OS page weights from "shared GPU memory" (host RAM) in place over PCIe. For the single-touch decode access pattern each weight is read exactly once per step, so copying it into VRAM only to evict it before any re-read is pure overhead (htod_bytes_per_token=3.94GB, 372 page-ins/token on qwen14b-zp). This is a policy change in memory_strategy.rs only - the residency cache, arena, and offload implementation are untouched. When inference would auto-enable weight streaming AND the platform provides the shared-memory fallback (cfg!(windows)) AND the operator did not explicitly request offload / a --vram-limit / a device-budget override AND the force knob is unset, the effective strategy becomes Compatibility with weight_offload_enabled=false and managed_no_spill=false (so the physical-handle pool does not cap the load below the weights and refuse them - the WDDM fallback must hold the over-budget remainder in host RAM). - Honored, not overridden: explicit ONNX_GENAI_WEIGHT_OFFLOAD=1, --vram-limit, or device-budget override still selects managed streaming. - New opt-in override ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1 forces the managed path; parsed so unrecognized values keep the fast fallback (deliberately not the ASYNC_PAGEIN opt-in-once-set trap shape). - Linux unchanged: no shared-memory fallback exists there, so managed streaming stays the only over-budget path (#783). - Loud startup warning + a serialized plan decision record the choice and the trade (OS-managed residency can thrash under low host RAM, #863). Measured A/B (qwen14b-zp, genuinely over budget, same binary/prompt, GPU clear before each run, token IDs byte-identical across all runs): WDDM default median 7.18 tok/s (n=4) vs forced managed 0.86 tok/s (n=3). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #874 +/- ##
==========================================
+ Coverage 77.70% 77.79% +0.08%
==========================================
Files 353 354 +1
Lines 147642 148003 +361
Branches 147642 148003 +361
==========================================
+ Hits 114727 115133 +406
+ Misses 28423 28378 -45
Partials 4492 4492
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
🔴 Benchmark Regression DetectedComparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).
Visual flags: Host infoWhat this cannot catch
|
Owner
Author
|
Validated end to end on hardware and merging. First, the thing I want to name: you reported 8.3× when my issue said 30.7×, and you were right to. Your explanation is correct — my numbers came from this box under its characteristic paging noise (#863), yours from a quiet box with On the judgement I delegated: your answer is correct and the reasoning is concrete. You cannot keep the no-spill managed arena while disabling weight offload, because the physical-handle pool is a hard cap at the resolved budget and Flagging the What I verified myselfGates, on a worktree with I also read Then on real hardware,
Three things that confirms, each of which was a stated requirement:
For context on this box: On the flagged failuresBoth check out. The Merging. #864 stays open for the hybrid (resident hot set + zero-copy cold reads), which is the durable direction; this PR takes the easy 8× now by no longer choosing the slow path by default. |
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…tructural cause (#874) (#876) Replaces the stale `27x slower than doing nothing` row with the measured result from #874 and, more importantly, the mechanism behind it. Measured on `qwen14b-zp`, solo with `nvidia-smi` verified empty before every run, byte-identical output: **WDDM 8.09/7.78 tok/s with `htod_bytes_per_token = 0`** (true zero-copy) against **managed streaming 0.11 tok/s**. `main` measured 0.05-0.08 tok/s on the same model minutes earlier, so the end-to-end effect on this box is ~100x. The row now carries the structural cause rather than a bare ratio: each weight is read **exactly once per decode step**, so both paths move the same bytes over the same link, but ours adds a CPU memcpy into pinned staging, a VRAM allocation, a `cuMemMap`, an eviction and a synchronize -- to buy VRAM residency discarded before it is ever re-read. **Copying to VRAM only pays off when the data is re-read from VRAM before eviction, and there is no intra-step reuse.** Also records three things a future reader needs and would otherwise have to rediscover: - **Keeping the no-spill arena while merely disabling offload is not an option.** The physical-handle pool is a hard cap and `cuMemCreate` cannot spill, so with nothing paging the over-budget remainder the load fails outright. - **The `Compatibility` arm has no device budget at all**, so on a host short of RAM an over-budget model can thrash and the governor cannot see system-wide pressure (#863). - **Linux is unchanged and must stay so** -- with no shared-memory fallback an over-budget model simply fails there, so managed streaming competes with 'does not run', not with something faster (#783's lesson about not inheriting a platform-specific conclusion). Supersedes the historical figures the row carried (6.01 tok/s WDDM, 27x behind, the h2d/staging/vram_free/vram_alloc split, capture disabled under offload): capture now runs under offload (#796) and the eviction-policy defect was fixed (#723). Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…877) Adds `scripts/zero_copy_host_map_probe.py`, which measures the two capabilities the #864 hybrid depends on — against the real weight blob, with no engine involvement. The hybrid is: keep the hot weight set resident in VRAM, and for the cold remainder do **not** copy — let the device read it in place from host memory, since each weight is read exactly once per decode step and a copy has no reuse to amortize. That rests on `cuMemHostRegister` over an existing read-only file mapping plus `cuMemHostGetDevicePointer`, which the tree does not currently use anywhere. **Measured** (RTX 4060 Laptop, 1 GiB slice of `qwen14b-zp`, best-of-3, CUDA-event timed, `can_map_host_memory = 1`): | path | bandwidth | |---|---:| | device reading mapped host memory (zero-copy) | **11.41 GB/s** | | explicit HtoD copy — what the managed path does today | 11.28 GB/s | | already-resident VRAM read — the ceiling | **110.46 GB/s** | `cuMemHostRegister` on a 1 GiB read-only mapping **succeeds in 3.1 ms**, so registration is cheap enough to do once at load over the whole blob rather than per page-in. **Two conclusions, and they reorder the hybrid's priorities — in the opposite direction to my assumption when I filed #864.** **Zero-copy is not a bandwidth win.** 11.41 vs 11.28 GB/s is 1.01×; PCIe limits both. What zero-copy removes is the copy's *second* pass and everything the copy drags along — the CPU memcpy into pinned staging, the VRAM allocation, the `cuMemMap`, the eviction, the synchronize. Those are exactly what #837 items 1–2 were separately trying to shave, so de-copying subsumes rather than competes with them. **The resident hot set is worth ~9.7×, and is the larger prize.** Every byte moved from the streamed fraction into the resident fraction gets ~10× faster; every byte moved from copied-stream to zero-copy-stream gets ~1× faster on bandwidth and only saves overhead. So the ordering should be: maximise residency first (#866 already moved the weight budget 6.12 → 7.14 GB and `htod_bytes_per_token` 3.94 → 2.35 GB), de-copy the cold remainder second. It also explains #874's result exactly: WDDM reaches 8.09 tok/s doing precisely this — zero-copy at PCIe rate, no copy machinery, `htod_bytes_per_token = 0`. We were paying the same PCIe pass *plus* the machinery. The hybrid beats WDDM only by holding a **larger and better-chosen** resident set than the driver picks blind, which is the one edge we have — and per #863 it is advisory rather than enforceable on WDDM. **Caveat, recorded in the docstring:** a `cuMemcpyDtoD` *from* mapped host memory measures the rate at which the device pulls those bytes across PCIe — the right proxy for a kernel reading in place, but not literally a strided GEMV. Treat the zero-copy figure as an upper bound on that arm. Also noted: registering all 8.33 GB of referenced weights page-locks that much host RAM (63.8 GB total / 34.4 GB free here), which is fine on this box but is a real claim on smaller hosts and needs a policy. Probe-only; no engine or runtime code touched. Verified it runs clean end to end (exit 0) after fixing the buffer-release ordering that `mmap.close()` requires. Refs #864 #837 #866 #874 #863. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…ems (#879) `PROGRESS.md` tracked the dispatch-bound work (models that **fit** device memory) but carried nothing about the other regime — models whose weights **do not** fit — despite that line producing the largest single change today. A reader looking for "what happened to large-model support" would have found nothing, while the answers were scattered across eight issues. ## Added milestone - **#874** — stop auto-enabling managed weight streaming on Windows/WDDM. `8.09 tok/s` with `htod_bytes_per_token = 0` against `0.11` managed; `main` immediately prior measured 0.05–0.08, so **~100× end to end**, byte-identical output. Records the *structural* cause rather than the ratio: each weight is read **exactly once per decode step**, so both paths move the same bytes over the same link, but ours added a CPU memcpy, a VRAM allocation, a `cuMemMap`, an eviction and a synchronize — to buy residency discarded before it is ever re-read. Notes that explicit requests are still honoured, that the force knob is parsed so a *typo* keeps the fast default, and that **Linux is unchanged and must stay so**. - **#866** — elastic weight budget (1.68× fewer bytes, +20.3 points byte-weighted hit rate, max-context reachability tested through the production reclaim path). - **#853/#856** — `total_weight_bytes` was **2.00× too large** (50% orphaned external-data prefix). Records *how it was found*: arithmetic that could not close, with measured traffic below its own theoretical floor. - **#863** — `no-spill` is an accounting guarantee, with the later refinement that it **does** hold physically when solo and under `managed_limit`; spill is specifically the system-wide over-commit case. - **#869** — byte-weighted hit rate, and why the count-based one misleads. - **#877** — hybrid feasibility, including that it **reversed the assumed priority order**: residency is worth ~9.7×, zero-copy is not a bandwidth win. ## Added framing **Two regimes, and wins do not transfer.** Weights fit ⇒ node-dispatch bound (#870). Weights do not fit ⇒ PCIe-streaming bound. Node-count reduction is invisible in the streaming regime; the streaming work is invisible on a model that fits. This is worth stating explicitly because both lines are active and a throughput number from one says nothing about the other. ## Added open items, with measured sizes - **The #864 hybrid**, with its unresolved risk stated plainly: #877's figure is a *sequential* `cuMemcpyDtoD`, not a strided GEMV, and a real kernel could be far worse — which would kill the zero-copy half. Capture compatibility with host-mapped pointers is untested. - **#837 item 3** — 1.97× above the streaming floor, **1.158 GB/step recoverable**, ~91 ms/step at measured bandwidths; `bypassed_page_ins` still unattributed. - **#750 batching** — with the condition stage 1 established: it amortizes **only** as one fused forward with `M = N`. - **#851** — parked with the three eliminated hypotheses and what unblocks it, so nobody repeats the search. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
#880) ## What this answers (#864) The task: the **11.4 GB/s** zero-copy figure on #864/#877 is an **upper bound** — it was measured with `cuMemcpyDtoD` *from* mapped host memory (a large, perfectly sequential, driver-optimised pull). A real decode kernel does strided, vectorised int4 GEMV reads. **Measure what a real kernel actually achieves reading a weight in place from host-mapped memory vs from VRAM, with outputs verified identical.** Then: does CUDA graph capture work with a host-mapped device pointer? This PR adds **measurement only** — two `#[ignore]`d tests in the `MatMulNBits` kernel module. **No engine behavior changes.** ## Method (only the weight pointer differs) `zerocopy_kernel_host_mapped_vs_vram_bandwidth` launches the **real** int4/fp16 decode GEMV — the exact kernel + launch config the engine selects for `qwen14b-zp` (block-32, fp16 scales, asymmetric zero-points) — twice per size, with **only the `packed` weight pointer differing**: - **VRAM arm:** weights from `cuMemAlloc` (what the managed path streams into). - **Host-mapped arm:** weights from `cuMemHostGetDevicePointer` over a `cuMemHostRegister(DEVICEMAP|READ_ONLY)` host buffer (the zero-copy cold path). Activation, scales, zero-points and output stay resident in VRAM in both arms, so the time delta is purely the weight read. Effective GB/s = `packed_bytes / kernel_time`, best-of-7, CUDA-event timed on the compute stream. Packed size is **swept** to separate L2-cached reads from cold reads (the ~11.9 MB average per-tensor page-in fits this GPU's L2; the real 8.33 GB layer walk reads each weight once, far exceeding L2). Outputs are asserted **bit-identical** between arms. `nvidia-smi --query-compute-apps` verified empty before every run. ## Results (RTX 4060 Laptop, 8 GiB, WDDM; best-of-7, two runs) | packed | N | VRAM GB/s | host GB/s | ratio | outputs | |---:|---:|---:|---:|---:|:--:| | 12 MiB (realistic per-op) | 4915 | 8–26 *(noisy)* | 3.1–3.4 | ~8× (run 1) | identical | | 64 MiB | 26214 | 6.5–34 *(noisy)* | 5.0–5.4 | — | identical | | 128 MiB | 52428 | 62–121 | 5.4–5.7 | ~11–21× | identical | | 256 MiB | 104857 | 95–125 | 5.7–6.1 | ~17–21× | identical | | 384 MiB (cold, ≫L2) | 157286 | 100–112 | 5.6–5.7 | ~18–20× | identical | **The reproducible signal is the host (PCIe-bound) arm: ~5.6 GB/s steady-state, stable across both runs and all sizes ≥64 MiB.** The VRAM arm at small (realistic-op) N is latency-bound and noisy on this WDDM box (#863) — I do **not** pin it; at large N it reaches the DRAM ceiling (~100–125 GB/s), matching #877's 111 GB/s resident figure and validating the harness. ### CUDA graph capture `zerocopy_kernel_capture_with_host_mapped_pointer`: capture **SUPPORTED**; the real GEMV with a host-mapped weight pointer captured into a CUDA graph and **replayed bit-identically** to the eager run (4 replays). So a host-mapped cold-path pointer is compatible with capture (relevant to the `captures > 0`, `fallbacks == 0` gates, #796). ## The number, and the recommendation **A real strided GEMV reads host-mapped weights at ~5.6 GB/s steady-state — about *half* the 11.3 GB/s sequential proxy, not a tenth of it.** - The proxy **was** optimistic: the strided/vectorised access pattern costs ~2× versus the ideal sequential pull. That is a genuine penalty #877's number masked, and **the planning figure for the hybrid's cold path should be revised from 11.3 GB/s down to ~5.6 GB/s.** - But it is **~2×, not the ~10× that would kill the design.** Zero-copy is **not dead.** It moves weights at PCIe rate, correctly (bit-identical every time), and is capture-compatible. - **Cross-check that this isn't an artifact:** ~5.6 GB/s is exactly consistent with the WDDM result on #874. WDDM reaches ~8 tok/s on `qwen14b-zp` by keeping ~7.7 GB resident and reading only the ~0.6 GB over-budget remainder from host RAM per step — 0.6 GB × 8/s ≈ 4.8 GB/s of PCIe traffic, the same order as this measurement. If zero-copy were really 10× worse (~1 GB/s), WDDM could not achieve 8 tok/s. The two numbers corroborate. - **The hot-set prize is confirmed and is the larger lever:** ~8× at the realistic per-op shape (matching the probe's 9.7×), rising toward ~18–20× as the resident read approaches DRAM bandwidth. Maximising the resident fraction remains priority #1, exactly as #877 concluded. **Verdict: build the hybrid.** Both halves are sound — but size the cold path against **~5.6 GB/s**, not 11.3, and treat the resident hot set as the dominant win. ## Honest caveats - **VRAM small-N is noisy** on this WDDM box; the clean signals are the host (PCIe) arm and the large-N VRAM ceiling. I report ranges, not a single VRAM tok-equivalent. - **`READ_ONLY` registration** was used deliberately (weights are immutable) — that is the flag the design would use, so it is the right thing to measure. - **Synthetic packed bytes** (deterministic LCG), not the real blob — but the physical location (host RAM over PCIe vs VRAM) is what governs bandwidth, and correctness is verified by arm-vs-arm bit-identity, not against an oracle. - Single device, WDDM, `can_map_host_memory=1`. ## Validation - `cargo fmt -p onnx-runtime-ep-cuda -- --check` ✅ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✅ (exact CI gate) - `cargo clippy -p onnx-runtime-ep-cuda --features cuda --tests` ✅ **for the changed file** — my new code is lint-clean. The run also surfaces pre-existing `-D warnings` errors in files I did **not** touch (`standard_attention.rs`, `optimizer.rs`, `tests/gqa_shared_prefix_parity_gpu.rs`, `tests/index_share_gpu.rs`); these exist on `origin/main` and are why the CI gate is lib-only / not `--all-targets` (the instructions note `--all-targets` fails here). - Both tests pass `--ignored` on the live device; outputs bit-identical in every arm. Measurement task per #864 — **not** merging, reporting back. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…#880) (#881) I published 11.41 GB/s for host-mapped weight reads on #877 and labelled it an upper bound. #880 measured the real thing and it was a meaningfully loose bound — correcting it here rather than leaving the optimistic number in the single-source doc. **What changed.** #877 used a sequential `cuMemcpyDtoD` *from* mapped host memory. #880 ran the **real strided int4 GEMV** — the exact kernel and launch config the engine selects for `qwen14b-zp` — with **only the weight pointer differing**, activation/scales/zero-points/output resident in both arms. Result: **~5.6 GB/s steady-state** host-mapped against **~100–133 GB/s** resident, outputs **bit-identical at every size**. I reproduced it independently before merging #880. **Size the hybrid's cold path at 5.6 GB/s, not 11.4.** **Why the correction is trustworthy.** It cross-checks against a measurement taken for an entirely different purpose: WDDM reaches ~8 tok/s on `qwen14b-zp` by keeping ~7.7 GB resident and reading the ~0.6 GB over-budget remainder from host each step — ~4.8 GB/s of PCIe traffic, the same order. If zero-copy were really ~1 GB/s, #874's result would be impossible. Two independent paths landing on the same order is much stronger than either alone. **Also recorded:** - **Sweeping the packed size mattered.** The realistic ~12 MiB per-tensor read fits this GPU's L2, so a single-point measurement there compares a *cached* read against a PCIe read and yields a badly wrong ratio. The real 8.33 GB layer walk reads each weight once and vastly exceeds L2, so the ≥64 MiB points are the honest ones. - **CUDA graph capture with a host-mapped weight pointer is SUPPORTED** and replays bit-identically to eager — measured, not assumed. The cold path therefore does not forfeit the `captures > 0` / `fallbacks == 0` gates (#796), which I had flagged as an open design risk. **The design survives.** A ~2× penalty is not the ~10× that would have killed it. Residency remains the dominant lever (~9.7× at the realistic per-op shape, ~15–24× at large N), so the priority order from #877 stands: maximise the resident fraction first, de-copy the remainder second. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
This was referenced Aug 13, 2026
Merged
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…ing amortization (#750) (#884) ## Stage 2a of #750 — batch-N fused forward + weight-streaming amortization (scoped by #844) Refs #750. Continues the staging plan in #844. **Do not merge; #750 stays open.** ### What landed A batch-N **input binding + fused forward** (`NativeDecodeSession::run_fused_batch_prefill`) that isolates the weight-streaming amortization mechanism (#844's premise) from the batch-1-pinned persistent KV bindings, plus the byte-identical guard #844 named and a profiler mode that leads with the deterministic counters. - `run_fused_batch_prefill(&[TokenId]) -> Vec<Vec<f32>>`: one eager `session.run` binding `input_ids`/`attention_mask`/`position_ids` `[N,1]` and an **empty past** `[N, heads, 0, head_dim]`; returns one `[vocab]` logits row per row. **Stateless** — never touches `self.past`, `current_len`, or the CUDA persistent bindings. - `make_empty_input_tensor_batched` (existing `make_empty_input_tensor` now delegates with `batch=1`, **byte-identical**) and `extract_batch_row_logits`. - `from_session_with_io` — public loader taking an explicit `ModelIoSpec` for graphs whose ports can't be disambiguated by shape/dtype. - `profile_native --fused-forward-amortization N,...` — resets the global weight-offload counters around one fused forward per batch size, with a per-run foreign-compute-app contention check (#851). ### Staging correction (well-evidenced, per #844's invitation) A genuine batched **decode step** is **inseparable** from a batched KV layout (stage 2b), so coupled assumptions **#1 (input binding)** and **#2 (KV batch)** cannot be split for any attention decoder: `persistent_state_shapes` pins KV batch axis 0 = 1, the mask is `[1, max_len]`, and ONNX attention couples the batch dim across QKV/mask/past-KV/present-KV. `cuda.rs` is **untouched** in this PR. The escape hatch #844 already anticipated ("validate the fused-forward amortization on a 2-row synthetic before touching KV growth"): the weight-offload residency is **EP-level, keyed purely by weight identity with no batch axis**, so a **stateless fused forward over N independent length-1 rows with an empty past** exercises the genuine batch axis and the weight stream **without any KV growth**. That is exactly what this PR measures. Assumptions #1 and #4 (symbolic-axis) are handled inside this probe; #2 and #3 remain stage 2b. ### Batch-1 byte-identical guard (#844) — PASS `cargo test -p onnx-genai-bench --features bench-native --test fused_batch_prefill` (CPU, synthetic cached decoder): - `native_fused_batch_prefill_row_identical_to_batch_one` — row `i` of a batch-N forward is **bit-for-bit** identical (`f32::to_bits`) to a batch-1 forward of `token_ids[i]`. - `native_fused_batch_prefill_is_stateless` — `current_len` stays 0; a subsequent ordinary `decode` is unaffected. - `native_fused_batch_prefill_rejects_empty`. ### Amortization table — LED BY `htod_bytes_per_token` and `page_ins_per_token` Model `qwen14b-zp` (16.65 GB weights, over-budget), streaming arm `ONNX_GENAI_WEIGHT_OFFLOAD=1` (4 GiB device budget), `ONNX_GENAI_CUDA_GRAPH=1`, RTX 4060 Laptop (8188 MiB). One stateless fused forward per batch size: | batch N | htod_bytes (total) | page_ins (total) | **htod_bytes_per_token** | **page_ins_per_token** | hit_rate | evictions | |--------:|-------------------:|-----------------:|-------------------------:|-----------------------:|---------:|----------:| | 1 | 3,575,949,312 | 301 | **3,575,949,312** | **301.0** | 65.28% | 0 | | 2 | 3,575,949,312 | 301 | **1,787,974,656** | **150.5** | 65.28% | 0 | | 4 | 3,575,949,312 | 301 | **893,987,328** | **75.2** | 65.28% | 0 | | 8 | 3,575,949,312 | 301 | **446,993,664** | **37.6** | 65.28% | 0 | The **totals are batch-invariant** (identical `htod_bytes`, `page_ins`, `hit_rate`, `evictions=0`) and the **per-token figures fall exactly `1/N`** — the precise signature that "amortization happened." One fused forward pages each weight in at most once (residency has no batch axis) and emits N rows. **Why the streaming arm, not the WDDM zero-copy default:** since #874 the Windows default is WDDM zero-copy (`htod_bytes = 0`), so there is no explicit copy for the counters to show shrinking — nothing to *measure* amortize in htod terms. The `ONNX_GENAI_WEIGHT_OFFLOAD=1` streaming arm is the arm where `htod_bytes` is nonzero and its `1/N` collapse is directly observable. The same batching amortizes the WDDM path's in-place PCIe reads identically in kind; the streaming arm is simply where it is instrumented. **Guarding against the #834/#851 error class (a knob producing its own number):** the `ONNX_GENAI_WEIGHT_OFFLOAD` knob sets *whether* weights stream, **not** the amortization ratio. `htod_bytes` is constant across all N; the `1/N` comes solely from dividing a batch-invariant stream by N rows. The knob cannot manufacture that. ### Contention disclosure (#851) A sibling agent's `profile_native` (wt-residency, a second 14B model) intermittently shared the 8 GB GPU. Runs are **labeled contended** by the built-in per-run check. Crucially, **`htod_bytes` and `page_ins` are byte-identical across a fully-contended run, a `batch=1`-solo run, and intermittently-contended runs** — direct empirical proof these counters are contention-invariant **process-local EP accounting**. Only wall-clock `effective_htod_gbps` moved with contention (2.1–3.9 GB/s), which is deliberately **not** headlined. The `batch=1` anchor of the `1/N` claim was captured with `gpu_contention_check: clear`. ### Capture behavior — explicit statement The stateless probe is an **eager** forward and by design **does not engage CUDA-graph capture**; it is not the decode path. The normal batch-1 **decode** path's capture is **provably unchanged** — `cuda.rs` is untouched (0 lines) — so the `captures > 0 && fallbacks == 0` gate (#796) for real decode is unaffected by this PR. Batch-N *decode* capture (a recapture per batch shape, to be **quantified**) is stage 2b/3 work, not exercised here. ### Memory gates `peak/oversubscribed` governance is untouched (empty past ⇒ no KV growth; `evictions=0`, `unaccounted`/`underflows` not perturbed). The mandatory Mobius parity gate applies only to memory/workspace-governance changes; none here. ### Validation - `cargo fmt -p onnx-genai-engine -p onnx-genai-bench -- --check` ✅ - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✅ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✅ - `cargo test -p onnx-genai-engine --features native-backend` ✅ except one **pre-existing** failure, `optional_modality_pipeline_e2e::undeclared_required_audio_input_never_receives_a_fallback` — a debug UB precondition in `onnx-genai-ort/value.rs` (audio pipeline, untouched here). **Reproduced on the clean base with my changes stashed**, so it is not introduced by this PR. - `cargo test -p onnx-runtime-session` ✅ - `cargo test -p onnx-genai-bench --features bench-native --test fused_batch_prefill` ✅ (byte-identical guard) Do not wait for CI; validated locally. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…ected (negative result) (#886) ## #837 item 3 — residency policy gap: attribution (the deliverable) + a rejected design, measured Refs #837 (item 3 only; items 1 & 2 landed in #842/#843). **Do not merge / do not close #837** — this is a report PR. A rejected design with its measurements attached is more useful than a verdict alone, so both are here. --- ## 1. The deliverable: the 1.158 GB/step, attributed `bypassed_page_ins` was **704** in the last measured run and had never been explained. It now is. A "bypass" is a page-in that `admit_committed_span` maps, H2D-fills, hands to the kernel transiently, then frees on `Drop` — **it is counted in `htod_bytes` but never joins the resident set**, so its bytes re-stream every decode step. I added byte-weighted accounting for it (`GlobalOffloadStats::bypassed_page_in_bytes`, `bypassed_byte_share()`, printed by `profile_native`). Measured on `qwen14b-zp`, managed path, **solo** (`nvidia-smi --query-compute-apps` verified empty immediately before the run), 16 tokens: | metric | value | |---|---| | `byte_hit_rate` | **70.16 %** | | `htod_bytes_per_token` | **2.349 GB** | | `bypassed_page_ins` | 336 (**11 %** of page-in *events*) | | `bypassed_page_in_bytes` | 16.77 GB / 16 tok | | **`bypassed_byte_share`** | **44.63 % of streamed bytes ≈ 1.05 GB/step** | | avg bypass size | 49.9 MB (large projections) | | avg retained page-in | 7.8 MB (norms / biases) | **Bypass traffic is 11 % of page-in events but 44.63 % of streamed bytes ≈ ~1.05 GB/step ≈ ~90 % of the 1.158 GB/step recoverable gap.** This converts an unexplained counter into the explanation for the whole 1.97×-above-floor figure. It is exactly the count-vs-byte trap #837 warned about: size-blind `StableResident` retains the small tensors (norms/biases) that win the first-fit race for leftover budget and re-streams the large projections transiently, every step. **This attribution stands on its own even with the policy change below abandoned, and is what redirects whoever picks item 3 up next.** --- ## 2. The obvious fix, implemented and rejected: byte-aware residency Env knob `ONNX_GENAI_WEIGHT_OFFLOAD_BYTE_AWARE` (default-**OFF**, opt-in). When enabled, admission keeps the largest tensors resident: retain an incoming tensor iff it is strictly larger than the smallest evictable resident, evicting smallest-first (strictly-smaller only, to avoid thrash when the large set alone exceeds budget). ### Before / after (managed arm, each run solo, `nvidia-smi` clear beforehand) | config | tokens | token IDs | offload engaged | `byte_hit_rate` | `htod/token` | |---|---|---|---|---|---| | byte-aware **OFF** (baseline), steady, graph ON | 16 ✓ | `[96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]` | yes (`page_ins=3008`) | 70.16 % | 2.349 GB | | byte-aware **ON**, steady, graph ON | **3 ✗** | — early EOS | yes | — (aborted) | — | | byte-aware **ON**, steady, graph OFF | **3 ✗** | — early EOS | yes | — (aborted) | — | | byte-aware **ON**, non-steady | 16 ✓ | identical to baseline | **no** (`page_ins=0`) | n/a | 0 | The "after" arm is being **abandoned**: the policy cannot be safely enabled, so **no bytes were moved**. The counters are attached because a measured rejection is the useful artifact. ### What went wrong: silent numeric corruption, not an error > **Byte-aware residency corrupts decode output — token-identity violated, generation collapses to 3 tokens — whenever offload actually engages on the managed streaming path, both with and without CUDA graph. The default-off path is unaffected.** This is **silent numeric corruption, not an admission error**, and that is *proven*, not inferred, from the profiler's error-propagation structure: `generate_with_callback` is `?`-propagated with `.context("steady measured generation")` **before** the `bail!("generation emitted N tokens")` check. The message we observed is the `bail!`, which means generation returned **`Ok` with only 3 tokens** (an early EOS from corrupted logits) — a surfaced `WeightHandleError` would have produced the `.context(...)` error instead. The runs that stayed byte-identical are exactly those where offload never engaged (`page_ins=0`), i.e. the branch never ran. **The token-identity hard constraint is what caught this.** That check exists precisely so a plausible optimisation cannot ship while silently producing wrong output — and this is the first time it has fired on a real proposal in this line of work. Without it, byte-aware would have "improved" the count-based hit-rate report while emitting garbage. ### Mechanism, as far as I got (and a defect worth flagging) A residency policy decides *what is resident*, not *what is computed*. Both the resident-hit and bypass paths fill the correct bytes at a **stable per-tensor VA**; every eviction here only targets `strong_count == 1` pages, and the admit loop **drains both the compute and copy streams before any eviction**. The only behavioural change vs baseline is the eviction *target*: `smallest_evictable` picks the smallest-by-bytes page, whereas `next_evictable_index` (used by both `Lru` and `StableResident`) picks the **front-of-order oldest** evictable page. Under those guards, a change of eviction *target* **should be value-neutral**. That it corrupts values implicates **state that depends on eviction *order*** — most plausibly the physical granule / retained-handle pool accounting — rather than the policy decision itself. Reproduction with **CUDA graph OFF** rules out captured-VA baking as the sole cause. I could **not** isolate, within this task, whether this is: - **(a)** a bug in the byte-aware admission loop, or - **(b)** a **latent defect in the existing offload path**, merely *exposed* (not caused) by reordering evictions — i.e. something downstream assumes the front-of-order eviction discipline and reads/reuses a physical granule it should not when that order changes. **Possibility (b) is the more valuable one and I cannot exclude it.** If it holds, it is a latent correctness defect independent of this knob, and should be filed and investigated separately (an instrumented run dumping the resident set + per-tensor checksums across steps under both eviction orders would decide it). Flagging rather than asserting. --- ## 3. Conclusion and where the effort should go The count-vs-byte residency gap **cannot be closed by an eviction-policy reorder**. Closing it needs the structural lever #837 deferred: a **dedicated transient staging zone** so a large tensor can be handed to the kernel *without* displacing a resident page (today even a bypass must evict a resident for physical room). That redirects effort toward **#864's hybrid** and **#750 batching** — as the issue anticipated, and as `batch-stage2a` independently corroborated (batch-N amortizes the weight stream exactly 1/N: 301→37.6 pages/token for N=1→8, and the byte counters are contention-invariant process-local EP accounting). ### #864 raises the stakes for item 3, it does not lower them Because the managed residency cache only governs Linux / forced-managed Windows / the #864 hybrid (see scope), one might think item 3 is now niche. The opposite: #880 measured the resident hot set at **~9.7× realistic / ~15–24× at large N** against ~5.6 GB/s host-mapped reads. **Which weights are resident is worth more than it was when this issue was filed** — a well-chosen resident hot set is exactly the edge the hybrid claims over the driver's blind WDDM choice. The attribution in §1 is the input to choosing that hot set well. --- ## Scope Since **#874** the **Windows default is the WDDM zero-copy path**, which has **no residency cache** (the driver chooses). So item 3 governs behaviour **only on the managed path**, which is the arm measured here (`ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`): **Linux** (managed streaming is the only way to run over-budget), **Windows with `ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`** / explicit `--vram-limit`, and prospectively the **#864 hybrid**. ## Hard constraints / gates - **Token identity:** default (byte-aware OFF) path byte-identical — `[96347, …, 752]`, matches prior baselines. (Byte-aware ON deliberately fails this; that is the finding, and it is default-OFF.) - Baseline: `captures=2` (>0), `fallbacks=0`, `oversubscribed_bytes=0`, `peak_committed_physical_bytes=7,725,907,968 < managed_limit=7,726,753,178`, `ref_underflows=0`, `byte_underflows=0`, `unaccounted_committed_bytes=0`. - No assertions added in `Drop`. - **Mandatory** `mobius_seqmajor_growth_parity_native_cuda` (`--ignored --test-threads=1`): **GREEN**, run **solo** with `nvidia-smi` verified clear (treated as binding per #851). - `cargo fmt` (per-package), `cargo clippy -p onnx-runtime-ep-cuda --features cuda -D warnings`, `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -D warnings`, `cargo test -p onnx-runtime-session`, `cargo test -p onnx-runtime-ep-cuda --lib weight_paging::tests` (18 pass): all green. - CPU unit tests exercise the size-greedy admission *decision function* on an abstract model and are explicitly documented **not** to validate the GPU path. ## Measurement hygiene Wall-clock is deliberately **not** headlined (WDDM paging our own memory ranges 3.9–28 tok/s here, #863). Each GPU run verified `nvidia-smi --query-compute-apps` empty immediately beforehand; GPU serialized with siblings `batch-stage2a` / `zerocopy-kernel`. One transient contender appeared during a short non-steady probe and is noted; it did not touch the steady A/B arms, which were clean. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…ication trade, measured (#750) (#891) ## Stage 2b of #750 — batched-KV fused forward + the KV-multiplication trade, measured (scoped by #844, continuing #884) Refs #750. Continues the staging plan in #844 / #884. **Do not merge; #750 stays open.** Stage 2a (#884) proved weight-streaming amortization with a **stateless, empty-past** batch-N fused forward. It explicitly owed stage 2b two things: (1) the **batched KV layout / N-reservation decision**, and (2) **recapture accounting — quantified, not assumed**. This PR delivers the decision and the measurement, lands the first batched forward that carries a **real (non-empty) batched KV past**, and reports an honest crossover analysis. It deliberately does **not** rewrite `cuda.rs` — see "Why this is where 2b splits." ### TL;DR - **Design decision (firm, evidenced): batched *dense* layout `[N, kv_heads, cap, head_dim]`, not N separate VMM reservations** — because the native decode binds past-KV to `session.run` and the ONNX attention op couples the batch dim across QKV/mask/past-KV. N disjoint VA reservations cannot be presented as one dense ONNX tensor; that requires a custom paged kernel, which the owner explicitly deferred to the paged/#721-stage-3 path. - **Amortization reconfirmed on hardware, and shown KV-content-invariant**: `htod_bytes_per_token` falls exactly `1/N` and is **identical at past_len 0, 512, 2048**. - **Recapture is *not* the net-loss risk for uniform fixed-N 2b** (it's ~1 recapture, amortized); it is the central risk for the ragged 2c path. - **The real cost is KV multiplication**, and it does **not** amortize: batched decode adds a per-token floor `L × kv_bytes_per_token` that is independent of N, plus a hard batch ceiling `N_max ≈ budget / (L × kv_bytes_per_token)`. Batching still wins per-token up to that ceiling; the win *saturates* at the KV floor. --- ### 1. Measurement — led by the deterministic per-token counters Model `qwen14b-zp` (8.33 GB referenced weights, genuinely over-budget), streaming arm `ONNX_GENAI_WEIGHT_OFFLOAD=1` (the arm where `htod_bytes` is nonzero and its `1/N` collapse is directly observable; the post-#874 WDDM default reports `htod_bytes = 0`), `ONNX_GENAI_CUDA_GRAPH=1`, RTX 4060 Laptop (8188 MiB). Each measurement window verified `gpu_contention_check: clear` **per run** (#851). **1a. Stage-2a amortization, reconfirmed byte-for-byte** (`--fused-forward-amortization 1,2,4,8`): | batch N | htod_bytes (total) | page_ins (total) | **htod_bytes_per_token** | **page_ins_per_token** | hit_rate | evictions | |--------:|-------------------:|-----------------:|-------------------------:|-----------------------:|---------:|----------:| | 1 | 3,575,949,312 | 301 | **3,575,949,312** | **301.0** | 65.28% | 0 | | 2 | 3,575,949,312 | 301 | **1,787,974,656** | **150.5** | 65.28% | 0 | | 4 | 3,575,949,312 | 301 | **893,987,328** | **75.2** | 65.28% | 0 | | 8 | 3,575,949,312 | 301 | **446,993,664** | **37.6** | 65.28% | 0 | **1b. NEW — the batched forward with a non-empty length-L KV past** (`--fused-forward-kv-sweep 1@0,2@0,4@0,8@0,1@512,…,8@2048`). The counters are **identical across every past_len** at each N: | batch N | past_len L | htod_bytes (total) | page_ins | **htod_bytes_per_token** | hit_rate | evictions | |--------:|-----------:|-------------------:|---------:|-------------------------:|---------:|----------:| | 1 | 0 / 512 / 2048 | 3,575,949,312 | 301 | **3,575,949,312** | 65.28% | 0 | | 2 | 0 / 512 / 2048 | 3,575,949,312 | 301 | **1,787,974,656** | 65.28% | 0 | | 4 | 0 / 512 / 2048 | 3,575,949,312 | 301 | **893,987,328** | 65.28% | 0 | | 8 | 0 / 512 / 2048 | 3,575,949,312 | 301 | **446,993,664** | 65.28% | 0 | **Reading this honestly.** The `1/N` amortization is real and robust, and it does **not depend on KV content** — a genuinely useful negative control (rules out any artifact where KV pressure was secretly producing the number). But it also means **this stateless probe cannot measure the weight-reclaim trade**: a transient host-supplied KV input does not compete with the *persistent* weight-residency budget the way persistent batched-KV bindings would. So the reclaim cost is real but is a property of the stage-2b-*impl* persistent bindings, and I quantify it analytically below rather than pretend the probe measured it. **1c. Batch-1 real *decode* baseline** (`--tokens 48 --steady`), the anchor for the gates and the recapture count: | metric | value | gate | |---|---:|---| | htod_bytes_per_token | 2,349,010,944 (~2.35 GB) | — | | cuda_graph captures / fallbacks / invalidations | **2 / 0 / 1** | `captures>0, fallbacks==0` ✅ | | peak_committed_physical_bytes | 7,725,907,968 | `< limit 7,726,753,178` ✅ (**99.99% full**) | | oversubscribed_bytes | 0 | ✅ | | ref_underflows / byte_underflows / unaccounted_committed_bytes | 0 / 0 / 0 | ✅ | --- ### 2. Design decision: batched dense layout, not N reservations The native decode path binds `past_key_values.*` as ONNX tensors into `session.run` and relies on the graph's attention op, under CUDA-graph capture. A single **fused** forward with `M = N` (the discipline #844 proved is the *only* implementation that amortizes) therefore needs past-KV as **one dense tensor `[N, kv_heads, cap, head_dim]`**. N separate VMM reservations (the owner's per-sequence contiguous-VA route recorded on #750) cannot be handed to a stock attention op as one tensor — that route needs a custom flat/paged kernel that reads each sequence from its own VA, which is exactly the paged path (#721 stage 3 / 2c) the owner deferred. **For stage 2b (uniform length, stock ORT attention decode), the dense `[N,…]` layout is the correct and only viable choice.** The per-sequence-VA route remains correct for the ragged/paged future; it is not this stage. ### 3. Recapture accounting — quantified From the capture state machine plus the batch-1 baseline (`captures=2, invalidations=1`): - **Initial capture:** 1, at the first decode step of a given batch shape. - **KV-growth recaptures:** identical to batch-1 and **independent of N** — head-major realloc invalidates+recaptures per bucket growth (the `invalidations=1` above); seq-major fixed-stride keeps device pointers stable and recaptures 0×. Batching does not change this axis. - **Batch-shape recaptures:** exactly **1 per distinct N**. Uniform fixed-N decode has one shape ⇒ folded into the initial capture ⇒ **~1 total, amortized over the whole run ⇒ negligible.** **Conclusion: recapture is not a net-loss risk for uniform fixed-N (2b).** The recapture *thrash* risk lives in **2c** (ragged/continuous batching), where N changes per step and the captured graph can be rebuilt up to `max_batch` times — potentially every step if N oscillates. That is the number to gate 2c on, and it is out of scope here. ### 4. The real trade: KV multiplication (honest crossover) Model per-step streaming under the elastic budget (#866), with `C` = weight bytes that miss the resident budget at batch-1, `kvpt` = kv_bytes_per_token: ``` htod_per_token(N, L) ≈ C/N + L·kvpt + activation_overhead(N)/N ``` - `C/N` — the weight base **amortizes** with N (the win). - `L·kvpt` — the KV you reclaim is **per-token constant** (KV scales with N, so KV-per-token does not), so it **never amortizes**. It is the floor the win saturates to. - **Hard ceiling:** KV must fit — `N·L·kvpt < budget − min_activations`. On this box (`budget ≈ 7.73 GB`, qwen14b `kvpt = 196,608 B/token`): `N_max ≈ 7.73 GB / (L·196,608)` → ~**19 at L=2048**, fewer at longer context. So batching **always improves per-token streaming up to `N_max`** (monotonic: `C/N + L·kvpt < C + L·kvpt`), but the benefit **saturates at `L·kvpt`** and is bounded by the KV-fit ceiling. At the measured 99.99%-full batch-1 commit, every KV byte reclaims a weight byte 1:1, so `htod_bytes_per_token` **rises toward that floor as N (and L) grow** — exactly the trade #866 predicts. **This is not a net loss at any reachable N** for uniform decode on this model; it is a *saturating* win with a firm N ceiling. It would only invert into a loss if recapture thrash were in play, which (per §3) it is not for uniform fixed-N — it is for 2c, informing the #864 hybrid direction there. ### 5. Why this is where 2b splits (evidence, per #844's invitation) A *correct* persistent batched-KV decode requires coordinated changes across ≥6 coupled sites in the 3,900-line `cuda.rs` (`persistent_state_shapes` axis0 `1→N`; all `kv_commit` geometry; mask `[1,max]→[N,max]`; `write_decode_inputs`; symbolic-axis pinning; capture signature) **plus** the engine loop carrying N sequences — landing all of it at once, against the 99.99%-full memory limit and the mandatory Mobius CUDA parity gate, is large and risky. Per the explicit guidance ("prefer a smaller increment that lands"), 2b lands the **decision + the on-hardware measurement + the batched-forward-with-real-past mechanism**, and the persistent-binding rewrite becomes **stage 2b-impl** (dense `[N,…]` bindings, guarded by the byte-identical batch-1 baseline in §1c and the recapture count in §3). ### What landed - `NativeDecodeSession::run_fused_batch_forward(token_ids, past_len)` — the stage 2a fused forward generalized to a non-empty length-L batched KV past; `run_fused_batch_prefill` now delegates with `past_len=0` (**byte-identical**). Still **stateless**: never touches `self.past`, `current_len`, or the CUDA persistent bindings. - `make_past_input_tensor_batched` (existing `make_empty_input_tensor_batched` delegates with `past_len=0`, byte-identical). - `profile_native --fused-forward-kv-sweep N@L,…` — resets the offload counters around each `N@L`, per-run contention check (#851), leads with the deterministic per-token counters. ### Batch-1 byte-identical — confirmed - **Structural:** `cuda.rs` is **0 lines changed** (`git diff --stat`: only `mod.rs`, `tensor.rs`, `profile_native.rs`, `fused_batch_prefill.rs`), so the batch-1 decode path is untouched. - **Test:** `native_fused_batch_forward_with_past_row_identical_to_batch_one` — row `i` of a batch-N forward at past_len L is **bit-for-bit** (`f32::to_bits`) equal to a batch-1 forward of the same token at the same L. `native_fused_batch_forward_past_zero_matches_prefill` — `run_fused_batch_forward(_, 0)` byte-identical to `run_fused_batch_prefill`. Plus the existing stage-2a guards. - **Empirical:** the batch-1 fused-forward counters (§1a/1b) and decode baseline (§1c) reproduce #884/#837 exactly. ### Gates / validation - `cargo fmt -p onnx-genai-engine -p onnx-genai-bench -- --check` ✅ - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✅ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✅ - `cargo clippy -p onnx-genai-bench --features bench-native,cuda --bin profile_native -- -D warnings` ✅ - `cargo test -p onnx-genai-engine --features native-backend` ✅ (472 + suites, 0 failed) - `cargo test -p onnx-runtime-session` ✅ (0 failed) - `cargo test -p onnx-genai-bench --features bench-native --test fused_batch_prefill` ✅ (6 guards incl. non-empty-past byte-identical) - **Mobius CUDA parity gate: N/A.** It gates memory/workspace-governance changes; `cuda.rs`, `kv_commit.rs`, and the memory governor are **0 lines changed** here (same reasoning accepted for #884). The stateful persistent batched-KV bindings of stage 2b-impl **will** require it. Do not wait for CI; validated locally. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…measurements (#902) `WEIGHT_OFFLOAD.md`'s north-star table carried two rows that today's measurements have overtaken. The doc's own preamble says stale figures should live in issue threads where they can be superseded, not in the design — so these two rows needed updating rather than leaving to drift. **Property 3 — "beats the OS"** said *"WDDM parity is unproven and remains separate work"*. It is now measured, and the cause is structural rather than a tuning gap. Measured on `qwen14b-zp`, byte-identical output, solo with `nvidia-smi` verified empty before each run: **WDDM 8.09 tok/s with `htod_bytes_per_token = 0`** against **managed streaming 0.11 tok/s**. That is why #874 made the OS path the Windows default and stopped auto-enabling managed streaming there. The mechanism is the part worth recording: each weight is read **exactly once per decode step**, so both paths move the same bytes over the same link, while ours adds a CPU memcpy into pinned staging, a VRAM allocation, a `cuMemMap`, an eviction and a synchronize — buying residency discarded before it is ever re-read. **Copying to VRAM only pays when the data is re-read from VRAM before eviction, and there is no intra-step reuse.** The row now also names the route that could actually beat the OS — the #864 hybrid, a resident hot set chosen knowing the layer walk plus **zero-copy** cold reads — and records that its two blocking risks are measured away (#880): a real strided GEMV reads host-mapped weights at ~5.6 GB/s against ~100–133 GB/s resident with bit-identical outputs, and CUDA graph capture with a host-mapped pointer is supported and replays bit-identically. Linux is called out as unaffected and required to stay so: with no shared-memory fallback an over-budget model simply fails there, so managed streaming competes with "does not run", not with something faster (#783's lesson about not letting a platform-specific conclusion propagate). **Property 4 — "one authority"** said only that the platform can allocate behind our back. #863 measured the stronger and more uncomfortable fact: WDDM pages out **our own VMM granules**. A `cuMemCreate`+`cuMemMap`+`cuMemSetAccess` allocator mirroring the engine arena committed *and touched every byte of* 9,984 MiB on an 8,188 MiB card, with device-resident capped at ~7,942 MiB while the process host working set reached 9.49 GB. So `peak_committed_physical_bytes < managed_limit_bytes` and `oversubscribed_bytes == 0` describe **our ledger, not physical residency** — refined by a later measurement to hold *physically* when solo and under `managed_limit` (`nvidia-smi` tracks us 1:1), with spill specifically the **system-wide over-commit** case, invisible to our counters because our own committed bytes do not change. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…e keys (#904) `/v1/resources` asserted that `memory_strategy` **carried** `strategy`, `weight_offload_enabled`, `managed_no_spill` and `auto_enabled`, and that the strategy was a string. It never checked what any of them **said**. #874 changed exactly those values on Windows — an inferred over-budget model now prefers the OS shared-memory path instead of auto-enabling managed weight streaming, measured at **8.09 tok/s against 0.11**. **A silent revert of that decision would not have failed a single test.** That is the same shape of gap that let `device_policy` go unnoticed for months (#678), and the shape the documented-env-var gate was built to catch — a knob or a decision that produces no error and no visible effect when it stops working. The server is the actual product entry point, and every measurement behind #874 was taken through `profile_native`, so this was the least-covered path carrying the most recently changed decision. **What is asserted:** the fitting-model case, which is platform-independent — `FullResident`, offload off, nothing auto-enabled from a vram limit. **What is deliberately not asserted:** the over-budget case, because that is where the platforms legitimately diverge. Windows/WDDM prefers `Compatibility`; Linux keeps managed streaming, since there is no shared-memory fallback there and the alternative is "does not run" (#783 — do not inherit a WDDM-specific conclusion). That asymmetry is written into the comment so the next reader does not mistake this test for a universal claim and "fix" it on the wrong platform. I verified the assertions hold on `main` before adding them, rather than writing an expectation and adjusting until it passed: `cargo test -p onnx-genai-server --lib resources` — 2 passed. Gates: `cargo fmt -p onnx-genai-server -- --check` pass, `cargo clippy --all-targets -p onnx-genai-server -- -D warnings` pass. Test-only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
This was referenced Aug 13, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 14, 2026
…he host-mapped cold recommendation (#915) Review pass on #907 before it goes to the wider ORT stack, with the disagreements applied directly rather than left as comments. The structure holds — contracts/invariants/flows is the right shape, the responsibility boundaries are clean, and the appendix's evidence table is the single best thing in the document because it lets a reader tell measurement from intent. My changes are where the document is either **missing a decision** or **recommending something we have since measured to fail**. ## The one I disagree with most The appendix told readers to prefer *"a resident hot set plus host-mapped cold, single-touch weights over copy-map-evict churn."* We built exactly that this week, and it lost twice (#912): - **Correctness.** Above ~0.44–0.65 GB of *distinct* host-mapped bytes read per decode step, the device silently returns stale data — generation collapsed 16 tokens to 3 with no error raised. A single host-mapped read is bit-identical at 1/8/16/32 cold weights, and a copy-instead A/B isolates the fault to the read, so it is an aggregate aperture ceiling, not a per-read bug. - **Throughput.** Even capped at a provably safe budget, 0.73 tok/s against WDDM demand paging's 7.84 on the same model in the same session. The safe aperture (0.26 GB/step) is smaller than the traffic it must displace (~0.6 GB/step) — too short by construction, not by tuning. Shipping a design doc that recommends this to other teams would have cost each of them the same week. Replaced with the narrower claim the evidence supports, and I kept what survived: the mechanism works and capture replays bit-identically, so this is a capacity verdict about one class of GPU, not a dead technique. Datacenter parts should be re-measured, not assumed to inherit it in either direction. ## The question the document never asked Every contract describes *how* to manage memory. None asks **whether managing a given class of bytes beats leaving it to the OS** — and the honest answer is sometimes it does not. That is not a caveat; it decides when the rest of the design pays for itself. The rule: **copying into device memory only pays when the data is re-read from device memory before eviction.** A weight read exactly once per decode step has no reuse to amortize, so we pay the same PCIe bytes as the OS *plus* a host copy, an allocation, a mapping, an eviction and a sync. Measured 30× slower than WDDM demand paging (#864); deferring recovered ~100× end-to-end (#874). We were not losing to a better algorithm — we were losing to not being in the path. Added as a decision table plus two obligations that are contracts, not advice: **a governor must be able to decline and report it** (that is what I3 exists for), and **the unmanaged path must stay runnable as a control**, since an A/B arm is the only thing that can tell you the control plane costs more than it saves. It also connects batching to memory management properly — batch-N is what *creates* the reuse that makes managing pay, which is why they are one lever and not two features (#884/#891, ceiling `N_max ≈ 19 @ 2048 ctx`). ## Corrections from measurements this design would otherwise repeat - **C7** must report host-mapped read capacity and resident-vs-host bandwidth — a tiering decision needs both and can infer neither. A platform that cannot report the first is treated as **zero**, not unbounded. - **I8b (new): an authority accounts charge, not residency.** On WDDM the two diverge with no notification (#863). A design that treats the ledger as physical truth is correct on TCC and quietly wrong on every consumer Windows machine. C8's numbers are now labelled accordingly, and residency claims must name their measurement source. - **I8c (new): reservations are elastic against occupancy.** A load-time `bytes_per_token × max_context` split is charged in full from the first token; making it elastic cut streamed bytes 1.68× with the max-context guarantee tested through the production reclaim path (#857/#866). - **I8**: bytes are *referenced extents*, not file or blob size — that error overstated one model by exactly 2.00× and made every derived budget wrong (#853/#856). - **C8**: policy rates must be byte-weighted. A count-based hit rate rose 57.09% → 81.31% while the byte gap *widened* 1.78× → 2.30× (#869). - **Risk table**: do not spend the dynamic-lending fix on eviction policy. Order is ruled out as the cause (#892) and bounded at ~10% of the recoverable gap (#901); ~90% is **admission** — 11% of events but 44.6% of streamed bytes. Without this a reader would reasonably spend a week on the wrong lever. ## What I deliberately did not change The Foundry/cluster layering, the transaction ordering rules, and the PagedAttention-as-a-view position all read correctly to me and I have no measurement that contradicts them. The `ClusterCoordinator` → `ServingMemoryCoordinator` split in "Relationship to existing design" is the right call. Every issue and PR number cited was verified to exist **and** to match the figure attributed to it. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 14, 2026
…missing edge in the diagram (#917) Two problems with taking #907 to the ORT team as written. ## The motivation was written for us, not for them It named **no ORT mechanism**, so an ORT reader could reasonably conclude this is a GenAI-side concern that does not touch their code. And it never answered the first objection anyone who knows ORT's memory model will raise: *these are configuration knobs — set `gpu_mem_limit`, choose an arena extend strategy, size `OrtArenaCfg`, share allocators through `OrtEnv`.* Rewritten around that objection, because it is a fair one and the answer is the whole point of the design: **ORT gives each pool a bound; it gives no holder a way to lend to another, and no way to reclaim from a holder that is over-provisioned right now.** It now names the concrete mechanisms — BFC high-water behaviour and precisely where end-of-`Run` shrinkage does and does not help, custom device allocators not being BFC-wrapped (so the governed path and the arena path are two implementations today, not one policy), and `RegisteredAllocator` leaking because ORT does not expose the last session user. Also puts **three measurements up front**. An architecture proposal with no evidence of harm reads as speculation, and we now have the harm: - a load-time `device_budget − kv_bytes_per_token × max_context` split that stranded **1.611 GB** from the first token; making it elastic cut streamed bytes **1.68×** (#857/#866) - weight bytes measured from the external-data file rather than referenced extents — wrong by exactly **2.00×**, so every derived budget was wrong (#853/#856) - managing single-touch weights measured **30× slower** than letting the OS demand-page them; deferring recovered ~**100×** (#864/#874) The first two are accounting failures that a single authority prevents *by construction*, which is the argument for the design. The third is why it also has to say when **not** to manage. ## The diagram had a real omission The appendix assigns **transient-peak reporting** to `InferenceSession`/EP, and I8 requires admission to derive exact bytes *including transient peaks* — but there was **no edge from `Session` to `ProcessMemoryManager` at all**. That is the principal ORT-side integration point, so leaving it out made the ORT-side work look smaller than it is and hid the one interface ORT actually has to add. Added the edge, added the matching entry to *Required architecture changes*, and stated explicitly that the session is a **reporter, not a holder**, so the new edge is not misread as an ownership change. Also split control-plane (solid) from data-plane (dotted) edges, since only the control plane is new work and a reviewer should be able to see that without reading the prose. ## One overclaim of my own, corrected I first wrote that the BFC arena "does not return memory". That is **wrong** — ORT supports arena shrinkage at a `Run` boundary, and an ORT reviewer would have caught it immediately and discounted everything around it. The accurate criticism is narrower and still holds: shrinkage is **caller-scheduled at a `Run` boundary, not demand-driven**, so it cannot answer pressure that arrives mid-generation. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 14, 2026
…l it the route (#919) `PROGRESS.md` and `WEIGHT_OFFLOAD.md` both still described the #864 hybrid as the open route to beating the OS, and the north-star row went as far as *"its two blocking risks are now measured away"*. True when written; now stale **in the worst direction**, because it points the next person at a lever that has already been built and measured to fail (#912). Corrected with what was actually found. The two blocking risks really were measured away first (#880: ~5.6 GB/s real strided GEMV, bit-identical capture with host-mapped pointers) — so what killed it was **neither of them**: - aggregate *distinct* host-mapped reads above ~0.44–0.65 GB/step **silently return stale data** (48 cold weights collapsed generation 16 → 3 tokens; a single read is bit-identical at 1/8/16/32, and a copy-instead A/B pins the fault on the read, so it is an aperture ceiling rather than a per-read bug); - capped at a provably safe 256 MiB, the arm ran **0.73 tok/s against WDDM's 7.84**. The safe aperture (0.26 GB/step) is smaller than the traffic it must displace (~0.6 GB/step) — short **by construction**, not by tuning. Row 3's honest status is now stated as such: **on WDDM consumer hardware we do not beat the OS, we correctly decline to compete with it, and no route currently on the table changes that.** It then names the two levers that do remain — batching (which creates the intra-step reuse that makes residency pay at all) and the admission side of the residency gap. Also in this pass: - Removes a justification the residency-policy item was **borrowing** from the hybrid, and adds what #892/#901 since established: eviction order is not the cause and is worth at most ~10% of the gap, so ~90% is admission. Anyone picking that up should not spend it on eviction policy. - Refreshes the #750 batching item, which still listed stage 2b as remaining after #891 and #913 landed. - Adds seven link-reference definitions (#783, #863, #864, #874, #877, #880, #912) that were used in bracket-link form but never defined, so they rendered as literal text. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 14, 2026
… capability negotiation (#926) Two corrections, the first prompted by a simple observation: **Linux has no WDDM.** ## The #912 closure was written as though it settled the hybrid everywhere It settles it on **one consumer GPU under WDDM**. On Linux neither half of the argument survives: - **The comparison arm does not exist.** There is no OS shared-memory fallback, so an over-budget model that does not stream fails outright. The hybrid competes with managed streaming (0.11–0.86 tok/s on this box) or with *not running* — not with WDDM's 7.84 tok/s. A result that loses badly on Windows can be the best available option on Linux. - **The ceiling that closed it is plausibly VidMm's, not the hardware's.** #863 already measured WDDM demoting our own `cuMemCreate`/`cuMemMap` granules invisibly; a host registration being silently remapped is the same family of behaviour. On Linux `cuMemHostRegister` pins pages in the driver with nothing above it. That second point is a **hypothesis**, so it is filed as #925 to be measured rather than asserted in either direction. This is #783's lesson turned on our own result. Worth noting: **the code was already right.** `shared_memory_weight_fallback` is `cfg!(windows)`, Linux still auto-enables managed streaming, unit-locked by `managed_default_no_flag_over_budget_model_auto_streams`. This is documentation catching up to implementation, which is the safer direction to be wrong in. The decision table now splits the single "defer to the platform" row **into two**, conditioned on whether a demand-paging path exists at all — because those two rows are the same workload on two operating systems with opposite correct answers. ## That split motivates the second change The runtime chooses between those rows using `cfg!(windows)`: a compile-time proxy for "is this WDDM" that is wrong for TCC-mode Windows and silent about any third EP. Serviceable as an interim, indefensible as a design. New section: **Platform and EP capability negotiation.** Every row traced to the measurement that established the need: | Reported fact | Evidence | Status today | |---|---|---| | Oversubscription behaviour (demand-page / fail / unified) | #864/#874 vs #783 | `cfg!(windows)` | | Ledger truthfulness | #863 | absent; assumed true | | Aggregate host-mapped read capacity | #912; unmeasured on Linux (#925) | absent; assumed unbounded, which corrupted output | | VMM capability and granule | 2 MiB CUDA granule | known to the allocator, not reported | | Resident vs host bandwidth for the access pattern | #877 → #880, a 2× error in the sizing input | absent | | KV layout as a stride descriptor | #783 | a runtime-side enum | | Reclaim capability | C5 | absent | The point that organises it: **these are quantities, not flags.** A boolean "supports host mapping" would not have prevented #912, whose entire finding was about *how much*. Today's channel is `ExecutionProviderCapabilities` — an opaque string set carrying exactly one entry, `"nxrt"`. ## Do we need new EP API? Three different answers - **Five reporting facts** want **one** versioned, `struct_size`-prefixed record rather than seven entry points, negotiated the way the plugin ABI already negotiates major/minor. A later capability then costs a field, not an ABI break. - **KV layout is a negotiation, not a report** — propose/accept/counter over a stride descriptor (#783) — because the runtime must still bind exactly one physical form per binding, so an EP that merely announces a preference leaves disagreements unresolvable. - **Reclaim is behavioural** — a `reclaim(target_bytes) -> released_bytes` callback, so C5 pressure can reach an EP-side holder, with the holder choosing victims (I6) and free to release zero. Plus four rules that keep this from fragmenting, of which the load-bearing one is: **an unreported capability degrades to its most conservative reading, never its most convenient one.** Unknown host-mapped capacity is zero, not unbounded. #912 is the cautionary case — the optimistic default produced silently wrong tokens rather than an error. ## A citation of my own, corrected Three documents referenced `#924` as the Linux issue **before I had created it**, and the number was taken by an unrelated PR in the meantime. Filed for real as #925 and the references repointed. Citing a number before verifying it exists is exactly what the measurement-discipline skill tells everyone else not to do. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 14, 2026
…n batch-N on (2b-impl-4, #750) (#930) ## What this is Increment **4 of 4** of native batch-N decode (issue #750), stage 2b-impl-4: pin the batch symbol, wire the token writer, turn batch-N on. Increments 1-3 (#913, #920, #924) were byte-identical by construction with `batch` constructor-fixed to `1`. **This is the increment that changes behaviour**, so it is measured rather than asserted. **#750 stays open** for the 2c caller. > **Correction (2nd commit).** The first version of this PR reported that > "capture and weight-streaming are mutually exclusive on this build." **That > was a harness artifact, not a runtime property**, and it has been corrected > in place. The sweep loaded the native session via `load_with_resolved_io`, > which passes `cuda_offload_policy: None`; `load_with_cuda_options_and_io` > then leaves `weight_offload_stable_va` at its conservative `false` default > (`native_decode/load.rs:793`), so the capture predicate > `weight_offload_enabled && !weight_offload_stable_va` declined capture *by > construction* whenever offload was on. The knob I set (the entry point) set > the flag that produced the decline that I read as a finding. The harness now > drives the **fully governed** `Engine` session, whose effective offload > policy carries the managed no-spill stable-VA authority — and **capture and > streaming coexist**, exactly as #796/#836 built and verified. All the > capture/streaming numbers below are re-measured on that governed path. ## What is on and under what flag - **`ONNX_GENAI_NATIVE_DECODE_BATCH`** (default `1`). At the default the decode path is **byte-identical to `main`**. Setting `N>1` binds a stable batch extent at session construction and drives N rows through the persistent captured decode path. - Batch symbol pinned for aux outputs via `persistent_output_shape` (axis-0 → `batch`); KV seq-axis pinning unchanged. - Token writer wired: `write_decode_inputs_batch` (N token IDs → N slots), `write_position_binding_batch` (rank-R mrope per sequence), `extend_mask` (N mask rows). Single-token entry points delegate to the batch writers (byte-identical at batch 1). - `decode_cuda_greedy_batch` / `NativeDecodeSession::decode_greedy_batch` is the batch-N greedy entry point; `native_decode_batch()` exposes the extent. - `Engine::native_decode_session_mut()` (new, native-backend) lets the measurement harness drive batch-N decode against the governed session. - `profile_native --native-decode-batch-sweep <list> [--native-decode-batch-context L]` builds the governed `Engine` per batch shape and exercises the captured decode path. **The batch-N generation driver (real per-sequence prompt seeding) is stage 2c and out of scope here** — the harness fans one identical token to all N rows, which is a strict correctness probe and, because the weight/page/capture counters are content-invariant (#884), the trustworthy measurement vehicle. ## Mandatory gates — all re-run locally after the correction | Gate | Result | |------|--------| | `cargo fmt -p onnx-genai-engine -- --check` (+ `-p onnx-genai-bench`) | **pass** | | `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` | **exit 0** | | `cargo test -p onnx-genai-engine --features native-backend,cuda --lib native_decode` | **96 passed, 0 failed** | | **mobius parity** `--test mobius_seqmajor_growth_parity_native_cuda -- --ignored --nocapture --test-threads=1` | **2 passed** (SOLO) | | **batch-1 byte-identity on hardware** | **exact match** (below) | ### Batch-1 byte-identity with counters Reference token IDs matched **exactly** with the feature compiled in and batch configured to 1 (`qwen14b-zp`, `--steady --tokens 16`): ``` [96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752] ``` Counters matched the documented baseline: `captures=2`, `replays=28`, `fallbacks=0`, `htod_bytes_per_token=2349010944`, `page_ins_per_token=188`, `hit_rate=78.32%`, `byte_hit_rate=70.16%`. ## Correctness of batch-N (row identity) `qwen14b-zp`, all N, every step, on the governed captured+streaming arm: **all N rows == row 0** (the #892 cross-row-corruption detector) and **row 0 == the batch-1 stream** at N=2,4,8 (the scoping-doc §3.8 guard). Token *values* at N>1 are content-arbitrary (real per-sequence seeding is 2c); the counters are not. ## Recapture per shape (#891) — **measured, capture survives batch-N WHILE streaming** `qwen14b-zp`, governed engine, managed streaming forced (`ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`, `ONNX_GENAI_CUDA_GRAPH=1`), empty past, measured window, per batch shape. **Non-zero htod confirms weights are actually streaming during these captured steps:** | N | graph enabled | captures | replays | fallbacks | invalidations | htod_bytes_per_token | |---|---------------|----------|---------|-----------|---------------|----------------------| | 1 | true | 1 | 6 | 0 | 0 | 2,994,222,848 | | 2 | true | 1 | 6 | 0 | 0 | 1,502,098,688 | | 4 | true | 1 | 6 | 0 | 0 | 736,831,040 | | 8 | true | 1 | 6 | 0 | 0 | 424,642,624 | **Capture survives batch-N as implemented, on the real streaming decode path** — one capture per shape, zero invalidations, zero fallbacks at every N, with ~3 GB/token of weights streaming underneath. This is the positive resolution of #891's open risk the line has been waiting for, and it is the part the original PR got wrong by measuring the wrong arm. ## 1/N weight-stream amortization on the captured decode path — **measured** Same governed captured+streaming arm as above (managed path, #874 note: the WDDM default is the OS shared-memory path with htod=0, so managed streaming is forced). Model `qwen14b-zp` (8.3 GB weights), empty past, `--tokens 8`. Emitted tokens = steps × N. | N | htod_bytes_per_token | page_ins_per_token | abs page_ins/forward | evictions | byte_hit_rate | |---|----------------------|--------------------|----------------------|-----------|---------------| | 1 | 2,994,222,848 | 276.8 | 2214 | 1273 | 61.96% | | 2 | 1,502,098,688 | 139.9 | 2239 | 1293 | 61.83% | | 4 | 736,831,040 | 66.6 | 2132 | 1185 | 62.55% | | 8 | 424,642,624 | 41.6 | 2662 | 1678 | 56.84% | `htod_bytes_per_token` falls **~1/N** (2.99 GB → 1.50 → 0.74 → 0.42; a 7.05× reduction N=1→8 against the ideal 8×). Amortization vs the N=1 ideal extrapolation: N=2 **+0.3%**, N=4 **−1.6%**, N=8 **+13.5%**. The N=8 shortfall is not noise — it is the #866 offset (next section). All four shapes ran SOLO with `nvidia-smi` verified empty; no contention flags. **The amortization is approximate here, not byte-exact.** On this governed managed arm the evictor's working set and page order vary per session load, so the per-forward `abs page_ins` drifts (2214 / 2239 / 2132 / 2662) rather than staying identical. That is the honest behaviour of the *captured streaming* path. For contrast, the earlier *eager* offload arm (offload on, capture declined, fixed `DEVICE_BYTES=6e9`) gave **byte-exact** 1/N with an identical `abs page_ins=1791` at every N — because a fixed budget with no capture is fully deterministic. Both arms agree that the weight stream is paged once per forward and the per-produced-token cost falls ~1/N; the captured arm is the one that matters for #750 and it now measures cleanly. Per the task instruction, **no throughput delta is reported as a finding** (this box ranges 3.9–28 tok/s across identical configs). The deterministic counters above are the evidence. ## The #866 offset — **measured within-sweep on the governed arm** #866 makes the weight budget elastic against occupancy, so batch-N (N× the working set) should shrink the weight residency and push `byte_hit_rate` down. The governed captured arm shows exactly this **within a single sweep**: - N = 1, 2, 4: `byte_hit_rate` ≈ **62%** (61.96 / 61.83 / 62.55, flat within run-to-run noise), evictions ≈ 1200–1300. - N = 8: `byte_hit_rate` drops to **56.84% — a −5.12-point offset**, evictions rise to 1678, and `abs page_ins/forward` rise 2214 → 2662. That offset is precisely why the N=8 per-token htod runs **+13.5% above** the ideal 1/N line: the N× working set ate into the elastic weight budget, so each forward had to re-stream more. This is the #866 offset the line predicted, measured on the real decode path rather than inferred. (The earlier PR could only show it as an *arm-to-arm* budget difference because the un-governed sweep never let the elastic budget respond to batch; the governed engine does.) ## Honest summary of what is and isn't proven - **Proven:** batch symbol pinned; token/mask/position writer wired; batch-N runs end-to-end N=1,2,4,8; batch-1 byte-identity; row identity and cross-batch identity; **capture survives batch-N *while weights stream*** (captures=1, invalidations=0, htod≈3 GB/token) on the governed engine path; ~1/N weight-stream amortization on the captured decode path; the within- sweep #866 byte-hit-rate offset (−5.12 pts at N=8). - **Corrected:** the original "capture and streaming are mutually exclusive" claim was a harness artifact of `load_with_resolved_io`'s conservative `weight_offload_stable_va=false` default; the governed path proves they coexist. - **Named limitation (in the spirit of #920's seq-major `bail!`):** on the governed managed-streaming arm the 1/N amortization is *approximate*, not byte-exact, because the evictor's per-load working set varies and the #866 offset grows the stream at N=8. This is a property of managed streaming, not a batch-N defect. - **Out of scope (2c):** real per-sequence prompt seeding / the batch-N generation driver. The harness fans one identical token to all rows, so token *values* at N>1 are content-arbitrary; the counters are content- invariant (#884) and are the measurement. **Do not merge** — leaving #750 open for the 2c caller. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
This was referenced Aug 14, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 14, 2026
Closes #947 Working as the governor/memory-strategy layer. Another agent owns `weight_paging.rs` / `cuda.rs`; this PR does not touch them. ## The principle `Unknown` and `a specific-looking number` are opposite facts and were rendering identically. On a Windows-ARM, unified-memory machine with **no NVIDIA GPU**, loading an 8.33 GB int4 model, the runtime fabricated an 8 GiB device and reported: ``` resolved_device_budget_bytes=Some(7730940928) # 8 GiB × 0.90, for a device that does not exist fits_resolved_device_budget=Some(false) # a model that fits the machine, "does not fit" WARN Raise the VRAM limit (--vram-limit ...) # unactionable: there is no device ceiling to raise ``` This PR stops fabricating and starts saying **unknown**. ## What is queried per platform, and how New module `crates/onnx-genai-engine/src/platform_capacity.rs`. Per the issue guidance I checked `Cargo.toml` first — the workspace carries no `sysinfo`/`libc` — and chose **small, targeted OS syscalls over adding a crate**. Every function returns `Option`; `None` means the platform could not report the figure. | Figure | Windows | Linux | macOS | |---|---|---|---| | Host RAM total/available | `GlobalMemoryStatusEx` | `/proc/meminfo` (`MemTotal`/`MemAvailable`) | `sysctlbyname("hw.memsize")` (total only; available needs `vm_stat`, out of scope → `None`) | | Disk total/free | `GetDiskFreeSpaceExW` on the spill dir | `statvfs` | `statvfs` | | VRAM | *(unchanged)* real `cudaMemGetInfo` via `real_cuda_vram_capacity`; **no DXGI/Metal/Vulkan probe** — deliberately out of scope | same | same | Disk is measured on the path that would actually be used for spill, not an arbitrary drive. ## What now reports unknown, and the downstream effect - `CapacityProvider::total_bytes()/free_bytes()` return `Option<u64>`; added `UnknownCapacity` (returns `None`). When there is no device query, the VRAM tier is `UnknownCapacity`, never a constant. - `resolve_limit` returns `Result<Option<u64>, _>`: a **fraction/auto of unknown capacity is `None`** (unknown is not a number); an explicit `Bytes` limit is always `Some`. - `ResolvedLimits.vram_bytes` and the memory-strategy plan's `resolved_device_budget_bytes` / `fits_resolved_device_budget` are `Option` and propagate `None` honestly (`None`, **not `Some(false)`**). - The plan labels each capacity/budget decision by source — `Inference` (measured), `ExplicitOverride` (configured), or the new `DecisionSource::Unavailable` — and renders unknown values as the literal `Unknown`. A reader never again has to divide by 0.9 to discover a constant. - The "Raise the VRAM limit" advice is suppressed when there is no device ceiling: `derive_kv_budget` takes `Option`, and the engine's reservation-does-not-fit error is gated on `vram_bytes.is_some()`, so the reported load stays advisory instead of hard-failing. An explicit `--vram-limit <bytes>` is still honoured **exactly**, device query or not — only fraction-of-capacity resolution needs a capacity. ## Before / after for the user's exact configuration No NVIDIA GPU (VRAM query unavailable), default `vram_limit = Fraction(0.90)`, 8.33 GB model: | | Before (`main`) | After | |---|---|---| | `resolved_device_budget_bytes` | `Some(7730940928)` (fabricated 8 GiB × 0.9) | `None` | | `fits_resolved_device_budget` | `Some(false)` | `None` | | Plan decision source | looked like an inference of a real number | `Unavailable`, value rendered `Unknown` | | "Raise the VRAM limit" WARN | emitted | suppressed (no device ceiling exists) | Encoded as the fail-on-`main` tests (per `measurement-discipline`): - `fraction_over_unmeasured_vram_is_unknown_not_a_number` — `resolve_vram_limit_bytes(&default, None) == Ok(None)`. On `main` this returns `Ok(~7.73e9)`. - `unknown_device_capacity_is_reported_as_unknown_not_a_false_fit` — a plan built with `resolved_vram_bytes: None` yields `resolved_device_budget_bytes == None`, `fits == None`, source `Unavailable`. On `main` the field was `Some(fabricated)`. The named tests the issue called out were updated (not deleted) to the new contract: `no_cuda_device_uses_provisional_vram_provider` → `no_cuda_device_reports_unknown_vram_capacity`; `fallback_vram_fraction_resolves_against_provisional_capacity` → `fraction_over_unmeasured_vram_is_unknown_not_a_number`. `real_cuda_capacity_lifts_fraction_above_provisional_cap` is kept (the portability property that motivated the real CUDA query). ## CUDA path is unchanged (verified on this box: Windows x86, RTX 4060) `nvidia-smi` showed the GPU idle before the run. ``` cargo build --release -p onnx-genai-bench --bin profile_native --features bench-native,cuda .\target\release\profile_native.exe --model ...\qwen14b-zp --ep cuda --tokens 16 --steady ``` ``` memory_strategy: strategy=Compatibility inferred=DynamicWeightResidency access=SequentialDense total_weight_bytes=8330595870 kv_bytes_per_token=Some(196608) resolved_device_budget_bytes=Some(7726753178) fits_resolved_device_budget=Some(false) generated_token_ids: [96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752] ``` `resolved_device_budget_bytes=Some(7726753178)` is the **real** `cudaMemGetInfo` query (≈ 0.90 × the 4060's true VRAM), not the old 8 GiB constant, and the token IDs match the expected sequence exactly. On a real device with a real ceiling, `fits=Some(false)` for an 8.33 GB model is the correct, measured answer. ## Gates 1. `cargo fmt -p onnx-genai-engine -p onnx-genai-scheduler -- --check` — clean. 2. `cargo clippy -p onnx-genai-engine --features native-backend -- -D warnings` — clean; `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` — clean. 3. `cargo test -p onnx-genai-engine --lib` — **377 passed**. `cargo test -p onnx-genai-scheduler --lib` — 66 passed. Server + CLI build clean. 4. CUDA real run above — unchanged. 5. No-device path proven by the two fail-on-`main` tests plus the scheduler `a_fraction_of_unmeasured_capacity_is_unknown_not_a_number` / `unknown_device_capacity_yields_no_device_kv_budget_without_advice`. ## Scope notes / what I could not do - **UMA single-authority modeling is still deferred** (per the issue). One pragmatic consequence had to be handled to avoid regressing CPU/ORT generation: the KV byte budget and the advisory device-authority ceiling are seeded from **measured host RAM** when no device can be measured, because on a device-less box the KV cache physically lives in host RAM. This is *not* device/host authority aliasing — no device lease is charged against host memory, and the plan still reports the device budget as `None`. It is the minimal honest bound so a device-less machine sizes KV from a real number instead of a fabricated one. Proper UMA unification remains a separate, larger change. - One cosmetic residual: on a device-less box the governor snapshot's live `vram.limit` tier reflects the host-RAM-bounded authority ceiling (a measured number), while `resolved_limits.vram_bytes` is `None`. These describe different things (live ledger tier vs resolved policy budget); flagged for follow-up. - No DXGI/Metal/Vulkan VRAM probe and no `cfg!(windows)` #874 proxy change — both explicitly out of scope. - The reported machine (Windows ARM) could not be tested here; the no-CUDA path was exercised on Windows x86 (CPU EP forces `cuda_index=None`) and via the fail-on-`main` unit tests.⚠️ This task touches the memory-governor layer and was flagged as needing review — please have a squad member review before merging. Commit trailers included: ``` Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624 ``` --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
This was referenced Aug 15, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 18, 2026
…e page cache, PCIe wall, what to build (#1333) ## MoE expert-offload architecture — synthesis of three converged workstreams Composes the mechanism, policy, and batching findings into one authoritative document, `docs/memory/MOE_OFFLOAD_ARCHITECTURE.md`, in the honesty register of `MEMORY_MANAGEMENT_MODEL_DESIGN.md` (measured vs inferred separated; hardware/model/regime on every figure; per-agent attribution so any claim traces to its evidence). ### The architectural conclusion of the week **Three regimes** decide whether a routing-aware cache is worth building (measured on this box — RTX 4060/WDDM, 63.8 GB DDR5-6400, gemma-3-27b, routing-skewed pattern under swept memory pressure): | regime | condition | OS captures | headroom | |---|---|---|---| | 1 | cache ≥ full bank | ~100% | ≈0 | | 2 | hot-set < cache < bank | ~88% (LRU exploits skew for free) | small | | 3 | cache ≲ hot-set | **33% hot-warm** | **~3× pin win** | - **Host/PCIe crossover:** warm host reads (~10.5 GB/s) exceed every host→device path (zero-copy 5.9, staged 10.9 GB/s), so in regimes 1–2 the host tier is not the bottleneck — **PCIe is**. Only regime-3 eviction-miss traffic drags host throughput below PCIe. - **CUDA UM is empirically closed on WDDM** (~0.6 GB/s, `concurrent=0`, `cuMemAdvise_v2`/`cuMemPrefetchAsync_v2` return `INVALID_DEVICE`) — recorded as a hard negative so nobody re-proposes it. - **Layout constraint gates the whole storage family:** only canonical MatMulNBits int4 / `BlockQuantizedMoE` are file-backed-usable; Marlin and MLAS-prepacked are excluded. - **DirectStorage/GDS** are PCIe-bound Regime-B backstops (bank > host RAM), not breakthroughs; the D3D12→CUDA interop link is flagged inferred-not-measured. ### Composed with the other workstreams (cited, not re-derived) - **Policy** (MoE agent, #1321/#1322/#1326/#1331): concentration milder than 80/20 (25% → 45.6%); static hot-pin within 17–27% of Belady; PCIe floor **1.63 GB/token → ~8 tok/s** (25% VRAM, batch 1, x8); no knee; **dynamic prefetch foreclosed** by a ≈0 intra-layer window. - **Batching**: **−61% bytes/token** (W=2→8) is the largest lever, but **blocked** behind a ~5.4× M=1→2 step overhead. ### What to build / not build 1. Routing-aware **static** hot-pin — **regime 3 only** (near-worthless in 1–2; dynamic prefetch foreclosed). 2. Target (hundreds-of-GB MoE) sits in regime 3 → worth building, but **cannot be validated on this 8 GB laptop**; the doc states what would (Linux datacentre GPU + real MoE). 3. **Quantisation** = best return per complexity (linear PCIe relief, no policy), at a stated accuracy cost. 4. DirectStorage/GDS = Regime-B backstop only. 5. Never re-propose CUDA UM on WDDM. Cross-links the result back into `MEMORY_MANAGEMENT_MODEL_DESIGN.md` (which said MoE "should be measured before any residency policy is designed"); its dense conclusions (`reads_per_step=1.000` #944, ~30× OS win #864/#874) kept intact and reframed as regime 1 with no reuse. Docs-only; nothing under `docs/` is consumed by a build script (verified). Measured on RTX 4060 / driver 591.55 / CUDA 13.1 / WDDM. Closes #1266. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 20, 2026
The wiki site is now published from [justinchuby/onnx-genai-wiki](https://github.com/justinchuby/onnx-genai-wiki), so the static site generator comes out of this repository. ## What moves `site/` (Quartz, its plugin lockfiles, the build and validation scripts) and `.github/workflows/wiki-pages.yml`. All of it is now in the publishing repository, unchanged except for what the move required. ## What stays `wiki/` — the notes themselves. This repository remains the source of truth for them, and they are still edited only here. ## How publishing works now ``` onnx-genai/wiki/ ──sync──▶ onnx-genai-wiki/content/zh/ ──translate──▶ content/en/ (source of truth) (mirror) (derived) ``` A scheduled workflow in the publishing repository mirrors `wiki/` into `content/zh/` and writes a manifest of this repository's tracked paths so links into `crates/` and `docs/` can still be resolved and checked without a checkout. An English edition is derived from the Chinese pages, and both are published at `/zh/` and `/en/` with a language switcher. The mirror is replaced wholesale on each sync, so a page deleted here disappears there. The consequence is worth stating plainly, and it is recorded in both `wiki/README.md` and the wiki maintenance note: **an edit made directly to the mirror is silently overwritten.** The site is at <https://www.justinchuby.com/onnx-genai-wiki/>. ## Why a sync rather than a submodule Measured, not assumed: this repository's `.git` is about 229 MB and `wiki/` is about 212 KB. A submodule would clone all of it on every build to reach a thousandth of it, and could only reference the whole repository rather than one directory. Copying also gives `content/zh` a real commit history in the publishing repository, which is what the translation staleness check reads. ## Other changes here - `.gitignore` loses the `site/quartz` build-state entries. - `ci.yml` no longer counts `site/*` as documentation-only. A path filter for a directory that cannot exist reads as intent to whoever changes it next. ## Checked Wikilink validation passes on `wiki/` — 145 links across 25 notes — using the validator that moved with the generator. The publishing repository builds both languages from these exact pages: 4159 links across 146 pages per locale, identical page sets in the two editions, and every note carrying the `lang` from its frontmatter. One defect turned up in that build and is worth knowing about, because no check of the Markdown could have found it. Obsidian reads `#word` as an inline tag when it follows whitespace. The Chinese text writes `、#864/#874(WDDM 回退)`, where the ideographic comma stops it being a tag; the natural English rendering is `, #864/#874 (WDDM fallback)`, which produced a tag page called `864/874` in the English site and nowhere else. The publishing build now requires both locales to emit the same set of pages, since every page derives from something identical across the two trees. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c80f8522-983c-47f7-8241-2155a823aabe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Windows/WDDM, auto-enabling managed weight streaming for an inferred over-budget model (the #798 default) is ~30× slower than letting the OS page weights from "shared GPU memory" (host RAM) in place over PCIe. This PR stops auto-enabling managed streaming on WDDM when that faster shared-memory fallback is available. Policy change only — the residency cache, arena, and offload implementation are untouched (per #864's request).
Closes #864 item 1.
The decision you asked me to make:
Compatibilityvs "offload-disabled but keep the arena"I chose
Compatibilitywithmanaged_no_spill=false— and keeping the managed arena is not viable here, for a concrete reason.For an over-budget model you cannot keep the no-spill managed arena while disabling weight offload. The no-spill physical-handle pool is a hard cap at the resolved budget (
cuMemCreatestructurally cannot spill, #798). With offload off there is nothing to page the over-budget remainder, so the pool refuses the weights and the load fails — exactly the "refuse, never spill" branch theqwen2_5_0_5b_managed_vmm_default_e2etest already documents. The whole point of the WDDM fast path is that the OS holds the over-budget remainder in host RAM, which is only reachable withmanaged_no_spill=false. So the correct target is the same arm #864 measured:strategy=Compatibility,weight_offload_enabled=false,managed_no_spill=false, no device budget.Does the
Compatibilityarm still enforce any device budget? No —managed_limit_bytes=None,device_budget_bytes=None. Residency is handed entirely to WDDM, which spills to host RAM. The trade, stated up front: on a host with little free RAM an over-budget model may thrash, and the governor cannot see system-wide pressure (#863). That is the price of the ~30× — and it is exactly what makes the single-touch pattern fast (read in place over PCIe, no copy, no eviction). The startup warning and the plan decision both say this.Requirements
forced_offload(ONNX_GENAI_WEIGHT_OFFLOAD=1),device_budget_override, and an explicit--vram-limitare all still honored → managed streaming. Unit-locked bywddm_explicit_offload_request_is_still_honored,wddm_explicit_device_budget_override_is_still_honored,wddm_explicit_vram_limit_is_still_honored.cfg!(windows)threaded in asshared_memory_weight_fallback. I chosecfg!(windows)over runtime WDDM-vs-TCC detection and I want to flag it: detecting TCC at the memory-strategy layer means threading a CUDA driver attribute up before provider construction — that crosses from "policy" into "architecture", which Managed offload is 30x SLOWER than WDDM demand paging on Windows — and it is the default: the copy-to-VRAM buys nothing for a single-touch access pattern #864 asked me not to do. Under TCCcuMemCreatefails at the physical limit rather than spilling, so the fast path wouldn't actually work there; TCC-mode Windows users who need the managed path use the force knob below. Linux is provably unchanged (shared_memory_weight_fallback=false⇒ the new branch is inert) — mandatory test:managed_default_no_flag_over_budget_model_auto_streamsproves Linux still auto-streams.tracing::warn!at startup plus aweight_streaming_platform_policydecision in the plan (surfaced in/v1/resources), both carrying the numbers and the force knob.ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1. Parsed opt-in-truthy so unrecognized values keep the fast fallback — deliberately not theASYNC_PAGEINtrap where an unrecognized value silently selects the slow path. The safe default (fast) is where unrecognized/typo values land.Measurement (this is the metric, per #864)
Direct A/B on
qwen14b-zp(8.33 GB weights vs 7.73 GB budget — genuinely over budget), same release binary, same prompt,nvidia-smiverified empty before every individual run,elastic-kv-budgetsibling agent held off the GPU.ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1)8.3× on medians. Token IDs byte-identical across all 7 runs (verified, not assumed). The deterministic counters match #864 exactly (
htod_bytes_per_token=3,943,690,240,page_ins_per_token=372on the managed arm; both zero on WDDM), confirming which path ran and that the knob is not manufacturing the number.Honest caveat: my 8.3× is smaller than the issue's 30.7×. That is because this box was quiet today (GPU cleared before each run); the managed arm's pathological 0.18 median in #864 was under this box's characteristic paging noise (#863). The direction, the structural cause, and the deterministic counters are identical. I did not reproduce a 30× on a quiet box, and I'm not claiming one — the clean, low-variance number is 8.3×.
Validation
cargo fmt -p onnx-genai-engine -- --check✅cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings✅cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings✅ (unchanged crate)cargo test -p onnx-genai-engine --features native-backend✅ (memory_strategy: 20 passed incl. 10 new; 471 lib passed)cargo test -p onnx-runtime-session✅mobius_seqmajor_growth_parity_native_cuda --ignored --test-threads=1— first solo run hit the known intermittent Latent gap: seq-major capture-retention proof covers KV binding pointers only, not weight buffers (original flaky-gate claim RETRACTED — contended measurement) #851ILLEGAL_ADDRESSasync fault; re-ran solo and it passed (2/2). Per Latent gap: seq-major capture-retention proof covers KV binding pointers only, not weight buffers (original flaky-gate claim RETRACTED — contended measurement) #851/WDDM pages out our VMM (cuMemCreate/cuMemMap) granules to host RAM under over-commit — #798 no-spill is an accounting guarantee, not physical residency, on WDDM #863 this is a pre-existing async-ordering phenomenon, not a regression, and my code path is inert for this fitting model.optional_modality_pipeline_e2e::undeclared_required_audio_input_never_receives_a_fallbackcrashes withSTATUS_STACK_BUFFER_OVERRUN— confirmed identical onorigin/mainwith my changes stashed (audio-modality Drop hazard, zero references to memory/vram/offload). Not touched here.