Repository navigation
Make managed no-spill VMM the default with automatic weight streaming (#755) - #798
Merged
Merged
Conversation
…#755) On native CUDA the authority-scoped managed no-spill VMM path is now selected by default, without a flag. Memory-strategy inference (#752) runs unconditionally and drives policy: a model that fits stays FullResident and does not page, and a load whose weights exceed the resolved device budget automatically enables weight streaming/offload instead of failing. `--vram-limit` becomes an override of the inferred budget rather than the trigger. The legacy allocator stays reachable for one release via `ONNX_GENAI_LEGACY_ALLOCATOR=1` (back-compat: `ONNX_GENAI_DYNAMIC_KV_WEIGHT_LENDING=0`). The resolved budget, chosen strategy and offload state are printed at startup and exposed in `/v1/resources` (`memory_strategy`) so the new default is observable. Capture stays ON under offload on the stable-VA path (#796); admission/refusal (#743 pre-header 429, #745 committed-granule, #748 growth grants) unchanged. This builds on #716 (capture-compatible weight offload), #723 (StableResident policy) and #704 (no silent WDDM spill). Adds an ignored live e2e (qwen2_5_0_5b_managed_vmm_default_e2e) that locks, in one process: fitting -> FullResident/no page-ins; synthetic over-budget -> streaming with bounded committed bytes (no spill); legacy opt-out observable; explicit limit overrides. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Owner
Author
|
Coordinator pre-merge validation on the merged branch: This closes the owner's original directive
It has been blocked on #716 the whole time, for a concrete reason: weight offload The two outcomes I flagged as legitimate failures did not occurI asked for these to be reported rather than worked around, so I checked both:
The new test names map one-to-one onto the behaviors that must not silently Scope discipline+587/-38 across 9 files, confined to policy selection and reporting. It does not Deterministic counters lead throughout and wall-clock is omitted with the reason Merging. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #798 +/- ##
=======================================
Coverage 79.40% 79.40%
=======================================
Files 329 329
Lines 135613 135613
Branches 135613 135613
=======================================
Hits 107686 107686
Misses 23591 23591
Partials 4336 4336
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
🔴 Benchmark Regression DetectedComparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).
Visual flags: Host infoWhat this cannot catch
|
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…801) ## Summary Builds native CUDA KV state to **grow along a fixed-stride BSNH byte layout** when the resolved layout is seq-major, closing the last engine-side gap to a real end-to-end seq-major generation — the documented next step after #797, and the shared prerequisite for token-major and #777 page-level prefix sharing. **Yes — a real seq-major generation is now possible.** On `qwen2.5-0.5b-q4_0-mobius` (stock BNSH vs a copy with `kv_layout=1` on its 24 GQA nodes), forcing growth with `ONNX_GENAI_KV_MIN_BUCKET=8` and generating 48 greedy tokens across 3 growth events, the head-major and seq-major token streams are **byte-identical**, capture requested OFF and ON, under both KV growth mechanisms. Rebased on current `origin/main` (through #798/#799/#800). Full `cargo test -p onnx-runtime-session` re-run green after the rebase (153 in the primary binary, 0 failures across all binaries). ## The precise structural finding The gap was **not** the shape build — it was the growth *byte geometry*. The CUDA GQA node validates `past_key`/`past_value` as BNSH `[batch, kv_heads, seq, head_dim]` and reads `present_capacity` from **axis 2** (`inputs[3].shape[2]` / `outputs[1].shape[2]`) **regardless** of the `kv_layout` attribute (only the kernel stride arithmetic re-specializes), and `DeviceIoBinding` requires logical/physical shapes to share axis order. So `persistent_state_shapes` **must stay BNSH for both layouts** — a permuted binding shape cannot express seq-major. Seq-major is realized purely in the growth/commit byte geometry. ## Changes (confined to `native_decode/cuda.rs` growth functions) - **`kv_growth_byte_layout()`** — maps the BNSH binding shape to the physical byte layout + grow axis the resolved `KvCommitLayout` stores it in. Head-major = declared shape, grow axis 2 (byte-identical to before). Seq-major = `[batch, capacity, kv_heads, head_dim]`, grow axis 1, capacity-independent `kv_heads*head_dim` per-token stride. - **`apply_vmm_growth` / `build_grown_buffers`** — drive the existing device copy/zero primitives over that byte layout instead of a hard-coded axis-2 scatter (which would corrupt BSNH). - **`ensure_capacity`** — in-place VMM growth accounts **0** moved KV bytes for seq-major (fixed stride keeps the prefix in place) vs `valid_len*bytes_per_token` for head-major. - **`persistent_state_shapes`** — documented the BNSH-metadata contract. **Head-major is byte-identical to today** (`default() == HeadMajor`, absent metadata unchanged), asserted explicitly. Edits are intentionally confined to KV shape/stride growth geometry so they merge cleanly with the (now-landed) managed-VMM-default (#798) and GEMM-workspace-governance (#799) work. ## Measurement (measured this session; interleaved, deterministic counters leading) Since #798 made managed no-spill VMM the **default**, the default native CUDA growth path is now **in-place VMM** — so seq-major grows moving **0 KV bytes by default**. The legacy reallocation allocator is reachable only via the #755 opt-out (`ONNX_GENAI_LEGACY_ALLOCATOR=1`). | Growth path | Capture | Tokens | Committed bytes (head = seq) | d2d head | d2d seq | |---|---|---|--:|--:|--:| | in-place VMM (default) | OFF | byte-identical | 100,663,296 | 688,576 | **0** | | in-place VMM (default) | ON (declined) | byte-identical | 100,663,296 | 688,576 | **0** | | legacy realloc | OFF | byte-identical | 786,432 | 688,576 | 688,576 | - **`FIXED-STRIDE RESULT: in-place VMM growth d2d KV bytes head-major=688,576 vs seq-major=0`** — the fixed-stride property #797 measured at the driver level, now at the engine level on a real model, on the default path. - **Committed physical bytes are equal head vs seq** in both mechanisms — the engine still commits flat bucket/granule ranges, so the `kv_heads×` residency win stays driver-level only (`vmm_kv_layout_residency_gpu`). Wiring `kv_commit.rs::live_prefix_ranges` into the live commit path is a scoped follow-on (shared with token-major and #777). - **Re-capture on growth not reproducible end to end**: this decoder's dynamic logical shape declines whole-step CUDA-graph capture (`captures=0`) — the same obstacle #794 hit. `invalidate_graph` still runs on every growth (retained because `present_capacity` is baked into any captured graph), so a capturing decoder would re-capture on each growth regardless of layout; the "0 re-captures" of #797 needs a fixed full-context stable-VA reservation the bucket-growth path does not inherit. - **Wall-clock: not reported.** The box was contended (identical runs ranged ~5 s to ~212 s across this session's re-runs), so no throughput number is trustworthy and none is extrapolated. ## Validation - 2 new engine unit tests (byte-layout mapping; fixed-stride-moves-no-bytes vs head-major re-strides). - New real-model e2e `mobius_seqmajor_growth_parity_native_cuda` (`#[ignore]`, model-gated) — the honest option-1 validation (patch metadata, run both ways, compare token IDs), extended to actually exercise **growth**, under both allocators. All device ops route through the engine's own stream ordering; run in isolation, single-threaded. - `gqa_seqmajor_parity_gpu` (independent CPU oracles), `group_query_attention_gpu` (25), `vmm_kv_layout_residency_gpu`, `vmm_kv_token_major_floor_gpu`, `vmm_stable_va_weight_slot_gpu` — all green single-threaded. - Full `cargo test -p onnx-runtime-session` — green post-rebase (153, 0 failures). - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` and `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` — green. `cargo fmt` (changed packages) — clean. Refs #797, #794, #782, #750, #777. **Does not flip any default; does not close any issue.** Out of scope: token-major across layers, #777 prefix sharing, converting remaining BNSH kernels, and wiring the dense-prefix residency commit. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
) ## Headline result `qwen2.5-0.5b-q4_0-mobius` does **not** structurally decline native CUDA graph capture on current `main`. The `captures=0` result in #801 was a policy artifact in its test harness: the capture-OFF phase set `ONNX_GENAI_CUDA_GRAPH=0` before the process-wide `RuntimeConfig` `OnceLock` was initialized, then the later capture-ON phase mutated the environment in the same process. That mutation cannot change the cached runtime policy, so predicate `ONNX_GENAI_CUDA_GRAPH` continued to decline capture. With each policy phase isolated in its own process, the real model captures successfully. The forced-growth run measured deterministic counters: - capture ON, head-major: `growth_events=3`, `captures=4`, `invalidations=4`, `replays=39`, no decline reason - capture ON, seq-major: `growth_events=3`, `captures=4`, `invalidations=4`, `replays=39`, no decline reason - capture OFF: `captures=0` now reports `predicate ONNX_GENAI_CUDA_GRAPH declined capture` This also supplies the missing end-to-end reproduction discussed in #801/#782/#797: growth currently triggers re-capture for both layouts. Fixed-stride seq-major removes VMM KV-copy bytes (`688,576 -> 0`) but does not remove the decoder's current growth invalidation. Changing that invalidation is correctness-sensitive because captured launches may bake the capacity geometry; this PR does not change eligibility or invalidation semantics. ## Reporting change - Replaces the decode-level boolean resolution with an invariant-carrying decision: every disabled decision owns a named predicate and reason. - Covers policy, structural, binding-shape, attention-mask, auxiliary-output, and runtime capture-audit declines. - Emits one warning per decoder session and exposes `decline_reason` through CUDA debug stats. - Prints `cuda_graph_decline_reason` beside the existing `cuda_graph` counters in `profile_native`. - Updates the #801 Mobius regression to run ON/OFF phases in separate processes and assert capture ON actually captures. - Adds an anti-regression test that every decode-level decline names its predicate. #796/#798 are reflected in the decision precedence: stable-VA managed no-spill offload does not decline capture; only `weight_offload_enabled && !weight_offload_stable_va` does. ## Validation - `cargo fmt -p onnx-genai-engine -p onnx-genai-bench -- --check` - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` - full `cargo test -p onnx-runtime-session` (153 primary tests; all secondary binaries passed) - targeted engine capture-decision tests (7 passed) - real-model Mobius seq-major growth regression (all three isolated phases passed) Wall-clock is intentionally not compared: the post-rebase real-model run was heavily contended (individual model runs ranged roughly 69-259 seconds). Only deterministic counters are used above. Refs #801, #782, #797, #796, #798. Does not close any issue. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 12, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
## What Documentation-only sync of five memory/KV design docs to the merged, executable-test-backed facts about VMM commit geometry, KV layout, weight offload, CUDA-graph capture, and plugin export. No code or tests changed. `MEMORY_ARCHITECTURE.md` is treated as the single source for KV geometry and the audit table; the other four link into it rather than restating (per #791). ## Most misleading claim found `MEMORY_ARCHITECTURE.md` — the document explicitly designated the single source of truth — still asserted the engine *"explicitly invalidates the graph on every KV bucket growth for both layouts"* and that *"the engine does not yet use that fact to retain the captured graph,"* with a table showing seq-major at `captures=4, invalidations=4`. That was falsified by **#811**: conditional invalidation has landed, seq-major fixed-stride growth now **keeps** the captured graph (growth-attributable invalidations **3 → 0**, `captures=1/invalidations=1`, matching the no-growth control), verified by `mobius_seqmajor_growth_parity_native_cuda.rs` including a negative oracle. Because this doc is the single source others link to, the stale claim had the widest blast radius. ## Changes - **MEMORY_ARCHITECTURE.md** — correct the #811 conditional-invalidation prose + table (verified against `cuda.rs` `binding_growth_signature`/`record_growth_decision` and the parity test); add the ORT-backend KV-routing + #762 plugin-export note (verified `allocator.rs:232` `for_session_device` selects by device, not implementation); thread #822 (granule queried), #812 (engine-side consumer gate), and a #800/#802 record correcting this doc's earlier fp16 sizing of Phase-2a scores (verified `attention.rs:368` uses `dtype.element_size()`). - **CUDA_GRAPH_CAPTURE.md** — replace the mutual-exclusion result with #796 (stable VA removes it), #798 (managed no-spill default), #811 (conditional invalidation), and the #804/#807 `captures=0` process-artifact correction. - **WEIGHT_OFFLOAD.md** — correct the "mutually exclusive today" claim (#796) and the north-star status for the managed no-spill default (#798). - **KV_INSERTION_DESIGN.md** (most stale, 2026-07-15) — add a synchronization banner linking to `MEMORY_ARCHITECTURE.md`; note the layout line post-dates the memo, prefix sharing is a landed primitive (#777), and block-table paged attention remains a proposal (verified: no `block_table` in any CUDA attention kernel). - **DESIGN.md** — forward-pointer in the already-superseded §39.5 noting layout (seq-major/token-major, #787/#792) is now the primary residency lever. ## Method Every retained performance claim cites the PR that measured it; measured fact / design intent / not-yet-built are distinguished; superseded designs are labelled, not deleted. The "still open" item (engine commits flat buckets, not the dense live-prefix ranges `kv_commit.rs` computes, so the `layers×2` floor is not yet realised through the engine) is described as in-flight, not delivered. ## Note on #762 The task framing had #762 (EP plugin export) as open; `gh pr view 762` shows it **merged** today. It is described accurately per that ground truth: the plugin-EP mechanism landed for the CPU EP (validated end-to-end), while the CUDA plugin shim compiles but is unvalidated on hardware — so plugin-EP-hosted VMM KV remains in-progress design intent, not a shipped capability. ## Validation Documentation only — `git diff --stat` touches exactly the five `.md` files; working tree otherwise clean. **Not merged; no issue closed.** Refs #736, #762, #776, #777, #783, #787, #791, #792, #794, #796, #797, #798, #801, #802, #804, #807, #811, #812, #822 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
## Summary - extend `profile_native --steady` observability with resolved memory strategy, CUDA graph counters/declines, residency activity, and committed VMM physical bytes - document the first genuinely over-budget Qwen 14B native CUDA measurement enabled by #384 - verify #796 and #798 at scale; record an honest weaker-than-historical #723 residency result with #704 no-spill evidence ## Measured result Capture was varied in isolated child processes (`ON -> OFF -> ON`) because `RuntimeConfig` is process-frozen. - **Capture ON:** `captures=2`, `replays=11`, `fallbacks=0`, `invalidations=2` - **Streaming in the same session:** `page_ins=5299`, `hits=8573`, `hit_rate=61.80%`, `evictions=3850`, `bypassed_page_ins=691` - **Default selection:** `DynamicWeightResidency`, `fits_resolved_device_budget=false`, `weight_offload_enabled=true`, `managed_no_spill=true`; no explicit `--vram-limit` - **Physical ceiling:** VMM peak committed physical bytes `6,996,099,072 / 7,730,940,928` (90.49%), zero oversubscription and zero unaccounted committed bytes - **Correctness:** identical greedy token IDs across both runs in every process and across capture ON/OFF - **Negative worth preserving:** the scale workload measured 61.80% hits and 3,850 evictions, not #723's earlier 74.18% / zero-eviction result. No old-policy A/B was run, so the document does not claim a regression ratio. Wall-clock was contended by concurrent GPU work and is intentionally not claimed. References #796, #798, #723, #704, and #384. ## Validation - `cargo fmt -p onnx-genai-bench -- --check` - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` - `cargo test -p onnx-runtime-session` - `cargo test -p onnx-genai-engine --features cuda,native-backend --test mobius_seqmajor_growth_parity_native_cuda -- --ignored --test-threads=1` - `cargo test -p onnx-genai-bench --features bench-native,cuda --bin profile_native` Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 12, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…#866) Closes #857 — do not merge yet (opened for review per the issue). ## TL;DR (read this first — the framing changed after #863/#864) This change is **correct, byte-identical, and safe**, and within the managed weight-offload path it recovers the idle full-context KV reservation and cuts weight streaming **1.68×** (`htod_bytes_per_token` 3.94 GB → 2.35 GB, `page_ins_per_token` 372 → 188, `hit_rate` 57% → 78%). But two hardware findings landed while this was in flight and they bound how much it is worth: - **#863:** WDDM pages out *our own* VMM-mapped granules under system-wide pressure, not just `cuMemAlloc`. So `oversubscribed_bytes == 0` / `peak_committed < managed_limit` are **ledger** guarantees, not proof of physical residency. I now measure `nvidia-smi` device-resident alongside the ledger (below) and treat residency as real only where the two agree. - **#864:** the managed *copying* offload path is ~30× slower than the legacy **WDDM zero-copy** path (5.53 vs 0.18 tok/s, n=5/4), because each weight is read exactly once per decode step, so copying it into VRAM pays alloc/map/evict/sync overhead that WDDM avoids. This PR makes a path that trails WDDM trail it by less; it does **not** close the path-level gap. So I am **not manufacturing a win**. I present this as a correct, well-evidenced *within-path* improvement plus an explicit statement (bottom) of where the real leverage now sits. If the reviewer prefers to hold it until the hybrid direction is decided, that is reasonable. ## What this does At load the weight-residency budget was derived once as `resolved_device_budget − kv_bytes_per_token × max_context` and never revisited, statically withholding the **full declared-max-context** KV reservation (1.611 GB on qwen14b-zp, ctx 8192) from weights even for short requests. This lends that reservation to weights and lets **KV growth reclaim it on demand**, still guaranteeing a sequence reaches its declared `max_context`, and **leaves a tunable device headroom** (load-bearing per #863). ## Design (points 1–4) **1. Reclaim path.** Budget/accounting change that unlocks already-built machinery. `reconcile_cuda_offload_budget_after_native_load` withholds only an **initial KV floor** (`kv_capacity_bucket(1, max)`, ~50 MB) instead of the full reservation. Bytes above the floor (less headroom) are lent to the weight cache. As KV grows, `ensure_capacity_seq_major_fixed → prepare_mapped_growth(KvCache)` drives `MappedGrowthAuthority`: it consumes KV's allowance, then unassigned capacity, then **transfers allowance from the weight holder** (`reclaim_priority()=0`) and calls `reclaim_mapped()` (LRU eviction of unreferenced pages). Transactional with rollback, serialized (no deadlock), requester filtered from its own victim list. Weights shed **in step** with KV growth; no refusal path. **2. Hysteresis.** The existing **monotonic KV bucket growth** (power-of-two) gives O(log max_context) reclaim events, each permanently lowering the weight ceiling. The lending high-water mark is a fixed budget set once at load, not re-lent per event. No thrash. **3. Capture.** Budget/limit changes move **no VA and no physical mapping**. Stable-VA weight paging (#716/#796) keeps captured graphs valid. **Measured:** `captures=2, fallbacks=0`. **4. Scope.** Single-sequence only. Serving/admission (#750) out of scope. **Gating.** Only when `managed_no_spill && commits_on_demand && dynamic_kv_weight_lending_enabled()` — the exact condition registering the weight cache as a reclaimable holder. Otherwise the static reservation is kept. Operator override unaffected. The non-override auto budget is recomputed from the elastic `available` (the plan-time `device_budget_bytes` carried the full reservation and would otherwise clamp the budget straight back — the one real trap, fixed). ## Safety: WDDM headroom (question RESOLVED by #863) #863 proved single-process and deterministically that this card pages out our own `cuMemCreate`/`cuMemMap` granules: an arena-mirroring allocator committed and *touched* all of 9,984 MiB on an 8,188 MiB card, but device-resident capped at ~7,942 MiB while host working set hit 9.49 GB — ~2 GB of our granules spilled to host RAM. `cuMemCreate` confers no pin. So this PR **does not lend to the last byte**: it keeps a tunable **device headroom** unlent below the managed limit (`ONNX_GENAI_ELASTIC_LENDING_HEADROOM_BYTES`, **default 512 MiB**), never dropping below the static-reservation baseline (never a regression). The headroom is now load-bearing, not precautionary. **Two nuances (from #863).** (a) The spill is a **latency** effect, transparent to kernel correctness — the driver makes a granule resident before a kernel uses it, so it costs tok/s, not correctness; it is not the #851 illegal address. (b) It is very likely **WDDM-specific**: on TCC / datacenter GPUs `cuMemCreate` should *fail* at the physical limit rather than spill, so the headroom is a WDDM-deployment safeguard and the right default may differ (smaller/zero on TCC, larger under concurrent GPU use). The tunable exists precisely so the default can be set per target. ## Results — qwen14b-zp, ctx 8192, 16 tokens, greedy (byte-identical) All runs solo, `nvidia-smi` verified clear before each (card coordinated with the #851 agent). `nvidia-smi peak` = max device-resident `memory.used` sampled at 300 ms during the run (includes ~229 MiB OS/display baseline). | metric | baseline (main, managed) | elastic 512 MiB (default) | elastic headroom=0 | **legacy WDDM** (`LEGACY_ALLOCATOR=1`) | |---|---:|---:|---:|---:| | htod_bytes_per_token | 3,943,690,240 | **2,349,010,944** | 2,349,010,944 | 0 (zero-copy) | | page_ins_per_token | 372 | 188 | 188 | 0 | | hit_rate | 57.09% | 78.32% | 78.32% | n/a | | evictions | 5,248 | 2,672 | 2,672 | 0 | | weight budget_bytes | 6,116,140,442 | 7,139,550,618 | 7,676,421,530 | — (offload off) | | weight mapped_physical | 5,662,310,400 | 6,406,799,360 | 6,406,799,360 | — | | peak_committed (ledger) | 6,983,516,160 | 7,725,907,968 | 7,725,907,968 | — | | **nvidia-smi peak resident MiB** | — | **7,479** | — | **7,951** | | oversubscribed / underflows | 0 | 0 | 0 | 0 | | throughput tok/s (single, noisy) | — | 0.72–1.42 | — | **7.42** | ### Reading the table honestly - **The byte win is real and byte-identical.** Token IDs identical across every column: `[96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]`. Elastic lending cuts managed-path streaming 1.68×. - **Solo, the lent residency is physically real.** `nvidia-smi` peaked 7,479 MiB while the ledger `peak_committed` was 7,368 MiB (7,725,907,968 B) and `physical_owned` 7,194 MiB — device-resident **tracked** our commitment rather than capping below it. No spill solo (7,368 MiB fits under the 7,959 MiB free with ~590 MiB to spare). The #863 spill is a **contention** effect our ledger cannot see; on this shared box that is the realistic exposure, hence the headroom and this caveat. Precise statement (refined jointly with #863): **#798 no-spill holds *physically* solo under managed_limit** — nvidia-smi tracks our commitment 1:1; **spill is specifically the *system-wide over-commit* case** (a 2nd process, or engine-at-limit + anything else), which is invisible to our ledger because our own committed bytes do not change. - **The right baseline is WDDM, and it wins decisively.** WDDM moves *zero* explicit HtoD bytes (it reads cold weights zero-copy from shared host memory, `nvidia-smi` sitting at the 7,951 MiB physical ceiling) and still runs 7.42 tok/s here vs the managed path's 0.72–1.42 — corroborating #864's 30.7×. **Wall-clock is not a headline** on this box (documented 3.9–28 tok/s spread), but the direction is unambiguous and structural, not noise. - The 512 MiB headroom costs nothing at 16 tokens (resident working set 6.4 GB sits below both ceilings); it caps the ceiling for high-residency runs, bounding our footprint under the contention where #863 spill occurs. ## max_context reachability (the property under risk) New governor test `kv_reclaims_the_full_lent_budget_from_weights_across_bucketed_growth` drives the **production reclaim code** (`prepare_mapped_growth`, allowance transfer, `reclaim_mapped`): weights hold the full budget, KV grows bucket-by-bucket and reclaims its **entire** reservation, reaching full reservation with `oversubscribed_bytes == 0` throughout, weights shedding exactly the reclaimed bytes. The mandatory GPU gate `mobius_seqmajor_growth_parity_native_cuda` passes **solo** with dynamic KV/weight lending active, head-major == seq-major byte-identical, `captures>0`, `fallbacks=0`. ## Where the real leverage sits (per #864) Weights are read **once per decode step**, so within a step there is no reuse to amortize a copy against; the only reuse is *across* steps, which is exactly what a resident cache captures — and what this PR enlarges. But the copying cache pays alloc/map/evict/sync per miss that WDDM's zero-copy reads avoid, so at the whole-path level WDDM dominates for the streamed (cold) fraction. The durable direction is a **hybrid**: keep a resident **hot** set (where a larger budget — this PR — genuinely helps) and serve the **cold** fraction by **zero-copy direct reads** rather than copy-then-discard. Lending more budget to a copying cache has real but bounded value: it helps the resident fraction, not the streamed one. This PR is a correct, safe increment for the resident fraction and an honest data point that the streamed fraction wants a different mechanism, not more copy budget. ## Validation (all local, per issue) - `cargo fmt -p {onnx-genai-engine, onnx-runtime-ep-cuda, onnx-runtime-memory-governor} -- --check` ✓ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✓ - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend` and `--features cuda,native-backend -- -D warnings` ✓ - `cargo test -p onnx-runtime-session` ✓; `cargo test -p onnx-runtime-memory-governor` ✓ (new reclaim test); `cargo test -p onnx-genai-engine --features cuda,native-backend --lib elastic` ✓ (4 tests) - **MANDATORY** `mobius_seqmajor_growth_parity_native_cuda --ignored --test-threads=1`, solo, GPU clear ✓ ## Files - `crates/onnx-runtime-ep-cuda/src/{provider,lib}.rs`: public `dynamic_kv_weight_lending_enabled()`. - `crates/onnx-genai-engine/src/engine/load.rs`: elastic KV floor, tunable headroom high-water mark, non-regression clamp, gating; unit tests. - `crates/onnx-runtime-memory-governor/src/lib.rs`: reclaim-to-full-reservation test. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
… by default (#864) (#874) ## Summary On Windows/WDDM, auto-enabling managed weight streaming for an *inferred* over-budget model (the #798 default) is **~30× slower** than letting the OS page weights from "shared GPU memory" (host RAM) in place over PCIe. This PR stops auto-enabling managed streaming on WDDM when that faster shared-memory fallback is available. **Policy change only** — the residency cache, arena, and offload implementation are untouched (per #864's request). Closes #864 item 1. ## The decision you asked me to make: `Compatibility` vs "offload-disabled but keep the arena" **I chose `Compatibility` with `managed_no_spill=false` — and keeping the managed arena is not viable here, for a concrete reason.** For an over-budget model you **cannot** keep the no-spill managed arena while disabling weight offload. The no-spill physical-handle pool is a hard cap at the resolved budget (`cuMemCreate` structurally cannot spill, #798). With offload off there is nothing to page the over-budget remainder, so the pool refuses the weights and the load fails — exactly the "refuse, never spill" branch the `qwen2_5_0_5b_managed_vmm_default_e2e` test already documents. The whole point of the WDDM fast path is that the OS holds the over-budget remainder in host RAM, which is only reachable with `managed_no_spill=false`. So the correct target is the same arm #864 measured: `strategy=Compatibility`, `weight_offload_enabled=false`, `managed_no_spill=false`, no device budget. **Does the `Compatibility` arm still enforce any device budget?** No — `managed_limit_bytes=None`, `device_budget_bytes=None`. Residency is handed entirely to WDDM, which spills to host RAM. **The trade, stated up front:** on a host with little free RAM an over-budget model may thrash, and the governor cannot see system-wide pressure (#863). That is the price of the ~30× — and it is exactly what makes the single-touch pattern fast (read in place over PCIe, no copy, no eviction). The startup warning and the plan decision both say this. ## Requirements 1. **Only the inferred default changes.** `forced_offload` (`ONNX_GENAI_WEIGHT_OFFLOAD=1`), `device_budget_override`, and an explicit `--vram-limit` are all still honored → managed streaming. Unit-locked by `wddm_explicit_offload_request_is_still_honored`, `wddm_explicit_device_budget_override_is_still_honored`, `wddm_explicit_vram_limit_is_still_honored`. 2. **Platform-gated.** Gated on `cfg!(windows)` threaded in as `shared_memory_weight_fallback`. **I chose `cfg!(windows)` over runtime WDDM-vs-TCC detection** and I want to flag it: detecting TCC at the memory-strategy layer means threading a CUDA driver attribute up before provider construction — that crosses from "policy" into "architecture", which #864 asked me not to do. Under TCC `cuMemCreate` fails at the physical limit rather than spilling, so the fast path wouldn't actually work there; TCC-mode Windows users who need the managed path use the force knob below. **Linux is provably unchanged** (`shared_memory_weight_fallback=false` ⇒ the new branch is inert) — mandatory test: `managed_default_no_flag_over_budget_model_auto_streams` proves Linux still auto-streams. 3. **Said loudly.** A `tracing::warn!` at startup plus a `weight_streaming_platform_policy` decision in the plan (surfaced in `/v1/resources`), both carrying the numbers and the force knob. 4. **Force override:** `ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`. Parsed opt-in-truthy so **unrecognized values keep the fast fallback** — deliberately *not* the `ASYNC_PAGEIN` trap where an unrecognized value silently selects the slow path. The safe default (fast) is where unrecognized/typo values land. ## Measurement (this is the metric, per #864) Direct A/B on `qwen14b-zp` (8.33 GB weights vs 7.73 GB budget — genuinely over budget), same release binary, same prompt, `nvidia-smi` verified empty **before every individual run**, `elastic-kv-budget` sibling agent held off the GPU. | arm | n | median | min | max | prefill | htod/tok | page_ins/tok | |---|---:|---:|---:|---:|---:|---:|---:| | **WDDM** (new default) | 4 | **7.18 tok/s** | 7.10 | 7.67 | ~191 ms | 0 | 0 | | **Managed** (`ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`) | 3 | **0.86 tok/s** | 0.84 | 0.88 | ~1097 ms | 3,943,690,240 | 372 | **8.3× on medians.** **Token IDs byte-identical across all 7 runs** (verified, not assumed). The deterministic counters match #864 exactly (`htod_bytes_per_token=3,943,690,240`, `page_ins_per_token=372` on the managed arm; both zero on WDDM), confirming which path ran and that the knob is not manufacturing the number. **Honest caveat:** my 8.3× is smaller than the issue's 30.7×. That is because this box was *quiet* today (GPU cleared before each run); the managed arm's pathological 0.18 median in #864 was under this box's characteristic paging noise (#863). The direction, the structural cause, and the deterministic counters are identical. I did **not** reproduce a 30× on a quiet box, and I'm not claiming one — the clean, low-variance number is 8.3×. ## Validation - `cargo fmt -p onnx-genai-engine -- --check` ✅ - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✅ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✅ (unchanged crate) - `cargo test -p onnx-genai-engine --features native-backend` ✅ (memory_strategy: 20 passed incl. 10 new; 471 lib passed) - `cargo test -p onnx-runtime-session` ✅ - `mobius_seqmajor_growth_parity_native_cuda --ignored --test-threads=1` — first solo run hit the known intermittent #851 `ILLEGAL_ADDRESS` async fault; **re-ran solo and it passed (2/2)**. Per #851/#863 this is a pre-existing async-ordering phenomenon, not a regression, and my code path is inert for this fitting model. - Pre-existing unrelated failure: `optional_modality_pipeline_e2e::undeclared_required_audio_input_never_receives_a_fallback` crashes with `STATUS_STACK_BUFFER_OVERRUN` — **confirmed identical on `origin/main` with my changes stashed** (audio-modality Drop hazard, zero references to memory/vram/offload). Not touched here.⚠️ This task was flagged as "needs review" territory (memory-governance policy). Please have a squad member review before merging. Do not merge — reporting back per instructions. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…ch-1 baseline (#905) ## #750 stage 2b-impl — scope + on-hardware batch-1 baseline **This is a well-evidenced scoping result, not the full rewrite.** After reading the persistent decode path end-to-end and measuring the batch-1 baseline on hardware, the honest conclusion is: **the smallest *correct* (token-producing) batch-N persistent-binding decode step does not fit in one reviewable PR.** This PR lands the split and the regression anchor that makes each following increment safe to attempt — the same deliverable shape you accepted for stage 1 (#844) and stage 2b (#891), and explicitly invited ("bindings first, mask second… take it"; "a truthful negative result is acceptable and valuable"). New doc: `docs/NATIVE_BATCH_DECODE_2B_IMPL_SCOPING.md` (+ a pointer from `docs/NATIVE_CUDA_DECODE.md`). **No runtime code changed** — so batch-1 is structurally byte-identical and the #796 capture gates are untouched. ### Batch-1 baseline — measured solo (`nvidia-smi` verified empty before the run) `qwen14b-zp`, RTX 4060, `ONNX_GENAI_CUDA_GRAPH=1`, `ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`, 16 tokens, current `main`: | metric | value | |---|---:| | `generated_token_ids` | `[96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]` — **exact** reference match ✓ | | `htod_bytes_per_token` | 2,349,010,944 | | `page_ins_per_token` | 188.0 | | `hit_rate` | 78.32% | | `cuda_graph_measured` | `captures=1 replays=14 fallbacks=0 invalidations=1` → **recapture ≈ 1** (confirms #891) | | `peak_committed_physical_bytes` | 7,725,907,968 `< managed_limit 7,726,753,178` ✓ | | `oversubscribed_bytes` / `ref_underflows` / `byte_underflows` / `unaccounted_committed_bytes` | 0 / 0 / 0 / 0 ✓ | Not headlining wall-clock (`1.72 GB/s`, `23.8 s/token`) — WDDM-paging noise (#863). Per-token bytes differ from #844's 4.11 GB because current `main` runs the elastic-budget (#866) managed path at the 7.73 GB auto budget, not the earlier fixed 4 GiB synthetic; this is the correct current-main anchor. ### Why one PR isn't achievable - The stage 2a/2b guards exercise the **stateless** `session.run` fused-forward (`run_fused_batch_forward`), which already takes `[N,…]`. The path 2b-impl must change is `DecodeCudaState` — **GPU-only, no CPU test seam**. - **No partial guard exists.** A token-producing decode needs *all* of: batched input binding, KV allocation, KV growth/commit, mask, logits read, greedy argmax, and capture-symbol pinning. Drop one → hard-fail or silent row corruption (#892). So increments are validated by batch-1 non-regression until the final one flips batch-N on with a row-identity guard. - The batch-aware **caller** (present N sequences, consume N rows) is the generation driver / `continuous_batch_manager` = **2c, out of scope**. ### Key finding The KV growth **stride** math (`kv_growth_byte_layout`, `copy_kv_prefix_device_to_device*`) is **already batch-general** (block-count based). The real blockers are shape allocation (`persistent_state_shapes` pins axis 0→1), the mask, the input/logits bindings, the **device argmax** (in `onnx-runtime-ep-cuda`, separate clippy gate), the seq-major commit scatter (`live_prefix_ranges` in `onnx-genai-kv`), and **capture-symbol pinning** (`collect_unit_symbols` collapses the batch symbol to 1; it must instead pin to N). Full file/line inventory in the doc. ### Proposed split (each holds batch-1 byte-identical + passes mobius solo) 1. **2b-impl-1** shapes & IO bindings (thread `batch`, constructor-fixed to 1). 2. **2b-impl-2** batched KV growth/commit (+ pure range-math unit tests). 3. **2b-impl-3** batched device argmax (`onnx-runtime-ep-cuda`). 4. **2b-impl-4** batch symbol pinning + entry points + batch-N row-identity guard + measured recapture-per-shape. ### KV-multiplication trade (restated vs #891) `htod/token ∝ 1/N` but with a never-amortizing floor `L × kv_bytes_per_token` and `N_max ≈ 19 @ 2048 ctx`. Under #866's elastic budget, batched KV growth reclaims weight residency, so 2b-impl-4 must report both the 1/N fall **and** the `hit_rate`-driven offset, with a crossover if batched decode ever loses at a reachable N. ### Validation `cargo fmt`/`clippy`/tests are unaffected (docs-only). The baseline was produced by the protocol's own `profile_native` build (`--features bench-native,cuda`), run solo. Not merging; #750 stays open. Refs #750 #844 #884 #891 #796 #798 #866 #892. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #755.
Makes the authority-scoped managed no-spill VMM the default on the native CUDA path — no flag required — with automatic weight streaming when a model exceeds the resolved device budget. Being larger than the budget becomes a supported configuration rather than an error.
Builds directly on the prerequisite #716 (PR #796), which made weight offload capture-compatible by paging weights under a stable virtual address, so flipping the default no longer disables CUDA graph capture for offloaded models.
What "default" now means
--vram-limitis an override, not the trigger.ONNX_GENAI_LEGACY_ALLOCATOR=1(back-compat:ONNX_GENAI_DYNAMIC_KV_WEIGHT_LENDING=0).Observability
The resolved budget, chosen strategy and offload state are printed at startup and exposed in
/v1/resourcesunder a newmemory_strategyblock, so the new default is observable rather than mysterious.Invariants preserved
cuMemCreatestructurally cannot spill (WDDM shared GPU memory lets cuMemAlloc exceed VRAM and the VMM arena cannot: the paths are not comparable on Windows #704).Drop.Measurement (native CUDA, deterministic counters lead)
Ignored live e2e
qwen2_5_0_5b_managed_vmm_default_e2e(Qwen2.5-0.5B int4 mobius, same process, interleaved managed-default vs. legacy opt-out):FullResident--vram-limit 268,435,456)DynamicWeightResidencyONNX_GENAI_LEGACY_ALLOCATOR=1)FullResident--vram-limit 12 GiBIn B, at that extreme synthetic cap the residency arithmetic refused cleanly ("no page is evictable") rather than spilling — the no-spill contract holds.
Caveats (measurement discipline):
qwen14b-zplacksinference_metadata.yaml(Large-model (27B) native E2E offload blocked: Unsqueeze rank bug + missing recurrent_state + mobius#432 #384) so it is not native-loadable. The over-budget condition was synthesized with a small explicit budget, as stated.Validation
cargo fmt -p onnx-genai-engine -p onnx-genai-server -- --checkclean (--all-targets --allfmt hits Windows os error 206 on this box's long worktree paths — a command-line-length artifact, not a formatting issue).-D warnings: engine (native-backend; alsocuda,native-backend), server,onnx-runtime-ep-cuda --features cuda— all clean.cargo test -p onnx-runtime-sessionpasses.memory_strategyunit tests (12, incl. 4 new) and server resources tests pass; cuda-gated engine tests incl. Page offloaded weights under a stable VA so offload keeps CUDA graph capture (#716) #796 capture arms pass.failed_runtime_shrink_preserves_policy_and_ledger_limit(server) fails identically onorigin/main(confirmed via stash); its error is on thehostauthority, untouched by this native-CUDA-only change. Not fixed here (out of scope).Do not merge and do not close the issue via this PR beyond the
Closes #755reference; leaving final disposition to review.