Repository navigation
Page offloaded weights under a stable VA so offload keeps CUDA graph capture (#716) - #796
Merged
Merged
Conversation
…capture (#716) Weight offload previously forced whole-step CUDA graph capture OFF: the pager returned a different device pointer on every page-in (alloc_raw/free_raw), and a captured graph bakes pointers into its recorded nodes, so the two were mutually exclusive. Large models that most need offload therefore got none of the capture-fragmentation wins from #708/#728 (35B-A3B decode 154 -> 34 segments). Give each retained weight key a reserved-once device virtual address and remap physical granules underneath on page-in/eviction, so a captured graph replays correctly across evict->repage. Then remove the offload/capture exclusion, gated on the paging path actually being stable-VA (managed no-spill authority). Mechanism (step 1, proven first, in isolation): - crates/onnx-runtime-cuda-memory/tests/vmm_stable_va_weight_slot_gpu.rs: a captured graph reading from a stable VA still produces correct results after the pages under that VA are swapped for different physical granules holding different weight values. Extends the #727 remap-at-same-VA proof. Stable-VA slot conversion (step 2, weight_paging.rs): - Per-key StableWeightSlot { va, len } reserved once via the existing #740 authority-scoped pool through allocate_committed/carve() suballocation; no second allocator, no per-weight reservation (the #733 mistake), charging stays on the global 0->1 / 1->0 granule-ref attribution. - CudaWeightPage::Drop decommits granules but KEEPS the VA for stable slots (deallocate_span only for transient bypass pages); never asserts in Drop. - Page-ins run entirely under the residency mutex, so eviction-driven decommit is serialized and cannot race a capture replay; capture is capture-once / replay-many with no page-ins during replay. GLOBAL_BYPASSED_PAGE_INS records non-retained page-ins as the engine's decline signal. - Residency GPU test vmm_retained_weight_key_keeps_a_stable_virtual_address_across_repage proves the VA is identical across an evict->repage cycle with one reused granule and no accounting regression. Exclusion removal (step 3): - resolve_graph_capture_enabled takes weight_offload_stable_va; keeps capture OFF only when weight_offload_enabled && !weight_offload_stable_va. Signal is policy.enabled && policy.managed_no_spill, threaded through NativeDecodeCudaOptions from the resolved offload policy. Measurement (step 4): - crates/onnx-genai-engine/tests/stable_va_capture_offload_e2e.rs drives the managed no-spill VMM stable-VA authority and verifies token-identical greedy decode with capture OFF and ON in one process, dumping the deterministic counters. On the 0.5B native model the managed no-spill window is pin-or-fail (weights fit resident, or a step cannot commit), so sustained-paging counters under a fully-successful decode were not obtainable here; the genuinely- offloaded 15.5GB model is not native-loadable (blocked on the #384 load-chain gap). The addressing mechanism itself is proven deterministically by the two GPU tests above. Refs #716, #755, #727, #740, #723. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #796 +/- ##
==========================================
+ Coverage 79.43% 80.02% +0.59%
==========================================
Files 331 331
Lines 137783 137783
Branches 137783 137783
==========================================
+ Hits 109442 110266 +824
+ Misses 23994 23164 -830
- Partials 4347 4353 +6
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Owner
Author
|
Coordinator pre-merge validation on the merged branch (Windows/RTX box, merged
This closes the largest structural defect on the offload lineUntil now weight offload and CUDA graph capture were mutually exclusive: The consequence was backwards: the models that most need offload — those that do The crux is now answered positively in isolation before any integration, which The residency test then shows VA identity preserved across evict -> repage with Removing the exclusion is correctly gated on On the missing large-model measurementReported honestly as not measured, with the reason, rather than estimated. I What was verified end to end is the property that matters most for correctness: What this unblocksThis is the hard prerequisite for #755 (managed no-spill VMM as the default It also removes the limitation #794 ran into an hour ago, where the 14B-scale run Merging. |
🔴 Benchmark Regression DetectedComparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).
Visual flags: Host infoWhat this cannot catch
|
justinchuby
pushed a commit
that referenced
this pull request
Aug 12, 2026
…ering The seq_major_fixed_stride_grows_under_captured_replay_without_recapture test in vmm_kv_layout_residency_gpu.rs failed ~50% of the time when run in isolation on a cold CUDA context (all-zero or partial readbacks with no CUDA error). Root cause is a test-harness ordering defect, not a product defect and not a merge interaction with #796 (which added no cuda-memory source): The baseline zero-fill and the residency readback issued synchronous copies on the legacy default stream, while the captured graph memset ran on a created non-blocking stream. Those two streams are mutually exempt from implicit synchronization, so on a cold context the readback raced the just-launched graph memset. It only appeared to pass because the sibling subtest runs first alphabetically and warms the context. Fix: route every device operation (baseline fill, captured memset, readback, tail write/read) through the same stream so a single cuStreamSynchronize is a total order. No assertion was relaxed; the product property (a captured graph replays unchanged across tail-granule growth at a stable VA — the #782/#794 0-re-captures win) is genuinely demonstrated and now reliable (40/40 in isolation). Also: - Replace cascading result.unwrap() (in commit_and_measure and the growth test) with std::panic::resume_unwind so the first failure reports cleanly instead of panicking again during teardown with 'Any { .. }'. - Add a mismatch_report diagnostic that names the failure mode (never-visible vs torn write) and a cuGraphGetNodes node-count assertion proving the captured graph holds exactly the memset node. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 12, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…#755) (#798) Closes #755. Makes the authority-scoped **managed no-spill VMM** the **default** on the native CUDA path — no flag required — with **automatic weight streaming** when a model exceeds the resolved device budget. Being larger than the budget becomes a supported configuration rather than an error. Builds directly on the prerequisite #716 (PR #796), which made weight offload capture-compatible by paging weights under a stable virtual address, so flipping the default no longer disables CUDA graph capture for offloaded models. ### What "default" now means 1. **Managed VMM is on without a flag.** Native CUDA loads run memory-strategy inference (#752) unconditionally and it *drives* policy: allocation goes through the physical-handle pool (#740) with committed-granule admission (#745), governed workspace planning (#747/#753/#795), transactional growth grants (#748), against one server-owned authority per device (#744). 2. **Over-budget is not an error.** When the resolved budget cannot hold the weights, the runtime auto-enables weight streaming/offload instead of failing. 3. **`--vram-limit` is an override**, not the trigger. 4. **The legacy allocator stays reachable** for one release via `ONNX_GENAI_LEGACY_ALLOCATOR=1` (back-compat: `ONNX_GENAI_DYNAMIC_KV_WEIGHT_LENDING=0`). ### Observability The resolved budget, chosen strategy and offload state are printed at startup and exposed in `/v1/resources` under a new `memory_strategy` block, so the new default is observable rather than mysterious. ### Invariants preserved - **No silent WDDM spill** — committed physical bytes stay within the managed cap; `cuMemCreate` structurally cannot spill (#704). - **Capacity refusal stays a pre-header HTTP 429** (#743); committed-granule admission (#745) and growth grants (#748) unchanged. - **A model that fits does not start paging** under the new default — tested explicitly (the most likely regression). - **Capture stays ON when offload is active on the stable-VA path** (#796 property), pinned by the existing unit tests. - No prefetch re-introduced (#715); no assert inside any `Drop`. ### Measurement (native CUDA, deterministic counters lead) Ignored live e2e `qwen2_5_0_5b_managed_vmm_default_e2e` (Qwen2.5-0.5B int4 mobius, **same process**, interleaved managed-default vs. legacy opt-out): | scenario | strategy | offload | managed_no_spill | resolved budget | committed device bytes | page-ins / evictions | |---|---|---|---|---|---|---| | **A** fits, managed default (no flag) | `FullResident` | false | true | 7,730,940,928 | 381,681,664 | **0 / 0** (does not page) | | **B** synthetic over-budget (`--vram-limit 268,435,456`) | `DynamicWeightResidency` | true | true | 268,435,456 | 90,177,536 (« cap; no spill) | **434 / 432** | | **C** legacy opt-out (`ONNX_GENAI_LEGACY_ALLOCATOR=1`) | `FullResident` | — | **false** | — | — | — | | **D** explicit `--vram-limit 12 GiB` | override applied | — | — | 12,884,901,888 | — | — | In **B**, at that extreme synthetic cap the residency arithmetic **refused cleanly** ("no page is evictable") rather than spilling — the no-spill contract holds. **Caveats (measurement discipline):** - Wall-clock / tok/s is **not** reported: this box ranged 3.9–28 tok/s across identical runs and other agents build concurrently. Deterministic counters only. - A genuine native large-model run that exceeds VRAM was **not measured**: `qwen14b-zp` lacks `inference_metadata.yaml` (#384) so it is not native-loadable. The over-budget condition was **synthesized** with a small explicit budget, as stated. - **CUDA-graph segment count** is not a public counter; capture-ON-under-offload on the stable-VA path is pinned by the #796 unit tests, not re-measured here. ### Validation - `cargo fmt -p onnx-genai-engine -p onnx-genai-server -- --check` clean (`--all-targets --all` fmt hits Windows os error 206 on this box's long worktree paths — a command-line-length artifact, not a formatting issue). - clippy `-D warnings`: engine (`native-backend`; also `cuda,native-backend`), server, `onnx-runtime-ep-cuda --features cuda` — all clean. - Full `cargo test -p onnx-runtime-session` passes. - Engine `memory_strategy` unit tests (12, incl. 4 new) and server resources tests pass; cuda-gated engine tests incl. #796 capture arms pass. - **Pre-existing unrelated failure:** `failed_runtime_shrink_preserves_policy_and_ledger_limit` (server) fails identically on `origin/main` (confirmed via stash); its error is on the `host` authority, untouched by this native-CUDA-only change. Not fixed here (out of scope). Do **not** merge and do **not** close the issue via this PR beyond the `Closes #755` reference; leaving final disposition to review. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
) ## Headline result `qwen2.5-0.5b-q4_0-mobius` does **not** structurally decline native CUDA graph capture on current `main`. The `captures=0` result in #801 was a policy artifact in its test harness: the capture-OFF phase set `ONNX_GENAI_CUDA_GRAPH=0` before the process-wide `RuntimeConfig` `OnceLock` was initialized, then the later capture-ON phase mutated the environment in the same process. That mutation cannot change the cached runtime policy, so predicate `ONNX_GENAI_CUDA_GRAPH` continued to decline capture. With each policy phase isolated in its own process, the real model captures successfully. The forced-growth run measured deterministic counters: - capture ON, head-major: `growth_events=3`, `captures=4`, `invalidations=4`, `replays=39`, no decline reason - capture ON, seq-major: `growth_events=3`, `captures=4`, `invalidations=4`, `replays=39`, no decline reason - capture OFF: `captures=0` now reports `predicate ONNX_GENAI_CUDA_GRAPH declined capture` This also supplies the missing end-to-end reproduction discussed in #801/#782/#797: growth currently triggers re-capture for both layouts. Fixed-stride seq-major removes VMM KV-copy bytes (`688,576 -> 0`) but does not remove the decoder's current growth invalidation. Changing that invalidation is correctness-sensitive because captured launches may bake the capacity geometry; this PR does not change eligibility or invalidation semantics. ## Reporting change - Replaces the decode-level boolean resolution with an invariant-carrying decision: every disabled decision owns a named predicate and reason. - Covers policy, structural, binding-shape, attention-mask, auxiliary-output, and runtime capture-audit declines. - Emits one warning per decoder session and exposes `decline_reason` through CUDA debug stats. - Prints `cuda_graph_decline_reason` beside the existing `cuda_graph` counters in `profile_native`. - Updates the #801 Mobius regression to run ON/OFF phases in separate processes and assert capture ON actually captures. - Adds an anti-regression test that every decode-level decline names its predicate. #796/#798 are reflected in the decision precedence: stable-VA managed no-spill offload does not decline capture; only `weight_offload_enabled && !weight_offload_stable_va` does. ## Validation - `cargo fmt -p onnx-genai-engine -p onnx-genai-bench -- --check` - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` - full `cargo test -p onnx-runtime-session` (153 primary tests; all secondary binaries passed) - targeted engine capture-decision tests (7 passed) - real-model Mobius seq-major growth regression (all three isolated phases passed) Wall-clock is intentionally not compared: the post-rebase real-model run was heavily contended (individual model runs ranged roughly 69-259 seconds). Only deterministic counters are used above. Refs #801, #782, #797, #796, #798. Does not close any issue. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
## What Documentation-only sync of five memory/KV design docs to the merged, executable-test-backed facts about VMM commit geometry, KV layout, weight offload, CUDA-graph capture, and plugin export. No code or tests changed. `MEMORY_ARCHITECTURE.md` is treated as the single source for KV geometry and the audit table; the other four link into it rather than restating (per #791). ## Most misleading claim found `MEMORY_ARCHITECTURE.md` — the document explicitly designated the single source of truth — still asserted the engine *"explicitly invalidates the graph on every KV bucket growth for both layouts"* and that *"the engine does not yet use that fact to retain the captured graph,"* with a table showing seq-major at `captures=4, invalidations=4`. That was falsified by **#811**: conditional invalidation has landed, seq-major fixed-stride growth now **keeps** the captured graph (growth-attributable invalidations **3 → 0**, `captures=1/invalidations=1`, matching the no-growth control), verified by `mobius_seqmajor_growth_parity_native_cuda.rs` including a negative oracle. Because this doc is the single source others link to, the stale claim had the widest blast radius. ## Changes - **MEMORY_ARCHITECTURE.md** — correct the #811 conditional-invalidation prose + table (verified against `cuda.rs` `binding_growth_signature`/`record_growth_decision` and the parity test); add the ORT-backend KV-routing + #762 plugin-export note (verified `allocator.rs:232` `for_session_device` selects by device, not implementation); thread #822 (granule queried), #812 (engine-side consumer gate), and a #800/#802 record correcting this doc's earlier fp16 sizing of Phase-2a scores (verified `attention.rs:368` uses `dtype.element_size()`). - **CUDA_GRAPH_CAPTURE.md** — replace the mutual-exclusion result with #796 (stable VA removes it), #798 (managed no-spill default), #811 (conditional invalidation), and the #804/#807 `captures=0` process-artifact correction. - **WEIGHT_OFFLOAD.md** — correct the "mutually exclusive today" claim (#796) and the north-star status for the managed no-spill default (#798). - **KV_INSERTION_DESIGN.md** (most stale, 2026-07-15) — add a synchronization banner linking to `MEMORY_ARCHITECTURE.md`; note the layout line post-dates the memo, prefix sharing is a landed primitive (#777), and block-table paged attention remains a proposal (verified: no `block_table` in any CUDA attention kernel). - **DESIGN.md** — forward-pointer in the already-superseded §39.5 noting layout (seq-major/token-major, #787/#792) is now the primary residency lever. ## Method Every retained performance claim cites the PR that measured it; measured fact / design intent / not-yet-built are distinguished; superseded designs are labelled, not deleted. The "still open" item (engine commits flat buckets, not the dense live-prefix ranges `kv_commit.rs` computes, so the `layers×2` floor is not yet realised through the engine) is described as in-flight, not delivered. ## Note on #762 The task framing had #762 (EP plugin export) as open; `gh pr view 762` shows it **merged** today. It is described accurately per that ground truth: the plugin-EP mechanism landed for the CPU EP (validated end-to-end), while the CUDA plugin shim compiles but is unvalidated on hardware — so plugin-EP-hosted VMM KV remains in-progress design intent, not a shipped capability. ## Validation Documentation only — `git diff --stat` touches exactly the five `.md` files; working tree otherwise clean. **Not merged; no issue closed.** Refs #736, #762, #776, #777, #783, #787, #791, #792, #794, #796, #797, #798, #801, #802, #804, #807, #811, #812, #822 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 12, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…ative) + #384 metadata generator (#834) ## Summary Demonstrates the KV-cache committed-**physical**-bytes floor end-to-end on **qwen14b** via the native CUDA decode path, comparing **head-major (BNSH)** vs **seq-major (BSNH)** layouts in the same session, swept across context capacities that bracket the 2 MiB-per-stripe granule threshold (#776). Token IDs are **byte-identical** between layouts at every point. This is the first time this line (chased since #797) has been driven through the *engine* on a model whose head stripe actually reaches a full granule (qwen14b @ 8192 ctx: `8192 x 128 x 2 = 2,097,152 B = exactly 2 MiB`), rather than the sub-granule qwen0.5b of #827. ## Headline result — honest negative (per #794 / #812 / #827 precedent) Head-major and seq-major commit **byte-identical** physical KV bytes at every swept capacity: | KV capacity (tokens) | committed bytes (both layouts) | granules | ratio seq/head | |---:|---:|---:|---:| | 1024 | 402,653,184 | 192 | 1.00x | | 2048 | 603,979,776 | 288 | 1.00x | | 4096 | 1,006,632,960 | 480 | 1.00x | | 8192 | 1,811,939,328 | 864 | 1.00x | Expected-if-it-separated: head-major `layers x 2 x kv_heads` = **768 granules (~1.5 GiB)** vs seq-major `layers x 2` = **96 granules (~192 MiB)** — the 8x that never appeared. ## Why it does not separate `growth_events == 0` and `committed_len == capacity` for **both** layouts: the engine commits the full KV bucket **eagerly** on reservation — reservation *is* the commit, so layout cannot change the total. Seq-major's fixed full-context stride *is* active and correct (`max_len == 8192` at every capacity vs head-major `max_len == capacity`, tokens match), which proves the seq-major kernel/layout resolution works — but a fixed stride only makes **growth** free (#797's 0 bytes moved) and preserves the captured graph (#811/#812); it does not shrink an eagerly-committed full bucket. The 768-vs-96 (8x) floor stays a **driver-level** property (`vmm_kv_layout_residency_gpu`) realized only by a **live-prefix commit-on-demand** path, which the engine does not do on initial reservation. The new guardrail assertions (`growth_events==0`, `committed_len==capacity`, `head==seq committed`) will **turn red** the moment a future change makes seq-major commit less — i.e. they are a tripwire for the desired win. Measurement used the in-place **VMM** allocator (`ONNX_GENAI_CUDA_VMM=1`), which reports granule-rounded **physical** bytes; the legacy/governor allocator reports nominal bytes and was deliberately avoided. Capture was declined throughout (`ONNX_GENAI_CUDA_GRAPH=0`); offload/weight-streaming (16.6 GB model vs 7.7 GB budget, #796) was active and did not perturb the KV commit counters. ## Deliverables 1. **`crates/onnx-genai-engine/tests/qwen14b_kv_floor_sweep_native_cuda.rs`** — child-process-isolated sweep harness (mirrors `mobius_seqmajor_growth_parity_native_cuda`), honoring process-frozen config (#804/#807). Self-execs per `(layout, capacity)`; asserts token parity + layout resolution + the deterministic guardrails above. `#[ignore]`, gated on `cuda,native-backend`. 2. **`scripts/gen_inference_metadata.py`** (#384) — emits `inference_metadata.yaml` from `genai_config.json` for any such export. Reproduces the hand-written qwen14b metadata **byte-for-byte**. Turns the one-off local `qwen14b-zp` fix (which unblocked native loading for this whole measurement) into a reproducible artifact; the model file itself stays local (models live outside the repo). 3. **Docs** — `MEMORY_ARCHITECTURE.md` floor-progression table + eager-full-bucket explanation; `MODEL_METADATA.md` #384 generator section. ## Method notes - Seq-major was obtained by patching a **copy** of the model (`kv_layout=1` on all 48 GQA nodes), as #801/#805 did for mobius — the original model directory was never modified. - Reported committed **physical** bytes throughout, never nominal content bytes. - Deterministic counters lead; wall-clock is omitted (this box's identical runs ranged 3.9-28 tok/s) and no number was extrapolated that was not measured. ## Unrelated pre-existing fix (disclosed) `crates/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs`: added a `#[cfg(not(target_arch = "aarch64"))] let _ = dot_kernel;` guard in `borrowed_affine_int4_matmul` to silence a pre-existing x86 unused-variable lint that blocked `-D warnings` (matches sibling functions at 3779/4272/4448). Behavior-neutral; the mobius ignored test still passes (2 passed). ## Validation - `cargo fmt -p onnx-genai-engine -p onnx-runtime-ep-cpu -- --check` (per-package; `--all` hits Windows `os error 206`) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` — pass - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` — pass - `cargo test -p onnx-runtime-session` (full) — pass - `cargo test -p onnx-genai-engine --features cuda,native-backend --test mobius_seqmajor_growth_parity_native_cuda -- --ignored --test-threads=1` (#814 policy) — 2 passed - New sweep harness run twice; all guardrails passed. Refs #797 #787 #827 #384 #777. Do **not** merge; does **not** close any issue. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
## Summary - extend `profile_native --steady` observability with resolved memory strategy, CUDA graph counters/declines, residency activity, and committed VMM physical bytes - document the first genuinely over-budget Qwen 14B native CUDA measurement enabled by #384 - verify #796 and #798 at scale; record an honest weaker-than-historical #723 residency result with #704 no-spill evidence ## Measured result Capture was varied in isolated child processes (`ON -> OFF -> ON`) because `RuntimeConfig` is process-frozen. - **Capture ON:** `captures=2`, `replays=11`, `fallbacks=0`, `invalidations=2` - **Streaming in the same session:** `page_ins=5299`, `hits=8573`, `hit_rate=61.80%`, `evictions=3850`, `bypassed_page_ins=691` - **Default selection:** `DynamicWeightResidency`, `fits_resolved_device_budget=false`, `weight_offload_enabled=true`, `managed_no_spill=true`; no explicit `--vram-limit` - **Physical ceiling:** VMM peak committed physical bytes `6,996,099,072 / 7,730,940,928` (90.49%), zero oversubscription and zero unaccounted committed bytes - **Correctness:** identical greedy token IDs across both runs in every process and across capture ON/OFF - **Negative worth preserving:** the scale workload measured 61.80% hits and 3,850 evictions, not #723's earlier 74.18% / zero-eviction result. No old-policy A/B was run, so the document does not claim a regression ratio. Wall-clock was contended by concurrent GPU work and is intentionally not claimed. References #796, #798, #723, #704, and #384. ## Validation - `cargo fmt -p onnx-genai-bench -- --check` - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` - `cargo test -p onnx-runtime-session` - `cargo test -p onnx-genai-engine --features cuda,native-backend --test mobius_seqmajor_growth_parity_native_cuda -- --ignored --test-threads=1` - `cargo test -p onnx-genai-bench --features bench-native,cuda --bin profile_native` Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…verdict for native batching (#750) (#844) Stage 1 of native batched decode (#750). Per the reprioritization comment, stage 1 is **(1) validate the premise, (2) scope the smallest honest increment, (3) land it or state the staging plainly.** This PR does not deliver batching — it lands the measurement instrument the effort needs and reports the verdict below. **Do not merge / do not close #750.** ## 1. Verdict on the premise: does native batching amortize the weight stream? **CONFIRMED — conditional on a single-fused-forward implementation.** Established by code reading plus the existing #837 counters (GPU is contended by three sibling agents; per the protocol I did the CPU-side work first and did not run a contended headline). **Evidence the weight-residency cache is *not* per-sequence:** - `CudaWeightResidency` (`crates/onnx-runtime-ep-cuda/src/weight_paging.rs`) is owned per-EP/per-device and keyed purely by weight identity: `pages: HashMap<u64, Arc<CudaWeightPage>>`. There is no sequence or batch dimension anywhere in the cache, its policy, or its stable-VA slot map. - The page-in entry point `resident_mapped(key, ...)` / `page_lazy_weight(key, ...)` (`provider.rs`) returns a cache **hit** for a resident key regardless of which row/sequence triggered it. A miss streams once and admits under the LRU/stable-resident budget. - The ORT batched path already exploits exactly this: `Engine::continuous_batch_manager` (`batched.rs:649`) binds `batch_size = N` into **one** `BatchedStaticCacheDecodeSession` run per step (GEMM `M = N` rows). Each weight op executes once per step and streams once for N tokens. The native path simply has no equivalent — it structurally pins batch to 1. **Corroborating counters (#837 baseline, qwen14b-zp, 16 tokens):** `htod_bytes = 65,772,419,072` ⇒ 4.11 GB/token; `page_ins = 5,535` ⇒ ~346 page-ins/step. ~346 distinct weight tensors streamed once per forward is consistent with "one forward per decode step, each offloaded weight paged exactly once." A single-fused batch-N forward keeps ~346 page-ins/step while emitting N tokens ⇒ `htod_bytes/token` and `page_ins/token` fall ~1/B. **The premise FAILS only if** batch-N is implemented as N independent forwards (or N per-sequence graph replays). On this over-budget model (16.65 GB weights vs 7.73 GB budget) the working set is evicted every step, so a per-sequence loop would miss each weight N times per step and amortize nothing. So the premise is a statement about **implementation discipline**: batch must be one fused forward with `M = N`, never a per-sequence loop. **The knee is unlocated.** As B grows, per-step compute grows and KV grows `B × context`; at some B the step becomes compute-bound (or KV+activations shrink the resident weight budget enough to lower `hit_rate`) and scaling flattens. Honest claim: "scales with B up to a knee we have not located," never "batch 8 = 8x." ## 2. Chosen scope, and why the batch-N increment itself is multi-PR The issue's suggested first increment (make the native path report `batch_supported = false`) is **already landed and tested** (#758 — `driver.rs:697-713`, `routes/admin.rs`, server test at `server/tests.rs:1876`). It is not available to re-land. The smallest *structural* increment — uniform-length batch-N (all sequences stepping together, no ragged admission, sidestepping the paged path) — is **genuinely multi-PR** on the native path. `native_decode/cuda.rs` (3,963 lines) assumes batch=1 structurally in at least four coupled places, each of which must change together to keep a valid decode: 1. **Input binding** collapses batch/query-seq to 1 (single-token `[1,1,·]` device bindings, `cuda.rs:574`). 2. **KV cache** is a single-sequence "batch-0 addressing" contiguous/VMM buffer with no row dimension (`cuda.rs:1782`, `ensure_capacity_*`); N sequences need N reservations or a batched KV layout. 3. **CUDA-graph capture** bakes the batch=1 shape; batch-N forces a recapture per batch shape (must be quantified against the `captures > 0 / fallbacks == 0` gate, #796). 4. **Symbolic-axis resolution** pins axis 0 to 1 (`cuda.rs:2371-2432`), rejecting any non-collapsible batch axis. Given the hard constraints (batch-1 byte-identical; capture preserved; `peak_committed < managed_limit` with KV now `B × context`; ledger counters at 0), landing a *correct* batch-N in one PR would be large and risky. Per the explicit guidance ("prefer a smaller increment that lands over a larger one that does not," and "a well-evidenced staging plan is a good stage-1 outcome"), I chose to land the **measurement instrument** now and stage the implementation: - **Stage 2a** (next): batch-N *input binding + fused forward* behind the existing admission contracts, uniform length only, with a batch-1 byte-identical guard test. No KV layout change yet — validate the fused-forward amortization on a 2-row synthetic before touching KV growth. - **Stage 2b**: batched KV (N reservations under the #740 pool, or batched layout) + recapture accounting. - **Stage 2c**: wire `continuous_batch_manager` to a native uniform-batch manager; ragged/continuous batching follows. ## 3. What this PR lands `weight_offload_amortization: emitted_tokens=… htod_bytes_per_token=… page_ins_per_token=…` on both native report paths in `profile_native`. These are the batch-invariant ratios the protocol mandates leading every batch comparison; `profile_native` previously printed only the raw accumulated counters, leaving the ratio to hand-computation and to be masked by wall-clock variance. Pure bench-only reporting — **runtime decode is byte-identical**. Unit test pins the ratio against the #837 baseline (65,772,419,072 / 16 = 4.11 GB/token). **Baseline (from #837, to be re-emitted by the new line once the GPU frees):** | metric | value | |---|---| | htod_bytes_per_token | 4.11 GB (65,772,419,072 / 16) | | page_ins_per_token | ~346 (5,535 / 16) | An "after (batch-N)" row is deliberately absent — this PR lands no batching. Stage 2a will populate it, and the ~1/B drop in these two rows is the pass/fail signal. ## Validation - `cargo fmt -p onnx-genai-bench -- --check` ✓ - `cargo clippy -p onnx-genai-bench --features bench-native,cuda --bin profile_native -- -D warnings` ✓ - `cargo test -p onnx-genai-bench --features bench-native,cuda --bin profile_native` (new `per_emitted_token_divides_counter_by_tokens`) ✓ No memory/workspace-governance code touched, so the mobius parity gate is N/A for this PR. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 12, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…#866) Closes #857 — do not merge yet (opened for review per the issue). ## TL;DR (read this first — the framing changed after #863/#864) This change is **correct, byte-identical, and safe**, and within the managed weight-offload path it recovers the idle full-context KV reservation and cuts weight streaming **1.68×** (`htod_bytes_per_token` 3.94 GB → 2.35 GB, `page_ins_per_token` 372 → 188, `hit_rate` 57% → 78%). But two hardware findings landed while this was in flight and they bound how much it is worth: - **#863:** WDDM pages out *our own* VMM-mapped granules under system-wide pressure, not just `cuMemAlloc`. So `oversubscribed_bytes == 0` / `peak_committed < managed_limit` are **ledger** guarantees, not proof of physical residency. I now measure `nvidia-smi` device-resident alongside the ledger (below) and treat residency as real only where the two agree. - **#864:** the managed *copying* offload path is ~30× slower than the legacy **WDDM zero-copy** path (5.53 vs 0.18 tok/s, n=5/4), because each weight is read exactly once per decode step, so copying it into VRAM pays alloc/map/evict/sync overhead that WDDM avoids. This PR makes a path that trails WDDM trail it by less; it does **not** close the path-level gap. So I am **not manufacturing a win**. I present this as a correct, well-evidenced *within-path* improvement plus an explicit statement (bottom) of where the real leverage now sits. If the reviewer prefers to hold it until the hybrid direction is decided, that is reasonable. ## What this does At load the weight-residency budget was derived once as `resolved_device_budget − kv_bytes_per_token × max_context` and never revisited, statically withholding the **full declared-max-context** KV reservation (1.611 GB on qwen14b-zp, ctx 8192) from weights even for short requests. This lends that reservation to weights and lets **KV growth reclaim it on demand**, still guaranteeing a sequence reaches its declared `max_context`, and **leaves a tunable device headroom** (load-bearing per #863). ## Design (points 1–4) **1. Reclaim path.** Budget/accounting change that unlocks already-built machinery. `reconcile_cuda_offload_budget_after_native_load` withholds only an **initial KV floor** (`kv_capacity_bucket(1, max)`, ~50 MB) instead of the full reservation. Bytes above the floor (less headroom) are lent to the weight cache. As KV grows, `ensure_capacity_seq_major_fixed → prepare_mapped_growth(KvCache)` drives `MappedGrowthAuthority`: it consumes KV's allowance, then unassigned capacity, then **transfers allowance from the weight holder** (`reclaim_priority()=0`) and calls `reclaim_mapped()` (LRU eviction of unreferenced pages). Transactional with rollback, serialized (no deadlock), requester filtered from its own victim list. Weights shed **in step** with KV growth; no refusal path. **2. Hysteresis.** The existing **monotonic KV bucket growth** (power-of-two) gives O(log max_context) reclaim events, each permanently lowering the weight ceiling. The lending high-water mark is a fixed budget set once at load, not re-lent per event. No thrash. **3. Capture.** Budget/limit changes move **no VA and no physical mapping**. Stable-VA weight paging (#716/#796) keeps captured graphs valid. **Measured:** `captures=2, fallbacks=0`. **4. Scope.** Single-sequence only. Serving/admission (#750) out of scope. **Gating.** Only when `managed_no_spill && commits_on_demand && dynamic_kv_weight_lending_enabled()` — the exact condition registering the weight cache as a reclaimable holder. Otherwise the static reservation is kept. Operator override unaffected. The non-override auto budget is recomputed from the elastic `available` (the plan-time `device_budget_bytes` carried the full reservation and would otherwise clamp the budget straight back — the one real trap, fixed). ## Safety: WDDM headroom (question RESOLVED by #863) #863 proved single-process and deterministically that this card pages out our own `cuMemCreate`/`cuMemMap` granules: an arena-mirroring allocator committed and *touched* all of 9,984 MiB on an 8,188 MiB card, but device-resident capped at ~7,942 MiB while host working set hit 9.49 GB — ~2 GB of our granules spilled to host RAM. `cuMemCreate` confers no pin. So this PR **does not lend to the last byte**: it keeps a tunable **device headroom** unlent below the managed limit (`ONNX_GENAI_ELASTIC_LENDING_HEADROOM_BYTES`, **default 512 MiB**), never dropping below the static-reservation baseline (never a regression). The headroom is now load-bearing, not precautionary. **Two nuances (from #863).** (a) The spill is a **latency** effect, transparent to kernel correctness — the driver makes a granule resident before a kernel uses it, so it costs tok/s, not correctness; it is not the #851 illegal address. (b) It is very likely **WDDM-specific**: on TCC / datacenter GPUs `cuMemCreate` should *fail* at the physical limit rather than spill, so the headroom is a WDDM-deployment safeguard and the right default may differ (smaller/zero on TCC, larger under concurrent GPU use). The tunable exists precisely so the default can be set per target. ## Results — qwen14b-zp, ctx 8192, 16 tokens, greedy (byte-identical) All runs solo, `nvidia-smi` verified clear before each (card coordinated with the #851 agent). `nvidia-smi peak` = max device-resident `memory.used` sampled at 300 ms during the run (includes ~229 MiB OS/display baseline). | metric | baseline (main, managed) | elastic 512 MiB (default) | elastic headroom=0 | **legacy WDDM** (`LEGACY_ALLOCATOR=1`) | |---|---:|---:|---:|---:| | htod_bytes_per_token | 3,943,690,240 | **2,349,010,944** | 2,349,010,944 | 0 (zero-copy) | | page_ins_per_token | 372 | 188 | 188 | 0 | | hit_rate | 57.09% | 78.32% | 78.32% | n/a | | evictions | 5,248 | 2,672 | 2,672 | 0 | | weight budget_bytes | 6,116,140,442 | 7,139,550,618 | 7,676,421,530 | — (offload off) | | weight mapped_physical | 5,662,310,400 | 6,406,799,360 | 6,406,799,360 | — | | peak_committed (ledger) | 6,983,516,160 | 7,725,907,968 | 7,725,907,968 | — | | **nvidia-smi peak resident MiB** | — | **7,479** | — | **7,951** | | oversubscribed / underflows | 0 | 0 | 0 | 0 | | throughput tok/s (single, noisy) | — | 0.72–1.42 | — | **7.42** | ### Reading the table honestly - **The byte win is real and byte-identical.** Token IDs identical across every column: `[96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]`. Elastic lending cuts managed-path streaming 1.68×. - **Solo, the lent residency is physically real.** `nvidia-smi` peaked 7,479 MiB while the ledger `peak_committed` was 7,368 MiB (7,725,907,968 B) and `physical_owned` 7,194 MiB — device-resident **tracked** our commitment rather than capping below it. No spill solo (7,368 MiB fits under the 7,959 MiB free with ~590 MiB to spare). The #863 spill is a **contention** effect our ledger cannot see; on this shared box that is the realistic exposure, hence the headroom and this caveat. Precise statement (refined jointly with #863): **#798 no-spill holds *physically* solo under managed_limit** — nvidia-smi tracks our commitment 1:1; **spill is specifically the *system-wide over-commit* case** (a 2nd process, or engine-at-limit + anything else), which is invisible to our ledger because our own committed bytes do not change. - **The right baseline is WDDM, and it wins decisively.** WDDM moves *zero* explicit HtoD bytes (it reads cold weights zero-copy from shared host memory, `nvidia-smi` sitting at the 7,951 MiB physical ceiling) and still runs 7.42 tok/s here vs the managed path's 0.72–1.42 — corroborating #864's 30.7×. **Wall-clock is not a headline** on this box (documented 3.9–28 tok/s spread), but the direction is unambiguous and structural, not noise. - The 512 MiB headroom costs nothing at 16 tokens (resident working set 6.4 GB sits below both ceilings); it caps the ceiling for high-residency runs, bounding our footprint under the contention where #863 spill occurs. ## max_context reachability (the property under risk) New governor test `kv_reclaims_the_full_lent_budget_from_weights_across_bucketed_growth` drives the **production reclaim code** (`prepare_mapped_growth`, allowance transfer, `reclaim_mapped`): weights hold the full budget, KV grows bucket-by-bucket and reclaims its **entire** reservation, reaching full reservation with `oversubscribed_bytes == 0` throughout, weights shedding exactly the reclaimed bytes. The mandatory GPU gate `mobius_seqmajor_growth_parity_native_cuda` passes **solo** with dynamic KV/weight lending active, head-major == seq-major byte-identical, `captures>0`, `fallbacks=0`. ## Where the real leverage sits (per #864) Weights are read **once per decode step**, so within a step there is no reuse to amortize a copy against; the only reuse is *across* steps, which is exactly what a resident cache captures — and what this PR enlarges. But the copying cache pays alloc/map/evict/sync per miss that WDDM's zero-copy reads avoid, so at the whole-path level WDDM dominates for the streamed (cold) fraction. The durable direction is a **hybrid**: keep a resident **hot** set (where a larger budget — this PR — genuinely helps) and serve the **cold** fraction by **zero-copy direct reads** rather than copy-then-discard. Lending more budget to a copying cache has real but bounded value: it helps the resident fraction, not the streamed one. This PR is a correct, safe increment for the resident fraction and an honest data point that the streamed fraction wants a different mechanism, not more copy budget. ## Validation (all local, per issue) - `cargo fmt -p {onnx-genai-engine, onnx-runtime-ep-cuda, onnx-runtime-memory-governor} -- --check` ✓ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✓ - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend` and `--features cuda,native-backend -- -D warnings` ✓ - `cargo test -p onnx-runtime-session` ✓; `cargo test -p onnx-runtime-memory-governor` ✓ (new reclaim test); `cargo test -p onnx-genai-engine --features cuda,native-backend --lib elastic` ✓ (4 tests) - **MANDATORY** `mobius_seqmajor_growth_parity_native_cuda --ignored --test-threads=1`, solo, GPU clear ✓ ## Files - `crates/onnx-runtime-ep-cuda/src/{provider,lib}.rs`: public `dynamic_kv_weight_lending_enabled()`. - `crates/onnx-genai-engine/src/engine/load.rs`: elastic KV floor, tunable headroom high-water mark, non-regression clamp, gating; unit tests. - `crates/onnx-runtime-memory-governor/src/lib.rs`: reclaim-to-full-reservation test. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…tructural cause (#874) (#876) Replaces the stale `27x slower than doing nothing` row with the measured result from #874 and, more importantly, the mechanism behind it. Measured on `qwen14b-zp`, solo with `nvidia-smi` verified empty before every run, byte-identical output: **WDDM 8.09/7.78 tok/s with `htod_bytes_per_token = 0`** (true zero-copy) against **managed streaming 0.11 tok/s**. `main` measured 0.05-0.08 tok/s on the same model minutes earlier, so the end-to-end effect on this box is ~100x. The row now carries the structural cause rather than a bare ratio: each weight is read **exactly once per decode step**, so both paths move the same bytes over the same link, but ours adds a CPU memcpy into pinned staging, a VRAM allocation, a `cuMemMap`, an eviction and a synchronize -- to buy VRAM residency discarded before it is ever re-read. **Copying to VRAM only pays off when the data is re-read from VRAM before eviction, and there is no intra-step reuse.** Also records three things a future reader needs and would otherwise have to rediscover: - **Keeping the no-spill arena while merely disabling offload is not an option.** The physical-handle pool is a hard cap and `cuMemCreate` cannot spill, so with nothing paging the over-budget remainder the load fails outright. - **The `Compatibility` arm has no device budget at all**, so on a host short of RAM an over-budget model can thrash and the governor cannot see system-wide pressure (#863). - **Linux is unchanged and must stay so** -- with no shared-memory fallback an over-budget model simply fails there, so managed streaming competes with 'does not run', not with something faster (#783's lesson about not inheriting a platform-specific conclusion). Supersedes the historical figures the row carried (6.01 tok/s WDDM, 27x behind, the h2d/staging/vram_free/vram_alloc split, capture disabled under offload): capture now runs under offload (#796) and the eviction-policy defect was fixed (#723). Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
#880) ## What this answers (#864) The task: the **11.4 GB/s** zero-copy figure on #864/#877 is an **upper bound** — it was measured with `cuMemcpyDtoD` *from* mapped host memory (a large, perfectly sequential, driver-optimised pull). A real decode kernel does strided, vectorised int4 GEMV reads. **Measure what a real kernel actually achieves reading a weight in place from host-mapped memory vs from VRAM, with outputs verified identical.** Then: does CUDA graph capture work with a host-mapped device pointer? This PR adds **measurement only** — two `#[ignore]`d tests in the `MatMulNBits` kernel module. **No engine behavior changes.** ## Method (only the weight pointer differs) `zerocopy_kernel_host_mapped_vs_vram_bandwidth` launches the **real** int4/fp16 decode GEMV — the exact kernel + launch config the engine selects for `qwen14b-zp` (block-32, fp16 scales, asymmetric zero-points) — twice per size, with **only the `packed` weight pointer differing**: - **VRAM arm:** weights from `cuMemAlloc` (what the managed path streams into). - **Host-mapped arm:** weights from `cuMemHostGetDevicePointer` over a `cuMemHostRegister(DEVICEMAP|READ_ONLY)` host buffer (the zero-copy cold path). Activation, scales, zero-points and output stay resident in VRAM in both arms, so the time delta is purely the weight read. Effective GB/s = `packed_bytes / kernel_time`, best-of-7, CUDA-event timed on the compute stream. Packed size is **swept** to separate L2-cached reads from cold reads (the ~11.9 MB average per-tensor page-in fits this GPU's L2; the real 8.33 GB layer walk reads each weight once, far exceeding L2). Outputs are asserted **bit-identical** between arms. `nvidia-smi --query-compute-apps` verified empty before every run. ## Results (RTX 4060 Laptop, 8 GiB, WDDM; best-of-7, two runs) | packed | N | VRAM GB/s | host GB/s | ratio | outputs | |---:|---:|---:|---:|---:|:--:| | 12 MiB (realistic per-op) | 4915 | 8–26 *(noisy)* | 3.1–3.4 | ~8× (run 1) | identical | | 64 MiB | 26214 | 6.5–34 *(noisy)* | 5.0–5.4 | — | identical | | 128 MiB | 52428 | 62–121 | 5.4–5.7 | ~11–21× | identical | | 256 MiB | 104857 | 95–125 | 5.7–6.1 | ~17–21× | identical | | 384 MiB (cold, ≫L2) | 157286 | 100–112 | 5.6–5.7 | ~18–20× | identical | **The reproducible signal is the host (PCIe-bound) arm: ~5.6 GB/s steady-state, stable across both runs and all sizes ≥64 MiB.** The VRAM arm at small (realistic-op) N is latency-bound and noisy on this WDDM box (#863) — I do **not** pin it; at large N it reaches the DRAM ceiling (~100–125 GB/s), matching #877's 111 GB/s resident figure and validating the harness. ### CUDA graph capture `zerocopy_kernel_capture_with_host_mapped_pointer`: capture **SUPPORTED**; the real GEMV with a host-mapped weight pointer captured into a CUDA graph and **replayed bit-identically** to the eager run (4 replays). So a host-mapped cold-path pointer is compatible with capture (relevant to the `captures > 0`, `fallbacks == 0` gates, #796). ## The number, and the recommendation **A real strided GEMV reads host-mapped weights at ~5.6 GB/s steady-state — about *half* the 11.3 GB/s sequential proxy, not a tenth of it.** - The proxy **was** optimistic: the strided/vectorised access pattern costs ~2× versus the ideal sequential pull. That is a genuine penalty #877's number masked, and **the planning figure for the hybrid's cold path should be revised from 11.3 GB/s down to ~5.6 GB/s.** - But it is **~2×, not the ~10× that would kill the design.** Zero-copy is **not dead.** It moves weights at PCIe rate, correctly (bit-identical every time), and is capture-compatible. - **Cross-check that this isn't an artifact:** ~5.6 GB/s is exactly consistent with the WDDM result on #874. WDDM reaches ~8 tok/s on `qwen14b-zp` by keeping ~7.7 GB resident and reading only the ~0.6 GB over-budget remainder from host RAM per step — 0.6 GB × 8/s ≈ 4.8 GB/s of PCIe traffic, the same order as this measurement. If zero-copy were really 10× worse (~1 GB/s), WDDM could not achieve 8 tok/s. The two numbers corroborate. - **The hot-set prize is confirmed and is the larger lever:** ~8× at the realistic per-op shape (matching the probe's 9.7×), rising toward ~18–20× as the resident read approaches DRAM bandwidth. Maximising the resident fraction remains priority #1, exactly as #877 concluded. **Verdict: build the hybrid.** Both halves are sound — but size the cold path against **~5.6 GB/s**, not 11.3, and treat the resident hot set as the dominant win. ## Honest caveats - **VRAM small-N is noisy** on this WDDM box; the clean signals are the host (PCIe) arm and the large-N VRAM ceiling. I report ranges, not a single VRAM tok-equivalent. - **`READ_ONLY` registration** was used deliberately (weights are immutable) — that is the flag the design would use, so it is the right thing to measure. - **Synthetic packed bytes** (deterministic LCG), not the real blob — but the physical location (host RAM over PCIe vs VRAM) is what governs bandwidth, and correctness is verified by arm-vs-arm bit-identity, not against an oracle. - Single device, WDDM, `can_map_host_memory=1`. ## Validation - `cargo fmt -p onnx-runtime-ep-cuda -- --check` ✅ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✅ (exact CI gate) - `cargo clippy -p onnx-runtime-ep-cuda --features cuda --tests` ✅ **for the changed file** — my new code is lint-clean. The run also surfaces pre-existing `-D warnings` errors in files I did **not** touch (`standard_attention.rs`, `optimizer.rs`, `tests/gqa_shared_prefix_parity_gpu.rs`, `tests/index_share_gpu.rs`); these exist on `origin/main` and are why the CI gate is lib-only / not `--all-targets` (the instructions note `--all-targets` fails here). - Both tests pass `--ignored` on the live device; outputs bit-identical in every arm. Measurement task per #864 — **not** merging, reporting back. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…#880) (#881) I published 11.41 GB/s for host-mapped weight reads on #877 and labelled it an upper bound. #880 measured the real thing and it was a meaningfully loose bound — correcting it here rather than leaving the optimistic number in the single-source doc. **What changed.** #877 used a sequential `cuMemcpyDtoD` *from* mapped host memory. #880 ran the **real strided int4 GEMV** — the exact kernel and launch config the engine selects for `qwen14b-zp` — with **only the weight pointer differing**, activation/scales/zero-points/output resident in both arms. Result: **~5.6 GB/s steady-state** host-mapped against **~100–133 GB/s** resident, outputs **bit-identical at every size**. I reproduced it independently before merging #880. **Size the hybrid's cold path at 5.6 GB/s, not 11.4.** **Why the correction is trustworthy.** It cross-checks against a measurement taken for an entirely different purpose: WDDM reaches ~8 tok/s on `qwen14b-zp` by keeping ~7.7 GB resident and reading the ~0.6 GB over-budget remainder from host each step — ~4.8 GB/s of PCIe traffic, the same order. If zero-copy were really ~1 GB/s, #874's result would be impossible. Two independent paths landing on the same order is much stronger than either alone. **Also recorded:** - **Sweeping the packed size mattered.** The realistic ~12 MiB per-tensor read fits this GPU's L2, so a single-point measurement there compares a *cached* read against a PCIe read and yields a badly wrong ratio. The real 8.33 GB layer walk reads each weight once and vastly exceeds L2, so the ≥64 MiB points are the honest ones. - **CUDA graph capture with a host-mapped weight pointer is SUPPORTED** and replays bit-identically to eager — measured, not assumed. The cold path therefore does not forfeit the `captures > 0` / `fallbacks == 0` gates (#796), which I had flagged as an open design risk. **The design survives.** A ~2× penalty is not the ~10× that would have killed it. Residency remains the dominant lever (~9.7× at the realistic per-op shape, ~15–24× at large N), so the priority order from #877 stands: maximise the resident fraction first, de-copy the remainder second. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…ing amortization (#750) (#884) ## Stage 2a of #750 — batch-N fused forward + weight-streaming amortization (scoped by #844) Refs #750. Continues the staging plan in #844. **Do not merge; #750 stays open.** ### What landed A batch-N **input binding + fused forward** (`NativeDecodeSession::run_fused_batch_prefill`) that isolates the weight-streaming amortization mechanism (#844's premise) from the batch-1-pinned persistent KV bindings, plus the byte-identical guard #844 named and a profiler mode that leads with the deterministic counters. - `run_fused_batch_prefill(&[TokenId]) -> Vec<Vec<f32>>`: one eager `session.run` binding `input_ids`/`attention_mask`/`position_ids` `[N,1]` and an **empty past** `[N, heads, 0, head_dim]`; returns one `[vocab]` logits row per row. **Stateless** — never touches `self.past`, `current_len`, or the CUDA persistent bindings. - `make_empty_input_tensor_batched` (existing `make_empty_input_tensor` now delegates with `batch=1`, **byte-identical**) and `extract_batch_row_logits`. - `from_session_with_io` — public loader taking an explicit `ModelIoSpec` for graphs whose ports can't be disambiguated by shape/dtype. - `profile_native --fused-forward-amortization N,...` — resets the global weight-offload counters around one fused forward per batch size, with a per-run foreign-compute-app contention check (#851). ### Staging correction (well-evidenced, per #844's invitation) A genuine batched **decode step** is **inseparable** from a batched KV layout (stage 2b), so coupled assumptions **#1 (input binding)** and **#2 (KV batch)** cannot be split for any attention decoder: `persistent_state_shapes` pins KV batch axis 0 = 1, the mask is `[1, max_len]`, and ONNX attention couples the batch dim across QKV/mask/past-KV/present-KV. `cuda.rs` is **untouched** in this PR. The escape hatch #844 already anticipated ("validate the fused-forward amortization on a 2-row synthetic before touching KV growth"): the weight-offload residency is **EP-level, keyed purely by weight identity with no batch axis**, so a **stateless fused forward over N independent length-1 rows with an empty past** exercises the genuine batch axis and the weight stream **without any KV growth**. That is exactly what this PR measures. Assumptions #1 and #4 (symbolic-axis) are handled inside this probe; #2 and #3 remain stage 2b. ### Batch-1 byte-identical guard (#844) — PASS `cargo test -p onnx-genai-bench --features bench-native --test fused_batch_prefill` (CPU, synthetic cached decoder): - `native_fused_batch_prefill_row_identical_to_batch_one` — row `i` of a batch-N forward is **bit-for-bit** identical (`f32::to_bits`) to a batch-1 forward of `token_ids[i]`. - `native_fused_batch_prefill_is_stateless` — `current_len` stays 0; a subsequent ordinary `decode` is unaffected. - `native_fused_batch_prefill_rejects_empty`. ### Amortization table — LED BY `htod_bytes_per_token` and `page_ins_per_token` Model `qwen14b-zp` (16.65 GB weights, over-budget), streaming arm `ONNX_GENAI_WEIGHT_OFFLOAD=1` (4 GiB device budget), `ONNX_GENAI_CUDA_GRAPH=1`, RTX 4060 Laptop (8188 MiB). One stateless fused forward per batch size: | batch N | htod_bytes (total) | page_ins (total) | **htod_bytes_per_token** | **page_ins_per_token** | hit_rate | evictions | |--------:|-------------------:|-----------------:|-------------------------:|-----------------------:|---------:|----------:| | 1 | 3,575,949,312 | 301 | **3,575,949,312** | **301.0** | 65.28% | 0 | | 2 | 3,575,949,312 | 301 | **1,787,974,656** | **150.5** | 65.28% | 0 | | 4 | 3,575,949,312 | 301 | **893,987,328** | **75.2** | 65.28% | 0 | | 8 | 3,575,949,312 | 301 | **446,993,664** | **37.6** | 65.28% | 0 | The **totals are batch-invariant** (identical `htod_bytes`, `page_ins`, `hit_rate`, `evictions=0`) and the **per-token figures fall exactly `1/N`** — the precise signature that "amortization happened." One fused forward pages each weight in at most once (residency has no batch axis) and emits N rows. **Why the streaming arm, not the WDDM zero-copy default:** since #874 the Windows default is WDDM zero-copy (`htod_bytes = 0`), so there is no explicit copy for the counters to show shrinking — nothing to *measure* amortize in htod terms. The `ONNX_GENAI_WEIGHT_OFFLOAD=1` streaming arm is the arm where `htod_bytes` is nonzero and its `1/N` collapse is directly observable. The same batching amortizes the WDDM path's in-place PCIe reads identically in kind; the streaming arm is simply where it is instrumented. **Guarding against the #834/#851 error class (a knob producing its own number):** the `ONNX_GENAI_WEIGHT_OFFLOAD` knob sets *whether* weights stream, **not** the amortization ratio. `htod_bytes` is constant across all N; the `1/N` comes solely from dividing a batch-invariant stream by N rows. The knob cannot manufacture that. ### Contention disclosure (#851) A sibling agent's `profile_native` (wt-residency, a second 14B model) intermittently shared the 8 GB GPU. Runs are **labeled contended** by the built-in per-run check. Crucially, **`htod_bytes` and `page_ins` are byte-identical across a fully-contended run, a `batch=1`-solo run, and intermittently-contended runs** — direct empirical proof these counters are contention-invariant **process-local EP accounting**. Only wall-clock `effective_htod_gbps` moved with contention (2.1–3.9 GB/s), which is deliberately **not** headlined. The `batch=1` anchor of the `1/N` claim was captured with `gpu_contention_check: clear`. ### Capture behavior — explicit statement The stateless probe is an **eager** forward and by design **does not engage CUDA-graph capture**; it is not the decode path. The normal batch-1 **decode** path's capture is **provably unchanged** — `cuda.rs` is untouched (0 lines) — so the `captures > 0 && fallbacks == 0` gate (#796) for real decode is unaffected by this PR. Batch-N *decode* capture (a recapture per batch shape, to be **quantified**) is stage 2b/3 work, not exercised here. ### Memory gates `peak/oversubscribed` governance is untouched (empty past ⇒ no KV growth; `evictions=0`, `unaccounted`/`underflows` not perturbed). The mandatory Mobius parity gate applies only to memory/workspace-governance changes; none here. ### Validation - `cargo fmt -p onnx-genai-engine -p onnx-genai-bench -- --check` ✅ - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✅ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✅ - `cargo test -p onnx-genai-engine --features native-backend` ✅ except one **pre-existing** failure, `optional_modality_pipeline_e2e::undeclared_required_audio_input_never_receives_a_fallback` — a debug UB precondition in `onnx-genai-ort/value.rs` (audio pipeline, untouched here). **Reproduced on the clean base with my changes stashed**, so it is not introduced by this PR. - `cargo test -p onnx-runtime-session` ✅ - `cargo test -p onnx-genai-bench --features bench-native --test fused_batch_prefill` ✅ (byte-identical guard) Do not wait for CI; validated locally. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…residency (#887) Two updates so PROGRESS carries the measurements rather than just the verdicts. **Batching stage 2a (#884) is measured.** A stateless batch-N fused forward over length-1 rows with an empty past gives **batch-invariant totals** — `htod_bytes` 3,575,949,312 and `page_ins` 301 identical at N = 1/2/4/8, `evictions` 0 — with **per-token figures falling exactly 1/N**. That is the signature of amortization, and the record now says *why the knob cannot have produced it*: `ONNX_GENAI_WEIGHT_OFFLOAD` controls whether weights stream, not the ratio, so the 1/N comes solely from dividing a batch-invariant stream by the row count. Also records the **staging correction**: assumptions #1 (input binding) and #2 (KV batch) are inseparable for an attention decoder — `persistent_state_shapes` pins KV batch axis 0 = 1, the mask is `[1, max_len]`, and ONNX attention couples the batch dim across QKV/mask/past/present — so stage 2a used the stateless escape hatch #844 had anticipated and left `cuda.rs` untouched, which is what keeps the #796 capture gates provably unaffected. **A rejected approach recorded under item 3, so nobody repeats it.** Byte-aware residency **corrupts decode output** — token identity violated, generation collapsing to 3 tokens — whenever offload actually engages on the managed streaming path, with and without CUDA graph; the default-off path is unaffected. The byte-identical token constraint is what caught it. A rejected design with its measurements attached is far more useful than one carrying only a verdict, and this is the first time today that constraint has fired on a real proposal. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
This was referenced Aug 13, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
…ch-1 baseline (#905) ## #750 stage 2b-impl — scope + on-hardware batch-1 baseline **This is a well-evidenced scoping result, not the full rewrite.** After reading the persistent decode path end-to-end and measuring the batch-1 baseline on hardware, the honest conclusion is: **the smallest *correct* (token-producing) batch-N persistent-binding decode step does not fit in one reviewable PR.** This PR lands the split and the regression anchor that makes each following increment safe to attempt — the same deliverable shape you accepted for stage 1 (#844) and stage 2b (#891), and explicitly invited ("bindings first, mask second… take it"; "a truthful negative result is acceptable and valuable"). New doc: `docs/NATIVE_BATCH_DECODE_2B_IMPL_SCOPING.md` (+ a pointer from `docs/NATIVE_CUDA_DECODE.md`). **No runtime code changed** — so batch-1 is structurally byte-identical and the #796 capture gates are untouched. ### Batch-1 baseline — measured solo (`nvidia-smi` verified empty before the run) `qwen14b-zp`, RTX 4060, `ONNX_GENAI_CUDA_GRAPH=1`, `ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`, 16 tokens, current `main`: | metric | value | |---|---:| | `generated_token_ids` | `[96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]` — **exact** reference match ✓ | | `htod_bytes_per_token` | 2,349,010,944 | | `page_ins_per_token` | 188.0 | | `hit_rate` | 78.32% | | `cuda_graph_measured` | `captures=1 replays=14 fallbacks=0 invalidations=1` → **recapture ≈ 1** (confirms #891) | | `peak_committed_physical_bytes` | 7,725,907,968 `< managed_limit 7,726,753,178` ✓ | | `oversubscribed_bytes` / `ref_underflows` / `byte_underflows` / `unaccounted_committed_bytes` | 0 / 0 / 0 / 0 ✓ | Not headlining wall-clock (`1.72 GB/s`, `23.8 s/token`) — WDDM-paging noise (#863). Per-token bytes differ from #844's 4.11 GB because current `main` runs the elastic-budget (#866) managed path at the 7.73 GB auto budget, not the earlier fixed 4 GiB synthetic; this is the correct current-main anchor. ### Why one PR isn't achievable - The stage 2a/2b guards exercise the **stateless** `session.run` fused-forward (`run_fused_batch_forward`), which already takes `[N,…]`. The path 2b-impl must change is `DecodeCudaState` — **GPU-only, no CPU test seam**. - **No partial guard exists.** A token-producing decode needs *all* of: batched input binding, KV allocation, KV growth/commit, mask, logits read, greedy argmax, and capture-symbol pinning. Drop one → hard-fail or silent row corruption (#892). So increments are validated by batch-1 non-regression until the final one flips batch-N on with a row-identity guard. - The batch-aware **caller** (present N sequences, consume N rows) is the generation driver / `continuous_batch_manager` = **2c, out of scope**. ### Key finding The KV growth **stride** math (`kv_growth_byte_layout`, `copy_kv_prefix_device_to_device*`) is **already batch-general** (block-count based). The real blockers are shape allocation (`persistent_state_shapes` pins axis 0→1), the mask, the input/logits bindings, the **device argmax** (in `onnx-runtime-ep-cuda`, separate clippy gate), the seq-major commit scatter (`live_prefix_ranges` in `onnx-genai-kv`), and **capture-symbol pinning** (`collect_unit_symbols` collapses the batch symbol to 1; it must instead pin to N). Full file/line inventory in the doc. ### Proposed split (each holds batch-1 byte-identical + passes mobius solo) 1. **2b-impl-1** shapes & IO bindings (thread `batch`, constructor-fixed to 1). 2. **2b-impl-2** batched KV growth/commit (+ pure range-math unit tests). 3. **2b-impl-3** batched device argmax (`onnx-runtime-ep-cuda`). 4. **2b-impl-4** batch symbol pinning + entry points + batch-N row-identity guard + measured recapture-per-shape. ### KV-multiplication trade (restated vs #891) `htod/token ∝ 1/N` but with a never-amortizing floor `L × kv_bytes_per_token` and `N_max ≈ 19 @ 2048 ctx`. Under #866's elastic budget, batched KV growth reclaims weight residency, so 2b-impl-4 must report both the 1/N fall **and** the `hit_rate`-driven offset, with a crossover if batched decode ever loses at a reachable N. ### Validation `cargo fmt`/`clippy`/tests are unaffected (docs-only). The baseline was produced by the protocol's own `profile_native` build (`--features bench-native,cuda`), run solo. Not merging; #750 stays open. Refs #750 #844 #884 #891 #796 #798 #866 #892. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 14, 2026
…#931) Third instance of failure mode 1 today, and the first where the "knob" was not something anyone set on purpose. #930 concluded that **capture and weight streaming are mutually exclusive on this build**, from a real decline message naming a real predicate: ``` predicate `weight_offload_enabled && !weight_offload_stable_va` declined capture ``` It was `!stable_va` that declined it. The sweep harness called `load_with_resolved_io`, which passes `cuda_offload_policy: None`, so `weight_offload_stable_va` fell to its deliberately conservative `unwrap_or(false)`. **The harness asserted pointer-instability; the runtime was never asked.** A run on the normal engine path, same box and model, had `captures=2 fallbacks=0` alongside `htod_bytes_per_token = 1,714,132,992`. They coexist — which is exactly what #796 built (stable-VA VMM paging keeps a baked weight pointer valid across evict → repage) and #836 verified at scale. Had it merged it would have been a load-bearing **false negative**, closing the most valuable open question on the #750 line: whether a captured batch-N decode can also stream weights. Adds the general check it implies, which is cheap and would have caught this at the source: **when a subsystem declines, print the inputs to the predicate rather than only its name.** `!stable_va` and "this hardware cannot do it" look identical in a log and mean opposite things. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
pushed a commit
that referenced
this pull request
Aug 14, 2026
… so capture and streaming coexist (2b-impl-4, #750) Correction to the 2b-impl-4 measurement harness. The first sweep loaded the native decode session via `load_with_resolved_io`, which passes `cuda_offload_policy: None`; `load_with_cuda_options_and_io` then leaves `weight_offload_stable_va` at its conservative `false` default (`load.rs:793`), so the CUDA-graph capture predicate `weight_offload_enabled && !weight_offload_stable_va` declined capture by construction whenever offload was on. The reported "capture and weight streaming are mutually exclusive" was therefore a harness artifact of the entry point, not a runtime property. Fix: the sweep now builds the fully governed `Engine::from_dir` and drives `decode_greedy_batch` on `Engine::native_decode_session_mut()`. That path resolves the effective offload policy from the memory-strategy plan (managed no-spill + stable-VA authority), so a captured graph that baked a weight pointer stays valid across evict->repage on the reserved-once VMM arena (#716/#796/#836). Measured on qwen14b-zp with managed streaming forced: capture stays enabled (captures=1, fallbacks=0, invalidations=0) while weights stream (htod ~= 3.0 GB/token at N=1), and the 1/N amortization and the within-sweep #866 byte-hit-rate offset now come from the real captured decode path. - Add `Engine::native_decode_session_mut()` (native-backend) so a harness can drive the governed session's batch-N entry points. - Rewire `run_native_decode_batch_sweep` to construct the governed engine per batch shape instead of the bare loader; drop the ad-hoc `set_weight_residency_budget` override (the governor now owns the budget). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 14, 2026
…n batch-N on (2b-impl-4, #750) (#930) ## What this is Increment **4 of 4** of native batch-N decode (issue #750), stage 2b-impl-4: pin the batch symbol, wire the token writer, turn batch-N on. Increments 1-3 (#913, #920, #924) were byte-identical by construction with `batch` constructor-fixed to `1`. **This is the increment that changes behaviour**, so it is measured rather than asserted. **#750 stays open** for the 2c caller. > **Correction (2nd commit).** The first version of this PR reported that > "capture and weight-streaming are mutually exclusive on this build." **That > was a harness artifact, not a runtime property**, and it has been corrected > in place. The sweep loaded the native session via `load_with_resolved_io`, > which passes `cuda_offload_policy: None`; `load_with_cuda_options_and_io` > then leaves `weight_offload_stable_va` at its conservative `false` default > (`native_decode/load.rs:793`), so the capture predicate > `weight_offload_enabled && !weight_offload_stable_va` declined capture *by > construction* whenever offload was on. The knob I set (the entry point) set > the flag that produced the decline that I read as a finding. The harness now > drives the **fully governed** `Engine` session, whose effective offload > policy carries the managed no-spill stable-VA authority — and **capture and > streaming coexist**, exactly as #796/#836 built and verified. All the > capture/streaming numbers below are re-measured on that governed path. ## What is on and under what flag - **`ONNX_GENAI_NATIVE_DECODE_BATCH`** (default `1`). At the default the decode path is **byte-identical to `main`**. Setting `N>1` binds a stable batch extent at session construction and drives N rows through the persistent captured decode path. - Batch symbol pinned for aux outputs via `persistent_output_shape` (axis-0 → `batch`); KV seq-axis pinning unchanged. - Token writer wired: `write_decode_inputs_batch` (N token IDs → N slots), `write_position_binding_batch` (rank-R mrope per sequence), `extend_mask` (N mask rows). Single-token entry points delegate to the batch writers (byte-identical at batch 1). - `decode_cuda_greedy_batch` / `NativeDecodeSession::decode_greedy_batch` is the batch-N greedy entry point; `native_decode_batch()` exposes the extent. - `Engine::native_decode_session_mut()` (new, native-backend) lets the measurement harness drive batch-N decode against the governed session. - `profile_native --native-decode-batch-sweep <list> [--native-decode-batch-context L]` builds the governed `Engine` per batch shape and exercises the captured decode path. **The batch-N generation driver (real per-sequence prompt seeding) is stage 2c and out of scope here** — the harness fans one identical token to all N rows, which is a strict correctness probe and, because the weight/page/capture counters are content-invariant (#884), the trustworthy measurement vehicle. ## Mandatory gates — all re-run locally after the correction | Gate | Result | |------|--------| | `cargo fmt -p onnx-genai-engine -- --check` (+ `-p onnx-genai-bench`) | **pass** | | `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` | **exit 0** | | `cargo test -p onnx-genai-engine --features native-backend,cuda --lib native_decode` | **96 passed, 0 failed** | | **mobius parity** `--test mobius_seqmajor_growth_parity_native_cuda -- --ignored --nocapture --test-threads=1` | **2 passed** (SOLO) | | **batch-1 byte-identity on hardware** | **exact match** (below) | ### Batch-1 byte-identity with counters Reference token IDs matched **exactly** with the feature compiled in and batch configured to 1 (`qwen14b-zp`, `--steady --tokens 16`): ``` [96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752] ``` Counters matched the documented baseline: `captures=2`, `replays=28`, `fallbacks=0`, `htod_bytes_per_token=2349010944`, `page_ins_per_token=188`, `hit_rate=78.32%`, `byte_hit_rate=70.16%`. ## Correctness of batch-N (row identity) `qwen14b-zp`, all N, every step, on the governed captured+streaming arm: **all N rows == row 0** (the #892 cross-row-corruption detector) and **row 0 == the batch-1 stream** at N=2,4,8 (the scoping-doc §3.8 guard). Token *values* at N>1 are content-arbitrary (real per-sequence seeding is 2c); the counters are not. ## Recapture per shape (#891) — **measured, capture survives batch-N WHILE streaming** `qwen14b-zp`, governed engine, managed streaming forced (`ONNX_GENAI_MANAGED_WEIGHT_STREAMING=1`, `ONNX_GENAI_CUDA_GRAPH=1`), empty past, measured window, per batch shape. **Non-zero htod confirms weights are actually streaming during these captured steps:** | N | graph enabled | captures | replays | fallbacks | invalidations | htod_bytes_per_token | |---|---------------|----------|---------|-----------|---------------|----------------------| | 1 | true | 1 | 6 | 0 | 0 | 2,994,222,848 | | 2 | true | 1 | 6 | 0 | 0 | 1,502,098,688 | | 4 | true | 1 | 6 | 0 | 0 | 736,831,040 | | 8 | true | 1 | 6 | 0 | 0 | 424,642,624 | **Capture survives batch-N as implemented, on the real streaming decode path** — one capture per shape, zero invalidations, zero fallbacks at every N, with ~3 GB/token of weights streaming underneath. This is the positive resolution of #891's open risk the line has been waiting for, and it is the part the original PR got wrong by measuring the wrong arm. ## 1/N weight-stream amortization on the captured decode path — **measured** Same governed captured+streaming arm as above (managed path, #874 note: the WDDM default is the OS shared-memory path with htod=0, so managed streaming is forced). Model `qwen14b-zp` (8.3 GB weights), empty past, `--tokens 8`. Emitted tokens = steps × N. | N | htod_bytes_per_token | page_ins_per_token | abs page_ins/forward | evictions | byte_hit_rate | |---|----------------------|--------------------|----------------------|-----------|---------------| | 1 | 2,994,222,848 | 276.8 | 2214 | 1273 | 61.96% | | 2 | 1,502,098,688 | 139.9 | 2239 | 1293 | 61.83% | | 4 | 736,831,040 | 66.6 | 2132 | 1185 | 62.55% | | 8 | 424,642,624 | 41.6 | 2662 | 1678 | 56.84% | `htod_bytes_per_token` falls **~1/N** (2.99 GB → 1.50 → 0.74 → 0.42; a 7.05× reduction N=1→8 against the ideal 8×). Amortization vs the N=1 ideal extrapolation: N=2 **+0.3%**, N=4 **−1.6%**, N=8 **+13.5%**. The N=8 shortfall is not noise — it is the #866 offset (next section). All four shapes ran SOLO with `nvidia-smi` verified empty; no contention flags. **The amortization is approximate here, not byte-exact.** On this governed managed arm the evictor's working set and page order vary per session load, so the per-forward `abs page_ins` drifts (2214 / 2239 / 2132 / 2662) rather than staying identical. That is the honest behaviour of the *captured streaming* path. For contrast, the earlier *eager* offload arm (offload on, capture declined, fixed `DEVICE_BYTES=6e9`) gave **byte-exact** 1/N with an identical `abs page_ins=1791` at every N — because a fixed budget with no capture is fully deterministic. Both arms agree that the weight stream is paged once per forward and the per-produced-token cost falls ~1/N; the captured arm is the one that matters for #750 and it now measures cleanly. Per the task instruction, **no throughput delta is reported as a finding** (this box ranges 3.9–28 tok/s across identical configs). The deterministic counters above are the evidence. ## The #866 offset — **measured within-sweep on the governed arm** #866 makes the weight budget elastic against occupancy, so batch-N (N× the working set) should shrink the weight residency and push `byte_hit_rate` down. The governed captured arm shows exactly this **within a single sweep**: - N = 1, 2, 4: `byte_hit_rate` ≈ **62%** (61.96 / 61.83 / 62.55, flat within run-to-run noise), evictions ≈ 1200–1300. - N = 8: `byte_hit_rate` drops to **56.84% — a −5.12-point offset**, evictions rise to 1678, and `abs page_ins/forward` rise 2214 → 2662. That offset is precisely why the N=8 per-token htod runs **+13.5% above** the ideal 1/N line: the N× working set ate into the elastic weight budget, so each forward had to re-stream more. This is the #866 offset the line predicted, measured on the real decode path rather than inferred. (The earlier PR could only show it as an *arm-to-arm* budget difference because the un-governed sweep never let the elastic budget respond to batch; the governed engine does.) ## Honest summary of what is and isn't proven - **Proven:** batch symbol pinned; token/mask/position writer wired; batch-N runs end-to-end N=1,2,4,8; batch-1 byte-identity; row identity and cross-batch identity; **capture survives batch-N *while weights stream*** (captures=1, invalidations=0, htod≈3 GB/token) on the governed engine path; ~1/N weight-stream amortization on the captured decode path; the within- sweep #866 byte-hit-rate offset (−5.12 pts at N=8). - **Corrected:** the original "capture and streaming are mutually exclusive" claim was a harness artifact of `load_with_resolved_io`'s conservative `weight_offload_stable_va=false` default; the governed path proves they coexist. - **Named limitation (in the spirit of #920's seq-major `bail!`):** on the governed managed-streaming arm the 1/N amortization is *approximate*, not byte-exact, because the evictor's per-load working set varies and the #866 offset grows the stream at N=8. This is a property of managed streaming, not a batch-N defect. - **Out of scope (2c):** real per-sequence prompt seeding / the batch-N generation driver. The harness fans one identical token to all rows, so token *values* at N>1 are content-arbitrary; the counters are content- invariant (#884) and are the measurement. **Do not merge** — leaving #750 open for the 2c caller. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Weight offload previously forced whole-step CUDA graph capture OFF: the pager returned a different device pointer on every page-in (
alloc_raw/free_raw), and a captured graph bakes pointers into its recorded nodes, so the two were mutually exclusive (resolve_graph_capture_enabledreturnedfalsewhenever offload was on). The models that most need offload — those that do not fit in VRAM — were exactly the ones that lost graph capture and all of the #708/#728 capture-fragmentation wins (35B-A3B decode 154 -> 34 segments).This PR gives each retained weight
keya reserved-once device virtual address and remaps physical granules underneath on page-in/eviction, so a captured graph replays correctly across evict->repage. It then removes the offload/capture mutual exclusion, gated on the paging path actually being stable-VA (the managed no-spill authority).Hard prerequisite for #755 (managed no-spill VMM as the default with automatic weight streaming).
What changed, by step
Step 1 - prove the mechanism in isolation first (
crates/onnx-runtime-cuda-memory/tests/vmm_stable_va_weight_slot_gpu.rs, passes on this HW): a captured graph that reads from a stable VA still produces correct results after the pages under that VA are swapped for different physical granules holding different weight values. Extends the #727 remap-at-same-VA proof to the weight-value-swap case.Step 2 - stable-VA slot conversion (
crates/onnx-runtime-ep-cuda/src/weight_paging.rs):StableWeightSlot { va, len }reserved once through the existing Pool CUDA VMM physical handles across reservations #740 authority-scoped pool viaallocate_committed/carve()suballocation. No second allocator, no per-weight reservation (the mistake that made Back weight offload pages with stable VMM slots #733 net-negative). Charging stays on the global granule-ref 0->1 / 1->0 attribution.CudaWeightPage::Dropdecommits granules but keeps the VA for stable slots (deallocate_spanonly for transient bypass pages). Never asserts inDrop.decommitof any stable slot - run entirely under the residency mutex; capture is capture-once/replay-many with no page-ins during replay; the engine additionally declines capture when a step reports a bypass/eviction (GLOBAL_BYPASSED_PAGE_INS). So nodecommitof a baked VA can occur while a replay is in flight - the case Probe CUDA graph replay across VMM remap #727 explicitly did not prove safe. NocuMemMap/grow inside a captured region.vmm_retained_weight_key_keeps_a_stable_virtual_address_across_repageproves the VA is identical across evict->repage with one reused granule and no accounting regression. The existingvmm_weight_admission_reuses_owned_granules_without_runtime_alloc_freeandweight_offload_gpusuite still pass (no regression).Step 3 - remove the exclusion, gated on stable-VA (
native_decode/{cuda,mod,load}.rs):resolve_graph_capture_enablednow takesweight_offload_stable_vaand keeps capture OFF only whenweight_offload_enabled && !weight_offload_stable_va. The signal ispolicy.enabled && policy.managed_no_spill(the explicit byte--vram-limitauthority that installs the VMM arena + physical pool). The pointer-unstablealloc_raw/free_rawcompatibility path keeps the old capture-off exclusion. #723'sStableResidenteviction policy is unchanged - this changes only the addressing mechanism underneath it.Step 4 - measure (
crates/onnx-genai-engine/tests/stable_va_capture_offload_e2e.rs): drives the managed no-spill VMM stable-VA authority and verifies token-identical greedy decode with capture OFF and ON in one process, dumping the deterministic offload counters (page_ins,hits,evictions,bypassed_page_ins,vram_alloc_ns/vram_free_ns).Honest measurement caveat
The full sustained-paging throughput/counter comparison the issue asks for could not be completed in this environment:
qwen14b-zp, 15.5 GB) is not native-loadable - it has noinference_metadata.yaml(model.ioports), which is the separate Large-model (27B) native E2E offload blocked: Unsqueeze rank bug + missing recurrent_state + mobius#432 #384 load-chain gap. The native decode path (where this change lives) cannot load it.What is verified end-to-end here: under the managed no-spill VMM stable-VA authority, decode with capture OFF and capture ON is byte-identical (32/32 greedy tokens), i.e. capture is transparent and no longer force-disabled. The addressing mechanism itself is proven deterministically by the two GPU tests in steps 1-2. A negative/partial result was an acceptable deliverable per the task; this is a positive mechanism result with a partial end-to-end result.
Validation (local; CI not waited on)
cargo fmt -p onnx-runtime-ep-cuda -p onnx-genai-engine -p onnx-runtime-cuda-memory -- --checkPASScargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warningsPASS (no--all-targets, the CI gate)cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warningsPASS (also linted the new cuda-gated test under--features cuda,native-backend)cargo test -p onnx-runtime-sessionPASS (full suite)vmm_stable_va_weight_slot_gpu,vmm_retained_weight_key_keeps_a_stable_virtual_address_across_repage,vmm_weight_admission_reuses_owned_granules_without_runtime_alloc_free,weight_offload_gpu(5/5)native_decodeunit tests PASS incl. newweight_offload_on_stable_va_path_keeps_graph_captureand updatedweight_offload_forces_graph_capture_offRefs #716, #755, #727, #740, #723. Do not merge; do not close #716.