Repository navigation
Probe: dummy-page mapping primitive for VMM KV cache (#759) - #776
Merged
Merged
Conversation
Self-contained driver-level GPU probe, no engine or KV-path change. Answers the five #759 questions each with a test, extending the proven patterns in vmm_graph_remap_gpu (#727) and vmm_kv_contiguous_tail_gpu (#772): - Q1 dummy_backed_tail_removes_the_padded_read_fault: one dummy handle mapped across the tail turns #772's faulting padded read (and a captured-graph read into the tail) into a successful read. - Q2 dummy_read_is_detectable_not_silent: fills the dummy with a 0xFF sentinel (NaN in f16/bf16/f32), not zeros. Corrects #759: a zero key is softmax-neutral (exp(0)=1), silently wrong; the dummy page is a fault-safety net, not a correctness net. - Q3 vmm_dummy_write_protect_gpu (own binary): a PROT_READ dummy rejects synchronous and asynchronous writes with CUDA_ERROR_INVALID_VALUE and the context survives -- non-sticky, viable defence on this hardware. - Q4 vmm_dummy_page_ledger_gpu: one handle at 96 VAs is one physical page (2 MiB, not 192 MiB) with all VAs aliasing; the real allocator/ledger charges a shared granule once, drops the mapping refcount to zero on free, and releases the handle on teardown. - Q5 dummy_to_real_growth_remap_cost: a 96-object fixed-stride token growth step is ~29 ms (~305 us/object), which is not free. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #776 +/- ##
==========================================
- Coverage 79.93% 79.40% -0.53%
==========================================
Files 331 331
Lines 137646 137646
Branches 137646 137646
==========================================
- Hits 110026 109303 -723
- Misses 23268 23996 +728
+ Partials 4352 4347 -5
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
…alysis Refs #759. Corrects the earlier NaN "sentinel" fill: NaN defeats additive masking (q*NaN=NaN, NaN+(-inf)=NaN, exp(NaN)=NaN), poisoning outputs that masking would have made correct. New rule is masking-conditional: zeros iff the decode kernel's tail masking is verified in the EP; never NaN; crash-safe only when masking is unverified. - Add dummy_fill_and_crossover.rs (CPU): IEEE-754 proof of the fill rule and the fixed-stride+dummy vs bucket-growth crossover derived from real KV geometry (qwen2.5-0.5b crossover 16384 tok = half ctx; qwen14b 8192 tok = full ctx, per-head stride == 1 granule so the dummy saves no memory there). - Drop the NaN detectability (Q2) GPU test; GPU probes now write only an arbitrary 0x5a readback marker, not a production fill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…C==2 MiB) Refs #759. The crossover = granule / (head_dim x elem_bytes), so it scales linearly with the VMM granule and is the design's highest-value lever. This round measures both granularities on-device and prices the finer one. Key finding: on this hardware CU_MEM_ALLOC_GRANULARITY_MINIMUM == RECOMMENDED == 2 MiB, so the hoped-for 64 KiB granule that would collapse the crossover ~32x (into the low hundreds of tokens) is NOT exposed here. The lever cannot be pulled on this box; the design stays at the 8K-16K-token crossover under the 2 MiB granule. - Add vmm_granularity_gpu.rs: reports both granularities; proves the #759 dummy pattern (aliased dummy handle, full-padded read, captured-graph read into the tail) works at the minimum granule including replay; measures per-mapping / prefill-commit / per-growth-step cost and captured-copy read throughput at both granules. - Extend dummy_fill_and_crossover.rs: prove the crossover is model-size independent (layers and kv_heads cancel; verified across a 480x size range and both real models) and publish the crossover table over {MIN,REC} granule x head_dim {64,128} x dtype {fp16,fp8}; fp8/int8 doubles the crossover. The 64 KiB row is labelled counterfactual since this device's minimum measured 2 MiB. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
pushed a commit
that referenced
this pull request
Aug 11, 2026
…ver-dead framing - Extend bench to head_dim<32 (32 B contiguous run) modelling int4/fp8 KV; measured seq/head-major ratio 0.83-1.00 -> no cache-line penalty even at a quarter cache line, closing the strided-read falsifier for quantized KV. - Doc: add #776 finding (2 MiB min==recommended granule; no 64 KiB lever), quantized KV doubles the head-major crossover, seq-major is the sole floor fix and the convergence point for three goals. - Reframe #759 fixed-stride+dummy-tail as a long-context-only optimization; seq-major makes the benefit unconditional; both coexist. - Elevate SS5 to an explicit GO/NO-GO (GO) decision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 11, 2026
🔴 Benchmark Regression DetectedComparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).
Visual flags: Host infoWhat this cannot catch
|
justinchuby
added a commit
that referenced
this pull request
Aug 11, 2026
## Summary A **truth pass** over the memory / offload / KV / capture documentation. Over the past days a series of hardware measurements (RTX 4060 Laptop GPU, Windows/WDDM) overturned several claims still asserted as fact in `docs/`. This PR finds those claims, corrects them against what was measured, and — where a document described an intended design in the present tense as though it were implemented — marks it as a design proposal with its current status. No wrong claim is silently deleted: each is replaced with what was measured and the PR/issue that superseded it, so a reader who remembers the old number can see it was tested and retired. **Docs-only. No code or tests changed.** ## Corrected claims | File | What it said | What is true | Superseded by | |---|---|---|---| | `ORT2.md` §30.4–30.8 | "Offloading doesn't mean stalling — prefetch overlaps with compute"; "If compute time > transfer time: zero visible overhead"; "Overlap compute + transfer: **Always**"; `// Async prefetch next layer's weights (overlaps with current compute)` | Sequential weight prefetch produced no usable overlap and was **deleted**; the bottleneck is disk/transfer bandwidth, not overlap. On a model that doesn't fit there isn't enough independent compute to hide the transfer behind. | PR #715; AirLLM analysis #718 (rates prefetch ~10%) | | `ORT2.md` §30.6 / §30.7 | "30% weights on CPU → prefetch mostly hidden by compute"; `memory.prefetch_layers` presented as an active knob | Estimate, not measured; prefetch removed. Managed offload was ~27× slower than doing nothing (WDDM) on a non-fitting model. | PR #715 | | `DESIGN.md` §39.5 | KV table "**tokens per granule**, 8B GQA (2048 B/token): 2 MiB → 1024"; floor `num_layers × 2 × 2 MiB` | KV is per-binding/head-major: a fixed full-context stride commits **objects × granule**, not tokens/granule. qwen2.5-0.5b commits **96 head-stripes × 2 MiB = 192 MiB for ~12 KiB** live. Crossover is `granule/(head_dim × sizeof(dtype))`, **model-size independent** (~8,192 tok @ head_dim 128/fp16; ~16,384 @ 64/fp16); quantized KV **doubles** it. 2 MiB is min **and** recommended — no finer granule on this device. | PR #776 | | `MEMORY_ARCHITECTURE.md` | h2d attribution 1.7% (already partly recorded); KV geometry, crossover, read-bounded commit, offload/capture exclusion not consolidated | Re-measured h2d transfer **18.8%**. Added the central KV reservation-geometry section: per-object granule floor, the crossover formula, 2 MiB min==recommended, the read-bounded-commit result (read—not reservation—forces the commit; #721 stage-4 32× regression = 1.5 GB vs 48 MB), offload/capture mutual exclusion, and prefetch deletion. | PR #715, #716, #772, #776 | | `MEMORY_ARCHITECTURE.md` (KV page storage row) | "still host-only" without VMM ownership | Device KV paging is owned by the CUDA VMM layer (`CudaVmmAllocator`, #740 pool), with committed-granule admission (#745) and growth grants (#748); `native_decode/cuda.rs` has no `PageTable` consumer and `paged_gqa.rs` is a batch-1 CPU primitive. | #740, #745, #748 | | `WEIGHT_OFFLOAD.md` §7.3, Phase 4 | "Prefetch the next exact/predicted expert wave … while the current wave computes"; Phase 4 predictive prefetch as the plan | Marked as a design proposal whose central premise (prefetch overlap) is measured false so far; plumbing (stream-ordered H2D, fences, double-buffer scheduler) is distinct from prefetch producing a throughput gain. | PR #715, #718 | | `WEIGHT_OFFLOAD.md` §0.4 | Did not mention offload/capture exclusion, the residency-cache defect, or committed-physical budgeting | Added: offload disables capture today (#716 is the fix; #708/#728 = 154→34 segments); residency cache had a **0% hit rate for its whole life**, fixed to 74.18% with evictions 6,286→0 (#723); a nominal 6 GB budget physically consumes **~6.51 GB**. | #716, #723 | | `CUDA_GRAPH_CAPTURE.md` | No mention of offload interaction or VMM remap survival | Added: offload/capture mutual exclusion (#716); a captured graph **survives a VMM remap at the same VA**, writing into the new physical pages — sentinel-proven; unmapping during in-flight replay and `cuMemMap`-during-capture are **not** proven. | #716, #727 | ## Deliberately left unchanged `GQA_KV_MATERIALIZATION_DESIGN.md`, `KV_INSERTION_DESIGN.md`, `DECODE_POOL_RESIDENCY_DESIGN.md`, `NATIVE_CUDA_DECODE.md`, `CUDA_STRATEGY.md`, `native-ort-kv-capacity.md` — checked against the relevant facts and found accurate/self-consistent for their scope (CPU-kernel data movement, CPU decode-pool threading, the native-decode milestone plan, the KV-capacity policy). Correct docs were not restructured, and subject areas outside this pass (CPU EP, quantization formats, CLI, diffusion) were not touched. ## Preventing recurrence Added a documentation-hygiene note to `MEMORY_ARCHITECTURE.md`: a performance claim in docs must cite the measurement that produced it; when measurement overturns a claim, correct it in place and name what superseded it rather than deleting it; and keep measured fact, design intent, and not-yet-built visibly distinct. Refs #715, #716, #718, #721, #723, #727, #740, #745, #748, #772, #776. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 11, 2026
## Seq-major (BSNH) KV layout — native fp16 decode pair (smallest next increment) Implements the owner-approved GO decision from the investigation in **#778**. This is the *narrow* increment: a per-backend KV **layout capability** plus conversion of **only the fused fp16 single-token decode pair**. It does not implement #777 prefix sharing, does not add a per-step transpose bridge, and leaves the other 7+ BNSH kernels untouched. Refs #750, #777, #778. **Do not merge / do not close any issue.** --- ### What changed 1. **`KvLayout` contract tag** (`onnx-genai-metadata`): `KvLayout { HeadMajorBnsh, SeqMajorBsnh }` added to `ModelIoSpec` next to `KvOwnership`, defaulting head-major. This records the per-backend capability — each backend owns its KV buffers and neither reads the other's KV bytes, so ORT stays BNSH and native *may* be BSNH. Committed JSON schema regenerated. 2. **Decode read kernel** (`gqa_decode_fp16.rs`): added a `kv_layout` arg; the KV base/stride now branches — BNSH `stride = head_dim`, BSNH `stride = kv_heads*head_dim`. `MODULE_KEY` bumped `v6 → v7`. 3. **Fused decode-prep write** (`group_query_attention.rs`, f32 kernel + half/bf16 template + macro): added `kv_layout`; the append destination offset branches BNSH `((b*kv_heads+h)*cap+past)*dim` vs BSNH `((b*cap+past)*kv_heads+h)*dim`. Prep module cache keys bumped. 4. **GQA op dispatch**: reads a `kv_layout` node attribute (`0`=BNSH default, `1`=BSNH), threads it to the fused prep write and the fp16 decode read, and **hard-gates** seq-major to that single path. A seq-major cache routed to any *unconverted* prep (unfused append/build/RoPE) or reader (flash / f32 decode / reference / phase2a) returns an actionable error rather than silently mis-indexing. **Design note — declared shape stays BNSH.** The physical element count is identical in both layouts, so present-capacity and workspace sizing are unchanged; only the native path's own kernels observe the physical index mapping. This matches the investigation finding that the physical layout may differ from the declared logical shape **iff only our kernels touch it** (KV never leaves the device on the native path). --- ### Validation (all green) New GPU test **`gqa_seqmajor_parity_gpu::seq_major_decode_is_bit_identical_to_head_major`** drives the real CUDA kernels for one fused fp16 decode step (append + read) and asserts: - **BNSH output ≡ BSNH output, byte-for-byte** (fp16 bits) on identical logical content — this exercises both the strided append write and the strided decode read; any offset error in either layout diverges. - The appended token lands at the **layout-correct slot** in each cache. - The head-major GPU run matches an **independent CPU oracle** (so the two GPU layouts cannot be symmetrically wrong). - Seq-major output **under CUDA-graph capture ≡ eager (capture-off)** output. Existing head-major GQA GPU tests pass unchanged (`gqa_fp16_gpu`, default path byte-for-byte unaffected). Exact CI gates, all passing: - `cargo fmt -p onnx-runtime-ep-cuda -p onnx-genai-metadata -p onnx-genai-engine -- --check` - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` (no `--all-targets`) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` - `cargo test -p onnx-genai-metadata` (incl. regenerated schema-sync test) --- ### Measurements **Committed physical bytes / granule floor (analytic, validated on-hardware by #772/#776).** Floor granule count under the corrected derivation (K and V remain separate bindings, so the unit count is `layers×2×kv_heads` head-major vs `layers×2` seq-major — a `kv_heads`× reduction, **not** to zero): | Model | layers | kv_heads | head_dim | head-major floor | seq-major floor | reduction | |---|---|---|---|---|---|---| | qwen2.5-0.5b | 24 | 2 | 64 | 96 granules → **192 MiB** | 48 → **96 MiB** | 2× | | qwen14b | 48 | 8 | 128 | 768 granules → **1.5 GiB** | 96 → **192 MiB** | 8× | The qwen0.5b head-major figure (96 × 2 MiB = 192 MiB) matches the #772 on-hardware measurement exactly. Seq-major crossover `= granule / (kv_heads × head_dim × sizeof)` → **1,024 tokens** (qwen14b), **8,192** (qwen0.5b). **Boundary-crossing remap burst (measured in #778, reused).** All buffers cross their granule boundary in lockstep, so cost arrives as a spike on one token: | Model | head-major burst | seq-major burst | |---|---|---| | qwen14b | ~49.5 ms | ~4.5 ms (**~11×**) | | qwen2.5-0.5b | ~4.9 ms | ~2.8 ms | (2 MiB granule, min == recommended per #776; ~48–64 µs per granule.) **Decode ms/tok + re-capture count (end-to-end): honestly deferred.** Seq-major is not yet wired through **prefill** — the flash prefill read still assumes BNSH and reads the *same* cache decode does, so a full seq-major generation cannot run until that read is converted (the next increment). What this PR *does* establish toward the "fixed stride removes growth-triggered re-capture" claim: the seq-major append offset is **sequence-length independent** (the stride is `kv_heads*head_dim`, constant), and the decode step is proven capture-safe and bit-identical with capture ON == OFF. The full end-to-end recapture-count and ms/tok comparison lands with the flash-prefill increment. --- ### Honest change inventory (what a full seq-major rollout still needs) - **Every other attention kernel's indexing** — `flash_attention`, `standard_attention`, `compressed_sparse_attention`, the varlen kernels, and the unfused append/build/RoPE prep chain — all still assume BNSH. The **flash prefill read** is the immediate blocker for end-to-end seq-major because it reads the same cache as decode. - **The engine KV mirror** (`read_present_kv`, `seed_prefix`) mirrors device KV to host BNSH byte-identically to ORT; under seq-major it must transpose at that boundary. Left head-major here (default), so mirror parity is unchanged. - **Validation oracle**: no byte-exact native-vs-ORT KV oracle exists; parity already compares **outputs**, so divergent physical layouts do not invalidate it (this is why the new test compares outputs, not KV bytes). - **Declared `present_key`/`present_value` shapes**: kept BNSH; confirmed only our kernels consume device KV on the native path, so the physical mapping can differ from the declared shape without breaking any external consumer. ### bench-seqmajor home The `bench-seqmajor/` microbenchmarks (strided-read + real-VMM remap burst) are the **evidence for #778** and live with that investigation record; they are not duplicated into this implementation PR. They should be folded into a durable home (e.g. a `benches/` target under `onnx-runtime-cuda-memory` / `onnx-runtime-ep-cuda`) when #778 lands, rather than remaining a workspace-excluded side crate. Flagging here so they don't rot. ### Recommendation / smallest next increment Land this layout capability, then convert the **flash prefill read** to be `kv_layout`-aware (the one remaining reader of the shared cache on the decode-eligible path). That unlocks end-to-end seq-major generation and the ms/tok + recapture-count measurement, and is the prerequisite for #777 page-level prefix sharing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
marked this pull request as ready for review
August 11, 2026 17:37
Owner
Author
|
Coordinator validation on the merged branch: merged current Merging so these become executable guards rather than prose in a draft. Two findings here are load-bearing for the whole memory line: The granule lever is closed on this hardware. The crossover derivation is verified, not assumed. The granule-floor unit really is per-head-stripe, so The dummy-page primitive itself works, including under captured-graph replay, and the ledger charges an aliased handle once while keeping it freeable. Its value is not memory saving — it does not reduce the |
justinchuby
pushed a commit
that referenced
this pull request
Aug 11, 2026
…aged Split the quantization claim into two verified directions (both models, fp16/fp8/int4): 1. TRUE/good: tokens-per-granule auto-scales because our quantum is byte-denominated (2 MiB granule) -- qwen14b 10.7/21.3/42.7 at fp16/fp8/int4, no tuning. 2. TRUE/against us: the per-sequence BYTE floor stays a fixed 2 MiB at any dtype, while a paged (token-denominated, block=16) design's waste shrinks -- ~3/1.5/0.75 MiB. So under aggressive KV quantization paged wastes fewer bytes per sequence than token-major. Stated plainly (absolute waste is <=1 partial granule, ~16 MiB at 8 concurrent -- a nuance, not a threat). The claim that survives intact: head-major makes quantization and residency ANTAGONISTIC (crossover = granule/(head_dim*dtype) doubles as bytes halve: 8192/16384/32768 tokens on qwen14b), while token-major makes them INDEPENDENT (no crossover -- floor already one granule per sequence). Also mark hardware-imposed (2 MiB granule, #776 MINIMUM==RECOMMENDED) vs design choices (vLLM block_size; token-major's flat-VA decision) so the comparison is fair. Internal fragmentation now stated in the same units for both designs. TLB remains the primary risk (measured, ratio 1.000 at 6 GiB). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 11, 2026
…sured (#750, #777, #783) (#787) ## Token-major-across-all-layers KV: viability probe A **measurement probe with one open risk settled** — no production kernel converted this round (microbenchmark + one isolating GPU floor test only, per scope). References #750 (multi-request batching), #777 (page-level prefix sharing), #783 (KV layout as a stride descriptor). Builds on the merged #772 head-major floor, #776 granularity probe, and the landed #782 seq-major decode pair / #778 investigation. Device: **RTX 4060 Laptop (sm_89, Ada), 8 GiB**, driver 591.55, CUDA 13. ### The question Under token-major, reading one layer's K strides by the **full per-token KV size** (~192 KB for qwen14b) instead of ~2 KB. Each head's run stays a 256-byte contiguous segment (#778 measured that as fine), so the risk is **not coalescing — it is TLB pressure**: each run lands on a different page. ### Result 1 — TLB risk is cleared (`bench-tokenmajor/`) Stride-only bandwidth sweep (identical bytes/occupancy/run; only the token stride changes via a plane-group parameter). Median GB/s, `ratio` = stride ÷ the 2 KB stride at the same L: | qwen14b L | working set | 2 KB | **192 KB (token-major)** | ratio | |---|---|---|---|---| | 8192 | 1.5 GiB | 206.3 | 206.1 | 0.999 | | 32768 | **6.0 GiB** | 207.1 | **207.1** | **1.000** | At the realistic working sets the 192 KB stride is **dead-flat**. Reads are DRAM-bandwidth-bound (~207 GB/s ≈ 80% of peak) *independent of stride* → **2 MiB device pages cover the 192 KB stride; the token-major read is free.** (`ncu` is not installed on this box, so the stride-only BW curve is the deterministic TLB proxy — every other variable held constant.) ### Result 2 — the floor payoff, measured (`vmm_kv_token_major_floor_gpu.rs`) Through the **real #740 authority-scoped pool via `carve()`** (no second allocator, no per-sequence physical reservation), reporting committed **physical** bytes: > head-major: 768 objects × 2 MiB granule = **1536 MiB** committed for 196608 B of live content. Token-major commits **2 MiB** for the identical content — a **768× reduction**. One token-major reservation commits **one granule per sequence** (stays one granule up to 10 tokens = `floor(2 MiB / 192 KiB)`). ### Implementation cost & #777 (in the doc) - Recommend **`KvLayout` → stride descriptor (#783)**, not a third enum variant — token-major collapses the per-(layer,side) buffer boundary the two existing variants keep; a descriptor shares the kernel edit with seq-major instead of duplicating it. - Per-layer bindings (`engine/model_io.rs` positional vectors) must survive as **views into one reservation** — the real structural work. - #777: a shared prefix is **one contiguous multi-map across all layers** (vs `layers×2` under seq-major). ~**2.56 GiB** saved for a 2,000-token prompt at 8 concurrent on qwen14b, robust to short prefixes (sharing granule floor is per-sequence, not per-stripe). ### Recommendation **GO**, staged: descriptor + binding-views first (validate on the existing output-level parity oracle), decode-pair kernel flip after, then bank #777. The read path is free; the cost is on the binding layer. ### Validation - `cargo fmt -p onnx-runtime-cuda-memory -- --check` ✅ · bench workspace `cargo fmt -- --check` ✅ - `cargo clippy -p onnx-runtime-cuda-memory --features gpu-tests --all-targets -- -D warnings` ✅ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✅ (baseline; unchanged) - Both GPU tests pass; bench runs the full sweep. > Do **not** merge; do **not** close any issue. This closes a real question either way — the answer happens to be positive. --- ### Update: vLLM-paging granularity comparison (section 6, verified) Corrected framing folded in — vLLM's real *per-sequence* allocation quantum is `block_size x all layers`, not the ~32 KB per-layer block: | | Per-sequence quantum | Kernel walks a page table? | Physical mem returnable? | |---|---|---|---| | vLLM paging | **~3 MiB** (16 tok x 48L, qwen14b) | **yes** | no — pool pre-committed | | **token-major + VMM** | **2 MiB** (~10.7 tok) | **no** | **yes** | Verified arithmetic (`granule / kv_bytes_per_token`): qwen14b **10.67** tok/granule, qwen0.5b **170.67**; vLLM = `block_size` (16). Internal fragmentation per sequence: token-major <= one 2 MiB granule (model-independent) vs paged <= one block (scales with model). Honest both ways — finer than vLLM on qwen14b (2 vs 3 MiB), coarser on qwen0.5b (2 vs 0.19 MiB). **Quantized KV doubles token-major tokens/granule (denser) while doubling head-major's crossover (worse)** — the asymmetry reinforces the quantized-KV direction. So on granularity token-major is **equivalent-or-finer than paging** while keeping flat kernels and on-demand release. --- ### Correction: quantization is a split result (verified fp16/fp8/int4, both models) Earlier "quantization is a uniform win for token-major" was half right. Two separate, verified claims: **1 (true, good) — tokens/granule auto-scale** (byte-denominated quantum, the 2 MiB hardware granule), no tuning: | qwen14b | fp16 | fp8 | int4 | |---|---|---|---| | token-major tokens/granule | 10.7 | 21.3 | 42.7 | **2 (true, against us) — the per-sequence BYTE floor stays fixed at 2 MiB**, while a paged (token-denominated, block=16) design's waste shrinks: | worst-case waste/seq | fp16 | fp8 | int4 | |---|---|---|---| | Paged (16 tok) | ~3 MiB | ~1.5 MiB | ~0.75 MiB | | **token-major + VMM** | **2 MiB** | **2 MiB** | **2 MiB** | So **under aggressive KV quantization a token-denominated paged design wastes fewer bytes per sequence than we do** — stated plainly (absolute waste ≤ one partial granule, ~16 MiB at 8 concurrent: a nuance, not a threat). **The claim that survives intact:** head-major makes quantization and residency **antagonistic** (crossover `granule/(head_dim·dtype)` doubles as bytes halve: 8192/16384/32768 tok on qwen14b), while **token-major makes them independent** (no crossover — floor is already one granule/sequence at any dtype). Also now marked **hardware-imposed** (2 MiB granule, #776 `MINIMUM==RECOMMENDED`) vs **design choice** (vLLM `block_size`; token-major's flat-VA decision), and internal fragmentation stated in the same units for both. TLB remains the primary risk — measured, ratio 1.000 at 6 GiB. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 11, 2026
…onesty check (#789) ## Problem **CUDA compile (Linux x86_64) has been failing on `main` since #776** (`237d3654`) and on every commit since (#786 `9e16e83e`, #784 `aaadd971`, …). It is **not** a compiler error — it is the `Verify CUDA test inventory and skip honesty` step (`.github/scripts/verify_cuda_test_honesty.py`): ``` CUDA test honesty check failed: - dummy_fill_and_crossover: 4 tests executed without gpu-tests; CUDA tests must be ignored, not pass - dummy_fill_and_crossover: 4 tests passed with gpu-tests on a no-CUDA host; CUDA tests must fail loud or remain ignored ``` ## Root cause The honesty checker requires every test target under the CUDA crates' `tests/` dirs to be **ignored (or fail loud) on a no-CUDA host** unless it is whitelisted in `ALWAYS_RUN`. #776 added `crates/onnx-runtime-cuda-memory/tests/dummy_fill_and_crossover.rs` — four **pure-CPU** deterministic design probes for the #759 dummy-page VMM KV scheme (safe dummy-fill value via additive-masking algebra; the fixed-stride+dummy vs bucket-growth memory crossover derived from real model KV geometry). They issue **no CUDA calls** and legitimately pass on the CPU lane (the file deliberately omits the `_gpu` suffix its device-bound siblings all carry), but it was never added to `ALWAYS_RUN`. ## Fix Add `dummy_fill_and_crossover` to `ALWAYS_RUN`, with the same documented rationale already carved out for `capture_sync_contract` (a pure-CPU static audit that also legitimately passes on the CPU lane). This is a one-line whitelist plus its justification comment — no test behavior changes. Unblocks the CUDA compile (Linux) lane for every PR based on current main. ## Note Discovered while rebasing #784 (DRY decoder-io refactor, now merged) onto current main. The failure is independent of that refactor — it reproduces on bare `main`. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 11, 2026
…robenchmark (#778) ## Seq-major (BSNH) KV layout — decision-critical investigation **Draft. Do not merge; closes no issue.** Investigation + measurement + two narrow microbenchmarks. No production kernel conversion in this round. Deliverable: `docs/SEQ_MAJOR_KV_INVESTIGATION.md` + standalone GPU benches under `bench-seqmajor/` (workspace-excluded; do not perturb the main build): `seqmajor_bench` (strided read) and `remap_burst` (real-VMM boundary spike). References #750, #759, **#776**, **#777**; builds on #522/#726 and #727/#740/#745. ### Owner corrections folded in (this revision) 1. **Seq-major shrinks the granule floor by `kv_heads`× — it does NOT remove it.** K and V stay separate per-layer bindings (verified: `model_io.rs` per-layer paired `key/value_cache_{inputs,outputs}`), so the floor unit goes `layers·2·kv_heads` → `layers·2` (qwen14b **768 → 96** granules), and the crossover becomes `granule/(kv_heads·head_dim·dtype)`: | model | head-major crossover | **seq-major crossover** | factor | |---|---|---|---| | qwen14b (kv8, hd128) | 8,192 tok | **1,024 tok** | **8×** = kv_heads | | qwen0.5b (kv2, hd64) | 16,384 tok | **8,192 tok** | **2×** = kv_heads | The seq-major crossover is therefore **model-dependent** (scales with kv_heads), not model-independent — big win on wide-GQA, modest on narrow. Token-major (1-granule floor) is *evaluated, not prototyped*: worth a future stride-sweep experiment given §2's cheap strided reads, gated on TLB/locality. 2. **Remap cost is a lockstep BURST, not a per-token tax — measured on HW** (`remap_burst.rs`, real `cuMemCreate`/`cuMemMap`/`cuMemSetAccess`): | scenario | N granules | measured burst | |---|---|---| | qwen14b head-major | 768 | **49.5 ms** | | qwen14b seq-major | 96 | **4.5 ms** | | qwen0.5b head-major | 96 | 4.9 ms | | qwen0.5b seq-major | 48 | 2.8 ms | Per-granule ~48–64 µs (below #776's 150 µs — that folded in first-touch); granule re-confirmed 2 MiB min==recommended. Seq-major cuts the single-token growth spike by `kv_heads`. Fully removable via **commit-ahead** (graph-safe because the VA is fixed — unlike bucket-growth realloc which forces re-capture) and/or **boundary staggering** (achievable; lockstep is not inherent). ### The three questions a go/no-go needs — answered 1. **ORT BSNH crux — definitively NO.** ORT GQA past/present KV is **BNSH-only on every dispatch path** (cuDNN-SDPA, FlashAttention, XQA, memory-efficient). `group_query_attention_helper.h` hardcodes `past_kv_format = Q_K_V_BNSH`; `CheckInputs` rejects `dim[1] != kv_num_heads` with a literal "BNSH Input" error. ⇒ Seq-major must be a **per-backend capability**: native seq-major, ORT BNSH, neither reads the other's KV bytes. Only a layout tag on the already-per-backend KV ownership contract; no unacceptable break. Per-step transpose explicitly rejected (full extra KV pass on the hot path). 2. **Strided-read falsifier — cleared (≤2%).** Fused online-softmax decode microbench, interleaved, sm_89, L ∈ {512…32K}. Seq/head ratio **0.80–1.02** everywhere. The `head_dim×dtype < 128 B` danger case does not materialize — including a **32 B contiguous run** (int4 KV), a quarter cache line: 0.83–1.00. Quantized KV is safe under seq-major. 3. **Prefix-sharing prize (#777, co-primary) — quantified & mechanism-confirmed.** qwen14b 2048-tok × 8 concurrent removes **2.62 GiB**; capacity **3 → 8 at 2 GiB, 10 → 30 at 6 GiB (≈2.7–3.0×)**. Mechanism verified against the **real** allocator/ledger: multi-map proven (#727), charge-once refcount-gated `granule_refs` (#740/#745) is the right shape; gap is 1:N handle bookkeeping (out of scope). Read-only `PROT_READ` shared prefix composes with the #759 dummy-page `cuMemSetAccess` (independent per-subrange descriptors). Head-major impossibility confirmed & quantified (needs P ≥ 8192/16384 tok/head-stripe; seq-major divides by kv_heads). New finding: page sharing has its own granule floor (14B P≥1024, 0.5b P≥8192). ### Context (#776): the granule lever is dead `GRANULARITY_MINIMUM == RECOMMENDED == 2 MiB` (re-confirmed by `remap_burst`) — no 64 KiB granule, so the crossover cannot be shrunk by a finer granule, and quantized KV *doubles* the head-major crossover. Seq-major is thus the only *layout* lever, cutting floor + crossover + growth-spike by `kv_heads`. #759 fixed-stride+dummy-tail is reframed as a long-context hardening layer that coexists with seq-major. ### Change inventory & recommendation: **GO**, staged 7+ BNSH kernels; no byte-exact KV oracle exists (parity already compares outputs); native KV is device-only so physical layout may diverge from the declared BNSH shape. Smallest next increment: add a `KvLayout` tag; convert only the decode pair (GQA append + `gqa_decode_fp16` read); reuse the output-level parity oracle on qwen0.5b; then add 1:N handle bookkeeping to bank #777. Not a big-bang rewrite. A negative result was in scope; this is a positive, staged one. ### Validation `bench-seqmajor` passes `cargo fmt -- --check` and `cargo clippy --release --all-targets -- -D warnings` clean. No existing package source changed, so the repo CI gates (`onnx-runtime-ep-cuda`, `onnx-genai-engine`) are unaffected. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 11, 2026
## Summary - make `docs/MEMORY_ARCHITECTURE.md` the single authoritative explanation of KV layout and residency - consolidate the measured floor progression, crossover formulas, platform granularity, read/remap costs, paging comparison, quantization result, prefix sharing, and per-EP ownership rule - distinguish landed head-major/seq-major behavior from measured token-major work and the proposed stride descriptor - turn the contiguous-VA and seq-major investigation docs into historical pointers so their copied conclusions cannot drift ## Validation - `git diff --check` - verified the commit modifies only Markdown files - documentation-only; no code tests required Refs #750, #759, #772, #776, #777, #778, #782, #783, #787. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 11, 2026
…ng (#777) (#793) ## VMM page-level prefix sharing — isolating GPU probe + design (#777) **Scope:** an isolating GPU probe in `crates/onnx-runtime-cuda-memory` plus a written design. **No KV integration, no engine or production-kernel change.** References #777, and builds on #727 (1-handle-at-2-VAs), #740 (physical-handle pool + 0→1/1→0 granule-ref attribution), #776 (2 MiB granule), #787 (token-major floor), #759 (dummy page). Read `docs/PREFIX_SHARE_INVESTIGATION.md` for the full measured answers, tables, and design; `docs/MEMORY_ARCHITECTURE.md` points to it. ### Result: the primitive is sound enough to build KV integration on — no kill finding All five questions cleared, each with its own GPU test binary (one question per binary so a poisoned context or a process-global counter cannot contaminate another): | Q | Test binary | Finding | |---|---|---| | **Q1** N-way sharing | `vmm_prefix_share_gpu` | One physical granule mapped into **N=8** sequences' reservations reads identically at every VA, including under **captured-graph replay** (the decode hot path). A write through any alias is visible at all — one physical page, not N copies. | | **Q2** ledger charge-once | `vmm_prefix_share_ledger_gpu` | Against the **real** `CudaVmmAllocator` + `LedgerGovernor` (no mock): a shared granule is charged **once**, stays alive until the **last** sharer leaves, and each additional sharer needs **0** incremental owned bytes. Admission arithmetic: **the Nth request costs only its private bytes.** | | **Q3** write protection | `vmm_prefix_share_write_protect_gpu` | A synchronous and an asynchronous write into a `PROT_READ` shared prefix both fault; the fault is **non-sticky** (context survives, later work succeeds) and **non-corrupting** (the other sharer's copy is intact). **The potential kill finding did not fire.** | | **Q4** compose with #759 | `vmm_prefix_share_compose_gpu` | A read-only shared prefix head, a read/write private live granule, and a read-only shared dummy tail **coexist in one reservation** with three distinct `cuMemSetAccess` postures. | | **Q5** copy-on-write | `vmm_prefix_share_gpu` | Boundary COW (copy 2 MiB + remap) is a **one-time burst**: ~**5.5 ms pooled** (#740 retained handle) / ~9.8 ms cold on WDDM — ~22% / ~39% of an assumed 25 ms decode step — never a per-token tax. | ### Concurrency-scaled saving (physical, granule-rounded) Under token-major the whole-model prefix is one contiguous range. Sharing an `S`-token prefix across `C` concurrent requests removes `(C−1)` duplicate copies. Example — qwen14b, **2,048-token** prefix, **8** concurrent: **2,688 MiB (≈2.6 GiB)** removed, matching the ~2.56 GiB figure in `MEMORY_ARCHITECTURE.md`. Full 512/2K/8K × 2/4/8/16 tables for qwen14b and qwen2.5-0.5b are in the doc. **Capacity consequence** (the number that matters): admissible concurrency rises from `floor(B / (prefix + private))` to `1 + floor((B − prefix) / private)`. For a 50/50 prefix/private split this roughly **doubles** concurrency on a fixed budget (e.g. 3.0 GiB KV budget: **4 → 8** requests), and the multiple grows as the shared prefix grows — reproducing the issue's "2 vs 8 concurrent" illustration. ### Design (documented, not implemented) - **Detection:** start with the **explicit pinned-prefix API** (unambiguous, covers the highest-value multi-tenant system-prompt scenario); hash-based detection is a later layer. - **Lifetime:** the **union** of all sharers' lifetimes — exactly the #740 granule refcount, plus an owner reference for a pinned prefix. - **Eviction:** a shared prefix is one unit affecting every sharer at once; the first increment makes a pinned prefix non-evictable for its lifetime. - **Layout:** impractical head-major, workable seq-major, **cleanest token-major** (one multi-map) — a further argument for #783/#787. ### Smallest next increment An explicit `commit_shared_prefix(handle, offset, len)` entry point on the token-major arena that maps a pooled physical handle into a new sequence's reservation at offset 0 and takes a granule-ref on it (owned axis → Nth sharer bills 0). Pinned, read-only, no detection/COW/eviction yet. Validate on the existing output-level parity oracle. The attention kernel and the flat per-sequence VA are unchanged. ### Constraints honoured Never unmap under a replay in flight (divergence syncs first); captures only read already-mapped stable VAs; no assertion inside any `Drop`; physical bytes reported from handle counts / the governor's owned axis, never nominal content bytes. ### Validation (local, this box — RTX 4060 Laptop, 8 GiB, WDDM) - `cargo fmt -p onnx-runtime-cuda-memory -- --check` ✅ - `cargo clippy -p onnx-runtime-cuda-memory --features gpu-tests --all-targets -- -D warnings` ✅ - New GPU tests + existing `vmm_graph_remap_gpu`, `vmm_dummy_page_gpu`, `vmm_kv_token_major_floor_gpu` — **13 tests, all pass**, single-threaded. ✅ **Do not merge, do not close #777.** Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 11, 2026
…capture (#716) (#796) ## Summary Weight offload previously forced whole-step CUDA graph capture **OFF**: the pager returned a different device pointer on every page-in (`alloc_raw`/`free_raw`), and a captured graph bakes pointers into its recorded nodes, so the two were mutually exclusive (`resolve_graph_capture_enabled` returned `false` whenever offload was on). The models that most need offload — those that do not fit in VRAM — were exactly the ones that lost graph capture and all of the #708/#728 capture-fragmentation wins (35B-A3B decode 154 -> 34 segments). This PR gives each retained weight `key` a **reserved-once device virtual address** and remaps physical granules underneath on page-in/eviction, so a captured graph replays correctly across evict->repage. It then removes the offload/capture mutual exclusion, **gated on the paging path actually being stable-VA** (the managed no-spill authority). Hard prerequisite for #755 (managed no-spill VMM as the default with automatic weight streaming). ## What changed, by step **Step 1 - prove the mechanism in isolation first** (`crates/onnx-runtime-cuda-memory/tests/vmm_stable_va_weight_slot_gpu.rs`, **passes on this HW**): a captured graph that reads from a stable VA still produces correct results after the pages under that VA are swapped for different physical granules holding **different weight values**. Extends the #727 remap-at-same-VA proof to the weight-value-swap case. **Step 2 - stable-VA slot conversion** (`crates/onnx-runtime-ep-cuda/src/weight_paging.rs`): - Per-key `StableWeightSlot { va, len }` reserved once through the **existing #740 authority-scoped pool** via `allocate_committed`/`carve()` suballocation. No second allocator, no per-weight reservation (the mistake that made #733 net-negative). Charging stays on the global granule-ref **0->1 / 1->0** attribution. - `CudaWeightPage::Drop` decommits granules but **keeps** the VA for stable slots (`deallocate_span` only for transient bypass pages). Never asserts in `Drop`. - Respects committed-granule admission (#745): reserving VA is free; only committed physical bytes are charged. 2 MiB granule sizing per #776. - **Enforceable ordering rule (not a comment):** page-ins - and thus eviction-driven `decommit` of any stable slot - run entirely under the residency mutex; capture is capture-once/replay-many with no page-ins during replay; the engine additionally declines capture when a step reports a bypass/eviction (`GLOBAL_BYPASSED_PAGE_INS`). So no `decommit` of a baked VA can occur while a replay is in flight - the case #727 explicitly did **not** prove safe. No `cuMemMap`/grow inside a captured region. - Residency GPU test `vmm_retained_weight_key_keeps_a_stable_virtual_address_across_repage` proves the VA is identical across evict->repage with one reused granule and **no accounting regression**. The existing `vmm_weight_admission_reuses_owned_granules_without_runtime_alloc_free` and `weight_offload_gpu` suite still pass (no regression). **Step 3 - remove the exclusion, gated on stable-VA** (`native_decode/{cuda,mod,load}.rs`): `resolve_graph_capture_enabled` now takes `weight_offload_stable_va` and keeps capture OFF only when `weight_offload_enabled && !weight_offload_stable_va`. The signal is `policy.enabled && policy.managed_no_spill` (the explicit byte `--vram-limit` authority that installs the VMM arena + physical pool). The pointer-unstable `alloc_raw`/`free_raw` compatibility path keeps the old capture-off exclusion. **#723's `StableResident` eviction policy is unchanged** - this changes only the addressing mechanism underneath it. **Step 4 - measure** (`crates/onnx-genai-engine/tests/stable_va_capture_offload_e2e.rs`): drives the managed no-spill VMM stable-VA authority and verifies **token-identical** greedy decode with capture OFF and ON in one process, dumping the deterministic offload counters (`page_ins`, `hits`, `evictions`, `bypassed_page_ins`, `vram_alloc_ns`/`vram_free_ns`). ## Honest measurement caveat The full sustained-paging throughput/counter comparison the issue asks for could **not** be completed in this environment: - The genuinely-offloaded model (`qwen14b-zp`, 15.5 GB) is **not native-loadable** - it has no `inference_metadata.yaml` (`model.io` ports), which is the separate #384 load-chain gap. The native decode path (where this change lives) cannot load it. - On the native-loadable 0.5B model, the managed no-spill authority is **pin-or-fail**: either the ~381 MB weight set fits fully resident (0 page-ins) or a step cannot commit its working set (admission/headroom failure after heavy eviction, ~504 evictions observed). There is no clean budget window that both pages sustainedly *and* completes a full decode - that is exactly what #755's automatic weight streaming is for. What **is** verified end-to-end here: under the managed no-spill VMM stable-VA authority, decode with capture OFF and capture ON is **byte-identical** (32/32 greedy tokens), i.e. capture is transparent and no longer force-disabled. The addressing mechanism itself is proven **deterministically** by the two GPU tests in steps 1-2. A negative/partial result was an acceptable deliverable per the task; this is a positive mechanism result with a partial end-to-end result. ## Validation (local; CI not waited on) - `cargo fmt -p onnx-runtime-ep-cuda -p onnx-genai-engine -p onnx-runtime-cuda-memory -- --check` PASS - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` PASS (no `--all-targets`, the CI gate) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` PASS (also linted the new cuda-gated test under `--features cuda,native-backend`) - `cargo test -p onnx-runtime-session` PASS (full suite) - GPU tests single-threaded PASS: `vmm_stable_va_weight_slot_gpu`, `vmm_retained_weight_key_keeps_a_stable_virtual_address_across_repage`, `vmm_weight_admission_reuses_owned_granules_without_runtime_alloc_free`, `weight_offload_gpu` (5/5) - engine `native_decode` unit tests PASS incl. new `weight_offload_on_stable_va_path_keeps_graph_capture` and updated `weight_offload_forces_graph_capture_off` Refs #716, #755, #727, #740, #723. **Do not merge; do not close #716.** Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…T answer (#777) (#822) Makes prefix sharing **layout-general** and answers, definitively and with code citations, whether the ORT backend can use it. Refs #777, #803, #809, #793, #776, #783. ## 1. The shareability decision is now arithmetic, not a layout (core work) Prior framing (mine) said prefix sharing "requires seq-major". That is a **granule-relative** claim stated absolutely, and it was wrong. The real gate: ``` fragment_bytes = prefix_len x (contiguous bytes per fragment in that layout) shareable = fragment_bytes >= granule shareable_granules_per_fragment = floor(fragment_bytes / granule) multi_map_ops = fragments x shareable_granules_per_fragment wasted_boundary_bytes/sequence = fragments x (fragment_bytes mod granule) ``` Layout sets `fragment_bytes` and the **cost**, not the **possibility**. The genuine requirements are (a) a VMM-backed KV buffer and (b) `fragment_bytes >= granule`. New tested predicate `onnx_runtime_memory_governor::shareability::evaluate_prefix_shareability` over `ModelKvGeometry` + `KvFragmentation` (`head_major_bnsh` / `seq_major_bsnh` / `token_major`, or an arbitrary #783 stride descriptor). Its `PrefixShareability` result carries the full accounting and a `refusal_reason()` so a KV path **refuses with a reason** instead of silently making N private copies. `gqa_shared_prefix_parity_gpu` now consults it to admit the share rather than assuming seq-major. Granularity is **queried** (2 MiB CUDA #776, ~64 KiB Level Zero/Vulkan, 4 KiB CPU mmap), never hardcoded. **Head-major becomes shareable when the arithmetic says so** — exactly 8,192 tokens at 2 MiB (`granule / (head_dim x dtype)`), realistic for RAG and long system prompts. ### Correction to the issue's worked table (a term to double-check) The 2 MiB seq-major cell reads "2 granules each". That is `ceil` (granules the prefix *touches* / residency). The **shareable** count is `floor(4,096,000 / 2,097,152) = 1`, giving `96 x 1 = 96` multi-map ops — which matches the design section's own "96 multi-maps per sequence". So the correct cell is **1 granule/fragment, 96 ops**. It is a ceil-vs-floor slip in one cell, not a missing term; every other verified cell matches. The invariant `fragments x contiguous_bytes_per_token = whole-model per-token bytes` holds for all layouts (encoded as a test). Full layout x granule x prefix-length grid for **qwen14b** and **qwen2.5-0.5b** is in `docs/PREFIX_SHARE_INVESTIGATION.md`. ## 2. ORT answer — definitive, with citations The question is not "can ORT share" (the primitive is layout-agnostic) but **"does ORT KV route through an allocator we can back with the VMM arena?"** - **Dynamic decode** (`DecodeKvMode::ZeroCopyRebind`, `decode/mod.rs:66`): ORT allocates `present.*` itself; no allocator of ours is involved. **Cannot route.** - **Shared-buffer decode** (`decode/shared_batch.rs:360`): uses `Value::empty_in` -> `CreateTensorAsOrtValue` with an allocator from `for_session_device` -> `CreateAllocator` (`allocator.rs:239`), which the doc says **"wraps the session's internal EP allocator"** (`allocator.rs:232`) — selects the *device*, not the *implementation*. - **`RegisterAllocator`** (`governed_allocator.rs:738`) installs our vtable but ORT reserves the arena kind for its own arenas (`allocator.rs:65`, `governed_allocator.rs:746`) and fronts a registered device allocator with its **own BFC arena** — each KV tensor is a byte-range inside an arena chunk, not a granule-aligned VMM reservation. **So `commit_shared_prefix` cannot attach to today's automatic ORT paths — a valuable negative.** The one viable route: allocate the shared-buffer KV on **our VMM arena** and bind it as external device memory via `CreateTensorWithDataAsOrtValue` (`value.rs:1103`, "so a caller can hand ORT memory it allocated itself") + `IoBinding`. Bounded by head-major arithmetic (>= 8,192-tok prefixes at 2 MiB). **Scoped, not implemented in this PR.** ## 3. Seam honesty preserved The #809 refusing defaults stay; the predicate supplies the reason so a non-shareable configuration errors rather than mis-maps. Shared regions stay `PROT_READ`, granules charged once, `incremental_owned = 0`, no second allocator. ## Validation (all green locally; CI not awaited, fix-forward) - `cargo fmt -p onnx-runtime-memory-governor -p onnx-runtime-ep-cuda -- --check` - `cargo clippy -p onnx-runtime-cuda-memory --features gpu-tests --all-targets -- -D warnings` - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` - `cargo test -p onnx-runtime-session` (all pass), predicate unit tests (8 pass) - GPU, single-threaded: `vmm_prefix_share_*` (5), `vmm_commit_shared_prefix_gpu` (4), `gqa_shared_prefix_parity_gpu` (1) — all pass - #814 policy: `mobius_seqmajor_growth_parity_native_cuda --ignored` — pass Does **not** merge or close #777. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
## What Documentation-only sync of five memory/KV design docs to the merged, executable-test-backed facts about VMM commit geometry, KV layout, weight offload, CUDA-graph capture, and plugin export. No code or tests changed. `MEMORY_ARCHITECTURE.md` is treated as the single source for KV geometry and the audit table; the other four link into it rather than restating (per #791). ## Most misleading claim found `MEMORY_ARCHITECTURE.md` — the document explicitly designated the single source of truth — still asserted the engine *"explicitly invalidates the graph on every KV bucket growth for both layouts"* and that *"the engine does not yet use that fact to retain the captured graph,"* with a table showing seq-major at `captures=4, invalidations=4`. That was falsified by **#811**: conditional invalidation has landed, seq-major fixed-stride growth now **keeps** the captured graph (growth-attributable invalidations **3 → 0**, `captures=1/invalidations=1`, matching the no-growth control), verified by `mobius_seqmajor_growth_parity_native_cuda.rs` including a negative oracle. Because this doc is the single source others link to, the stale claim had the widest blast radius. ## Changes - **MEMORY_ARCHITECTURE.md** — correct the #811 conditional-invalidation prose + table (verified against `cuda.rs` `binding_growth_signature`/`record_growth_decision` and the parity test); add the ORT-backend KV-routing + #762 plugin-export note (verified `allocator.rs:232` `for_session_device` selects by device, not implementation); thread #822 (granule queried), #812 (engine-side consumer gate), and a #800/#802 record correcting this doc's earlier fp16 sizing of Phase-2a scores (verified `attention.rs:368` uses `dtype.element_size()`). - **CUDA_GRAPH_CAPTURE.md** — replace the mutual-exclusion result with #796 (stable VA removes it), #798 (managed no-spill default), #811 (conditional invalidation), and the #804/#807 `captures=0` process-artifact correction. - **WEIGHT_OFFLOAD.md** — correct the "mutually exclusive today" claim (#796) and the north-star status for the managed no-spill default (#798). - **KV_INSERTION_DESIGN.md** (most stale, 2026-07-15) — add a synchronization banner linking to `MEMORY_ARCHITECTURE.md`; note the layout line post-dates the memo, prefix sharing is a landed primitive (#777), and block-table paged attention remains a proposal (verified: no `block_table` in any CUDA attention kernel). - **DESIGN.md** — forward-pointer in the already-superseded §39.5 noting layout (seq-major/token-major, #787/#792) is now the primary residency lever. ## Method Every retained performance claim cites the PR that measured it; measured fact / design intent / not-yet-built are distinguished; superseded designs are labelled, not deleted. The "still open" item (engine commits flat buckets, not the dense live-prefix ranges `kv_commit.rs` computes, so the `layers×2` floor is not yet realised through the engine) is described as in-flight, not delivered. ## Note on #762 The task framing had #762 (EP plugin export) as open; `gh pr view 762` shows it **merged** today. It is described accurately per that ground truth: the plugin-EP mechanism landed for the CPU EP (validated end-to-end), while the CUDA plugin shim compiles but is unvalidated on hardware — so plugin-EP-hosted VMM KV remains in-progress design intent, not a shipped capability. ## Validation Documentation only — `git diff --stat` touches exactly the five `.md` files; working tree otherwise clean. **Not merged; no issue closed.** Refs #736, #762, #776, #777, #783, #787, #791, #792, #794, #796, #797, #798, #801, #802, #804, #807, #811, #812, #822 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…r); floor is granule-bound on qwen0.5b (#797 #801 #812 #811 #787 #777) (#827) ## Summary Realises the `layers × 2` KV floor **through the engine** by wiring the seq‑major fixed‑stride KV commit path through the shared `kv_commit::live_prefix_ranges` geometry, the last increment in the chain named by #797, #801 and #812. `DecodeCudaState::seq_major_kv_commit_requests` now *consumes* the same `kv_commit.rs` geometry the driver‑level residency test (`vmm_kv_layout_residency_gpu`) and the `kv_commit.rs` unit tests exercise, instead of duplicating the byte arithmetic inline — so the live commit geometry and the measured floor are single‑sourced and cannot drift (previously `kv_commit.rs` carried a `#![cfg_attr(not(test), allow(dead_code))]` because nothing in a non‑test build called it). Head‑major stays **byte‑identical**: on a growing packed bucket its per‑head stride *is* the bucket, so the `kv_heads` live‑prefix fragments tile the same contiguous `[0, bucket_bytes)` run — head‑major's dense ranges equal its bucket ranges. `vmm_growth_requests` is left as the flat bucket commit, now with a doc comment saying exactly that. Refs #797 #801 #812 #811 #787 #777. ## Headline: committed‑bytes comparison `physical_committed_bytes`, process‑isolated, `mobius_seqmajor_growth_parity_native_cuda`, qwen2.5‑0.5b: | Config (vmm‑inplace) | head‑major | seq‑major | |---|---:|---:| | capture ON, forced growth | 100,663,296 | 100,663,296 | | capture OFF, forced growth | 100,663,296 | 100,663,296 | | no‑growth control (`min_bucket=64`) | 100,663,296 | 100,663,296 | | realloc, forced growth | 786,432 | 786,432 | **The floor does not move on this model, and that is the correct result, not an un‑wired commit.** At `head_dim=64`/fp16 with a 512‑token hard max, a single binding's *entire reservation* is `512 × 2 × 64 × 2 = 131,072 B = 128 KiB`, far below the 2 MiB CUDA granule. So **both** layouts pin at exactly one granule per binding: `48 bindings × 2 MiB = 100,663,296 B`, which *is* the `layers × 2 = 48`‑granule floor. Seq‑major already sits on that floor; head‑major coincides because its 64‑token bucket is also sub‑granule. The `kv_heads×` separation (`vmm_kv_layout_residency_gpu`, qwen14b, 32,768‑token stride) only materialises once a **single head's live stripe crosses a granule** — `capacity × head_dim × elem ≥ 2 MiB`, i.e. ≈8,192 tokens at head_dim 128/fp16 (the measured head‑major crossover, #776). A 512‑max, 48‑token qwen0.5b run never approaches it, so #797's driver‑level `kv_heads×` win is **not reproducible end‑to‑end on this harness** — its geometry never leaves a single granule. This is the same class of precise negative result as #794 (identical bytes) and #812, stated with the same precision. ## Full per‑config counters (measured) | Config | growth_events | kv_bytes_moved | captures | replays | invalidations | growth_keeps | |---|---:|---:|---:|---:|---:|---:| | head vmm ON, growth | 3 | 688,576 | 4 | 39 | 4 | 0 | | seq vmm ON, growth | 3 | **0** | 1 | 45 | 1 | **3** | | head vmm ON, no‑growth control | 0 | 0 | 1 | 45 | 1 | 0 | | seq vmm ON, no‑growth control | 0 | 0 | 1 | 45 | 1 | 0 | | head vmm OFF, growth | 3 | 688,576 | 0 | 0 | 4 | 0 | | seq vmm OFF, growth | 3 | 0 | 0 | 0 | 1 | 3 | | head realloc OFF, growth | 3 | 688,576 | 0 | 0 | 4 | 0 | | seq realloc OFF, growth | 3 | 688,576 | 0 | 0 | 4 | 0 | - **Token IDs byte‑identical** across every configuration (`" Paris. It is the largest city…"`, 48 tokens). - Seq‑major keeps its captured graph across all 3 growths (`growth_keeps=3`, forced‑growth graph accounting `1/45/1` collapses onto the no‑growth control) — **the #811 keep decision is preserved**; the dense‑range commit changes neither device pointers nor physical shapes, so the four keep dependencies still hold. - Head‑major still moves 688,576 B and re‑captures per growth (`4/39/4`, `growth_keeps=0`) — unchanged. Wall‑clock intentionally omitted (deterministic counters answer the question; this box's identical runs vary 3.9–28 tok/s). ## Invariants held - Idempotent re‑commit; governor bills only newly mapped granules (via `seq_major_kv_commit_requests` → `commit_binding_ranges_with_mapped_growth`). - #748 `MappedGrowthGrant` growth path, #745 committed‑granule admission, #814 governor claim ordering, pre‑header 429 refusal (#743) — all unchanged. - No map/grow inside a captured region; no unmap under in‑flight replay; **no assert inside any `Drop`**. ## What this unblocks / does not - The dense‑prefix commit is now floor‑faithful and single‑sourced, so the `layers × 2` geometry is realised end‑to‑end **wherever a per‑head stripe exceeds a granule** (large `head_dim`/context). Demonstrating the `kv_heads×` win end‑to‑end needs such a model, **not** a change to this commit path. - It does **not** address the seq‑major seed/host‑mirror stride arithmetic (`read_present_kv` / `seed_prefix`), which still refuse under seq‑major (#812). Prefix sharing under seq‑major therefore still needs the BSNH seed as a separate follow‑on; the doc bullet is corrected to say so. ## Validation (all green, on an RTX 4060 Laptop, ORT 1.27, CUDA 13 wheels) - `cargo fmt -p onnx-genai-engine -- --check` ✔ (`--all-targets` full‑workspace fmt hits Windows os error 206, a command‑length artifact, not a formatting problem) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✔ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✔ - `cargo test -p onnx-runtime-session` ✔ (277 passed) - `cargo test -p onnx-genai-engine --features native-backend --lib` ✔ (454 passed; `kv_commit` 9/9) - **Mandatory** `mobius_seqmajor_growth_parity_native_cuda --ignored --test-threads=1` ✔ (2 passed, incl. the head‑major negative oracle) - `gqa_seqmajor_parity_gpu` ✔ (2), `gqa_shared_prefix_parity_gpu` ✔ (1), `group_query_attention_gpu` ✔ (26), `vmm_kv_contiguous_tail_gpu` ✔ (3), `vmm_kv_layout_residency_gpu` ✔ (2), `vmm_kv_token_major_floor_gpu` ✔ (2) — all `--ignored --test-threads=1`. Do not merge / do not close any issue. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…ative) + #384 metadata generator (#834) ## Summary Demonstrates the KV-cache committed-**physical**-bytes floor end-to-end on **qwen14b** via the native CUDA decode path, comparing **head-major (BNSH)** vs **seq-major (BSNH)** layouts in the same session, swept across context capacities that bracket the 2 MiB-per-stripe granule threshold (#776). Token IDs are **byte-identical** between layouts at every point. This is the first time this line (chased since #797) has been driven through the *engine* on a model whose head stripe actually reaches a full granule (qwen14b @ 8192 ctx: `8192 x 128 x 2 = 2,097,152 B = exactly 2 MiB`), rather than the sub-granule qwen0.5b of #827. ## Headline result — honest negative (per #794 / #812 / #827 precedent) Head-major and seq-major commit **byte-identical** physical KV bytes at every swept capacity: | KV capacity (tokens) | committed bytes (both layouts) | granules | ratio seq/head | |---:|---:|---:|---:| | 1024 | 402,653,184 | 192 | 1.00x | | 2048 | 603,979,776 | 288 | 1.00x | | 4096 | 1,006,632,960 | 480 | 1.00x | | 8192 | 1,811,939,328 | 864 | 1.00x | Expected-if-it-separated: head-major `layers x 2 x kv_heads` = **768 granules (~1.5 GiB)** vs seq-major `layers x 2` = **96 granules (~192 MiB)** — the 8x that never appeared. ## Why it does not separate `growth_events == 0` and `committed_len == capacity` for **both** layouts: the engine commits the full KV bucket **eagerly** on reservation — reservation *is* the commit, so layout cannot change the total. Seq-major's fixed full-context stride *is* active and correct (`max_len == 8192` at every capacity vs head-major `max_len == capacity`, tokens match), which proves the seq-major kernel/layout resolution works — but a fixed stride only makes **growth** free (#797's 0 bytes moved) and preserves the captured graph (#811/#812); it does not shrink an eagerly-committed full bucket. The 768-vs-96 (8x) floor stays a **driver-level** property (`vmm_kv_layout_residency_gpu`) realized only by a **live-prefix commit-on-demand** path, which the engine does not do on initial reservation. The new guardrail assertions (`growth_events==0`, `committed_len==capacity`, `head==seq committed`) will **turn red** the moment a future change makes seq-major commit less — i.e. they are a tripwire for the desired win. Measurement used the in-place **VMM** allocator (`ONNX_GENAI_CUDA_VMM=1`), which reports granule-rounded **physical** bytes; the legacy/governor allocator reports nominal bytes and was deliberately avoided. Capture was declined throughout (`ONNX_GENAI_CUDA_GRAPH=0`); offload/weight-streaming (16.6 GB model vs 7.7 GB budget, #796) was active and did not perturb the KV commit counters. ## Deliverables 1. **`crates/onnx-genai-engine/tests/qwen14b_kv_floor_sweep_native_cuda.rs`** — child-process-isolated sweep harness (mirrors `mobius_seqmajor_growth_parity_native_cuda`), honoring process-frozen config (#804/#807). Self-execs per `(layout, capacity)`; asserts token parity + layout resolution + the deterministic guardrails above. `#[ignore]`, gated on `cuda,native-backend`. 2. **`scripts/gen_inference_metadata.py`** (#384) — emits `inference_metadata.yaml` from `genai_config.json` for any such export. Reproduces the hand-written qwen14b metadata **byte-for-byte**. Turns the one-off local `qwen14b-zp` fix (which unblocked native loading for this whole measurement) into a reproducible artifact; the model file itself stays local (models live outside the repo). 3. **Docs** — `MEMORY_ARCHITECTURE.md` floor-progression table + eager-full-bucket explanation; `MODEL_METADATA.md` #384 generator section. ## Method notes - Seq-major was obtained by patching a **copy** of the model (`kv_layout=1` on all 48 GQA nodes), as #801/#805 did for mobius — the original model directory was never modified. - Reported committed **physical** bytes throughout, never nominal content bytes. - Deterministic counters lead; wall-clock is omitted (this box's identical runs ranged 3.9-28 tok/s) and no number was extrapolated that was not measured. ## Unrelated pre-existing fix (disclosed) `crates/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs`: added a `#[cfg(not(target_arch = "aarch64"))] let _ = dot_kernel;` guard in `borrowed_affine_int4_matmul` to silence a pre-existing x86 unused-variable lint that blocked `-D warnings` (matches sibling functions at 3779/4272/4448). Behavior-neutral; the mobius ignored test still passes (2 passed). ## Validation - `cargo fmt -p onnx-genai-engine -p onnx-runtime-ep-cpu -- --check` (per-package; `--all` hits Windows `os error 206`) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` — pass - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` — pass - `cargo test -p onnx-runtime-session` (full) — pass - `cargo test -p onnx-genai-engine --features cuda,native-backend --test mobius_seqmajor_growth_parity_native_cuda -- --ignored --test-threads=1` (#814 policy) — 2 passed - New sweep harness run twice; all guardrails passed. Refs #797 #787 #827 #384 #777. Do **not** merge; does **not** close any issue. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 18, 2026
…1308) ## What Measures **per-expert MoE weight-paging churn** on the RTX 4060, the "measure map/unmap cost as part of the plumbing, before a policy hides it" deliverable for the MoE expert-weight streaming slice. **Neutral + test-only** — no executor/kernel behaviour changes. - `crates/onnx-runtime-ep-cuda/tests/weight_offload_churn_gpu.rs` — GPU-gated (`gpu-tests`) benchmark that drives the real `CudaWeightResidency` VMM arena (stable-VA slots #716, 2 MiB granule #776) per-expert, using the `granite-3.0-1b-a400m` routing shape (32 experts, top-8) and expert byte sizes spanning the granule. - `docs/benchmarks/2026-08-18-moe-per-expert-paging-churn.md` — records the numbers (house rule §32.2). ## Why Today `bind_block_quantized_moe` pages the whole expert bank as **one key**, so a QMoE run reports whole-bank `reads_per_step ≈ 1.0` (dense-like) and the measured router skew is invisible. Per-expert paging would expose it, but at the price of more, smaller VMM regions — and VMM map/unmap churn is a known binding limiter here. This quantifies that price before the structural executor wiring is attempted. ## Measured (RTX 4060 8 GB, i7-13800H 14C/20T, driver 591.55, CUDA 13.1) - **Per-expert keying is neutral** — one-time page-ins then all hits, ≤0.37 ms/step even at 16 MiB experts. - **Per-expert paging churn is real, bandwidth-dominated** — 3–6 ms/step of pure VMM map/unmap at GLM/DeepSeek-class 16 MiB experts (14.3 ms/step total skewed). - **Skew is a bankable win** — skewed routing cuts page-ins **~46%** vs uniform, consistently across all expert sizes, matching the independently-measured granite top-8 read share. - **2 MiB granule is a hard floor** — sub-granule experts (granite int4 ~0.75 MiB) can't be paged individually and waste 2.7× physical; per-expert VMM paging is a large-expert technique. ## Validation `cargo test -p onnx-runtime-ep-cuda --features gpu-tests --test weight_offload_churn_gpu -- --nocapture` — passes on the 4060 (real VMM map/unmap path). CPU-only runs report it `ignored`. The test does not change any shipping code path. ## Scope The remaining per-expert **executor dispatch** wiring (router top-k → per-expert `resident()` in the decode loop, the deferred #82/#87 seam) is **structural** and reported to the owner separately, per the plumbing-before-policy ordering. This is our own work (streaming slice). No other team's PRs touched. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Probe: the #759 "dummy page mapping" primitive for VMM KV cache
Refs #759. Self-contained GPU probe in
crates/onnx-runtime-cuda-memory— no engine and no KV-path change. Establishes or kills the primitive at the driver level before KV integration, extendingvmm_graph_remap_gpu(#727) andvmm_kv_contiguous_tail_gpu(#772: a padded read one byte into the uncommitted tail faultsCUDA_ERROR_INVALID_VALUE).Do not merge, do not close #759.
Fill value — masking decides it, and it forbids NaN (corrected)
An earlier revision filled the dummy with a NaN "sentinel" for detectability. That is wrong. #721 stage 4's decode kernel reads the full padded shape and relies on additive masking (
score -> -infpast the live length). NaN defeats additive masking —q·NaN = NaN,NaN + (-inf) = NaN,exp(NaN) = NaN— so a NaN dummy poisons the softmax output masking would otherwise have made correct, including the live positions.dummy_fill_and_crossover::masking_determines_the_safe_dummy_fillmeasures the rule (host-side IEEE-754, deterministic):1.0000(tail weight 0)NaN0.8808(tail weightedexp(0)=1)Rule: fill with ZERO once the decode kernel's tail masking is verified effective in the EP; never NaN. If masking is unverified, the dummy page is crash-safe only and the kernel must be bounded. Whether our production decode kernel masks those tail positions is an EP fact this memory-crate probe cannot exercise (no attention kernel here) — it must be verified there before zeros are trusted for correctness. The driver-level tests therefore write only an arbitrary
0x5areadback marker (a test probe, not the production fill).The five questions (measured on this hardware, granule = 2 MiB)
vmm_dummy_page_gpu::dummy_backed_tail_removes_the_padded_read_faultdummy_fill_and_crossover::masking_determines_the_safe_dummy_fillvmm_dummy_write_protect_gpu::read_only_dummy_write_fault_is_non_sticky(own binary)PROT_READdummy rejects a synchronouscuMemcpyHtoD_v2and an asynccuMemsetD8AsyncwithCUDA_ERROR_INVALID_VALUE(validated at enqueue); context survives both, confirmed by a fresh allocate/write/read.vmm_dummy_page_ledger_gpuLedgerGovernorcharge a shared granule once (2 MiB for 128 allocations), drop the mapping refcount to zero on free, release the handle on teardown (governor → 0).vmm_dummy_page_gpu::dummy_to_real_growth_remap_costCrossover is model-size independent — and the granule is its only lever
The crossover between fixed-stride+dummy and bucket growth reduces to a closed form that does not depend on model size:
crossover_is_model_size_independentverifies the cancellation against a deliberately 480× size range (4 layers/1 head vs 120 layers/16 heads — identical crossover) and both real models. So the crossover is a property of the layout unit (head_dim, KV dtype, granule), not of the model. It scales linearly with the granule, making the granule the single highest-value lever.Granularity measured on-device (
vmm_granularity_gpu::report_both_allocation_granularities):The hoped-for finer granule (e.g. 64 KiB, which would collapse the crossover ~32× into the low hundreds of tokens) is not exposed on this box. The lever cannot be pulled here; the design is stuck at the 2 MiB crossover.
granularity_cost_minimum_vs_recommendedconsequently commits and reads identically at both flags (~2.4 ms / 32 MiB region either way, ~105 GiB/s read both) — because both flags return the same granule. On a device that does expose a finer minimum, the same binary would price the trade-off (32× morecuMemMap/cuMemUnmapcalls, TLB pressure).Crossover table (tokens) =
granule / (head_dim × elem_bytes), model-size independent (granularity_is_the_crossover_lever):fp8/int8 KV halves bytes/token, so it doubles the crossover in tokens. At the realizable 2 MiB granule the crossover is 8K–16K tokens — a high bar that loses fixed-stride+dummy against bucket growth in most serving windows.
Per-model crossover at 2 MiB (
fixed_stride_plus_dummy_crossover_vs_bucket_growth, geometry read from eachgenai_config.json):Minimum-granule dummy pattern works (when a finer granule exists)
minimum_granularity_dummy_pattern_works_including_captured_replaybuilds the #759 primitive at whatever the driver reports as MINIMUM (2 MiB here): one dummy handle aliased across 64 tail granules, a full-padded read, and a captured-graph device-to-device read into the dummy tail — all succeed and read back correctly. So if a device exposed a finer granule, the pattern would carry it; the blocker on this box is that no finer granule is offered.What this does not change
The dummy page does not reduce the
objects × granulefloor. #772 measured 96×2 MiB = 192 MiB for ~12 KiB live content (0.5b, head-major fixed stride); those granules hold real, distinct live data. Anyone reading #759 as a fix for the 32× regression is mistaken.Accounting caveat for integration (Q4)
The production commit path maps a distinct handle per granule; a genuine dummy integration maps one handle at N VAs, so
mapped_bytes(VA coverage) would count each dummy VA whiletotal_owned_bytes(physical) counts the one handle once. Charge on the owned/physical axis — which the governor already bills againstTier::Device.Ordering rules relied on (per #727)
cuMemMap/growth inside a captured region.Recommendation
The primitive is sound as a fault-safety net: it removes the tail fault (Q1), its write-protection is non-sticky (Q3), and it charges once and frees cleanly against the real ledger (Q4). But three constraints bound where it is worth building:
Do not build it under a fixed full-context head-major stride (no memory win for 14b; per-token remap cost prohibitive; crossover too high at the only available granule). Build it, if at all, under a bucketed growable path — one contiguous dummy-backed VA per sequence, real granules committed per bucket (remaps rare, only at bucket boundaries), fill = zeros gated on verified masking, dummy
PROT_READ.Smallest next increment: (a) verify in the CUDA EP that the decode attention kernel actually masks positions
>= live_len(this decides zeros-vs-bounded-kernel); (b) prototype a single bucketed dummy-backed KV buffer and measure end-to-end committed bytes and per-token step time against bucket growth, gating on remaps-per-token staying O(few). Re-runvmm_granularity_gpuon any target GPU whose MINIMUM granule is finer than 2 MiB before betting on the fixed-stride variant there.Validation run locally
cargo fmt -p onnx-runtime-cuda-memory -- --check✅cargo clippy -p onnx-runtime-cuda-memory --features gpu-tests --all-targets -- -D warnings✅cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings✅ (exact CI gate)vmm_dummy_page_gpu(Q1, Q5),vmm_dummy_page_ledger_gpu(Q4 ×2),vmm_dummy_write_protect_gpu(Q3),vmm_granularity_gpu(granularity report + min-granule dummy pattern + cost), plus existingvmm_graph_remap_gpu✅. CPU:dummy_fill_and_crossover(masking rule + crossover + model-size-independence + granularity table) ✅. (vmm_kv_contiguous_tail_gpulives on Isolate the contiguous-VA KV crux (VMM per-sequence KV): read pattern forces commit, head-major floor stands #772's branch, not onmain.)