From f330ecdcfea5c1b584b3227e53951cf23d6a260a Mon Sep 17 00:00:00 2001 From: justinchuby <223556219+Copilot@users.noreply.github.com> Date: Tue, 11 Aug 2026 09:10:13 -0700 Subject: [PATCH] docs: correct stale memory/offload/KV claims against measurement Truth pass over the memory/offload/KV/capture docs, correcting claims that hardware measurement has overturned and marking aspirational design text as proposals rather than implemented capability. - ORT2.md 30.4-30.8: prefetch/compute overlap was measured ineffective and removed (PR #715, corroborated by AirLLM #718); h2d transfer corrected from the withdrawn 1.7% to 18.8%. - DESIGN.md 39.5: replace the wrong 'tokens per granule' KV framing with the per-object granule floor and the model-size-independent crossover granule/(head_dim*sizeof(dtype)) (PR #776). - MEMORY_ARCHITECTURE.md: add the central KV reservation-geometry section (facts on granule floor, crossover, 2 MiB min==recommended, read-bounded commit #772, offload/capture mutual exclusion #716, prefetch deletion #715); augment KV-page-storage row with VMM device ownership (#740/#745/#748); add a documentation-hygiene note on citing measurements. - WEIGHT_OFFLOAD.md: mark prefetch as superseded design (#715/#718); add offload/capture mutual exclusion (#716), residency-cache 0%-hit fix (#723), and committed-physical-bytes budgeting. - CUDA_GRAPH_CAPTURE.md: add offload/capture mutual exclusion (#716) and captured-graph VMM remap survival (#727). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/CUDA_GRAPH_CAPTURE.md | 25 ++++++++++ docs/DESIGN.md | 35 ++++++++++---- docs/MEMORY_ARCHITECTURE.md | 94 ++++++++++++++++++++++++++++++++++++- docs/ORT2.md | 51 ++++++++++++++++---- docs/WEIGHT_OFFLOAD.md | 46 ++++++++++++++++-- 5 files changed, 229 insertions(+), 22 deletions(-) diff --git a/docs/CUDA_GRAPH_CAPTURE.md b/docs/CUDA_GRAPH_CAPTURE.md index 4fc9a5c151..bddd4af335 100644 --- a/docs/CUDA_GRAPH_CAPTURE.md +++ b/docs/CUDA_GRAPH_CAPTURE.md @@ -55,3 +55,28 @@ hybrids decline through the same structural gate until their capture lane lands. The process-global counter `NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES` increments once per captured routed decode step, providing a non-vacuous capture-hit signal for tests. + +## Interaction with weight offload and VMM remap + +Two measured facts constrain how capture composes with the memory system; the +full analysis lives in +[`MEMORY_ARCHITECTURE.md`](./MEMORY_ARCHITECTURE.md). + +- **Weight offload and CUDA graph capture are mutually exclusive today.** The + weight pager's alloc/copy/free operations are capture-illegal, so enabling + offload disables capture (module docs in + `crates/onnx-genai-engine/src/native_decode/cuda.rs`). A model large enough to + need offload therefore gets **none** of the capture-fragmentation wins that + #708 and #728 landed (they took 35B-A3B decode from **154 to 34** graph + segments). Issue **#716** is the fix: page weights under a stable virtual + address so page-in remaps physical granules at the same VA instead of returning + a new pointer that would invalidate the capture. +- **A captured graph survives a VMM remap at the same virtual address (#727).** + A graph instantiated before `cuMemUnmap`/`cuMemCreate`/`cuMemMap` at the same + VA replays correctly afterwards and writes into the **new** physical pages — + sentinel-proven in + `crates/onnx-runtime-cuda-memory/tests/vmm_graph_remap_gpu.rs`; one physical + handle mapped at two VAs is readable through either. **Not proven, treated as + unsafe:** unmapping while a replay is in flight; and `cuMemMap` *during* capture + returns `CUDA_SUCCESS` but is not proven replayable, so growth is issued outside + the captured segment. This survivability is the premise under #716. diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 3f6512c5fb..f9efa9cb72 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -8002,16 +8002,33 @@ Both are standard ONNX ops. No custom ops needed. The "paging" is expressed pure > where this is cheap. ONNX exposes one `past_key`/`past_value` per layer, so a > range is reserved per tensor and layers cannot share one: > -> | mapping | granularity | tokens per granule, 8B GQA (2048 B/token) | -> |---|---|---| -> | CUDA VMM (min and recommended) | 2 MiB | 1024 | -> | Windows host | 64 KiB | 32 | +> | mapping | granularity | +> |---|---| +> | CUDA VMM (min **and** recommended — no finer granule on this device, PR #776) | 2 MiB | +> | Windows host | 64 KiB | > -> On the host, granularity is as fine as paging, so virtual contiguity is -> strictly better than both copying and a paged kernel. On CUDA the 2 MiB granule -> floors a sequence at `num_layers × 2 × 2 MiB` regardless of its length, which -> only matters with many concurrent short sequences — a workload this project has -> decided not to target. +> **Correction (PR #776).** An earlier version of this table added a "tokens per +> granule" column (e.g. "2 MiB / 2048 B ≈ 1024 tokens") and claimed the CUDA +> floor was `num_layers × 2 × 2 MiB` regardless of length. That framing was +> measured wrong and was publicly corrected. KV is **per-binding / head-major**, +> so a fixed full-context stride does not pack tokens into a granule — it commits +> `objects × granule`, where `objects` is the number of head-stripe bindings. +> Measured: qwen2.5-0.5b commits **96 head-stripes × 2 MiB = 192 MiB to hold +> ~12 KiB of live content**. The relevant question is therefore not "how many +> tokens fit in a granule" but "at what context length does a stable full-context +> VA commit *less* than bucket-growth reservation". +> +> **The crossover is `granule / (head_dim × sizeof(dtype))`** under head-major, +> and it is **model-size independent** — `layers` and `kv_heads` cancel. On this +> device (2 MiB granule): ~**8,192 tokens** at head_dim 128 / fp16, ~**16,384 +> tokens** at head_dim 64 / fp16. Verified on hardware across a 480× model-size +> range (PR #776). Quantized KV makes it **worse**: fp8/int8 halves bytes per +> element and so **doubles** the crossover in tokens. Below the crossover, a +> stable full-context VA over-commits (this is the mechanism behind the #721 +> stage-4 regression — a full-context VA committed 1.5 GB where bucket growth +> commits 48 MB); above it, virtual contiguity wins. See +> [`MEMORY_ARCHITECTURE.md`](./MEMORY_ARCHITECTURE.md) for the full KV-geometry +> analysis and the read-bounded-commit result (PR #772). > > **Phase 3 is dropped.** ORT's `PagedAttention` > (`contrib_ops/cuda/bert/paged_attention.*`) exists, but it is CUDA-only — zero diff --git a/docs/MEMORY_ARCHITECTURE.md b/docs/MEMORY_ARCHITECTURE.md index 6a7dc8a16a..bf0af9618a 100644 --- a/docs/MEMORY_ARCHITECTURE.md +++ b/docs/MEMORY_ARCHITECTURE.md @@ -36,7 +36,7 @@ when in fact it is 1955 lines. | Weight offload vs the OS | — | **27x slower than doing nothing; cause identified 2026-08-06** | WDDM demand-paging reaches 6.01 tok/s on a model that does not fit; our managed offload path is far behind. The gap now has a named mechanism rather than a mystery: the residency cache never served a single weight without copying it (see the row below), so every decode step re-staged 7.33 GiB and paid 6,936 `cuMemAlloc`/`cuMemFree` pairs. Measured with CUDA events, that places **~39% of step time downstream of one eviction-policy defect** — `h2d_copy` 18.8%, `staging_fill` 9.0%, `vram_free` 9.0%, `vram_alloc` 2.3%. The remainder is not yet accounted for, and graph capture — worth **2.6x** in isolation, ranges non-overlapping — is disabled on **every** offload runffload reaches 0.22. The largest known cause is stated only in a warning: *"weight offload is incompatible with CUDA graph capture; capture disabled"*. On a Windows consumer GPU a user is currently better off leaving offload off (#705) | | `--vram-limit` | — | **enforced; explicit bytes select managed no-spill mode** | The limit is resolved before native CUDA session construction. If package weights exceed it, weight offload is enabled and its allowance is derived as `limit − governed device state`; the same authority owns physical VMM handles, mapped weights, KV, and workspace. A 6 GiB qwen2.5-14b int4 live run loaded, generated the expected `Paris`, stayed at 5,534,384,128 owned bytes, rejected an 8K request pre-header with 429, and completed four queued short requests without overlap-only rejection. Crossing the first physical KV-growth boundary transferred 201,326,592 bytes through one grant and reported 201,326,592 mapped KV bytes; later governed capacity exhaustion returned 429 rather than 500/OOM. Managed initialization failure is an early arithmetic/provider error, while `ONNX_GENAI_DYNAMIC_KV_WEIGHT_LENDING=0` restores the prior non-VMM/WDDM-capable compatibility fallback. Limitation: the native server's FIFO per-request engine queue verified ordering and absence of overlap-only rejection, but did not demonstrate simultaneous live KV accumulation. Caveat carried, not closed: `activations_bytes=unknown` and `runtime_overhead_bytes=unknown` are not subtracted (#514). PRs #717, #736. | | Weight residency cache (CUDA) | — | **had a 0% hit rate for its entire life; fixed 2026-08-06** | `evict_to_fit` was a plain LRU evicting from the least-recently-used end, while the decode weight walk is a **cyclic sequential scan** over the layers — the pessimal pairing, which returns the hit rate to zero at *every* capacity. Measured: **6,936 page-ins, 0 hits, at both 3 GB and 6 GB budgets**, with staged bytes identical (7,870,916,608 per step) because miss traffic under this pathology is invariant to capacity. Rivals eliminated on hardware: the budget knob works (peak resident 3.0 vs 6.0 GB) and staged bytes reconcile with `page_ins × page size`. A stable-resident-subset policy measures **74.18%** against a `B/W ≈ 76%` ceiling, page-ins 6,936 → 1,791, evictions 6,286 → **0**. #720, PR #723 | -| KV page storage | — | **contract opened 2026-08-06; still host-only** | `onnx-genai-kv` implements paging, ref-counting/CoW, prefix sharing and quantization layout, and **never touched device memory**: storage was host `Vec`, `Device::Gpu(0)` a label on a struct field, and a tier migration one enum assignment moving zero bytes. Documented as a placeholder in `tiered.rs` from the start — the seam was designed in and the GPU backend behind it was never built. `PageTable` now holds `Box` so a third party can supply a store without patching the crate; host stores expose slices, device stores expose an opaque span, and a host view of a device page requires explicit materialization. Stages 2–5 in #721. PR #726 | +| KV page storage | — | **contract opened 2026-08-06; still host-only** | `onnx-genai-kv` implements paging, ref-counting/CoW, prefix sharing and quantization layout, and **never touched device memory**: storage was host `Vec`, `Device::Gpu(0)` a label on a struct field, and a tier migration one enum assignment moving zero bytes. Documented as a placeholder in `tiered.rs` from the start — the seam was designed in and the GPU backend behind it was never built. `PageTable` now holds `Box` so a third party can supply a store without patching the crate; host stores expose slices, device stores expose an opaque span, and a host view of a device page requires explicit materialization. Stages 2–5 in #721. PR #726. Device KV paging is **not** owned by `onnx-genai-kv`: under the VMM design it is owned by the CUDA VMM layer (`CudaVmmAllocator`, the #740 physical-handle pool), with committed-granule admission (#745) and growth grants (#748). `native_decode/cuda.rs` still has no `PageTable`/`PagedKvCache` consumer, and `paged_gqa.rs` is a batch-1 CPU primitive. | | Captured graphs across a VMM remap | — | **verified on hardware 2026-08-06** | A CUDA graph instantiated before `cuMemUnmap`/`cuMemCreate`/`cuMemMap` at the same virtual address replays correctly afterwards and writes into the **new** physical pages — sentinel-proven, closing the page-recycling confound. The growth-shaped case passes, and one physical handle mapped at two virtual addresses is readable by captured work through either. Untested and treated as unsafe: unmapping while a replay is in flight. `cuMemMap` during capture returns `CUDA_SUCCESS` but is **not** proven replayable, so growth is issued outside the captured segment. This is the premise under #721 stage 4 and under re-scoping #716, and its stated falsifier did not fire. PR #727 | | Activation planning | — | **wired for measurement; not yet allocating** | `crates/onnx-runtime-memory` now has a consumer: the session executor builds a `ViewMap`, runs the planner, and reports peak vs naive. Measured 2.4x-2.7x on qwen2.5-0.5b, though see #671 for a review finding that may inflate that. Sharing slots for real is #670 | | Native KV page size | — | **wrong unit** | `governor_kv_config` puts a token count in `page_size_bytes` when the geometry is unknown, so the native `bytes_per_token` is 1 (#628) | @@ -145,6 +145,78 @@ pooled dense-attention scores slot — govern `standard_attention.rs:602` and same `workspace_requirement` + prepared-workspace contract, then fold in the legacy `attention.rs:711` per-call path. +### KV reservation geometry: the crossover, the granule floor, and read-bounded commit + +This is the single home for the KV VMM geometry. Other documents should link +here rather than restate it, because the numbers below were each measured to +overturn an earlier plausible-looking claim, and a copied number drifts. + +**The granule floor is per-object, not per-token (#776).** KV is stored +per-binding, head-major, so a fixed full-context stride commits `objects × +granule`, not `ceil(content / granule)`. There is no "tokens per granule": you +cannot pack multiple tokens of one head-stripe into a shared granule when each +head-stripe is its own binding with the full-context stride. Measured on +qwen2.5-0.5b: **96 head-stripes × 2 MiB = 192 MiB physically committed to hold +~12 KiB of live content.** An earlier `docs/DESIGN.md` §39.5 table framed this as +"~1024 tokens per 2 MiB granule" and floored a sequence at `num_layers × 2 × +2 MiB`; that reasoning was wrong and was publicly corrected. + +**The crossover is `granule / (head_dim × sizeof(dtype))` and is model-size +independent (#776).** Under head-major layout, `layers` and `kv_heads` cancel +between the full-context-VA cost and bucket-growth cost, so the context length at +which a stable full-context VA commits *less* than power-of-two bucket growth +depends only on the granule, the head dimension, and the KV element size: + +``` +crossover_tokens = granule / (head_dim × sizeof(dtype)) +``` + +On this device (2 MiB granule): ~**8,192 tokens** at head_dim 128 / fp16, +~**16,384 tokens** at head_dim 64 / fp16. Verified on hardware across a **480× +model-size range** (PR #776). Quantized KV makes the crossover *worse*: fp8/int8 +halves bytes-per-element and therefore **doubles** the crossover in tokens, so a +full-context VA is the wrong default for exactly the long-context/quantized-KV +regime it looks most attractive for. Below the crossover a stable full-context VA +over-commits; above it, virtual contiguity wins and growth becomes remap rather +than copy. + +**`CU_MEM_ALLOC_GRANULARITY_MINIMUM == RECOMMENDED == 2 MiB` on this device +(#776).** There is no finer device granule available here, so "shrink the +granule" is not an available lever on this hardware — the 2 MiB floor is a +hardware constant to design around, not a tunable. + +**The read pattern forces the commit, not the reservation (#772).** A read +bounded to the live length leaves the reservation tail *uncommitted*, including +under captured replay; reading one byte past the live length faults +`CUDA_ERROR_INVALID_VALUE`. So a stable full-context VA does not itself commit +the tail — an over-broad read does. This is the correct attribution of the #721 +stage-4 **32× regression**: a stable full-context VA committed **1.5 GB** where +bucket growth commits **48 MB**, because the read spanned the full context rather +than the live prefix. Bound the read to the live length and the tail stays free. + +**Weight offload and CUDA graph capture are mutually exclusive today.** The +pager's alloc/copy/free operations are capture-illegal, so enabling offload +disables capture (see the module docs in +`crates/onnx-genai-engine/src/native_decode/cuda.rs`). Large models that need +offload therefore get **none** of the capture-fragmentation wins that #708 and +#728 landed — those took 35B-A3B decode from **154 to 34** graph segments. The +fix is **#716**: page weights under a stable virtual address so page-in remaps +physical granules at the same VA (proven survivable, #727 — see the status +table) instead of returning a new pointer that would invalidate the capture. +Until #716 lands, offload-on implies capture-off, and that is a large part of why +managed offload trails WDDM demand paging on a model that does not fit. + +**Sequential weight prefetch does not work and was deleted (#715, #718).** It was +implemented, measured to produce no usable compute/transfer overlap on the dev +box, and removed in PR #715. AirLLM's own analysis (#718) reaches the same +conclusion independently — the bottleneck is disk/transfer bandwidth, not a lack +of overlap — and rates prefetch at ~10% by their own account. On a model that +does not fit there is not enough independent compute to hide the transfer behind, +so any doc presenting prefetch/compute overlap as the plan, the lever, or an +existing capability is stale. The withdrawn `h2d_enqueue_copy_ms` = 1.7% figure +that once retired this line of work is corrected to **18.8%** by CUDA-event +measurement (see the instrument-failure narrative below). + ### The platform is part of the memory system On Windows WDDM, `cudaMalloc` does not fail when it exceeds dedicated VRAM — the @@ -300,6 +372,26 @@ After the third instance the response should be a guard, not more vigilance. The `Challenger` role (`.squad/agents/challenger/charter.md`) exists to apply this systematically to any result that would change technical direction. +### Keeping the documentation from going stale + +The same discipline applies to the prose, not just the code and the counters. +Most of the staleness this subsystem accumulates is a *performance claim written +in the present tense that a later measurement overturned* — prefetch overlap +(#715), the 1.7% h2d figure (#718), the 0% residency cache (#723), the +"tokens per granule" KV framing (#776). Three rules keep it from recurring: + +- **A performance claim in docs must cite the measurement that produced it.** A + number with no PR/issue behind it is a guess, and a guess ages into a false + fact. If you cannot name where it was measured, mark it as an estimate. +- **When measurement overturns a claim, correct it in place and say what + superseded it** — do not silently delete the old claim. A reader who remembers + the old number must be able to see it was tested and retired, with the PR/issue + that did it. That traceability is what makes the docs trustworthy. +- **Distinguish measured fact from design intent from not-yet-built.** Aspirational + design written in the present tense is the most common failure mode: a design + document may describe an intended design, but must not claim it is implemented + or measured when it is not. + The last five rows are newer than the layers above them and move fastest, so each says where it actually is. "Decided" and "in `main`" are different states and a reader should not have to guess which one a row means. diff --git a/docs/ORT2.md b/docs/ORT2.md index 4c5b70be9f..46bb66a78b 100644 --- a/docs/ORT2.md +++ b/docs/ORT2.md @@ -3528,10 +3528,25 @@ Priority heuristic (customizable via `execution_hints.json` or `onnx_runtime.mem For long-sequence prefill where activations exceed GPU memory: +> **Measured correction (2026-08, PR #715, issue #718).** The +> "prefetch next layer's weights so the transfer hides behind compute" idea +> sketched below was *implemented and then deleted*: sequential weight prefetch +> was measured ineffective on the RTX 4060 Laptop dev box and removed in +> PR #715. AirLLM's own analysis (issue #718) reaches the same conclusion — the +> bottleneck is disk/transfer bandwidth, not a lack of compute/transfer overlap, +> and they rate prefetch at ~10% by their own account. On a model that does not +> fit, the transfer cannot be hidden because there is not enough independent +> compute to hide it behind. Treat the code below as **falsified design intent**, +> retained to show what was tried. See +> [`MEMORY_ARCHITECTURE.md`](./MEMORY_ARCHITECTURE.md) for the measurement +> narrative. + ```rust fn execute_with_activation_spill(&mut self, layers: &[Layer]) -> Result<()> { for (i, layer) in layers.iter().enumerate() { - // Async prefetch next layer's weights (overlaps with current compute) + // NOTE: prefetch/compute overlap was measured ineffective and removed + // (PR #715, corroborated by AirLLM in #718). Kept only to illustrate the + // discarded design; it is not the plan and is not implemented. if i + 1 < layers.len() { self.prefetch_weights_async(i + 1); } @@ -3577,17 +3592,29 @@ impl PagedKvCache { ### 30.5 Overlap: Hiding Transfer Latency -Offloading doesn't mean stalling — **prefetch overlaps with compute:** +> **Superseded by measurement (PR #715, issue #718).** This section asserted that +> prefetch overlaps with compute and that offloading therefore need not stall. +> That was measured false on the dev box: sequential weight prefetch produced no +> usable overlap and was removed in PR #715; issue #718 (AirLLM) independently +> attributes the cost to disk/transfer bandwidth, not to missing overlap. When a +> model does not fit, per-layer compute is far smaller than the per-layer +> transfer, so there is nothing to hide the transfer behind — "zero visible +> overhead" does not occur. Re-measured with CUDA events, host-to-device transfer +> is **18.8%** of decode step time (not the withdrawn 1.7% `h2d_enqueue_copy_ms` +> figure; see [`MEMORY_ARCHITECTURE.md`](./MEMORY_ARCHITECTURE.md)). The timeline +> below is retained only as a record of the discarded design. ``` -Timeline (layer-pipeline offloading): +Timeline (layer-pipeline offloading) — DISCARDED DESIGN, prefetch did not overlap: GPU compute: [== Layer N ==] [== Layer N+1 ==] [== Layer N+2 ==] H2D stream: [-- prefetch N+1 weights --] [-- prefetch N+2 --] D2H stream: [-- spill N-1 activation --] -If compute time > transfer time: zero visible overhead. -If not: partially hidden, still better than blocking. +Intended claim (falsified): if compute time > transfer time, zero visible +overhead. In practice on a model that does not fit, transfer time dominates +compute time and the prefetch stream cannot be hidden — this is why PR #715 +deleted it. ``` Implementation: separate CUDA streams for compute vs transfer, fence synchronization: @@ -3621,10 +3648,18 @@ impl OverlappedExecutor { ### 30.6 Performance Expectation +> **Caveat (PR #715, #718).** The "prefetch mostly hidden by compute" column +> below was written before measurement and is not what the hardware showed — +> prefetch was measured ineffective and removed (PR #715). Read these as an +> original optimistic estimate, not a measured result. On the dev box, managed +> weight offload was **~27x slower than doing nothing** (WDDM demand paging) on a +> model that does not fit; see the status table in +> [`MEMORY_ARCHITECTURE.md`](./MEMORY_ARCHITECTURE.md). + | Scenario | Relative Speed | What's happening | |----------|---------------|------------------| | 100% GPU-resident | 1.0× | Ideal | -| 30% weights on CPU | ~0.7× | Prefetch mostly hidden by compute | +| 30% weights on CPU | ~0.7× (estimate; prefetch overlap unproven) | Transfer, not overlap, dominates | | 70% weights on CPU | ~0.3-0.4× | Transfer becomes bottleneck | | Activation spill | ~0.5× | Extra D2H + H2D per layer | | KV pages on CPU | ~0.7-0.9× | Only evicted pages need fetch | @@ -3648,7 +3683,7 @@ Namespaced options for fine control: session = nxrt.load("model.onnx", options={ "memory.gpu_budget_mb": "12000", # 12GB GPU budget "memory.offload_strategy": "layerwise", # vs "tensorwise" - "memory.prefetch_layers": "2", # prefetch 2 layers ahead + "memory.prefetch_layers": "2", # prefetch 2 layers ahead (NOT active: sequential prefetch removed in PR #715 as ineffective) "memory.kv_gpu_pages": "1024", # max KV pages on GPU "memory.activation_spill": "auto", # auto/always/never }) @@ -3662,7 +3697,7 @@ session = nxrt.load("model.onnx", options={ | Weight placement at load time | **Yes (static)** | Avoids runtime jitter. Re-plan only on explicit resize. | | Activation spill | **Dynamic** | Can't predict at load time (depends on input shapes). | | KV cache eviction | **LRU per-page** | Matches paged attention. Old context evicted first. | -| Overlap compute + transfer | **Always** | Separate streams, fence sync. Zero overhead when compute-bound. | +| Overlap compute + transfer | **Removed (PR #715)** | Sequential prefetch was measured ineffective and deleted; #718 corroborates the bottleneck is transfer bandwidth, not overlap. Not "always", and not currently a lever. | | Disk tier (mmap) | **Tier 2 fallback** | For 405B on 32GB RAM laptops. Not primary path. | | Offload granularity | **Per-layer (weights), per-page (KV), per-tensor (activation)** | Each has different lifecycle. | diff --git a/docs/WEIGHT_OFFLOAD.md b/docs/WEIGHT_OFFLOAD.md index 29cc80e6a4..a86a4474ce 100644 --- a/docs/WEIGHT_OFFLOAD.md +++ b/docs/WEIGHT_OFFLOAD.md @@ -86,6 +86,26 @@ Two structural notes that are not merely "not done yet": past dedicated VRAM into system RAM and `cuMemGetInfo` cannot see it. Device capacity is therefore not a number we can read, and floor comparisons taken on such a machine do not mean what they appear to ([#704]). +- **Weight offload and CUDA graph capture are mutually exclusive today.** The + pager's alloc/copy/free operations are capture-illegal, so enabling offload + disables capture (module docs in + `crates/onnx-genai-engine/src/native_decode/cuda.rs`). Large models that need + offload therefore forfeit **all** of the capture-fragmentation wins #708/#728 + landed (35B-A3B decode from **154 to 34** graph segments). The fix is **#716** + — page weights under a stable virtual address so page-in remaps physical + granules at the same VA (survivable per #727) rather than returning a new + pointer. Until it lands, offload-on implies capture-off, which is a large part + of why managed offload trails WDDM demand paging. +- **The residency cache had a 0% hit rate for its entire life** — `evict_to_fit` + was a plain LRU evicting against a cyclic sequential layer scan, so it always + evicted the page needed soonest (0 hits across 6,936 page-ins, at both 3 GB and + 6 GB budgets). Fixed by a scan-resistant stable-subset policy in **PR #723** + (74.18% hit rate; evictions 6,286 → 0). This is the named mechanism behind the + "beats the OS" failure. See [`MEMORY_ARCHITECTURE.md`](./MEMORY_ARCHITECTURE.md). +- **Public budgets are committed physical bytes, not nominal content bytes.** A + nominal 6 GB content budget was measured to physically consume ~**6.51 GB** + (~8.5% hidden), because reservation is committed at the 2 MiB granule. Size and + report budgets in committed bytes. What is confirmed working, and worth not re-litigating: **prefix reuse**, and **multi-request concurrency** up to the point where admission becomes the limit. @@ -418,8 +438,13 @@ binding. - Keep compressed blocks in VRAM unless a measured device kernel requires another representation. - Keep a bounded pinned-host staging ring. -- Prefetch the next exact/predicted expert wave on a transfer stream while the current - wave computes. +- ~~Prefetch the next exact/predicted expert wave on a transfer stream while the current + wave computes.~~ **Superseded (#715, #718):** sequential weight prefetch was + implemented and measured to produce no usable compute/transfer overlap on the + dev box, and was removed in PR #715; AirLLM's own analysis (#718) attributes + the cost to disk/transfer bandwidth rather than missing overlap. Do not present + prefetch as an existing lever. Expert-wave prefetch remains a *design proposal* + contingent on a measured win — see Phase 4. - If all planned weights fit, eagerly load and pin them, disable eviction, and match a conventional resident runtime's hot path. @@ -625,12 +650,25 @@ VRAM. On a fitting model, fully resident performance must remain near baseline. ### Phase 4 — asynchronous and predictive prefetch -**Ship independently:** +> **Status: design proposal, and its central premise is measured false so far.** +> Sequential weight prefetch was implemented and **removed in PR #715** because it +> produced no usable compute/transfer overlap on the dev box; AirLLM's own +> analysis (#718) independently attributes the cost to disk/transfer bandwidth, +> not to a lack of overlap, and rates prefetch at ~10% by their own account. On a +> model that does not fit there is not enough independent compute to hide the +> transfer behind. Treat everything below as a proposal that must clear a measured +> end-to-end win before it is built, not as a planned or landed capability. The +> "landed" annotations below refer only to the low-level plumbing (stream-ordered +> H2D, awaitable fences, a standalone double-buffer scheduler), **not** to +> prefetch producing a throughput gain. + +**Ship independently (plumbing only; predictive prefetch itself is unproven):** - implement true stream-ordered H2D and awaitable fences; **[landed — issue #87]** - double-buffer expert panels; **[strategy landed as a standalone, unit-tested scheduler; live MoE wiring pending Phase-3b device binding]** -- add exact-next-wave, heat-based, then opt-in router-predicted prefetch; +- add exact-next-wave, heat-based, then opt-in router-predicted prefetch + — **only if measured to help; sequential prefetch was measured not to (#715)**; - budget reservations and cancel low-value work under pressure. **Measure:** prefetch hit/late/waste, hidden transfer percentage, p50/p99 token latency,