perf(cuda-ep): make GatedDeltaNet Split capture-safe (C3) — collapse 30 decode graph seams by justinchuby · Pull Request #708 · justinchuby/onnx-genai

justinchuby · 2026-08-06T17:57:21Z

Summary

The 35B-A3B QMoE (GatedDeltaNet hybrid) decode step fragments into 184 CUDA-graph segments / 183 eager seams. Using the runtime's own ONNX_GENAI_LOG_CAPTURE_SEGMENTS=1 diagnostic (steady-state, warmed, time-windowed to decode), the fragmentation attributes to three independent capture blockers — Split is one of them:

Root cause	Op(s)	Eager seams/decode	File	Status
C1 shared elementwise fixed-decode signature rejects head-major shapes	Sqrt, Div, Mul, Sigmoid	~120	`elementwise.rs` (`is_fixed_decode_shape`)	follow-up (dominant)
C2 kernel trailing host `synchronize()` aborts capture	LinearAttention	~30	`linear_attention.rs`	rejected — see below
C3 dynamic Split host-read + trailing `synchronize()`	Split	30	`movement.rs` `SplitKernel`	✅ this PR

What C3 fixes

The 30 GatedDeltaNet Split nodes take their sizes from a Constant node (value_ints=[2048,2048,4096], axis=-1). The default OptimizationLevel::None never runs ConstantFolding, so those sizes are never materialized into a graph initializer → the kernel fell to the dynamic path: host-read the split-size input, then a trailing synchronize(). Both are illegal inside CUDA-graph capture, so every GDN Split de-captured the surrounding decode block.

But the executor pre-allocates each Split output at its statically-inferred shape, so the split sizes are fully determined by outputs[i].shape[axis]. SplitKernel now:

derives the split axis+sizes from the pre-allocated output shapes (plan_from_outputs) — no host read, no sync;
caches the derived plan so capture_support() reports Supported after warmup;
copies via the same stream-ordered launch_copies — byte-identical to the host-read path.

Genuinely data-dependent splits (outputs not statically resolved) fall through to the existing host-read path and are structurally declined for capture by the executor, so correctness is preserved for the dynamic case.

Results (35B-A3B QMoE, GPU0 `CUDA_VISIBLE_DEVICES=0 taskset -c 1`, `--steady --warmups 2 --runs 3 --tokens 128 --temperature 0`, `ONNX_GENAI_CUDA_KV_MAX_LEN=4096`)

	decode ms/tok	tok/s	segments / eager seams	Split seams
Before (`origin/main` `0753608d`)	13.415	74.54	184 / 183	30
After (C3)	12.132	82.43	154 / 153	0

−9.6% decode latency, +10.6% tok/s. All 30 GDN Split eager seams collapse (−30 segments).

Byte-exact confirmation

Baseline decode is deterministic run-to-run. With C3 the generated token stream is identical to baseline, and the token@119 == 33803 autoregressive lock holds. C3 changes only where the split sizes come from — no kernel math changes.

C2 investigated and rejected (correctness gate)

Eliding the LinearAttention trailing synchronize() on the fixed-decode path was implemented and tested. It broke byte-exactness: the decode stream diverged before position 119 (token@119 became 46283; 33803 disappeared). The trailing synchronize() is load-bearing, not the conservative barrier hypothesized — removing it exposes a real ordering/aliasing hazard the same-stream assumption doesn't cover. Per "we do not trade correctness for capture," C2 is dropped from this PR and its sync-elision change/test reverted. It needs a deeper root-cause pass (what actually consumes the LinearAttention output/present_state, and whether an event/stream-wait can replace the host sync) before any retry.

Residual & follow-ups (honest framing)

154 segments / 153 eager seams remain. This PR removes the Split host-sync seam class; it does not collapse the segment count to a small number — that requires:

C1 (dominant follow-up): the shared elementwise is_fixed_decode_shape gate rejects GDN head-major shapes ([1,1,16,128], [32,128]), keeping ~120 Sqrt/Div/Sigmoid/Mul seams eager. High blast radius (shared by bitwise/pointwise/prelu/elementwise) → isolated PR (squad/elementwise-capture-seqindep). Must derive seq-independence from graph metadata, not runtime shape values ([32,128] head-major is indistinguishable from [tokens,feat] by shape alone).
C2 retry: after root-causing the LinearAttention output consumer.

Test

split_runtime_sizes_with_resolved_outputs_warms_and_captures (construction_gpu.rs) exercises the exact GDN case — a runtime split-size input with constant_split_input=false: the cold kernel declines capture, warms via the output-derived plan, flips to Supported, then captures + replays byte-identically. Existing Split parity/capture tests unchanged and green. cargo fmt --all --check and cargo clippy clean on the changed crate.

Coordination

No persistent workspace is introduced → memory-governor registration (double-claim avoidance vs VMM #682) is N/A for this lever. No VMM/KV code touched.

Refs the roofline + GDN-Split investigation (.squad/decisions/inbox/cohaagen-gdn-split-capture.md).

Do not self-merge.

…30 decode graph seams The 35B-A3B QMoE (GatedDeltaNet hybrid) decode step fragments into 184 CUDA-graph segments / 183 eager seams. Steady-state capture logging (ONNX_GENAI_LOG_CAPTURE_SEGMENTS) attributed these to three independent blockers: C1 shared elementwise fixed-decode signature (~120), C2 LinearAttention trailing host synchronize (~30), C3 dynamic Split host-read + synchronize (30). This lands C3. The 30 GatedDeltaNet `Split` nodes take their sizes from a `Constant` node (value_ints=[2048,2048,4096]) that the default OptimizationLevel::None never folds into an initializer, so the kernel fell to the dynamic path: host-read the split-size input + trailing synchronize, both illegal during capture. But the executor pre-allocates each Split output at its statically-inferred shape, so the split sizes are fully determined by the output shapes. SplitKernel now derives the plan from the output shapes (no host read, no sync), caches it so capture_support() reports Supported after warmup, and copies via the same stream-ordered launch_copies — byte identical to the host-read path. Genuinely data-dependent splits (outputs not statically resolved) fall through to the old path and are structurally declined for capture, so correctness is preserved. Byte-exact (35B-A3B QMoE, GPU0, temp=0, KV_MAX_LEN=4096): generated token stream identical to baseline, token@119 == 33803 autoregressive lock holds. Decode 13.415 -> 12.132 ms/tok (-9.6%, 74.5 -> 82.4 tok/s). Segments 184 -> 154 (all 30 Split seams collapse). C2 (LinearAttention sync elision) was implemented and REJECTED: removing the trailing synchronize diverged the decode stream (token@119 became 46283) — the barrier is load-bearing, not conservative. Per "we do not trade correctness for capture," C2 is dropped pending a deeper root-cause pass. C1 (shared elementwise capture gate) remains the dominant follow-up. Test: split_runtime_sizes_with_resolved_outputs_warms_and_captures exercises the GDN case (runtime split input, constant_split_input=false) — cold Split declines capture, warms via the output-derived plan, then captures + replays byte-identically. Refs roofline + GDN-Split investigation (.squad/decisions/inbox/cohaagen-gdn-split-capture.md). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov · 2026-08-06T18:55:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.11%. Comparing base (0753608) to head (a82bdd0).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #708      +/-   ##
==========================================
- Coverage   80.48%   80.11%   -0.37%     
==========================================
  Files         328      329       +1     
  Lines      125517   130334    +4817     
  Branches   125517   130334    +4817     
==========================================
+ Hits       101024   104419    +3395     
- Misses      20318    21683    +1365     
- Partials     4175     4232      +57

Flag	Coverage Δ
cli-ort-linux	`82.21% <ø> (ø)`
cli-ort-windows	`81.68% <ø> (?)`
offline	`80.03% <ø> (-0.39%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 24 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…symbol classification (C1) + re-anchor 35B QMoE oracle lock on fp32 teacher-forcing (#722) (#728) ## Summary Ships **C1** — the build-time growing-symbol pointwise-capture classifier — collapsing the 35B-A3B QMoE hybrid GDN decode from **154 → 34** CUDA-graph segments, together with a **re-anchor of the 35B QMoE oracle correctness lock onto fp32 teacher-forcing (#722)** so the lock is capture-independent. This is the culmination of the 35B-A3B CUDA-graph capture-fragmentation wave (roofline investigation → GDN-Split C3 #708 → this C1). C3 removed the host-sync seams; C1 collapses the dominant pointwise segment count. ## The classifier (pinned-vs-growing rationale) Classify the graph's **growing** (KV-length) symbols once at build/graph-load time — structurally, from attention `past`/`present` KV sequence axes only (reusing #700's `recurrent_past_names()` / `is_recurrent_state_shape()` / KV-length-symbol machinery, DRY). Collect them into a `HashSet<SymbolId>`. A pointwise/elementwise/bitwise/prelu op is **`seq_independent` (capture-eligible) iff NONE of its symbolic output dims is in the growing set.** Pinned symbolic dims — batch=1, query seq_len=1, heads — are admitted; anything carrying a KV-length symbol stays eager. - Why not strict all-`Dim::Static`? It was proven a no-op: GDN pointwise ops carry `[Symbolic(batch=1), Symbolic(seq=1), Static(16), Static(128)]` — both symbolic dims are *pinned*, never all-Static — so strict-C1 kept every GDN op eager and collapsed nothing. - **Heads-vs-tokens ambiguity is resolved by the growing SET, not dim position:** `[Symbolic(heads),Static(128)]`, heads ∉ growing → capturable; `[Symbolic(seq_kv),Static(128)]`, seq_kv ∈ growing → eager. Runtime shape values cannot distinguish these; the KV-axis set can. - ⚠️ The **query `sequence_len` symbol is NOT marked growing** — it is symbolic but PINNED (=1, held constant across every decode replay). Only the KV-cache length axis actually grows. This distinction is the entire win. **EP-API / build-time plumbing:** the flag is set at kernel CREATION (`KernelFactory::create`), before the single warm step — no runtime diff, no 2nd warm step, never gates the main-executor arm. (Earlier runtime-diff designs were structurally wrong: main arms at warm#1, so a warm#2 diff corrupts decode.) One shared helper feeds all 5 pointwise kernels; no per-op copies, no per-model hardcoding. **seq_len>1 guard:** the capture region is keyed to the concrete pinned-symbol binding (batch=1, query seq_len=1). A seq_len>1 entry forms a different (or uncaptured) region and cannot replay a seq_len==1 capture. The flag only asserts "no growing-symbol dependence," so the pinned query symbol is never misclassified. ## The #722 re-anchor (why it's needed and why it's a stronger invariant) C1 legitimately flips the 35B-A3B greedy **token@119 from 33803 → 46283** — an inherent fp16 inter-layer near-tie (top-2 gap < 0.1 logit), both within fp16 noise of the fp32 oracle. Heavy accumulators are already fp32; this is an inter-layer fp16 reassociation/rounding coin-flip, not a reduction bug (diagnosed in #722, same class as DeepSeek-R1 #158). The OLD lock asserted the *autoregressive captured* token@119 == 33803 — capture-dependent, so it broke under C1 (which moves the captured stream onto the eager stream). Per #158 precedent, an fp16 near-tie satisfied by BOTH paths is not a bug; the lock belongs on the fp32 oracle. Re-anchored `qwen36_35b_a3b_qmoe_divergence.rs`: - **Primary lock** = fp32-oracle argmax via **fresh-engine teacher-forcing of a FIXED canonical context** (`prompt + SHARED_GENERATED_PREFIX`, the recorded 119-token 33803-path prefix). Teacher-forcing is prefill-dominated → capture-independent → **both capture-on and capture-off re-derive 33803.** The context is fixed/recorded, NOT reconstructed from the live stream (under C1 that stream is the 46283-path). - Dense int4 teacher-forced argmax == 33803 + margin (replaces the broken autoregressive dense assertions; measured margin 0.078125 matches the recorded DENSE row +0.0781). - fp32-oracle CPU cross-check at the same context (env-gated, capture-independent). - The autoregressive token@119 is preserved as a **DOCUMENTED, NON-FATAL #722 tripwire** (logs 33803-or-not) so the captured-vs-eager fragility stays visible; comment cites #722. - #695 continuation test: 33803 pin demoted to a #722 note; now asserts the capture-independent `reused_argmax == fresh_argmax` invariant. The fp32-teacher-forcing anchor is **stronger**: it's the ground-truth correctness invariant that both capture arms satisfy, instead of pinning on which side of a coin-flip one autoregressive stream lands. ## Results (post-rebase onto origin/main, KV_MAX_LEN=4096, GPU0) | metric | baseline (154-seg) | C1 | delta | |---|---|---|---| | CUDA-graph segments | 154 | **34** | −120 | | decode ms/tok | 12.307 | **11.959** | −2.9% | | throughput tok/s | 81.26 | **83.62** | +2.9% | (Isolated earlier run measured 12.263→11.763, +4.3%; this re-measure ran on a busier box. The 154→34 collapse is the robust invariant; the 33 residual seams are LinearAttention's load-bearing C2 sync, out of scope.) ## Correctness — byte-exact proof on BOTH models AND capture on/off - Re-anchored 35B oracle test **PASSES capture-ON and capture-OFF** (`ONNX_GENAI_CUDA_GRAPH=0/1`) — the fp32-teacher-forcing anchor holds regardless of capture. - Pure-attention `qwen2.5-0.5b-int4-onnx-native`: decode **byte-identical**, native CUDA == ORT CUDA == expected, capture-ON and capture-OFF (KV-length-dependent ops stay eager). - **Negative-assertion unit test:** fully-pinned-symbolic op → capturable; op carrying a growing KV-length symbol → stays eager (metadata-driven synthetic nodes, no model files). 71 classifier/session tests pass. - `cargo fmt --all --check` clean; `cargo clippy` clean on changed crates. ## Blast radius This touches the SHARED elementwise/bitwise/pointwise/prelu capture gate used by ALL models — a wrong broadening would silently corrupt any model. Mitigations: the classifier keys off graph metadata (the growing KV-axis symbol set), never runtime shape values; the negative-assertion unit test guards the growing→eager direction; byte-exact validation on both a hybrid (35B) and a pure-attention (qwen2.5-0.5b) model, capture on and off. ## Notes - **user-visible:** this shifts the 35B-A3B greedy token@119 from **33803→46283**, an inherent fp16 near-tie both within fp16 noise of the fp32 oracle — see #722. - **Closes #722** (implements the recommended fp32-teacher-forcing re-anchor). - No overlap with VMM/KV #682: C1 adds no persistent workspace (governor registration N/A). Builds on C3 #708 (154-seg baseline). Do not self-merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

…ecisions (#756) Records two owner decisions that redirect in-flight work, each with the measured counter-evidence it has to beat. **Contiguous-VA KV for batching (#750).** Each sequence gets its own contiguous device virtual address with physical granules mapped underneath on demand; the attention kernel keeps seeing a flat buffer and never learns page tables. This supersedes #721 stage 3 unless the route fails. The record carries the result it must beat: #721 stage 4 measured a stable full-context VA committing 1.5 GB where bucket growth commits 48 MB, because the decode kernel reads the full padded shape and masks, forbidding tail decommit. **Managed no-spill VMM as default (#755)**, with automatic weight streaming instead of hard OOM. Sequenced explicitly behind #716, because weight offload and CUDA graph capture are mutually exclusive today — flipping the default first would silently disable capture for exactly the models that need offload, giving back the 154 -> 34 segment collapse from #708/#728. Docs only, no code paths touched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

## Summary A **truth pass** over the memory / offload / KV / capture documentation. Over the past days a series of hardware measurements (RTX 4060 Laptop GPU, Windows/WDDM) overturned several claims still asserted as fact in `docs/`. This PR finds those claims, corrects them against what was measured, and — where a document described an intended design in the present tense as though it were implemented — marks it as a design proposal with its current status. No wrong claim is silently deleted: each is replaced with what was measured and the PR/issue that superseded it, so a reader who remembers the old number can see it was tested and retired. **Docs-only. No code or tests changed.** ## Corrected claims | File | What it said | What is true | Superseded by | |---|---|---|---| | `ORT2.md` §30.4–30.8 | "Offloading doesn't mean stalling — prefetch overlaps with compute"; "If compute time > transfer time: zero visible overhead"; "Overlap compute + transfer: **Always**"; `// Async prefetch next layer's weights (overlaps with current compute)` | Sequential weight prefetch produced no usable overlap and was **deleted**; the bottleneck is disk/transfer bandwidth, not overlap. On a model that doesn't fit there isn't enough independent compute to hide the transfer behind. | PR #715; AirLLM analysis #718 (rates prefetch ~10%) | | `ORT2.md` §30.6 / §30.7 | "30% weights on CPU → prefetch mostly hidden by compute"; `memory.prefetch_layers` presented as an active knob | Estimate, not measured; prefetch removed. Managed offload was ~27× slower than doing nothing (WDDM) on a non-fitting model. | PR #715 | | `DESIGN.md` §39.5 | KV table "**tokens per granule**, 8B GQA (2048 B/token): 2 MiB → 1024"; floor `num_layers × 2 × 2 MiB` | KV is per-binding/head-major: a fixed full-context stride commits **objects × granule**, not tokens/granule. qwen2.5-0.5b commits **96 head-stripes × 2 MiB = 192 MiB for ~12 KiB** live. Crossover is `granule/(head_dim × sizeof(dtype))`, **model-size independent** (~8,192 tok @ head_dim 128/fp16; ~16,384 @ 64/fp16); quantized KV **doubles** it. 2 MiB is min **and** recommended — no finer granule on this device. | PR #776 | | `MEMORY_ARCHITECTURE.md` | h2d attribution 1.7% (already partly recorded); KV geometry, crossover, read-bounded commit, offload/capture exclusion not consolidated | Re-measured h2d transfer **18.8%**. Added the central KV reservation-geometry section: per-object granule floor, the crossover formula, 2 MiB min==recommended, the read-bounded-commit result (read—not reservation—forces the commit; #721 stage-4 32× regression = 1.5 GB vs 48 MB), offload/capture mutual exclusion, and prefetch deletion. | PR #715, #716, #772, #776 | | `MEMORY_ARCHITECTURE.md` (KV page storage row) | "still host-only" without VMM ownership | Device KV paging is owned by the CUDA VMM layer (`CudaVmmAllocator`, #740 pool), with committed-granule admission (#745) and growth grants (#748); `native_decode/cuda.rs` has no `PageTable` consumer and `paged_gqa.rs` is a batch-1 CPU primitive. | #740, #745, #748 | | `WEIGHT_OFFLOAD.md` §7.3, Phase 4 | "Prefetch the next exact/predicted expert wave … while the current wave computes"; Phase 4 predictive prefetch as the plan | Marked as a design proposal whose central premise (prefetch overlap) is measured false so far; plumbing (stream-ordered H2D, fences, double-buffer scheduler) is distinct from prefetch producing a throughput gain. | PR #715, #718 | | `WEIGHT_OFFLOAD.md` §0.4 | Did not mention offload/capture exclusion, the residency-cache defect, or committed-physical budgeting | Added: offload disables capture today (#716 is the fix; #708/#728 = 154→34 segments); residency cache had a **0% hit rate for its whole life**, fixed to 74.18% with evictions 6,286→0 (#723); a nominal 6 GB budget physically consumes **~6.51 GB**. | #716, #723 | | `CUDA_GRAPH_CAPTURE.md` | No mention of offload interaction or VMM remap survival | Added: offload/capture mutual exclusion (#716); a captured graph **survives a VMM remap at the same VA**, writing into the new physical pages — sentinel-proven; unmapping during in-flight replay and `cuMemMap`-during-capture are **not** proven. | #716, #727 | ## Deliberately left unchanged `GQA_KV_MATERIALIZATION_DESIGN.md`, `KV_INSERTION_DESIGN.md`, `DECODE_POOL_RESIDENCY_DESIGN.md`, `NATIVE_CUDA_DECODE.md`, `CUDA_STRATEGY.md`, `native-ort-kv-capacity.md` — checked against the relevant facts and found accurate/self-consistent for their scope (CPU-kernel data movement, CPU decode-pool threading, the native-decode milestone plan, the KV-capacity policy). Correct docs were not restructured, and subject areas outside this pass (CPU EP, quantization formats, CLI, diffusion) were not touched. ## Preventing recurrence Added a documentation-hygiene note to `MEMORY_ARCHITECTURE.md`: a performance claim in docs must cite the measurement that produced it; when measurement overturns a claim, correct it in place and name what superseded it rather than deleting it; and keep measured fact, design intent, and not-yet-built visibly distinct. Refs #715, #716, #718, #721, #723, #727, #740, #745, #748, #772, #776. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

…capture (#716) (#796) ## Summary Weight offload previously forced whole-step CUDA graph capture **OFF**: the pager returned a different device pointer on every page-in (`alloc_raw`/`free_raw`), and a captured graph bakes pointers into its recorded nodes, so the two were mutually exclusive (`resolve_graph_capture_enabled` returned `false` whenever offload was on). The models that most need offload — those that do not fit in VRAM — were exactly the ones that lost graph capture and all of the #708/#728 capture-fragmentation wins (35B-A3B decode 154 -> 34 segments). This PR gives each retained weight `key` a **reserved-once device virtual address** and remaps physical granules underneath on page-in/eviction, so a captured graph replays correctly across evict->repage. It then removes the offload/capture mutual exclusion, **gated on the paging path actually being stable-VA** (the managed no-spill authority). Hard prerequisite for #755 (managed no-spill VMM as the default with automatic weight streaming). ## What changed, by step **Step 1 - prove the mechanism in isolation first** (`crates/onnx-runtime-cuda-memory/tests/vmm_stable_va_weight_slot_gpu.rs`, **passes on this HW**): a captured graph that reads from a stable VA still produces correct results after the pages under that VA are swapped for different physical granules holding **different weight values**. Extends the #727 remap-at-same-VA proof to the weight-value-swap case. **Step 2 - stable-VA slot conversion** (`crates/onnx-runtime-ep-cuda/src/weight_paging.rs`): - Per-key `StableWeightSlot { va, len }` reserved once through the **existing #740 authority-scoped pool** via `allocate_committed`/`carve()` suballocation. No second allocator, no per-weight reservation (the mistake that made #733 net-negative). Charging stays on the global granule-ref **0->1 / 1->0** attribution. - `CudaWeightPage::Drop` decommits granules but **keeps** the VA for stable slots (`deallocate_span` only for transient bypass pages). Never asserts in `Drop`. - Respects committed-granule admission (#745): reserving VA is free; only committed physical bytes are charged. 2 MiB granule sizing per #776. - **Enforceable ordering rule (not a comment):** page-ins - and thus eviction-driven `decommit` of any stable slot - run entirely under the residency mutex; capture is capture-once/replay-many with no page-ins during replay; the engine additionally declines capture when a step reports a bypass/eviction (`GLOBAL_BYPASSED_PAGE_INS`). So no `decommit` of a baked VA can occur while a replay is in flight - the case #727 explicitly did **not** prove safe. No `cuMemMap`/grow inside a captured region. - Residency GPU test `vmm_retained_weight_key_keeps_a_stable_virtual_address_across_repage` proves the VA is identical across evict->repage with one reused granule and **no accounting regression**. The existing `vmm_weight_admission_reuses_owned_granules_without_runtime_alloc_free` and `weight_offload_gpu` suite still pass (no regression). **Step 3 - remove the exclusion, gated on stable-VA** (`native_decode/{cuda,mod,load}.rs`): `resolve_graph_capture_enabled` now takes `weight_offload_stable_va` and keeps capture OFF only when `weight_offload_enabled && !weight_offload_stable_va`. The signal is `policy.enabled && policy.managed_no_spill` (the explicit byte `--vram-limit` authority that installs the VMM arena + physical pool). The pointer-unstable `alloc_raw`/`free_raw` compatibility path keeps the old capture-off exclusion. **#723's `StableResident` eviction policy is unchanged** - this changes only the addressing mechanism underneath it. **Step 4 - measure** (`crates/onnx-genai-engine/tests/stable_va_capture_offload_e2e.rs`): drives the managed no-spill VMM stable-VA authority and verifies **token-identical** greedy decode with capture OFF and ON in one process, dumping the deterministic offload counters (`page_ins`, `hits`, `evictions`, `bypassed_page_ins`, `vram_alloc_ns`/`vram_free_ns`). ## Honest measurement caveat The full sustained-paging throughput/counter comparison the issue asks for could **not** be completed in this environment: - The genuinely-offloaded model (`qwen14b-zp`, 15.5 GB) is **not native-loadable** - it has no `inference_metadata.yaml` (`model.io` ports), which is the separate #384 load-chain gap. The native decode path (where this change lives) cannot load it. - On the native-loadable 0.5B model, the managed no-spill authority is **pin-or-fail**: either the ~381 MB weight set fits fully resident (0 page-ins) or a step cannot commit its working set (admission/headroom failure after heavy eviction, ~504 evictions observed). There is no clean budget window that both pages sustainedly *and* completes a full decode - that is exactly what #755's automatic weight streaming is for. What **is** verified end-to-end here: under the managed no-spill VMM stable-VA authority, decode with capture OFF and capture ON is **byte-identical** (32/32 greedy tokens), i.e. capture is transparent and no longer force-disabled. The addressing mechanism itself is proven **deterministically** by the two GPU tests in steps 1-2. A negative/partial result was an acceptable deliverable per the task; this is a positive mechanism result with a partial end-to-end result. ## Validation (local; CI not waited on) - `cargo fmt -p onnx-runtime-ep-cuda -p onnx-genai-engine -p onnx-runtime-cuda-memory -- --check` PASS - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` PASS (no `--all-targets`, the CI gate) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` PASS (also linted the new cuda-gated test under `--features cuda,native-backend`) - `cargo test -p onnx-runtime-session` PASS (full suite) - GPU tests single-threaded PASS: `vmm_stable_va_weight_slot_gpu`, `vmm_retained_weight_key_keeps_a_stable_virtual_address_across_repage`, `vmm_weight_admission_reuses_owned_granules_without_runtime_alloc_free`, `weight_offload_gpu` (5/5) - engine `native_decode` unit tests PASS incl. new `weight_offload_on_stable_va_path_keeps_graph_capture` and updated `weight_offload_forces_graph_capture_off` Refs #716, #755, #727, #740, #723. **Do not merge; do not close #716.** Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

…detail (#825) Justin asked to update `docs/PROGRESS.md` with the latest progress and drop stale info. It had grown to **1166 lines** of day-by-day changelog spanning 2026-07-15 → 2026-07-27, with the entire recent arc missing. ## Changes (1166 → 303 lines) - **Added recent milestones (2026-07-28 → 2026-08-12), newest-first:** - EP plugins run inside ORT via the plugin-EP C ABI (#762) and ship on PyPI — **`nxrt-ep-cpu` + `nxrt-ep-cuda` 0.1.0.dev5 both LIVE** (#819/#824/#820). - Absent-optional-output memory-safety wave (#762 review; Miri-clean). - VMM / KV-layout / offload / batching residency wave — managed no-spill VMM default, layout-governed KV floor (768× reduction), offload+capture coexist, #736 over-reservation audit (#755–#814). - Issue triage + autonomous fixes (#784/#785/#786/#788/#789); CPU-EP TopK correctness+perf (#774/#775). - Qwen3.5/3.6-27B hybrid Gated-DeltaNet native CUDA (#779); GLM-4-9B + DeepSeek-V2-Lite native + ORT 1.28 3-config fairness (#766/#770/#771). - Fused QMoE decode kernel + QMoE arc concluded; CUDA-graph capture trilogy, version-selectable CUDA (#708/#728/#757/#760), megakernel study (#769). - **Compressed stale detail:** the ~40 dated July day-by-day sections + verbose diffusion/LoRA/ORT2 narrative collapsed into a ~35-line "Foundational milestones" spine; superseded perf-wave numbers dropped; §1–§40 table condensed to a compact grid. - **Header refreshed:** last-updated → 2026-08-12; Published line now lists both EP plugins on PyPI + Miri CI. - **Preserved & updated** the DeepSeek/GLM known-gaps checklists and added new open items. Docs-only; full narrative remains in `.squad/decisions-archive/` and `docs/benchmarks/`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby enabled auto-merge (squash) August 6, 2026 18:38

justinchuby merged commit 178392b into main Aug 6, 2026
12 of 15 checks passed

justinchuby deleted the squad/gdn-split-capture branch August 6, 2026 21:43

justinchuby mentioned this pull request Aug 7, 2026

perf(cuda-ep): capture-eligible pointwise ops via build-time growing-symbol classification (C1) + re-anchor 35B QMoE oracle lock on fp32 teacher-forcing (#722) #728

Merged

justinchuby mentioned this pull request Aug 11, 2026

docs: correct stale memory/offload/KV claims against measurement #781

Merged

This was referenced Aug 11, 2026

Infer memory strategy plan at load #752

Merged

Page offloaded weights under a stable VA so offload keeps CUDA graph capture (#716) #796

Merged

This was referenced Aug 12, 2026

Make managed no-spill VMM the default with automatic weight streaming (#755) #798

Merged

docs: refresh PROGRESS.md — Aug 2026 milestones, compress stale July detail #825

Merged

justinchuby mentioned this pull request Aug 22, 2026

研读 FreeToken (arXiv:2608.16157):把 MoE cache miss 当作可调度的工作,而非要搬的字节 #1759

Open

Navigation Menu

Repository navigation

perf(cuda-ep): make GatedDeltaNet Split capture-safe (C3) — collapse 30 decode graph seams - #708

perf(cuda-ep): make GatedDeltaNet Split capture-safe (C3) — collapse 30 decode graph seams#708
justinchuby merged 1 commit into
mainfrom
squad/gdn-split-capture

justinchuby commented Aug 6, 2026

Uh oh!

codecov Bot commented Aug 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justinchuby commented Aug 6, 2026

Summary

What C3 fixes

Results (35B-A3B QMoE, GPU0 CUDA_VISIBLE_DEVICES=0 taskset -c 1, --steady --warmups 2 --runs 3 --tokens 128 --temperature 0, ONNX_GENAI_CUDA_KV_MAX_LEN=4096)

Byte-exact confirmation

C2 investigated and rejected (correctness gate)

Residual & follow-ups (honest framing)

Test

Coordination

Uh oh!

codecov Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Results (35B-A3B QMoE, GPU0 `CUDA_VISIBLE_DEVICES=0 taskset -c 1`, `--steady --warmups 2 --runs 3 --tokens 128 --temperature 0`, `ONNX_GENAI_CUDA_KV_MAX_LEN=4096`)

codecov Bot commented Aug 6, 2026 •

edited

Loading