Repository navigation
Seq-major 4 invalidations = head-major 4: growth policy still re-captures - #805
Merged
Merged
Conversation
Run every layout/configuration in its own process, interleave head-major and seq-major, and add a no-growth control that attributes three extra invalidations and captures to three bucket growths for both layouts. Record the result and physical-byte/copy counters in the memory architecture documentation. Refs #782, #797, #801, #804. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #805 +/- ##
==========================================
- Coverage 79.86% 79.21% -0.65%
==========================================
Files 331 331
Lines 138203 138203
Branches 138203 138203
==========================================
- Hits 110369 109484 -885
- Misses 23483 24372 +889
+ Partials 4351 4347 -4
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Owner
Author
|
Coordinator validation on the merged branch: This is outcome 2, and it is worth more than the claim it refutesI framed three acceptable outcomes for this measurement. This is the second: if
So the stride theory is correct and irrelevant at the same time. Seq-major That converts a two-week-old open question into a one-line policy problem. #782 Why the sequence of failures is worth recordingThis measurement was blocked three times, each for a different wrong reason:
Only the last one is the actual cause. Two of the three intermediate explanations The follow-up this createsMaking growth invalidation conditional on an address actually changing would It is correctness-sensitive — invalidating too little is far worse than Merging the measurement and the documentation update. |
🔴 Benchmark Regression DetectedComparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).
Visual flags: Host infoWhat this cannot catch
|
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…blished (#808) A durable record of the load-bearing results from this push, with every claim tied to a merged executable test — and, deliberately, the claims that were **refuted**. Three of the most useful outcomes were corrections to things I had stated as achieved: - **seq-major had not realised the 8x floor.** #794 measured head-major and seq-major committing *identical* physical bytes, because the bindings did not consume the layout descriptor. Fixed in #797; I corrected the floor table before merging. - **The decoder was not declining capture.** \captures=0\ came from a cached \ONNX_GENAI_CUDA_GRAPH=0\ in a long-lived test process (#804) — two PRs had misattributed a failed measurement to a model gate. - **A fixed KV stride does not remove growth-triggered re-capture.** True in mechanism (seq-major moves 0 bytes vs 688,576), irrelevant in practice, because the engine invalidates unconditionally on growth (#805). Also records the #736 audit's recurring finding — **four of five slices found over-reservation** rather than an ungoverned allocation — and the guidance that follows: start from *use*, not from allocation, because governing a bypass without sizing it to use converts invisible waste into charged waste, which tightens admission and costs concurrency. Docs only, no code paths touched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…ajor-only KV consumers (#777 #801 #797 #794 #809) (#812) # Remove the last blocker to seq-major KV being usable in production: gate the fourth place the layout lives **Outcome reached: #3 — "it works only under a condition."** Seq-major decode works end-to-end and is byte-identical to head-major, but only on the **pure decode path where our own GQA kernel is the sole consumer of the device KV buffer**. This PR makes the runtime **refuse rather than mis-map** when that condition does not hold, and pins down — with citations — exactly what remains between "growth byte geometry is seq-major" (#801) and "the `layers × 2` floor realised + prefix sharing available." Refs #777 #797 #794 #801 #809 (and #787 for token-major). Does **not** merge, does **not** close any issue. ## The question this PR answers > What remains between "growth byte geometry is seq-major" and "a real model runs seq-major end to end with the floor realised and prefix sharing available"? A fourth place the layout lives, or can the declared BNSH shape and a seq-major physical layout coexist provided only our own kernels read the buffer? **Both.** The declared BNSH shape and a seq-major physical layout *do* coexist on the native decode path (KV never leaves the device; only our GQA kernel indexes it). But there **is a fourth place the layout lives** — two engine-side *non-kernel* consumers of the device KV buffer that still stride it with hard-coded head-major (BNSH) arithmetic: 1. **`DecodeCudaState::read_present_kv`** (`native_decode/cuda.rs`) — the device present-KV **host mirror** used by prefix/paged read-out and diagnostics. It compacts per head with `capacity_head_stride = physical_shape[2] * head_dim * elem` and a per-head fragment read. 2. **`DecodeCudaState::seed_prefix`** (`native_decode/cuda.rs`) — the **paged prefix-reuse device seed** (#777/#809's device counterpart). It writes each head at `head * capacity_head_stride * elem_size`, the same head-major capacity offset. Under a seq-major (BSNH) physical buffer, heads are **interleaved per token** (`slot * kv_heads * head_dim + h * head_dim + d`), not laid out as `max_len × head_dim` capacity stripes. So both offsets index the wrong bytes: the host mirror would **silently return mis-indexed KV**, and the prefix seed would **corrupt the cache**. Neither had a seq-major guard — a real latent-corruption path. This is the fourth place the layout lives, beyond the three that are already converted: 1. kernel indexing — done (#792/#794); 2. commit geometry — done (#797); 3. growth byte geometry — done (#801); 4. **engine-side non-kernel KV consumers — gated here.** ## What this PR changes `crates/onnx-genai-engine/src/native_decode/cuda.rs`: - `read_present_kv` and `seed_prefix` now **error under `kv_layout.is_seq_major()`** with a precise message stating the condition and why (head-major byte arithmetic vs interleaved seq-major bytes). This matches the kernel-level "unsupported readers/writers fail rather than silently mis-index" gate (`kv_stride.rs::require_converted_path_support`) at the engine level. - **Head-major is untouched and byte-identical** — the gate is a pure `if is_seq_major() { bail! }` before the existing code. Absent metadata ⇒ head-major ⇒ no gate. `docs/MEMORY_ARCHITECTURE.md` (§ "KV layout and residency"): records the fourth layout location and the refuse-not-mismap gate as a bullet in the existing residency-caveat block. ## The structural constraint (#801), re-verified The BNSH binding **metadata** shape must stay for both layouts. The CUDA GQA node validates `past_key`/`past_value` as `[batch, kv_heads, seq, head_dim]` and reads `present_capacity` from **axis 2 regardless of `kv_layout`** (`group_query_attention.rs`), and `DeviceIoBinding` requires matching axis order. `persistent_state_shapes` is therefore *correctly* BNSH for seq-major too; seq-major lives in the byte geometry, not a permuted shape. Confirmed still true after merging the concurrent GQA-workspace change. ## Why the `layers × 2` floor is still not realised through the engine (and what it needs) Measured directly, **before** the merge broke the harness, with the validated #801/#805 setup — stock `qwen2.5-0.5b-q4_0-mobius` (BNSH) vs a copy whose 24 GQA nodes carry `kv_layout=1`, forcing growth with `ONNX_GENAI_KV_MIN_BUCKET=8`, 48 greedy tokens, 3 growth events (`mobius_seqmajor_growth_parity_native_cuda`, this branch @ pre-merge): | Growth path | Capture | Output | Physical committed bytes (head = seq) | d2d KV copy bytes head | d2d KV copy bytes seq | |---|---|---|---:|---:|---:| | in-place VMM (default) | OFF | **byte-identical** | 100,663,296 | 688,576 | **0** | | in-place VMM (default) | ON | **byte-identical** | 100,663,296 | 688,576 | **0** | | legacy realloc | OFF | **byte-identical** | 786,432 | 688,576 | 688,576 | Capture-ON, per-process: head-major and seq-major both `(growth 3, captures 4, replays 39, invalidations 4)`. So on this branch the seq-major decode path is **byte-identical head vs seq, capture ON and OFF, and moves 0 KV bytes on in-place growth** — but **`physical_committed_bytes` is identical between layouts**, i.e. the floor is *not* realised through the engine. Two cited reasons: 1. **`vmm_growth_requests` commits the full bucket** (`(index, 0, full_bucket_bytes)`) for every KV binding, for both layouts, instead of the dense live-prefix ranges `native_decode/kv_commit.rs::live_prefix_ranges` computes (those helpers are unit-tested and mirror the driver GPU test but are `allow(dead_code)` — not wired into the engine). 2. **The `kv_heads×` difference only becomes *measurable* under a large fixed stride.** With bucket growth the bucket ≈ 2× valid, so on 0.5b both layouts land in **1 granule/binding** (48 granules = 96 MiB). The `kv_heads=2` gap appears only when a head stripe crosses a granule boundary, i.e. `capacity × head_dim × elem ≥ granule` (`ONNX_GENAI_CUDA_KV_MAX_LEN=32768` in the driver test `vmm_kv_layout_residency_gpu`): head-major = `layers × 2 × kv_heads` granules, seq-major = `layers × 2` (the measured 192 MiB → 96 MiB, 2× = `kv_heads`; 1536 MiB → 192 MiB, 8× on qwen14b). Realising the measurable floor therefore requires **a fixed full-context stride + dense-prefix commit + per-step commit-ahead**, which eliminates growth-driven bucket recapture. That change lives in the **growth/capture-invalidation decision** — the explicit lane of the concurrent `conditional-invalidation` / `bsnh-fixed-stride` work — and outside this PR's shape/binding-construction lane. It is left as the documented next step (shared with token-major #787 and prefix sharing #777), consistent with the existing doc note: *"wiring the dense-prefix commit into the live path is the documented next step."* ## Prefix sharing (#777/#809) under seq-major It does **not** fall out naturally. #809's device consumer is `seed_prefix`, a head-major byte writer; a shared prefix is only physically contiguous under seq-major, but the seed that would install it cannot yet address the seq-major buffer. This PR makes that explicit (the seed errors under seq-major) rather than letting prefix reuse silently corrupt seq-major KV. Realising it means teaching `seed_prefix` (and `read_present_kv`) the BSNH byte geometry — same dense-prefix work as the floor. ## Validation - `cargo fmt -p onnx-genai-engine -- --check` ✓ (`--all` fails with os error 206 on this box — a command-line-length artifact, not formatting) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✓ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✓ (post-merge) - full `cargo test -p onnx-runtime-session` ✓ (post-merge) - `mobius_seqmajor_growth_parity_native_cuda` (byte-identical head vs seq, capture ON/OFF, 0 bytes moved) ✓ **both pre-merge and post-merge** (post-merge re-run includes the #814 governor fix, see below); the gate does not regress it. All four configs identical to the pre-merge numbers above (`physical_committed_bytes` 100,663,296 / 786,432; seq `kv_bytes_moved=0`). ### Cross-agent regression surfaced by `merge origin/main` — now fixed by #814 Merging `origin/main` (which brought the `gqa-bnsh-transpose` GQA transpose-scratch change #810) initially broke the mobius end-to-end test **during prefill workspace prep**, before any decode: ``` Error: prepare native CUDA prefill workspace Caused by: cannot reserve 2097152 bytes of device memory for Workspace { step_scoped: false }: 0 of 2097152 bytes are already leased, leaving 0 ``` #810 reserves a new **session-persistent** BNSH transpose-scratch `Workspace { step_scoped: false }` (`WorkspaceLifetime::SessionPersistent`) during prefill; the `vram-limit dynamic KV/weight lending` strategy did not provision headroom for it (GPU otherwise 100% free — 0 MiB used). This was a governor role-budgeting interaction in the **GQA/memory lane**, independent of this PR's gate (which only adds error paths in `read_present_kv`/`seed_prefix`, never on the prefill workspace path). I attributed it precisely and reported it to the `gqa-bnsh-transpose` sibling rather than fixing out of lane. **It has since been fixed forward in #814** (commit `54f89a03`: "mapped growth now claims unassigned authority capacity before holder reclaim, with rollback/overestimate accounting"). Merging #814 locally on top of this branch, the mobius test **passes all four VMM/realloc × capture configurations, byte-identical head vs seq**, confirming this PR's gate and #814's governor fix coexist cleanly. #814 is not yet on `main`, so this branch is left at its clean two-file state (the fix will arrive via `main`); the post-merge validation above was run against `this branch + #814` locally. ## Files - `crates/onnx-genai-engine/src/native_decode/cuda.rs` — seq-major refuse-not-mismap gate on the two head-major-only KV consumers. - `docs/MEMORY_ARCHITECTURE.md` — fourth-layout-location note. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…: 4 -> 0 growth invalidations) (#811) ## Headline: seq-major KV growth no longer needlessly re-captures the decode graph Measured, process-isolated, on `qwen2.5-0.5b-q4_0-mobius` (+ the `kv_layout=1` patched copy from #801/#805), capture ON, forced growth (3 bucket growths, output byte-identical throughout): | layout (capture ON, forced growth) | captures | replays | invalidations | growth_keeps | |---|---:|---:|---:|---:| | head-major (**before = after**, moves bytes) | 4 | 39 | 4 | 0 | | seq-major **before** (#805) | 4 | 39 | 4 | 0 | | seq-major **after** (this PR) | **1** | 45 | **1** | **3** | Seq-major's forced-growth graph accounting now matches its **no-growth control exactly** (`captures=1, replays=45, invalidations=1`): all **3 growth-attributable invalidations are eliminated** and recorded as keeps. The single residual invalidation is a baseline (present in the no-growth control too), not growth-attributable. Head-major is unchanged — it genuinely re-strides every head stripe on growth (688,576 bytes moved), so it must re-capture. Output is **byte-identical** to `main` across both layouts, capture ON and OFF, VMM and legacy realloc. ## Why this is sound (the dependency set I check, and why it is complete) The CUDA GQA decode kernel bakes exactly two KV-dependent things at capture: - **device pointers** into the recorded graph nodes, and - the **physical shape** (kernel argument extents / strides), which the executor's `replay_device_graph` folds into a per-binding replay **signature** (device_ptr + physical_shape + dtype + I/O identity) and *rejects* replay on any change. Everything else the decode graph reads at replay is **data**, not baked: the key extent and launch geometry derive from the `total_lengths` **device pointer** (read at replay), never from `cache_capacity` (kernel comment `gqa_decode_fp16.rs:388`). And `cache_capacity` enters seq-major addressing (`kv_stride.rs`) only through the **batch** term of `GQA_KV_BASE` — which is `0` in native decode (batch=1, b=0). So at b=0, seq-major addressing is **capacity-independent**: token `t` always lands at `t*kv_heads*head_dim`. Therefore, if I keep the device pointers and physical shapes identical across a growth, **no captured dependency changes**. I achieve that by pinning the seq-major fixed-stride path's KV physical shape (and mask island) at the **hard-max** context length from construction, committing only the initial dense token prefix, and growing by committing more granules on demand at the **same base VA** (in-place VMM, proven at the driver level by #797 and byte-identical with 0 bytes moved by #805). Growth changes neither pointer nor shape nor addressing. **Default-to-invalidate, defense in depth:** before keeping, I snapshot every binding's `(device_ptr, physical_shape)` and re-check it after the commit; if anything moved, I **invalidate** and say so. The executor's replay signature check is a further backstop that fails safe (errors, never silently corrupts) if a pointer ever changed unexpectedly. Head-major keeps invalidating (its per-head stride *is* `cache_capacity`), and the legacy realloc path (`ONNX_GENAI_LEGACY_ALLOCATOR=1`, `commits_on_demand == false`) keeps invalidating — verified: seq-major realloc still moves 688,576 bytes and invalidates 4×, identical to head-major. ## Reporting (#804) The decision is surfaced through named reporting so an operator can see *why* a graph was kept or invalidated: - `CudaGraphDebugStats.growth_keeps` (count) and `CudaGraphDebugStats.growth_decision` (named rationale) on `CudaKvDebugStats`; - a `tracing::info!` event per decision. A silent "kept" is as dangerous as the silent `captures=0` that cost prior PRs, so every keep is counted and logged. ## Tests - **Negative oracle (the test that matters most)** — `head_major_growth_that_moves_addresses_must_invalidate_not_keep`: forces a growth that genuinely relocates KV addresses (head-major, 688,576 bytes moved) and asserts the graph is **invalidated and re-captured, never kept** (`growth_keeps == 0`, one extra invalidation + one extra capture per growth vs. the no-growth control). Fails red if a graph is ever kept when a dependency changed. - The #805 harness `mobius_seqmajor_growth_parity_native_cuda` is extended to assert the conditional-keep counts above and that seq-major forced-growth accounting equals its no-growth control, while keeping the byte-identical parity assertions. - `gqa_seqmajor_parity_gpu` (independent CPU prefill + decode oracles) stays green. ## Validation run (all green) - `cargo fmt -p onnx-genai-engine -- --check` - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` - `cargo clippy --locked --all-targets -p onnx-genai-engine --features cuda,native-backend -- -D warnings` - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` - full `cargo test -p onnx-runtime-session` — all `ok`, 0 failed (153 unit + all integration/doctests) - GPU, single-threaded: `mobius_seqmajor_growth_parity_native_cuda` (2 passed, incl. negative oracle), `gqa_seqmajor_parity_gpu` (2), `group_query_attention_gpu` (26), `vmm_kv_contiguous_tail_gpu` / `vmm_kv_layout_residency_gpu` / `vmm_kv_token_major_floor_gpu` (7 total) ## Scope / notes - Edits are confined to the capture/invalidation decision path (`native_decode/cuda.rs`) and the growth harness, to merge cleanly alongside concurrent work. - Never asserts inside a `Drop`; never maps/grows inside a captured region (commit happens between decode steps); never unmaps a live mapping (VMM keeps the base VA stable). - Per-run tok/s on this shared box is not published — identical runs range widely and other agents build concurrently; only the deterministic counters above are reported. Refs #805 #782 #797 #801 #804. Does **not** merge and does **not** close any issue. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…ative) + #384 metadata generator (#834) ## Summary Demonstrates the KV-cache committed-**physical**-bytes floor end-to-end on **qwen14b** via the native CUDA decode path, comparing **head-major (BNSH)** vs **seq-major (BSNH)** layouts in the same session, swept across context capacities that bracket the 2 MiB-per-stripe granule threshold (#776). Token IDs are **byte-identical** between layouts at every point. This is the first time this line (chased since #797) has been driven through the *engine* on a model whose head stripe actually reaches a full granule (qwen14b @ 8192 ctx: `8192 x 128 x 2 = 2,097,152 B = exactly 2 MiB`), rather than the sub-granule qwen0.5b of #827. ## Headline result — honest negative (per #794 / #812 / #827 precedent) Head-major and seq-major commit **byte-identical** physical KV bytes at every swept capacity: | KV capacity (tokens) | committed bytes (both layouts) | granules | ratio seq/head | |---:|---:|---:|---:| | 1024 | 402,653,184 | 192 | 1.00x | | 2048 | 603,979,776 | 288 | 1.00x | | 4096 | 1,006,632,960 | 480 | 1.00x | | 8192 | 1,811,939,328 | 864 | 1.00x | Expected-if-it-separated: head-major `layers x 2 x kv_heads` = **768 granules (~1.5 GiB)** vs seq-major `layers x 2` = **96 granules (~192 MiB)** — the 8x that never appeared. ## Why it does not separate `growth_events == 0` and `committed_len == capacity` for **both** layouts: the engine commits the full KV bucket **eagerly** on reservation — reservation *is* the commit, so layout cannot change the total. Seq-major's fixed full-context stride *is* active and correct (`max_len == 8192` at every capacity vs head-major `max_len == capacity`, tokens match), which proves the seq-major kernel/layout resolution works — but a fixed stride only makes **growth** free (#797's 0 bytes moved) and preserves the captured graph (#811/#812); it does not shrink an eagerly-committed full bucket. The 768-vs-96 (8x) floor stays a **driver-level** property (`vmm_kv_layout_residency_gpu`) realized only by a **live-prefix commit-on-demand** path, which the engine does not do on initial reservation. The new guardrail assertions (`growth_events==0`, `committed_len==capacity`, `head==seq committed`) will **turn red** the moment a future change makes seq-major commit less — i.e. they are a tripwire for the desired win. Measurement used the in-place **VMM** allocator (`ONNX_GENAI_CUDA_VMM=1`), which reports granule-rounded **physical** bytes; the legacy/governor allocator reports nominal bytes and was deliberately avoided. Capture was declined throughout (`ONNX_GENAI_CUDA_GRAPH=0`); offload/weight-streaming (16.6 GB model vs 7.7 GB budget, #796) was active and did not perturb the KV commit counters. ## Deliverables 1. **`crates/onnx-genai-engine/tests/qwen14b_kv_floor_sweep_native_cuda.rs`** — child-process-isolated sweep harness (mirrors `mobius_seqmajor_growth_parity_native_cuda`), honoring process-frozen config (#804/#807). Self-execs per `(layout, capacity)`; asserts token parity + layout resolution + the deterministic guardrails above. `#[ignore]`, gated on `cuda,native-backend`. 2. **`scripts/gen_inference_metadata.py`** (#384) — emits `inference_metadata.yaml` from `genai_config.json` for any such export. Reproduces the hand-written qwen14b metadata **byte-for-byte**. Turns the one-off local `qwen14b-zp` fix (which unblocked native loading for this whole measurement) into a reproducible artifact; the model file itself stays local (models live outside the repo). 3. **Docs** — `MEMORY_ARCHITECTURE.md` floor-progression table + eager-full-bucket explanation; `MODEL_METADATA.md` #384 generator section. ## Method notes - Seq-major was obtained by patching a **copy** of the model (`kv_layout=1` on all 48 GQA nodes), as #801/#805 did for mobius — the original model directory was never modified. - Reported committed **physical** bytes throughout, never nominal content bytes. - Deterministic counters lead; wall-clock is omitted (this box's identical runs ranged 3.9-28 tok/s) and no number was extrapolated that was not measured. ## Unrelated pre-existing fix (disclosed) `crates/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs`: added a `#[cfg(not(target_arch = "aarch64"))] let _ = dot_kernel;` guard in `borrowed_affine_int4_matmul` to silence a pre-existing x86 unused-variable lint that blocked `-D warnings` (matches sibling functions at 3779/4272/4448). Behavior-neutral; the mobius ignored test still passes (2 passed). ## Validation - `cargo fmt -p onnx-genai-engine -p onnx-runtime-ep-cpu -- --check` (per-package; `--all` hits Windows `os error 206`) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` — pass - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` — pass - `cargo test -p onnx-runtime-session` (full) — pass - `cargo test -p onnx-genai-engine --features cuda,native-backend --test mobius_seqmajor_growth_parity_native_cuda -- --ignored --test-threads=1` (#814 policy) — 2 passed - New sweep harness run twice; all guardrails passed. Refs #797 #787 #827 #384 #777. Do **not** merge; does **not** close any issue. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Headline: fixed-stride seq-major still has 4 invalidations, equal to head-major
The #782 prediction is not closed by the current end-to-end engine path. On the default managed in-place VMM allocator, process-isolated and interleaved capture-ON runs measured:
The 48 generated tokens and text were byte-identical. Every individual configuration ran in its own process, avoiding #804's cached-policy failure. Runs were interleaved head-major then seq-major. Wall-clock is intentionally omitted because deterministic counters answer the question and this shared box has high timing variance.
A no-growth control (
min_bucket=64) measured1 capture, 45 replays, 1 invalidationfor both layouts. Thus three forced growths add exactly three invalidations and three captures in both layouts. Code attribution agrees:DecodeCudaState::ensure_capacitycallsinvalidate_graphunconditionally after VMM growth, before the layout-specific moved-byte accounting. Fixed stride removes KV movement but the engine does not yet retain the graph based on that fact.The harness now process-isolates every layout/configuration, emits structured measurements, asserts output parity, and keeps the legacy realloc check (
786,432physical bytes;688,576moved for both). Documentation records the actual invalidator rather than claiming end-to-end zero re-captures.Validation
cargo fmt -p onnx-genai-engine -- --checkcargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warningscargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warningscargo test -p onnx-runtime-session(153 primary tests plus all secondary binaries)Refs #782, #797, #801, #804. Does not close or merge any issue.