Enable stride-described seq-major flash prefill by justinchuby · Pull Request #794 · justinchuby/onnx-genai

justinchuby · 2026-08-11T22:47:59Z

Summary

convert flash/prefill KV reads, prefill cache writes, append, and K-RoPE indexing to the shared KvCacheStrides descriptor
keep NVRTC specialization static per layout and bump the affected module keys
centralize converted-path support in kv_stride.rs, preserving hard errors for unsupported descriptors/paths
add full prefill + decode seq-major parity coverage with independent CPU prefill and decode oracles, both with CUDA graph capture enabled and disabled

This completes the CUDA compute path needed for a full seq-major generation through flash prefill and fp16 split-K decode. Head-major remains the default, and absent metadata still selects head-major.

Related: #782, #783, #792, #750, #777.

Correctness

The new GPU test runs an actual prefill followed by decode and verifies:

head-major and seq-major output bytes are identical
each layout independently matches CPU causal-prefill and decode oracles
capture-on and capture-off results agree
capture installs one graph segment

I also patched the available native int4 qwen2.5 0.5B model with kv_layout=1; head-major and seq-major produced the same 32 token IDs and identical text.

The exact requested qwen2.5-0.5b-int4-onnx-native directory was not present on this machine, so this check used the available qwen2.5-0.5b-q4_0-mobius native model instead.

Measurements

Measurements were taken in the same local session and interleaved. These are committed physical bytes, not nominal KV content bytes.

Model	Head-major committed	Seq-major committed	Result
qwen2.5 0.5B native int4	100,663,296 B	100,663,296 B	equal
qwen14b-zp	402,653,184 B	402,653,184 B	equal

For the 0.5B capture run, both layouts reported one installed segment, one initial capture, 29 replays, zero post-install re-captures, and zero fallbacks.

This refutes the expected residency improvement at the current engine boundary: native CUDA bindings still allocate bucket-sized packed shapes and commit flat bucket ranges without consuming the KV layout metadata. Therefore seq-major currently changes kernel indexing but not reservation/commit geometry. For the same reason, there was no distinct end-to-end layout-specific boundary-crossing event from which to reproduce the 49.5 ms versus 4.5 ms microbenchmark result; boundary and normal steps use the same current growth geometry.

The 14B model required weight offload on this GPU, which disables CUDA graph capture, so graph segment/re-capture measurements are not available for that scale run. Both layouts generated token 96347 (Polit). Wall-clock throughput and decode ms/token varied too widely on this box to report as a meaningful layout comparison; deterministic byte, residency, and capture counters are reported above instead.

Remaining work

Seq-major is now usable end to end for the flash-prefill + fp16-decode compute path. The expected physical-memory and growth-boundary benefits still require layout-aware native binding/residency allocation. Non-flash/reference, f32 decode, phase2a, and unfused decode-prep paths remain deliberately rejected for seq-major until converted. Token-major and prefix sharing remain out of scope.

Validation

cargo fmt -p onnx-runtime-ep-cuda -- --check
cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings
cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings
cargo test -p onnx-runtime-ep-cuda --features cuda kv_stride --lib
cargo test -p onnx-runtime-ep-cuda --features cuda --test gqa_seqmajor_parity_gpu -- --test-threads=1
cargo test -p onnx-runtime-ep-cuda --features cuda --test group_query_attention_gpu -- --test-threads=1 (25 passed, one benchmark ignored)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov · 2026-08-11T22:54:54Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.40%. Comparing base (3e85d55) to head (0de1a01).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #794      +/-   ##
==========================================
- Coverage   80.08%   79.40%   -0.68%     
==========================================
  Files         331      329       -2     
  Lines      137782   135612    -2170     
  Branches   137782   135612    -2170     
==========================================
- Hits       110339   107685    -2654     
- Misses      23091    23590     +499     
+ Partials     4352     4337      -15

Flag	Coverage Δ
cli-ort-linux	`83.50% <ø> (ø)`
cli-ort-windows	`83.09% <ø> (ø)`
mlas	`?`
offline	`79.24% <ø> (-0.69%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

PR #794 measured head-major and seq-major committing identical physical bytes on both models, because native CUDA bindings still commit flat bucket ranges without consuming the KV layout metadata. The floor table described the layout geometry, which a reader would take as current behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

justinchuby · 2026-08-11T23:07:15Z

Coordinator validation (merged branch, Windows/RTX box): cargo fmt -p onnx-runtime-ep-cuda -- --check clean, cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings clean (exact CI gate), gqa_seqmajor_parity_gpu
2/2 including the new seq_major_full_generation_prefill_and_decode_is_bit_identical,
and group_query_attention_gpu head-major regression 25/25.

The most valuable thing in this PR is the negative result

head-major and seq-major commit identical physical bytes — 100,663,296 B on
qwen2.5-0.5b and 402,653,184 B on qwen14b

Because native CUDA bindings still allocate bucket-sized packed shapes and commit
flat bucket ranges without consuming the KV layout metadata, seq-major today
changes kernel indexing only — not reservation or commit geometry. So the
expected kv_headsx residency improvement does not materialize yet.

This matters because I had been describing #782 as having delivered an 8x floor
reduction on qwen14b. It has not. The kernels index correctly; the allocator still
commits bucket-flat ranges. #787 reached the same conclusion from the other
direction — the read path is free, the cost is on the binding layer — and this
measurement confirms it from the engine side.

I have added a note to the floor table in docs/MEMORY_ARCHITECTURE.md before
merging, because the table as written ("Decode pair landed in #782") would lead a
reader to believe ~192 MiB is what the runtime commits today. The floors are
properties of the layout geometry, not of current behavior, and that
distinction is exactly the class of overstatement #781 existed to correct. The row
now reads "Geometry only — not realized today" and cites the measured equality.

What did land

The CUDA compute path for a full seq-major generation: flash/prefill KV reads,
prefill cache writes, append, and K-RoPE indexing all now go through the shared
KvCacheStrides descriptor, with static per-layout NVRTC specialization and no
runtime branch. Converted-path support stays centralized in kv_stride.rs, so
unsupported descriptors still error rather than mis-index — the property that
lets seq-major exist in the tree without being able to half-work.

The new GPU test runs a real prefill followed by decode and checks four things:
byte-identical output between layouts, each layout independently matching
separate CPU causal-prefill and decode oracles, capture-on equals capture-off,
and one installed graph segment. The independent oracles are the load-bearing part
— without them a shared indexing error in both layouts would still agree
byte-for-byte and pass.

Two honest limitations, correctly reported rather than papered over: the requested
qwen2.5-0.5b-int4-onnx-native directory is not on this machine so the
end-to-end check used qwen2.5-0.5b-q4_0-mobius (both layouts produced identical
32 token IDs and text); and the 14B run needed weight offload, which still
disables capture, so no segment/re-capture numbers exist at that scale. Wall-clock
was not reported as a layout comparison, which is right — the box was running
three concurrent CUDA agents and its throughput variance was already extreme.

That offload-disables-capture limitation is #716, in progress on another branch,
and is the hard prerequisite for #755.

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	51.67 µs	97.79 µs	+89.3%
🔴	`matmul/medium_generic_f32_threads=8/32x512x512`	1.25 ms	1.81 ms	+45.1%
🔴	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	892.34 µs	1.21 ms	+35.1%
⚠️	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	3.61 ms	4.32 ms	+19.6%
⚠️	`kv_cache/alloc_dealloc_pages`	38.75 µs	45.94 µs	+18.6%
✅	`qwen3_sampling_processors/top_k_top_p_fast`	677.02 µs	774.49 µs	+14.4%
✅	`gather/large_bf16_threads=1-internal/131072`	15.87 µs	18.09 µs	+14.0%
✅	`logit_processing/seven_processor_chain_per_step`	321.40 µs	365.93 µs	+13.9%
✅	`gather/large_f16_threads=1-internal/131072`	15.80 µs	17.95 µs	+13.6%
✅	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	5.94 ms	6.63 ms	+11.7%
✅	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	53.59 µs	59.28 µs	+10.6%
✅	`qwen3_sampling_processors/top_p_fast_after_top_k`	523.61 µs	576.46 µs	+10.1%
✅	`qwen3_sampling_processors/top_k_partial_selection`	140.28 µs	151.49 µs	+8.0%
✅	`gather/large_f32_threads=1-internal/131072`	45.62 µs	48.70 µs	+6.7%
✅	`matmul/small_generic_f32_threads=8/1x256x256`	51.58 µs	55.05 µs	+6.7%
✅	`sampling_latency/top_p_per_token`	390.49 µs	409.51 µs	+4.9%
✅	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	83.14 µs	86.98 µs	+4.6%
✅	`gather/small_f16_threads=1-internal/4096`	480.2 ns	501.3 ns	+4.4%
✅	`matmul/large_generic_f16_threads=8/32x1024x1024`	103.81 µs	108.36 µs	+4.4%
✅	`matmul/large_generic_f32_threads=1/32x1024x1024`	9.51 ms	9.88 ms	+3.9%
✅	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	459.47 µs	469.46 µs	+2.2%
✅	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.20 ms	2.24 ms	+2.1%
✅	`matmul/large_generic_bf16_threads=1/32x1024x1024`	2.04 ms	2.07 ms	+1.2%
✅	`matmul/large_generic_f32_threads=8/32x1024x1024`	6.84 ms	6.92 ms	+1.1%
✅	`matmul/large_generic_bf16_threads=8/32x1024x1024`	2.31 ms	2.34 ms	+1.1%
✅	`sampling_latency/min_p_per_token`	227.49 µs	229.90 µs	+1.1%
✅	`grammar_masking/llguidance_compute_mask/32`	78.51 µs	79.31 µs	+1.0%
✅	`tokenization/decode_tokens_per_second`	6.73 ms	6.78 ms	+0.8%
✅	`matmul/small_generic_bf16_threads=8/1x256x256`	39.15 µs	39.18 µs	+0.1%
✅	`sampling_latency/greedy_per_token`	3.26 µs	3.24 µs	-0.4%
✅	`matmul/medium_generic_f16_threads=8/32x512x512`	31.11 µs	30.95 µs	-0.5%
✅	`matmul/medium_generic_f16_threads=1/32x512x512`	33.04 µs	32.86 µs	-0.5%
✅	`matmul/small_generic_bf16_threads=1/1x256x256`	36.50 µs	36.07 µs	-1.2%
✅	`tokenization/encode_tokens_per_second`	398.67 µs	392.82 µs	-1.5%
✅	`gather/small_bf16_threads=1-internal/4096`	486.2 ns	478.2 ns	-1.6%
✅	`matmul/small_generic_f16_threads=8/1x256x256`	38.60 µs	37.86 µs	-1.9%
✅	`gather/medium_f16_threads=1-internal/32768`	2.52 µs	2.41 µs	-4.1%
✅	`add/small_bf16_threads=1-internal/1024`	14.02 µs	13.33 µs	-4.9%
✅	`matmul/small_generic_f16_threads=1/1x256x256`	38.86 µs	36.90 µs	-5.1%
✅	`matmul/large_generic_f16_threads=1/32x1024x1024`	97.54 µs	92.46 µs	-5.2%
✅	`reduce_mean/medium_f32_threads=1-internal/65536`	263.81 µs	249.11 µs	-5.6%
✅	`matmul/medium_generic_bf16_threads=1/32x512x512`	572.13 µs	535.24 µs	-6.4%
✅	`sampling_latency/top_k_per_token`	56.09 µs	52.44 µs	-6.5%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	46.62 µs	42.78 µs	-8.2%
✅	`reduce_mean/large_f32_threads=1-internal/262144`	1.13 ms	1.03 ms	-8.8%
✅	`add/medium_f16_threads=1-internal/262144`	2.87 ms	2.59 ms	-9.7%
✅	`add/small_f16_threads=1-internal/1024`	14.61 µs	13.02 µs	-10.9%
✅	`add/medium_f32_threads=1-internal/262144`	2.89 ms	2.54 ms	-12.2%
✅	`gather/medium_bf16_threads=1-internal/32768`	2.81 µs	2.39 µs	-15.0%
🟢	`add/large_f16_threads=1-internal/4194304`	49.35 ms	41.79 ms	-15.3%
🟢	`matmul/medium_generic_f32_threads=1/32x512x512`	2.77 ms	2.34 ms	-15.6%
🟢	`gather/small_f32_threads=1-internal/4096`	858.1 ns	692.6 ns	-19.3%
🟢	`matmul/medium_generic_bf16_threads=8/32x512x512`	644.48 µs	516.37 µs	-19.9%
🟢	`add/large_f32_threads=1-internal/4194304`	52.83 ms	40.75 ms	-22.9%
🟢	`add/small_f32_threads=1-internal/1024`	276.5 ns	208.3 ns	-24.7%
🟢	`add/medium_bf16_threads=1-internal/262144`	3.55 ms	2.62 ms	-26.2%
🟢	`reduce_mean/small_f32_threads=1-internal/4096`	21.44 µs	15.37 µs	-28.3%
🟢	`add/large_bf16_threads=1-internal/4194304`	61.49 ms	43.83 ms	-28.7%
🟢	`gather/medium_f32_threads=1-internal/32768`	7.66 µs	4.17 µs	-45.6%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.48 3.27 5.39 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

…ering The seq_major_fixed_stride_grows_under_captured_replay_without_recapture test in vmm_kv_layout_residency_gpu.rs failed ~50% of the time when run in isolation on a cold CUDA context (all-zero or partial readbacks with no CUDA error). Root cause is a test-harness ordering defect, not a product defect and not a merge interaction with #796 (which added no cuda-memory source): The baseline zero-fill and the residency readback issued synchronous copies on the legacy default stream, while the captured graph memset ran on a created non-blocking stream. Those two streams are mutually exempt from implicit synchronization, so on a cold context the readback raced the just-launched graph memset. It only appeared to pass because the sibling subtest runs first alphabetically and warms the context. Fix: route every device operation (baseline fill, captured memset, readback, tail write/read) through the same stream so a single cuStreamSynchronize is a total order. No assertion was relaxed; the product property (a captured graph replays unchanged across tail-granule growth at a stable VA — the #782/#794 0-re-captures win) is genuinely demonstrated and now reliable (40/40 in isolation). Also: - Replace cascading result.unwrap() (in commit_and_measure and the growth test) with std::panic::resume_unwind so the first failure reports cleanly instead of panicking again during teardown with 'Any { .. }'. - Add a mismatch_report diagnostic that names the failure mode (never-visible vs torn write) and a cuGraphGetNodes node-count assertion proving the captured graph holds exactly the memset node. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Headline: layout now moves committed physical bytes PR #794 converted flash prefill and decode to the shared `KvCacheStrides` descriptor and then measured that head-major and seq-major commit **identical** physical bytes — `100,663,296 B` on qwen2.5-0.5b and `402,653,184 B` on qwen14b. The cause was on the binding layer: the native CUDA KV binding always committed a single flat bucket range `0..(capacity × kv_heads × head_dim × elem)` regardless of layout, so seq-major changed **kernel indexing only**, not reservation or commit geometry. #787 reached the same conclusion from the allocator side: the read path is free; the cost is on the binding layer. This PR builds the layout-aware commit **geometry**, makes the binding layer **consume** the KV layout descriptor, and **measures** the residency delta directly at the CUDA driver level. ### Measured committed physical bytes (deterministic, `vmm_kv_layout_residency_gpu`) On a fixed full-context stride (the stable-VA regime), at the near-empty floor: | Model | Per-binding head-major | Per-binding seq-major | Ratio | Model-wide head-major | Model-wide seq-major | |---|---:|---:|---:|---:|---:| | qwen2.5-0.5b | 4 MiB (2 granules) | 2 MiB (1 granule) | **2× = kv_heads** | 192 MiB (96 granules) | 96 MiB (48 granules) | | qwen14b | 16 MiB (8 granules) | 2 MiB (1 granule) | **8× = kv_heads** | 1536 MiB (768 granules) | 192 MiB (96 granules) | This is the number #794 measured as *equal*. It moves by the `kv_heads` factor once committed geometry follows the descriptor — the `layers × 2` floor materialising, on real 2 MiB CUDA granules. ### Growth / re-capture The same GPU test grows a seq-major fixed-stride binding **under a captured replay** — committing tail granules at a stable VA and stride (granule `0 → 3`) — and the originally captured graph replays unchanged: **0 re-captures**. This is the stable-stride win #782 could not demonstrate end to end, shown here on hardware. Bucket growth, by contrast, re-strides and forces a re-capture. ### Boundary-crossing step cost Not measured end to end in this PR, and reported as such rather than extrapolated. #778's 49.5 ms vs 4.5 ms microbenchmark needs a seq-major **generation** through the engine to reproduce, which requires the BSNH fixed-stride shape build described below (there is no seq-major model on record to drive it). The driver-level residency measurement is the correctness-independent result this PR stands on. ## What changed - **`crates/onnx-genai-engine/src/native_decode/kv_commit.rs`** (new) — pure, unit-tested `KvCommitLayout` geometry. `live_prefix_ranges` returns the commit ranges a layout implies on a fixed full-context stride: seq-major is one dense run `0..ceil(live_bytes/granule)`; head-major is one live fragment per head stripe. `KvCommitLayout::default() == HeadMajor`, so **absent metadata means head-major, byte-identical to today**. 9 unit tests. - **Engine consumes the descriptor** — `resolve_cuda_kv_layout` maps `ModelIoSpec.kv_layout` (`#792` view-carrying descriptor) to `KvCommitLayout` (env override `ONNX_GENAI_CUDA_KV_LAYOUT` for diagnostics only), threaded from `load.rs` into `DecodeCudaState`, surfaced as `CudaKvDebugStats.kv_layout_seq_major`. Head-major keeps its exact behavior; the growth path (`apply_vmm_growth` / `vmm_growth_requests`, which #716 edits) is **untouched** for a clean merge. - **`crates/onnx-runtime-cuda-memory/tests/vmm_kv_layout_residency_gpu.rs`** (new) — driver-level GPU measurement (2 tests), the headline above. Uses the #740 pool's granule model; commits/reads back real committed bytes. - **`docs/MEMORY_ARCHITECTURE.md`** — floor table + note updated from "not realized today" to the measured driver-level result, plus the precise structural boundary. ## Honest structural boundary (per the task's "stop and report" clause) The layout-aware commit *geometry* is built and measured, and the engine now consumes the layout metadata. What is **not** wired is the seq-major dense-prefix commit on the end-to-end **generation** path, because: - `persistent_state_shapes` hard-codes a **BNSH** physical shape whose axis-2 (seq) grows; there is no **BSNH fixed-stride** physical-shape build to hang the dense-prefix commit on, and - no model on this machine declares seq-major, so a byte-identical seq-major *generation* cannot be validated here. The per-layer `key_cache_*`/`value_cache_*` bindings must survive as views (they do — this PR does not collapse them). The BSNH fixed-stride shape build is the shared prerequisite for token-major and #777 page-level prefix sharing, and is the documented next step. This is reported as information, not a failure. ## Correctness `gqa_seqmajor_parity_gpu::seq_major_full_generation_prefill_and_decode_is_bit_identical` (independent CPU prefill + decode oracles) passes — decode output is byte-identical between head-major and seq-major for a full generation, capture on and off. ## Validation (all green locally; not waiting for CI) - `cargo fmt -p onnx-genai-engine -p onnx-runtime-cuda-memory -- --check` ✓ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✓ - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✓ - full `cargo test -p onnx-runtime-session` ✓ - GPU single-threaded: `vmm_kv_layout_residency_gpu`, `vmm_kv_contiguous_tail_gpu`, `vmm_kv_token_major_floor_gpu`, `gqa_seqmajor_parity_gpu` (2), and `group_query_attention_gpu` (25 passed, 1 ignored H200 bench) ✓ - `kv_commit` unit tests (9) + `cuda_kv_layout_resolves_from_metadata_with_env_override` ✓ ## Notes / caveats - The box is heavily contended; wall-clock is unreliable (identical runs have ranged 3.9–28 tok/s). Results above lead with **byte and granule counters**. - **qwen14b** needs weight offload on this 8 GiB GPU, which still disables capture (#716, separate) — so capture counters are unavailable at that scale. The re-capture demonstration is at the driver level and on the small model. Refs #794, #787, #782, #750, #777. **Do not merge; does not close any issue.** --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

Make native CUDA KV state grow along a fixed-stride BSNH byte layout when the resolved layout is seq-major, closing the last engine-side gap to a real end-to-end seq-major generation (the documented next step after #797). The precise gap turned out not to be the *shape build* but the *growth byte geometry*. The CUDA GQA node validates past_key/past_value as BNSH [batch, kv_heads, seq, head_dim] and reads present_capacity from axis 2 regardless of the kv_layout attribute (only the kernel stride arithmetic re-specializes), and DeviceIoBinding requires logical/physical shapes to share axis order. So persistent_state_shapes must stay BNSH for both layouts; a permuted binding shape cannot express seq-major. Seq-major is realized purely in the growth/commit byte geometry. Changes (confined to native_decode/cuda.rs growth functions, to merge cleanly with the concurrent managed-VMM and GEMM-workspace work): - kv_growth_byte_layout(): map the BNSH binding shape to the physical byte layout + grow axis the resolved KvCommitLayout stores it in. Head-major = declared shape, grow axis 2 (one stripe per head, byte-identical to before). Seq-major = [batch, capacity, kv_heads, head_dim], grow axis 1, with a capacity-independent kv_heads*head_dim per-token stride. - apply_vmm_growth / build_grown_buffers: drive the existing device copy/zero primitives over that byte layout instead of a hard-coded axis-2 scatter. - ensure_capacity: in-place VMM growth accounts 0 moved KV bytes for seq-major (fixed stride keeps the prefix in place) vs valid_len*bytes_per_token for head-major. - persistent_state_shapes: documented the BNSH-metadata contract. head-major is byte-identical to today (default() == HeadMajor and absent metadata unchanged), asserted explicitly in a unit test. Validation: - 2 new engine unit tests: head-major stays BNSH / seq-major permutes to BSNH; seq-major growth is fixed-stride and moves no bytes while head-major re-strides. - New real-model e2e (mobius_seqmajor_growth_parity_native_cuda, #[ignore]): patches qwen2.5-0.5b-q4_0-mobius GQA nodes with kv_layout=1, forces growth with ONNX_GENAI_KV_MIN_BUCKET=8, generates 48 greedy tokens over 3 growth events, and asserts byte-identical head-major vs seq-major output, capture requested OFF and ON, under both growth mechanisms. Measured end to end: in-place VMM seq-major growth moved 0 KV bytes vs head-major 688,576. - gqa_seqmajor_parity_gpu (independent CPU oracles), group_query_attention_gpu (25), vmm_kv_layout_residency_gpu, vmm_kv_token_major_floor_gpu, vmm_stable_va_weight_slot_gpu all green single-threaded; full onnx-runtime-session green; engine + ep-cuda clippy gates green. Honest caveats (orthogonal, out of scope): committed physical bytes remain equal head vs seq (flat bucket/granule commit; the kv_heads-x residency win is still driver-level only, #797); this decoder's dynamic logical shape declines whole-step graph capture (captures=0) so the boundary re-capture cost could not be reproduced end to end (the obstacle #794 also hit); wall-clock not reported (box contended, 7 s-212 s spread this session). Refs #797, #794, #782, #750, #777. Does not flip any default; no issue closed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…801) ## Summary Builds native CUDA KV state to **grow along a fixed-stride BSNH byte layout** when the resolved layout is seq-major, closing the last engine-side gap to a real end-to-end seq-major generation — the documented next step after #797, and the shared prerequisite for token-major and #777 page-level prefix sharing. **Yes — a real seq-major generation is now possible.** On `qwen2.5-0.5b-q4_0-mobius` (stock BNSH vs a copy with `kv_layout=1` on its 24 GQA nodes), forcing growth with `ONNX_GENAI_KV_MIN_BUCKET=8` and generating 48 greedy tokens across 3 growth events, the head-major and seq-major token streams are **byte-identical**, capture requested OFF and ON, under both KV growth mechanisms. Rebased on current `origin/main` (through #798/#799/#800). Full `cargo test -p onnx-runtime-session` re-run green after the rebase (153 in the primary binary, 0 failures across all binaries). ## The precise structural finding The gap was **not** the shape build — it was the growth *byte geometry*. The CUDA GQA node validates `past_key`/`past_value` as BNSH `[batch, kv_heads, seq, head_dim]` and reads `present_capacity` from **axis 2** (`inputs[3].shape[2]` / `outputs[1].shape[2]`) **regardless** of the `kv_layout` attribute (only the kernel stride arithmetic re-specializes), and `DeviceIoBinding` requires logical/physical shapes to share axis order. So `persistent_state_shapes` **must stay BNSH for both layouts** — a permuted binding shape cannot express seq-major. Seq-major is realized purely in the growth/commit byte geometry. ## Changes (confined to `native_decode/cuda.rs` growth functions) - **`kv_growth_byte_layout()`** — maps the BNSH binding shape to the physical byte layout + grow axis the resolved `KvCommitLayout` stores it in. Head-major = declared shape, grow axis 2 (byte-identical to before). Seq-major = `[batch, capacity, kv_heads, head_dim]`, grow axis 1, capacity-independent `kv_heads*head_dim` per-token stride. - **`apply_vmm_growth` / `build_grown_buffers`** — drive the existing device copy/zero primitives over that byte layout instead of a hard-coded axis-2 scatter (which would corrupt BSNH). - **`ensure_capacity`** — in-place VMM growth accounts **0** moved KV bytes for seq-major (fixed stride keeps the prefix in place) vs `valid_len*bytes_per_token` for head-major. - **`persistent_state_shapes`** — documented the BNSH-metadata contract. **Head-major is byte-identical to today** (`default() == HeadMajor`, absent metadata unchanged), asserted explicitly. Edits are intentionally confined to KV shape/stride growth geometry so they merge cleanly with the (now-landed) managed-VMM-default (#798) and GEMM-workspace-governance (#799) work. ## Measurement (measured this session; interleaved, deterministic counters leading) Since #798 made managed no-spill VMM the **default**, the default native CUDA growth path is now **in-place VMM** — so seq-major grows moving **0 KV bytes by default**. The legacy reallocation allocator is reachable only via the #755 opt-out (`ONNX_GENAI_LEGACY_ALLOCATOR=1`). | Growth path | Capture | Tokens | Committed bytes (head = seq) | d2d head | d2d seq | |---|---|---|--:|--:|--:| | in-place VMM (default) | OFF | byte-identical | 100,663,296 | 688,576 | **0** | | in-place VMM (default) | ON (declined) | byte-identical | 100,663,296 | 688,576 | **0** | | legacy realloc | OFF | byte-identical | 786,432 | 688,576 | 688,576 | - **`FIXED-STRIDE RESULT: in-place VMM growth d2d KV bytes head-major=688,576 vs seq-major=0`** — the fixed-stride property #797 measured at the driver level, now at the engine level on a real model, on the default path. - **Committed physical bytes are equal head vs seq** in both mechanisms — the engine still commits flat bucket/granule ranges, so the `kv_heads×` residency win stays driver-level only (`vmm_kv_layout_residency_gpu`). Wiring `kv_commit.rs::live_prefix_ranges` into the live commit path is a scoped follow-on (shared with token-major and #777). - **Re-capture on growth not reproducible end to end**: this decoder's dynamic logical shape declines whole-step CUDA-graph capture (`captures=0`) — the same obstacle #794 hit. `invalidate_graph` still runs on every growth (retained because `present_capacity` is baked into any captured graph), so a capturing decoder would re-capture on each growth regardless of layout; the "0 re-captures" of #797 needs a fixed full-context stable-VA reservation the bucket-growth path does not inherit. - **Wall-clock: not reported.** The box was contended (identical runs ranged ~5 s to ~212 s across this session's re-runs), so no throughput number is trustworthy and none is extrapolated. ## Validation - 2 new engine unit tests (byte-layout mapping; fixed-stride-moves-no-bytes vs head-major re-strides). - New real-model e2e `mobius_seqmajor_growth_parity_native_cuda` (`#[ignore]`, model-gated) — the honest option-1 validation (patch metadata, run both ways, compare token IDs), extended to actually exercise **growth**, under both allocators. All device ops route through the engine's own stream ordering; run in isolation, single-threaded. - `gqa_seqmajor_parity_gpu` (independent CPU oracles), `group_query_attention_gpu` (25), `vmm_kv_layout_residency_gpu`, `vmm_kv_token_major_floor_gpu`, `vmm_stable_va_weight_slot_gpu` — all green single-threaded. - Full `cargo test -p onnx-runtime-session` — green post-rebase (153, 0 failures). - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` and `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` — green. `cargo fmt` (changed packages) — clean. Refs #797, #794, #782, #750, #777. **Does not flip any default; does not close any issue.** Out of scope: token-major across layers, #777 prefix sharing, converting remaining BNSH kernels, and wiring the dense-prefix residency commit. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

…804) (#807) ## Why This class of defect produced **two wrong conclusions this week**, both of which cost real work before being caught. Both are the same family: **state that is frozen or warmed once per process, making results depend on test order.** On a measurement-driven subsystem a wrong number does not just fail — it redirects the design. - **#797** — a GPU residency test failed ~50% of the time on a cold context and looked exactly like a correctness bug in growth-under-capture. Root cause: a readback on the **legacy default stream** raced a memset on a created **non-blocking stream** (the two are mutually exempt from implicit synchronization), so the readback returned partial fills **with no CUDA error**. `cuCtxSynchronize` did *not* fix it, which refuted the obvious race theory. It only "passed" because an alphabetically-earlier sibling subtest warmed the context. - **#801 / #804** — `captures=0` was reported as the decoder structurally declining CUDA graph capture, and that attribution reached a merged PR body. Root cause (#804): the capture-OFF phase set `ONNX_GENAI_CUDA_GRAPH=0` before the process-wide `RuntimeConfig` `OnceLock` was first read, and that config is frozen on first read — so every later phase in the same process inherited it. #794 and #801 failed to reproduce an end-to-end measurement and misattributed the reason. This is the same telemetry-that-misleads lineage as #719 / #715 (documented in `MEMORY_ARCHITECTURE.md`'s "third failure mode"). After the second instance in a week, the response is a **guard**, not more vigilance. This PR is **test infrastructure + documentation only**. No production behavior changes. ## What ### 1. A guard for process-frozen configuration (`onnx-genai-runtime-config`) `runtime_config()` records the environment it froze the `RuntimeConfig` snapshot from (captured through the same lookup `from_env` uses, so the set can never drift from the parsed fields). On every later call it re-reads those variables and **panics loudly** — with the frozen and current values, and a pointer to #804 — if any now differs. A test can no longer silently believe it set a knob that was already frozen. - **Gated on `debug_assertions`, not `cfg(test)`.** The #804 mutation lives in an *integration* test of a downstream crate, where `cfg(test)` for this crate is never set. `debug_assertions` is enabled for dependencies under the `dev`/`test` profile, so the guard is live for every `cargo test` in the workspace, and it is compiled out of `--release` — **the production path is byte-for-byte unchanged and pays nothing** (a single `OnceLock` read). - The comparison is a pure `find_changed()` helper with unit tests for the unchanged, changed, newly-set (the #804 shape), and cleared cases. ### 2. A helper for stream-correct device test code (`onnx-runtime-cuda-memory::test_support::TestStream`) A shared, test-only (`#[cfg(any(test, feature = "gpu-tests"))]`) RAII wrapper around **one** non-blocking stream. Every device op a test needs — `fill`, `write`, `read`, plus `raw()` for graph launches/memsets — flows through that stream, so a single `cuStreamSynchronize` is a total order. The module doc explains the legacy-default-vs-non-blocking exemption **at the call site** so the next author does not rediscover #797 the hard way. `vmm_kv_layout_residency_gpu.rs` (the exact #797 site) is converted to it and its four local stream helpers removed; **8/8 reliable in isolation** and green in the full single-threaded suite. ### 3. The pattern, documented where it will be read (`docs/MEMORY_ARCHITECTURE.md`) A tight "**a fourth failure mode: order-dependent test state**" subsection next to the existing third-failure-mode material: both case studies, the two mechanisms (frozen process config; context warmed by an earlier sibling), the two remedies (process isolation for anything that reads process-frozen config; single-stream discipline for device tests), and the one-line discipline that names the defect — *run any test you touch in isolation as well as in the full suite*. Cites #797/#801/#804/#794/#719/#715 and links rather than duplicating (#791 consolidated this doc to stop drift). ## Inventory of other order-dependent tests found (not fixed here) Per scope discipline, these are **listed, not converted**. The `TestStream` helper and the config guard make converting them mechanical follow-ups. ### Device tests that mix a created stream with default-stream copies (#797 hazard) Each creates a non-blocking stream **and** issues default-stream `cuMemcpyHtoD_v2`/`cuMemcpyDtoH_v2` copies. Where those touch the same buffers as stream-ordered work on a cold context, they are exposed to the exact #797 race. They should migrate to `test_support::TestStream`: - `crates/onnx-runtime-cuda-memory/tests/vmm_commit_shared_prefix_gpu.rs` - `crates/onnx-runtime-cuda-memory/tests/vmm_dummy_page_gpu.rs` - `crates/onnx-runtime-cuda-memory/tests/vmm_dummy_write_protect_gpu.rs` - `crates/onnx-runtime-cuda-memory/tests/vmm_granularity_gpu.rs` - `crates/onnx-runtime-cuda-memory/tests/vmm_graph_remap_gpu.rs` - `crates/onnx-runtime-cuda-memory/tests/vmm_kv_contiguous_tail_gpu.rs` - `crates/onnx-runtime-cuda-memory/tests/vmm_prefix_share_gpu.rs` - `crates/onnx-runtime-cuda-memory/tests/vmm_prefix_share_write_protect_gpu.rs` - `crates/onnx-runtime-cuda-memory/tests/vmm_stable_va_weight_slot_gpu.rs` (Non-vmm siblings `virtual_memory_gpu.rs` and `physical_pool_production_gpu.rs` also use default-stream copies; they pass but are the same latent shape.) ### Tests that mutate a `RuntimeConfig`-feeding env var within a process These change a variable that feeds the frozen snapshot (`ONNX_GENAI_CUDA_GRAPH`, `ONNX_GENAI_EP`) across phases. The correct pattern is **child-process-per-phase**; the new guard will now fire loudly on any that mutate in-process and then read `runtime_config()`: - `crates/onnx-genai-engine/tests/mobius_seqmajor_growth_parity_native_cuda.rs` — **already correct** (spawns a child per mode; its own doc comment states `ONNX_GENAI_CUDA_GRAPH` is parsed once). This is the reference pattern. - `crates/onnx-genai-engine/tests/stable_va_capture_offload_e2e.rs` — sets `ONNX_GENAI_CUDA_GRAPH` per capture phase. - `crates/onnx-genai-engine/tests/gemma4_e2b_native_cuda_pipeline_bench.rs` — sets `ONNX_GENAI_CUDA_GRAPH` to `1` then `0`. - `crates/onnx-genai-engine/tests/qwen2_5_1_5b_divergence.rs` — switches `ONNX_GENAI_EP` between `cpu` and `cuda`. - `crates/onnx-genai-engine/tests/qwen3_0_6b_capture_step_inputs_decline.rs` and `gemma3n_native_cuda_capture_realmodel.rs` — toggle the native-decoder capture env per phase. Session-crate tests that toggle env (`ONNX_GENAI_DECODE_MEMO`, `ONNX_GENAI_SCAN_INLINE_SINGLE_TRIP`, `ONNX_RT_SIBLING_MERGE`, the EP-CPU weight-offload knob) read those variables **directly at their call sites**, not through `RuntimeConfig`, so they are *not* affected by the freeze guard and the full `onnx-runtime-session` suite stays green. ## Validation (all local; not waiting for CI) - `cargo fmt -p onnx-genai-runtime-config -p onnx-runtime-cuda-memory -- --check` ✅ (`--all` hits the Windows `os error 206` long-command artifact, not a formatting issue) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✅ - `cargo clippy -p onnx-runtime-cuda-memory --features gpu-tests --all-targets -- -D warnings` ✅ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✅ - `cargo test -p onnx-runtime-session` ✅ (full suite) - `cargo test -p onnx-runtime-cuda-memory --features gpu-tests -- --test-threads=1` ✅ (all vmm_*/gpu suites, RTX 4060) - The converted `seq_major_fixed_stride_grows_under_captured_replay_without_recapture` run **8× in isolation**: 8/8 pass — the property this PR is about. ## Notes - Kept strictly to test infra + the docs subsection so the two other active workstreams (GQA pooled QKV staging in `group_query_attention.rs`; head-major vs seq-major re-capture measurement) merge cleanly — no overlap. - Does **not** merge and does **not** close any issue. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

…blished (#808) A durable record of the load-bearing results from this push, with every claim tied to a merged executable test — and, deliberately, the claims that were **refuted**. Three of the most useful outcomes were corrections to things I had stated as achieved: - **seq-major had not realised the 8x floor.** #794 measured head-major and seq-major committing *identical* physical bytes, because the bindings did not consume the layout descriptor. Fixed in #797; I corrected the floor table before merging. - **The decoder was not declining capture.** \captures=0\ came from a cached \ONNX_GENAI_CUDA_GRAPH=0\ in a long-lived test process (#804) — two PRs had misattributed a failed measurement to a model gate. - **A fixed KV stride does not remove growth-triggered re-capture.** True in mechanism (seq-major moves 0 bytes vs 688,576), irrelevant in practice, because the engine invalidates unconditionally on growth (#805). Also records the #736 audit's recurring finding — **four of five slices found over-reservation** rather than an ungoverned allocation — and the guidance that follows: start from *use*, not from allocation, because governing a bypass without sizing it to use converts invisible waste into charged waste, which tightens admission and costs concurrency. Docs only, no code paths touched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

…809) ## What this does (#777) PR #803 landed the prefix-sharing primitive (`create_shared_prefix` / `commit_shared_prefix` / `incremental_owned_bytes_for_shared_prefix`) on the production `CudaVmmAllocator`, but **nothing in production called it** — only its definition and GPU tests. An unconsumed contract reads as working infrastructure while proving nothing; that is exactly the shape that left #721 stage 3 stranded. This PR gives the primitive a **first real, deliberately narrow consumer**. ### What is now shared in production - **An allocator-agnostic seam on the `DeviceAllocator` trait** (`onnx-runtime-memory-governor`): `create_shared_prefix`, `incremental_owned_bytes_for_shared_prefix`, `commit_shared_prefix`, plus an opaque `dyn SharedDevicePrefix` handle and `SharedPrefixCommitInfo`. A caller holding only `dyn DeviceAllocator` can now pin a token prefix once and have subsequent sequences map it. Default impls **refuse** (`InvalidRequest`) rather than mis-map, so every existing non-VMM allocator keeps compiling and refuses safely (unsupported requests error rather than mismap, per #745/#750). - **`CudaVmmAllocator` implements the seam** by delegating to the merged inherent methods (#803) and downcasting the opaque handle via `as_any()`. Granules come from the existing #740 authority-scoped pool through `carve()`; no second allocator, no per-sequence physical reservation. - **First consumer:** the real **seq-major** fused fp16 GQA decode kernel (#782/#792/#794) reads shared-prefix VMM KV through the `dyn DeviceAllocator` seam. The prefix is physically contiguous as `layers × 2` ranges (seq-major, the gate #793 requires); token-major (one multi-map/sequence) is measured (#787) but not built here. The API is **explicit** (a caller declares the shared prefix), as #793 recommended for the highest-value case (a shared system prompt), and is **restricted to prefixes that are provably read-only for the sharers' lifetime** — copy-on-write at divergence is out of scope. ### Proof + measured numbers New GPU parity test `crates/onnx-runtime-ep-cuda/tests/gqa_shared_prefix_parity_gpu.rs` drives the real seq-major GQA decode kernel over shared-prefix VMM KV and proves two sequences sharing one pinned prefix produce **byte-identical** output to two independent sequences. Measured (KV_HEADS=8, HEAD_DIM=128, f16, 1024-token prefix + 1024-token private tail per sequence; 2 MiB granule → 1024 tokens/granule): | quantity | value | | --- | --- | | independent (2 sequences) committed **physical** bytes | **16,777,216 B** (8 granules) | | shared (2 sequences) committed **physical** bytes | **12,582,912 B** (6 granules) | | removed by sharing | **4,194,304 B** (2 granules = `(C−1)×(K_prefix+V_prefix)`) | | second-sharer admission cost | **4,194,304 B** — private tails only | | prefix `incremental_owned_bytes_for_shared_prefix` | **0** (charged once) | | output | **byte-identical** for both sequences | Deterministic counters lead; wall-clock is omitted (this box's identical runs range 3.9–28 tok/s and other agents build concurrently — no number is extrapolated). ### Validation (all local; not waiting for CI, fix-forward) - `cargo fmt -p onnx-runtime-memory-governor -p onnx-runtime-cuda-memory -p onnx-runtime-ep-cuda -- --check` — clean (`--all` fails on this box with os error 206, a Windows command-line-length artifact, not formatting). - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` — clean. - `cargo clippy -p onnx-runtime-cuda-memory --features gpu-tests --all-targets -- -D warnings` — clean. - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` — clean. - Full `cargo test -p onnx-runtime-session` — pass. - Single-threaded GPU suites — pass: `vmm_commit_shared_prefix_gpu` (4), `vmm_prefix_share_gpu` (2), `vmm_prefix_share_ledger_gpu` (1), `vmm_prefix_share_write_protect_gpu` (1), `vmm_prefix_share_compose_gpu` (1), `gqa_seqmajor_parity_gpu` (2), and the new `gqa_shared_prefix_parity_gpu` (1). Harness respects the known traps: VMM allocator built on the EP's own CUDA context; all device ops route through the single EP stream (#797); pool env set before any allocator is constructed and the process is isolated (#804); no assertions inside any `Drop`; live reservations explicitly deallocated before allocators drop. ### What remains (structural / out of scope) - **Auto engine use is blocked structurally.** The engine generation loop cannot call this seam automatically until `native_decode/cuda.rs` can build a seq-major fixed-stride physical KV shape — `persistent_state_shapes` is hard-coded BNSH and no model declares a seq-major end-to-end physical shape (#794 showed seq-major changed only kernel indexing, not commit geometry). A BNSH/seq-major fixed-stride physical-shape build is the prerequisite. This PR delivers the seam + the seq-major GQA-kernel consumer + proof; the automatic decode-path wiring is the next structural increment. - Hash-based automatic prefix detection (later; explicit path proven first). - Token-major (one multi-map per sequence; measured #787, not built). - Copy-on-write at divergence (#793 measured ~5.5 ms pooled / ~9.8 ms cold, once). Docs updated: `docs/MEMORY_ARCHITECTURE.md` (concurrency-scaled prize section) and `docs/PREFIX_SHARE_INVESTIGATION.md` (smallest next increment) now record the first production consumer, the measured numbers, and the remaining structural gap. Refs #777 #803 #793 #782 #745 #750 > Please do **not** merge and do **not** close #777 from this PR. Edits are > confined to the KV/prefix integration path to merge cleanly with concurrent work. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

…ajor-only KV consumers (#777 #801 #797 #794 #809) (#812) # Remove the last blocker to seq-major KV being usable in production: gate the fourth place the layout lives **Outcome reached: #3 — "it works only under a condition."** Seq-major decode works end-to-end and is byte-identical to head-major, but only on the **pure decode path where our own GQA kernel is the sole consumer of the device KV buffer**. This PR makes the runtime **refuse rather than mis-map** when that condition does not hold, and pins down — with citations — exactly what remains between "growth byte geometry is seq-major" (#801) and "the `layers × 2` floor realised + prefix sharing available." Refs #777 #797 #794 #801 #809 (and #787 for token-major). Does **not** merge, does **not** close any issue. ## The question this PR answers > What remains between "growth byte geometry is seq-major" and "a real model runs seq-major end to end with the floor realised and prefix sharing available"? A fourth place the layout lives, or can the declared BNSH shape and a seq-major physical layout coexist provided only our own kernels read the buffer? **Both.** The declared BNSH shape and a seq-major physical layout *do* coexist on the native decode path (KV never leaves the device; only our GQA kernel indexes it). But there **is a fourth place the layout lives** — two engine-side *non-kernel* consumers of the device KV buffer that still stride it with hard-coded head-major (BNSH) arithmetic: 1. **`DecodeCudaState::read_present_kv`** (`native_decode/cuda.rs`) — the device present-KV **host mirror** used by prefix/paged read-out and diagnostics. It compacts per head with `capacity_head_stride = physical_shape[2] * head_dim * elem` and a per-head fragment read. 2. **`DecodeCudaState::seed_prefix`** (`native_decode/cuda.rs`) — the **paged prefix-reuse device seed** (#777/#809's device counterpart). It writes each head at `head * capacity_head_stride * elem_size`, the same head-major capacity offset. Under a seq-major (BSNH) physical buffer, heads are **interleaved per token** (`slot * kv_heads * head_dim + h * head_dim + d`), not laid out as `max_len × head_dim` capacity stripes. So both offsets index the wrong bytes: the host mirror would **silently return mis-indexed KV**, and the prefix seed would **corrupt the cache**. Neither had a seq-major guard — a real latent-corruption path. This is the fourth place the layout lives, beyond the three that are already converted: 1. kernel indexing — done (#792/#794); 2. commit geometry — done (#797); 3. growth byte geometry — done (#801); 4. **engine-side non-kernel KV consumers — gated here.** ## What this PR changes `crates/onnx-genai-engine/src/native_decode/cuda.rs`: - `read_present_kv` and `seed_prefix` now **error under `kv_layout.is_seq_major()`** with a precise message stating the condition and why (head-major byte arithmetic vs interleaved seq-major bytes). This matches the kernel-level "unsupported readers/writers fail rather than silently mis-index" gate (`kv_stride.rs::require_converted_path_support`) at the engine level. - **Head-major is untouched and byte-identical** — the gate is a pure `if is_seq_major() { bail! }` before the existing code. Absent metadata ⇒ head-major ⇒ no gate. `docs/MEMORY_ARCHITECTURE.md` (§ "KV layout and residency"): records the fourth layout location and the refuse-not-mismap gate as a bullet in the existing residency-caveat block. ## The structural constraint (#801), re-verified The BNSH binding **metadata** shape must stay for both layouts. The CUDA GQA node validates `past_key`/`past_value` as `[batch, kv_heads, seq, head_dim]` and reads `present_capacity` from **axis 2 regardless of `kv_layout`** (`group_query_attention.rs`), and `DeviceIoBinding` requires matching axis order. `persistent_state_shapes` is therefore *correctly* BNSH for seq-major too; seq-major lives in the byte geometry, not a permuted shape. Confirmed still true after merging the concurrent GQA-workspace change. ## Why the `layers × 2` floor is still not realised through the engine (and what it needs) Measured directly, **before** the merge broke the harness, with the validated #801/#805 setup — stock `qwen2.5-0.5b-q4_0-mobius` (BNSH) vs a copy whose 24 GQA nodes carry `kv_layout=1`, forcing growth with `ONNX_GENAI_KV_MIN_BUCKET=8`, 48 greedy tokens, 3 growth events (`mobius_seqmajor_growth_parity_native_cuda`, this branch @ pre-merge): | Growth path | Capture | Output | Physical committed bytes (head = seq) | d2d KV copy bytes head | d2d KV copy bytes seq | |---|---|---|---:|---:|---:| | in-place VMM (default) | OFF | **byte-identical** | 100,663,296 | 688,576 | **0** | | in-place VMM (default) | ON | **byte-identical** | 100,663,296 | 688,576 | **0** | | legacy realloc | OFF | **byte-identical** | 786,432 | 688,576 | 688,576 | Capture-ON, per-process: head-major and seq-major both `(growth 3, captures 4, replays 39, invalidations 4)`. So on this branch the seq-major decode path is **byte-identical head vs seq, capture ON and OFF, and moves 0 KV bytes on in-place growth** — but **`physical_committed_bytes` is identical between layouts**, i.e. the floor is *not* realised through the engine. Two cited reasons: 1. **`vmm_growth_requests` commits the full bucket** (`(index, 0, full_bucket_bytes)`) for every KV binding, for both layouts, instead of the dense live-prefix ranges `native_decode/kv_commit.rs::live_prefix_ranges` computes (those helpers are unit-tested and mirror the driver GPU test but are `allow(dead_code)` — not wired into the engine). 2. **The `kv_heads×` difference only becomes *measurable* under a large fixed stride.** With bucket growth the bucket ≈ 2× valid, so on 0.5b both layouts land in **1 granule/binding** (48 granules = 96 MiB). The `kv_heads=2` gap appears only when a head stripe crosses a granule boundary, i.e. `capacity × head_dim × elem ≥ granule` (`ONNX_GENAI_CUDA_KV_MAX_LEN=32768` in the driver test `vmm_kv_layout_residency_gpu`): head-major = `layers × 2 × kv_heads` granules, seq-major = `layers × 2` (the measured 192 MiB → 96 MiB, 2× = `kv_heads`; 1536 MiB → 192 MiB, 8× on qwen14b). Realising the measurable floor therefore requires **a fixed full-context stride + dense-prefix commit + per-step commit-ahead**, which eliminates growth-driven bucket recapture. That change lives in the **growth/capture-invalidation decision** — the explicit lane of the concurrent `conditional-invalidation` / `bsnh-fixed-stride` work — and outside this PR's shape/binding-construction lane. It is left as the documented next step (shared with token-major #787 and prefix sharing #777), consistent with the existing doc note: *"wiring the dense-prefix commit into the live path is the documented next step."* ## Prefix sharing (#777/#809) under seq-major It does **not** fall out naturally. #809's device consumer is `seed_prefix`, a head-major byte writer; a shared prefix is only physically contiguous under seq-major, but the seed that would install it cannot yet address the seq-major buffer. This PR makes that explicit (the seed errors under seq-major) rather than letting prefix reuse silently corrupt seq-major KV. Realising it means teaching `seed_prefix` (and `read_present_kv`) the BSNH byte geometry — same dense-prefix work as the floor. ## Validation - `cargo fmt -p onnx-genai-engine -- --check` ✓ (`--all` fails with os error 206 on this box — a command-line-length artifact, not formatting) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✓ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✓ (post-merge) - full `cargo test -p onnx-runtime-session` ✓ (post-merge) - `mobius_seqmajor_growth_parity_native_cuda` (byte-identical head vs seq, capture ON/OFF, 0 bytes moved) ✓ **both pre-merge and post-merge** (post-merge re-run includes the #814 governor fix, see below); the gate does not regress it. All four configs identical to the pre-merge numbers above (`physical_committed_bytes` 100,663,296 / 786,432; seq `kv_bytes_moved=0`). ### Cross-agent regression surfaced by `merge origin/main` — now fixed by #814 Merging `origin/main` (which brought the `gqa-bnsh-transpose` GQA transpose-scratch change #810) initially broke the mobius end-to-end test **during prefill workspace prep**, before any decode: ``` Error: prepare native CUDA prefill workspace Caused by: cannot reserve 2097152 bytes of device memory for Workspace { step_scoped: false }: 0 of 2097152 bytes are already leased, leaving 0 ``` #810 reserves a new **session-persistent** BNSH transpose-scratch `Workspace { step_scoped: false }` (`WorkspaceLifetime::SessionPersistent`) during prefill; the `vram-limit dynamic KV/weight lending` strategy did not provision headroom for it (GPU otherwise 100% free — 0 MiB used). This was a governor role-budgeting interaction in the **GQA/memory lane**, independent of this PR's gate (which only adds error paths in `read_present_kv`/`seed_prefix`, never on the prefill workspace path). I attributed it precisely and reported it to the `gqa-bnsh-transpose` sibling rather than fixing out of lane. **It has since been fixed forward in #814** (commit `54f89a03`: "mapped growth now claims unassigned authority capacity before holder reclaim, with rollback/overestimate accounting"). Merging #814 locally on top of this branch, the mobius test **passes all four VMM/realloc × capture configurations, byte-identical head vs seq**, confirming this PR's gate and #814's governor fix coexist cleanly. #814 is not yet on `main`, so this branch is left at its clean two-file state (the fix will arrive via `main`); the post-merge validation above was run against `this branch + #814` locally. ## Files - `crates/onnx-genai-engine/src/native_decode/cuda.rs` — seq-major refuse-not-mismap gate on the two head-major-only KV consumers. - `docs/MEMORY_ARCHITECTURE.md` — fourth-layout-location note. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

## What Documentation-only sync of five memory/KV design docs to the merged, executable-test-backed facts about VMM commit geometry, KV layout, weight offload, CUDA-graph capture, and plugin export. No code or tests changed. `MEMORY_ARCHITECTURE.md` is treated as the single source for KV geometry and the audit table; the other four link into it rather than restating (per #791). ## Most misleading claim found `MEMORY_ARCHITECTURE.md` — the document explicitly designated the single source of truth — still asserted the engine *"explicitly invalidates the graph on every KV bucket growth for both layouts"* and that *"the engine does not yet use that fact to retain the captured graph,"* with a table showing seq-major at `captures=4, invalidations=4`. That was falsified by **#811**: conditional invalidation has landed, seq-major fixed-stride growth now **keeps** the captured graph (growth-attributable invalidations **3 → 0**, `captures=1/invalidations=1`, matching the no-growth control), verified by `mobius_seqmajor_growth_parity_native_cuda.rs` including a negative oracle. Because this doc is the single source others link to, the stale claim had the widest blast radius. ## Changes - **MEMORY_ARCHITECTURE.md** — correct the #811 conditional-invalidation prose + table (verified against `cuda.rs` `binding_growth_signature`/`record_growth_decision` and the parity test); add the ORT-backend KV-routing + #762 plugin-export note (verified `allocator.rs:232` `for_session_device` selects by device, not implementation); thread #822 (granule queried), #812 (engine-side consumer gate), and a #800/#802 record correcting this doc's earlier fp16 sizing of Phase-2a scores (verified `attention.rs:368` uses `dtype.element_size()`). - **CUDA_GRAPH_CAPTURE.md** — replace the mutual-exclusion result with #796 (stable VA removes it), #798 (managed no-spill default), #811 (conditional invalidation), and the #804/#807 `captures=0` process-artifact correction. - **WEIGHT_OFFLOAD.md** — correct the "mutually exclusive today" claim (#796) and the north-star status for the managed no-spill default (#798). - **KV_INSERTION_DESIGN.md** (most stale, 2026-07-15) — add a synchronization banner linking to `MEMORY_ARCHITECTURE.md`; note the layout line post-dates the memo, prefix sharing is a landed primitive (#777), and block-table paged attention remains a proposal (verified: no `block_table` in any CUDA attention kernel). - **DESIGN.md** — forward-pointer in the already-superseded §39.5 noting layout (seq-major/token-major, #787/#792) is now the primary residency lever. ## Method Every retained performance claim cites the PR that measured it; measured fact / design intent / not-yet-built are distinguished; superseded designs are labelled, not deleted. The "still open" item (engine commits flat buckets, not the dense live-prefix ranges `kv_commit.rs` computes, so the `layers×2` floor is not yet realised through the engine) is described as in-flight, not delivered. ## Note on #762 The task framing had #762 (EP plugin export) as open; `gh pr view 762` shows it **merged** today. It is described accurately per that ground truth: the plugin-EP mechanism landed for the CPU EP (validated end-to-end), while the CUDA plugin shim compiles but is unvalidated on hardware — so plugin-EP-hosted VMM KV remains in-progress design intent, not a shipped capability. ## Validation Documentation only — `git diff --stat` touches exactly the five `.md` files; working tree otherwise clean. **Not merged; no issue closed.** Refs #736, #762, #776, #777, #783, #787, #791, #792, #794, #796, #797, #798, #801, #802, #804, #807, #811, #812, #822 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…r); floor is granule-bound on qwen0.5b (#797 #801 #812 #811 #787 #777) (#827) ## Summary Realises the `layers × 2` KV floor **through the engine** by wiring the seq‑major fixed‑stride KV commit path through the shared `kv_commit::live_prefix_ranges` geometry, the last increment in the chain named by #797, #801 and #812. `DecodeCudaState::seq_major_kv_commit_requests` now *consumes* the same `kv_commit.rs` geometry the driver‑level residency test (`vmm_kv_layout_residency_gpu`) and the `kv_commit.rs` unit tests exercise, instead of duplicating the byte arithmetic inline — so the live commit geometry and the measured floor are single‑sourced and cannot drift (previously `kv_commit.rs` carried a `#![cfg_attr(not(test), allow(dead_code))]` because nothing in a non‑test build called it). Head‑major stays **byte‑identical**: on a growing packed bucket its per‑head stride *is* the bucket, so the `kv_heads` live‑prefix fragments tile the same contiguous `[0, bucket_bytes)` run — head‑major's dense ranges equal its bucket ranges. `vmm_growth_requests` is left as the flat bucket commit, now with a doc comment saying exactly that. Refs #797 #801 #812 #811 #787 #777. ## Headline: committed‑bytes comparison `physical_committed_bytes`, process‑isolated, `mobius_seqmajor_growth_parity_native_cuda`, qwen2.5‑0.5b: | Config (vmm‑inplace) | head‑major | seq‑major | |---|---:|---:| | capture ON, forced growth | 100,663,296 | 100,663,296 | | capture OFF, forced growth | 100,663,296 | 100,663,296 | | no‑growth control (`min_bucket=64`) | 100,663,296 | 100,663,296 | | realloc, forced growth | 786,432 | 786,432 | **The floor does not move on this model, and that is the correct result, not an un‑wired commit.** At `head_dim=64`/fp16 with a 512‑token hard max, a single binding's *entire reservation* is `512 × 2 × 64 × 2 = 131,072 B = 128 KiB`, far below the 2 MiB CUDA granule. So **both** layouts pin at exactly one granule per binding: `48 bindings × 2 MiB = 100,663,296 B`, which *is* the `layers × 2 = 48`‑granule floor. Seq‑major already sits on that floor; head‑major coincides because its 64‑token bucket is also sub‑granule. The `kv_heads×` separation (`vmm_kv_layout_residency_gpu`, qwen14b, 32,768‑token stride) only materialises once a **single head's live stripe crosses a granule** — `capacity × head_dim × elem ≥ 2 MiB`, i.e. ≈8,192 tokens at head_dim 128/fp16 (the measured head‑major crossover, #776). A 512‑max, 48‑token qwen0.5b run never approaches it, so #797's driver‑level `kv_heads×` win is **not reproducible end‑to‑end on this harness** — its geometry never leaves a single granule. This is the same class of precise negative result as #794 (identical bytes) and #812, stated with the same precision. ## Full per‑config counters (measured) | Config | growth_events | kv_bytes_moved | captures | replays | invalidations | growth_keeps | |---|---:|---:|---:|---:|---:|---:| | head vmm ON, growth | 3 | 688,576 | 4 | 39 | 4 | 0 | | seq vmm ON, growth | 3 | **0** | 1 | 45 | 1 | **3** | | head vmm ON, no‑growth control | 0 | 0 | 1 | 45 | 1 | 0 | | seq vmm ON, no‑growth control | 0 | 0 | 1 | 45 | 1 | 0 | | head vmm OFF, growth | 3 | 688,576 | 0 | 0 | 4 | 0 | | seq vmm OFF, growth | 3 | 0 | 0 | 0 | 1 | 3 | | head realloc OFF, growth | 3 | 688,576 | 0 | 0 | 4 | 0 | | seq realloc OFF, growth | 3 | 688,576 | 0 | 0 | 4 | 0 | - **Token IDs byte‑identical** across every configuration (`" Paris. It is the largest city…"`, 48 tokens). - Seq‑major keeps its captured graph across all 3 growths (`growth_keeps=3`, forced‑growth graph accounting `1/45/1` collapses onto the no‑growth control) — **the #811 keep decision is preserved**; the dense‑range commit changes neither device pointers nor physical shapes, so the four keep dependencies still hold. - Head‑major still moves 688,576 B and re‑captures per growth (`4/39/4`, `growth_keeps=0`) — unchanged. Wall‑clock intentionally omitted (deterministic counters answer the question; this box's identical runs vary 3.9–28 tok/s). ## Invariants held - Idempotent re‑commit; governor bills only newly mapped granules (via `seq_major_kv_commit_requests` → `commit_binding_ranges_with_mapped_growth`). - #748 `MappedGrowthGrant` growth path, #745 committed‑granule admission, #814 governor claim ordering, pre‑header 429 refusal (#743) — all unchanged. - No map/grow inside a captured region; no unmap under in‑flight replay; **no assert inside any `Drop`**. ## What this unblocks / does not - The dense‑prefix commit is now floor‑faithful and single‑sourced, so the `layers × 2` geometry is realised end‑to‑end **wherever a per‑head stripe exceeds a granule** (large `head_dim`/context). Demonstrating the `kv_heads×` win end‑to‑end needs such a model, **not** a change to this commit path. - It does **not** address the seq‑major seed/host‑mirror stride arithmetic (`read_present_kv` / `seed_prefix`), which still refuse under seq‑major (#812). Prefix sharing under seq‑major therefore still needs the BSNH seed as a separate follow‑on; the doc bullet is corrected to say so. ## Validation (all green, on an RTX 4060 Laptop, ORT 1.27, CUDA 13 wheels) - `cargo fmt -p onnx-genai-engine -- --check` ✔ (`--all-targets` full‑workspace fmt hits Windows os error 206, a command‑length artifact, not a formatting problem) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` ✔ - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` ✔ - `cargo test -p onnx-runtime-session` ✔ (277 passed) - `cargo test -p onnx-genai-engine --features native-backend --lib` ✔ (454 passed; `kv_commit` 9/9) - **Mandatory** `mobius_seqmajor_growth_parity_native_cuda --ignored --test-threads=1` ✔ (2 passed, incl. the head‑major negative oracle) - `gqa_seqmajor_parity_gpu` ✔ (2), `gqa_shared_prefix_parity_gpu` ✔ (1), `group_query_attention_gpu` ✔ (26), `vmm_kv_contiguous_tail_gpu` ✔ (3), `vmm_kv_layout_residency_gpu` ✔ (2), `vmm_kv_token_major_floor_gpu` ✔ (2) — all `--ignored --test-threads=1`. Do not merge / do not close any issue. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

…ative) + #384 metadata generator (#834) ## Summary Demonstrates the KV-cache committed-**physical**-bytes floor end-to-end on **qwen14b** via the native CUDA decode path, comparing **head-major (BNSH)** vs **seq-major (BSNH)** layouts in the same session, swept across context capacities that bracket the 2 MiB-per-stripe granule threshold (#776). Token IDs are **byte-identical** between layouts at every point. This is the first time this line (chased since #797) has been driven through the *engine* on a model whose head stripe actually reaches a full granule (qwen14b @ 8192 ctx: `8192 x 128 x 2 = 2,097,152 B = exactly 2 MiB`), rather than the sub-granule qwen0.5b of #827. ## Headline result — honest negative (per #794 / #812 / #827 precedent) Head-major and seq-major commit **byte-identical** physical KV bytes at every swept capacity: | KV capacity (tokens) | committed bytes (both layouts) | granules | ratio seq/head | |---:|---:|---:|---:| | 1024 | 402,653,184 | 192 | 1.00x | | 2048 | 603,979,776 | 288 | 1.00x | | 4096 | 1,006,632,960 | 480 | 1.00x | | 8192 | 1,811,939,328 | 864 | 1.00x | Expected-if-it-separated: head-major `layers x 2 x kv_heads` = **768 granules (~1.5 GiB)** vs seq-major `layers x 2` = **96 granules (~192 MiB)** — the 8x that never appeared. ## Why it does not separate `growth_events == 0` and `committed_len == capacity` for **both** layouts: the engine commits the full KV bucket **eagerly** on reservation — reservation *is* the commit, so layout cannot change the total. Seq-major's fixed full-context stride *is* active and correct (`max_len == 8192` at every capacity vs head-major `max_len == capacity`, tokens match), which proves the seq-major kernel/layout resolution works — but a fixed stride only makes **growth** free (#797's 0 bytes moved) and preserves the captured graph (#811/#812); it does not shrink an eagerly-committed full bucket. The 768-vs-96 (8x) floor stays a **driver-level** property (`vmm_kv_layout_residency_gpu`) realized only by a **live-prefix commit-on-demand** path, which the engine does not do on initial reservation. The new guardrail assertions (`growth_events==0`, `committed_len==capacity`, `head==seq committed`) will **turn red** the moment a future change makes seq-major commit less — i.e. they are a tripwire for the desired win. Measurement used the in-place **VMM** allocator (`ONNX_GENAI_CUDA_VMM=1`), which reports granule-rounded **physical** bytes; the legacy/governor allocator reports nominal bytes and was deliberately avoided. Capture was declined throughout (`ONNX_GENAI_CUDA_GRAPH=0`); offload/weight-streaming (16.6 GB model vs 7.7 GB budget, #796) was active and did not perturb the KV commit counters. ## Deliverables 1. **`crates/onnx-genai-engine/tests/qwen14b_kv_floor_sweep_native_cuda.rs`** — child-process-isolated sweep harness (mirrors `mobius_seqmajor_growth_parity_native_cuda`), honoring process-frozen config (#804/#807). Self-execs per `(layout, capacity)`; asserts token parity + layout resolution + the deterministic guardrails above. `#[ignore]`, gated on `cuda,native-backend`. 2. **`scripts/gen_inference_metadata.py`** (#384) — emits `inference_metadata.yaml` from `genai_config.json` for any such export. Reproduces the hand-written qwen14b metadata **byte-for-byte**. Turns the one-off local `qwen14b-zp` fix (which unblocked native loading for this whole measurement) into a reproducible artifact; the model file itself stays local (models live outside the repo). 3. **Docs** — `MEMORY_ARCHITECTURE.md` floor-progression table + eager-full-bucket explanation; `MODEL_METADATA.md` #384 generator section. ## Method notes - Seq-major was obtained by patching a **copy** of the model (`kv_layout=1` on all 48 GQA nodes), as #801/#805 did for mobius — the original model directory was never modified. - Reported committed **physical** bytes throughout, never nominal content bytes. - Deterministic counters lead; wall-clock is omitted (this box's identical runs ranged 3.9-28 tok/s) and no number was extrapolated that was not measured. ## Unrelated pre-existing fix (disclosed) `crates/onnx-runtime-ep-cpu/src/kernels/matmul_nbits.rs`: added a `#[cfg(not(target_arch = "aarch64"))] let _ = dot_kernel;` guard in `borrowed_affine_int4_matmul` to silence a pre-existing x86 unused-variable lint that blocked `-D warnings` (matches sibling functions at 3779/4272/4448). Behavior-neutral; the mobius ignored test still passes (2 passed). ## Validation - `cargo fmt -p onnx-genai-engine -p onnx-runtime-ep-cpu -- --check` (per-package; `--all` hits Windows `os error 206`) - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` — pass - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` — pass - `cargo test -p onnx-runtime-session` (full) — pass - `cargo test -p onnx-genai-engine --features cuda,native-backend --test mobius_seqmajor_growth_parity_native_cuda -- --ignored --test-threads=1` (#814 policy) — 2 passed - New sweep harness run twice; all guardrails passed. Refs #797 #787 #827 #384 #777. Do **not** merge; does **not** close any issue. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

… baseline (#841) (#849) Follow-up to #841, correcting the single-source doc. #841 measured, end to end on qwen14b with on-demand commit **active**, that head-major and seq-major commit **byte-identical** physical KV. The mechanism is in `native_decode/cuda.rs`: head-major growth sets `new_shape[2] = new_capacity`, so its per-head stride is *the current bucket*, not a fixed full context. It re-packs dense and already sits on the `layers × 2` floor (96 granules) — **0.25× of the 768-granule figure this line of work has quoted since #797**. So the `kv_heads ×` ("8×") saving seq-major was pursued for does not exist on the path the engine actually runs, and no further layout work would have produced it. What this changes in the doc: - The floor table now distinguishes **head-major with a fixed full-context stride** (768 granules — a geometry the engine never instantiates) from **head-major as the engine actually runs it** (96 granules, measured). - The driver-level `vmm_kv_layout_residency_gpu` 8× ratio is still correct *as measured*, but is now explicitly marked as **not transferring to the engine**, since it is measured on that fixed stride. - The `layers × 2 × kv_heads` prose earlier in the section is now qualified with the stride condition it silently assumed. - Seq-major's real, measured advantage is recorded plainly: **growth cost, not residency** — 0 bytes moved and capture retained on growth, against head-major's 100,667,392 bytes moved plus invalidation. That is what the layout chain (#782/#792/#794/#797/#801/#812/#827) bought, and it is worth having; it is simply not a residency saving. Lesson recorded in the doc: **before quoting a floor, confirm the code instantiates the geometry that floor assumes.** The arithmetic was right throughout; the baseline was not. Notably `cuda.rs` has said head-major grows a *packed* bucket since #827 — the fact was in the tree before the measurement caught up to it. Docs only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

Convert flash prefill KV reads to stride descriptors

762c13c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby merged commit ef7f6fc into main Aug 11, 2026
1 check passed

justinchuby deleted the squad/flash-prefill-layout branch August 11, 2026 23:07

This was referenced Aug 11, 2026

Page offloaded weights under a stable VA so offload keeps CUDA graph capture (#716) #796

Merged

Make native CUDA KV binding commit geometry layout-aware #797

Merged

justinchuby mentioned this pull request Aug 12, 2026

Build KV growth byte geometry under a BSNH fixed stride (seq-major) #801

Merged

justinchuby mentioned this pull request Aug 12, 2026

docs(squad): record what the VMM/offload/streaming/batching push established #808

Merged

justinchuby mentioned this pull request Aug 12, 2026

Give the prefix-sharing primitive a first production consumer (#777) #809

Merged

justinchuby mentioned this pull request Aug 12, 2026

Gate the fourth place the KV layout lives: refuse seq-major on head-major-only KV consumers (#777 #801 #797 #794 #809) #812

Merged

This was referenced Aug 12, 2026

Make CUDA-graph invalidation on KV growth conditional (seq-major keep: 4 -> 0 growth invalidations) #811

Merged

docs: sync memory design docs to VMM/KV-layout/offload facts #826

Merged

justinchuby mentioned this pull request Aug 12, 2026

Wire dense live-prefix KV commit through kv_commit geometry (seq-major); floor is granule-bound on qwen0.5b (#797 #801 #812 #811 #787 #777) #827

Merged

justinchuby mentioned this pull request Aug 12, 2026

qwen14b KV committed-bytes floor: head-major vs seq-major (honest negative) + #384 metadata generator #834

Merged

This was referenced Aug 12, 2026

test(engine): decisive qwen14b KV committed-bytes floor experiment — 8x separation does NOT appear (mechanism identified) #841

Merged

docs(memory): the 768-granule head-major floor was never the engine's baseline (#841) #849

Merged

Navigation Menu

Repository navigation

Enable stride-described seq-major flash prefill - #794

Enable stride-described seq-major flash prefill#794
justinchuby merged 2 commits into
mainfrom
squad/flash-prefill-layout

justinchuby commented Aug 11, 2026

Uh oh!

codecov Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

justinchuby commented Aug 11, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

justinchuby commented Aug 11, 2026

Summary

Correctness

Measurements

Remaining work

Validation

Uh oh!

codecov Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

justinchuby commented Aug 11, 2026

The most valuable thing in this PR is the negative result

What did land

Next

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026

🔴 Benchmark Regression Detected

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Aug 11, 2026 •

edited

Loading