[justinchuby]

Summary

Realises the layers × 2 KV floor through the engine by wiring the seq‑major fixed‑stride KV commit path through the shared kv_commit::live_prefix_ranges geometry, the last increment in the chain named by #797, #801 and #812. DecodeCudaState::seq_major_kv_commit_requests now consumes the same kv_commit.rs geometry the driver‑level residency test (vmm_kv_layout_residency_gpu) and the kv_commit.rs unit tests exercise, instead of duplicating the byte arithmetic inline — so the live commit geometry and the measured floor are single‑sourced and cannot drift (previously kv_commit.rs carried a #![cfg_attr(not(test), allow(dead_code))] because nothing in a non‑test build called it).

Head‑major stays byte‑identical: on a growing packed bucket its per‑head stride is the bucket, so the kv_heads live‑prefix fragments tile the same contiguous [0, bucket_bytes) run — head‑major's dense ranges equal its bucket ranges. vmm_growth_requests is left as the flat bucket commit, now with a doc comment saying exactly that.

Refs #797 #801 #812 #811 #787 #777.

Headline: committed‑bytes comparison

physical_committed_bytes, process‑isolated, mobius_seqmajor_growth_parity_native_cuda, qwen2.5‑0.5b:

Config (vmm‑inplace)	head‑major	seq‑major
capture ON, forced growth	100,663,296	100,663,296
capture OFF, forced growth	100,663,296	100,663,296
no‑growth control (min_bucket=64)	100,663,296	100,663,296
realloc, forced growth	786,432	786,432

The floor does not move on this model, and that is the correct result, not an un‑wired commit. At head_dim=64/fp16 with a 512‑token hard max, a single binding's entire reservation is 512 × 2 × 64 × 2 = 131,072 B = 128 KiB, far below the 2 MiB CUDA granule. So both layouts pin at exactly one granule per binding: 48 bindings × 2 MiB = 100,663,296 B, which is the layers × 2 = 48‑granule floor. Seq‑major already sits on that floor; head‑major coincides because its 64‑token bucket is also sub‑granule.

The kv_heads× separation (vmm_kv_layout_residency_gpu, qwen14b, 32,768‑token stride) only materialises once a single head's live stripe crosses a granule — capacity × head_dim × elem ≥ 2 MiB, i.e. ≈8,192 tokens at head_dim 128/fp16 (the measured head‑major crossover, #776). A 512‑max, 48‑token qwen0.5b run never approaches it, so #797's driver‑level kv_heads× win is not reproducible end‑to‑end on this harness — its geometry never leaves a single granule. This is the same class of precise negative result as #794 (identical bytes) and #812, stated with the same precision.

Full per‑config counters (measured)

Config	growth_events	kv_bytes_moved	captures	replays	invalidations	growth_keeps
head vmm ON, growth	3	688,576	4	39	4	0
seq vmm ON, growth	3	0	1	45	1	3
head vmm ON, no‑growth control	0	0	1	45	1	0
seq vmm ON, no‑growth control	0	0	1	45	1	0
head vmm OFF, growth	3	688,576	0	0	4	0
seq vmm OFF, growth	3	0	0	0	1	3
head realloc OFF, growth	3	688,576	0	0	4	0
seq realloc OFF, growth	3	688,576	0	0	4	0

• Token IDs byte‑identical across every configuration (" Paris. It is the largest city…", 48 tokens).
• Seq‑major keeps its captured graph across all 3 growths (growth_keeps=3, forced‑growth graph accounting 1/45/1 collapses onto the no‑growth control) — the Make CUDA-graph invalidation on KV growth conditional (seq-major keep: 4 -> 0 growth invalidations) #811 keep decision is preserved; the dense‑range commit changes neither device pointers nor physical shapes, so the four keep dependencies still hold.
• Head‑major still moves 688,576 B and re‑captures per growth (4/39/4, growth_keeps=0) — unchanged.

Wall‑clock intentionally omitted (deterministic counters answer the question; this box's identical runs vary 3.9–28 tok/s).

Invariants held

• Idempotent re‑commit; governor bills only newly mapped granules (via seq_major_kv_commit_requests → commit_binding_ranges_with_mapped_growth).
• Add transactional mapped growth grants #748 MappedGrowthGrant growth path, Use committed granules for VMM weight admission #745 committed‑granule admission, Fix GQA workspace growth from unassigned device capacity #814 governor claim ordering, pre‑header 429 refusal (Return overload response for KV admission refusal #743) — all unchanged.
• No map/grow inside a captured region; no unmap under in‑flight replay; no assert inside any Drop.

What this unblocks / does not

• The dense‑prefix commit is now floor‑faithful and single‑sourced, so the layers × 2 geometry is realised end‑to‑end wherever a per‑head stripe exceeds a granule (large head_dim/context). Demonstrating the kv_heads× win end‑to‑end needs such a model, not a change to this commit path.
• It does not address the seq‑major seed/host‑mirror stride arithmetic (read_present_kv / seed_prefix), which still refuse under seq‑major (Gate the fourth place the KV layout lives: refuse seq-major on head-major-only KV consumers (#777 #801 #797 #794 #809) #812). Prefix sharing under seq‑major therefore still needs the BSNH seed as a separate follow‑on; the doc bullet is corrected to say so.

Validation (all green, on an RTX 4060 Laptop, ORT 1.27, CUDA 13 wheels)

• cargo fmt -p onnx-genai-engine -- --check ✔ (--all-targets full‑workspace fmt hits Windows os error 206, a command‑length artifact, not a formatting problem)
• cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings ✔
• cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings ✔
• cargo test -p onnx-runtime-session ✔ (277 passed)
• cargo test -p onnx-genai-engine --features native-backend --lib ✔ (454 passed; kv_commit 9/9)
• Mandatory mobius_seqmajor_growth_parity_native_cuda --ignored --test-threads=1 ✔ (2 passed, incl. the head‑major negative oracle)
• gqa_seqmajor_parity_gpu ✔ (2), gqa_shared_prefix_parity_gpu ✔ (1), group_query_attention_gpu ✔ (26), vmm_kv_contiguous_tail_gpu ✔ (3), vmm_kv_layout_residency_gpu ✔ (2), vmm_kv_token_major_floor_gpu ✔ (2) — all --ignored --test-threads=1.

Do not merge / do not close any issue.

[justinchuby]

Coordinator validation on the merged branch: cargo fmt -p onnx-genai-engine -- --check clean; full cargo test -p onnx-runtime-session 153 passed; and per
the #814 policy, mobius_seqmajor_growth_parity_native_cuda --ignored 2/2.

The negative result is correct, and I checked the arithmetic myself

The floor does not move on qwen2.5-0.5b, and the explanation is precise rather
than a shrug:

qwen2.5-0.5b: 24 layers x 2 = 48 bindings, head_dim 64, kv_heads 2, fp16, max 512 tokens
per binding = 512 x 2 x 64 x 2 = 131,072 B = 128 KiB  <  2 MiB granule

The entire reservation of a single binding is sub-granule, so both layouts pin
at exactly one granule per binding = 48 granules — which is the layers x 2
floor. qwen2.5-0.5b is already sitting on it; there is nothing for this change to
recover on that model.

The kv_heads x separation only appears once a single head's stripe crosses a
granule, i.e. at roughly 8,192 tokens at head_dim 128/fp16 — the same threshold
#822 derived for head-major prefix shareability, and the same 2 MiB granule
measured in #776. The result is internally consistent with the arithmetic we
already established, which is the main reason I believe it rather than suspecting
an un-wired commit path.

physical_committed_bytes = 100,663,296 for both layouts across all four
vmm-inplace configs (786,432 for realloc), token IDs byte-identical everywhere.

What did land, and why it still matters

seq_major_kv_commit_requests now goes through the shared
kv_commit::live_prefix_ranges geometry instead of duplicated inline byte
arithmetic — those helpers were previously dead_code in non-test builds, meaning
the driver-level residency measurement and the live commit path were computing
the same thing twice, separately. That is precisely the drift risk that has
already produced two wrong claims on this line. Single-sourcing them is worth doing
independently of whether the floor moves today.

Head-major is left byte-identical on its flat bucket commit, with the reason
documented (its per-head bucket stride makes dense ranges equal bucket ranges), and
the #811 keep decision, idempotent re-commit, #748 grant path and #814 ordering are
all preserved. Counters confirm #811 still holds: seq-major growth_keeps=3 with
invalidations collapsing to 1/45/1, head-major moving 688,576 B and re-capturing
4/39/4.

What this tells us about demonstrating the 8x

The layers x 2 floor claim now needs a model whose per-head stripe exceeds a
granule to be shown end to end — qwen14b at head_dim 128 would, but it is not
native-loadable (no inference_metadata.yaml, the #384 gap). So the remaining
gap is not in the commit path; it is that we lack a native-loadable model large
enough to exhibit the effect.

That reframes #384 from "a missing metadata file" to "the blocker on demonstrating
the largest measured memory win end to end". Worth raising its priority
accordingly.

Merging.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	49.69 µs	78.15 µs	+57.3%
🔴	matmul/medium_generic_f16_threads=8/32x512x512	29.55 µs	43.56 µs	+47.4%
🔴	matmul/large_generic_bf16_threads=1/32x1024x1024	2.06 ms	2.73 ms	+32.7%
⚠️	matmul/large_generic_f16_threads=8/32x1024x1024	88.31 µs	109.17 µs	+23.6%
⚠️	matmul/small_generic_f16_threads=1/1x256x256	41.08 µs	48.90 µs	+19.0%
⚠️	matmul/medium_generic_f16_threads=1/32x512x512	31.66 µs	37.22 µs	+17.6%
⚠️	sampling_latency/min_p_per_token	221.08 µs	258.26 µs	+16.8%
⚠️	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	693.30 µs	808.18 µs	+16.6%
⚠️	matmul/small_generic_f32_threads=1/1x256x256	40.67 µs	46.89 µs	+15.3%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	389.13 µs	444.56 µs	+14.2%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.48 ms	6.24 ms	+13.8%
✅	tokenization/encode_tokens_per_second	399.86 µs	453.47 µs	+13.4%
✅	kv_cache/alloc_dealloc_pages	41.57 µs	46.84 µs	+12.7%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	543.73 µs	610.54 µs	+12.3%
✅	reduce_mean/small_f32_threads=1-internal/4096	16.77 µs	18.78 µs	+12.0%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.37 ms	2.65 ms	+11.8%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.52 ms	3.93 ms	+11.4%
✅	logit_processing/seven_processor_chain_per_step	353.83 µs	392.21 µs	+10.8%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	89.14 µs	98.75 µs	+10.8%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	504.39 µs	558.37 µs	+10.7%
✅	gather/large_f32_threads=1-internal/131072	29.53 µs	32.52 µs	+10.1%
✅	gather/medium_bf16_threads=1-internal/32768	2.30 µs	2.52 µs	+9.7%
✅	sampling_latency/top_p_per_token	401.46 µs	438.59 µs	+9.2%
✅	add/medium_f16_threads=1-internal/262144	2.77 ms	2.98 ms	+7.7%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.18 ms	2.34 ms	+7.6%
✅	matmul/small_generic_bf16_threads=8/1x256x256	35.32 µs	37.57 µs	+6.4%
✅	gather/large_f16_threads=1-internal/131072	12.39 µs	13.09 µs	+5.7%
✅	qwen3_sampling_processors/top_k_partial_selection	143.94 µs	150.60 µs	+4.6%
✅	sampling_latency/top_k_per_token	54.55 µs	57.02 µs	+4.5%
✅	qwen3_sampling_processors/top_k_top_p_fast	681.65 µs	706.16 µs	+3.6%
✅	grammar_masking/llguidance_compute_mask/32	80.39 µs	83.25 µs	+3.6%
✅	add/medium_bf16_threads=1-internal/262144	2.60 ms	2.66 ms	+2.4%
✅	add/medium_f32_threads=1-internal/262144	2.74 ms	2.80 ms	+2.2%
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	48.86 µs	49.30 µs	+0.9%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.91 ms	9.91 ms	-0.0%
✅	gather/medium_f16_threads=1-internal/32768	2.49 µs	2.48 µs	-0.4%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	77.77 µs	76.98 µs	-1.0%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	525.70 µs	513.56 µs	-2.3%
✅	add/small_f32_threads=1-internal/1024	220.0 ns	210.5 ns	-4.3%
✅	matmul/small_generic_f32_threads=8/1x256x256	59.25 µs	56.21 µs	-5.1%
✅	gather/medium_f32_threads=1-internal/32768	5.05 µs	4.71 µs	-6.6%
✅	gather/small_bf16_threads=1-internal/4096	510.7 ns	472.2 ns	-7.5%
✅	sampling_latency/greedy_per_token	3.34 µs	3.07 µs	-8.0%
✅	add/large_f32_threads=1-internal/4194304	47.36 ms	43.43 ms	-8.3%
✅	matmul/medium_generic_f32_threads=8/32x512x512	1.30 ms	1.18 ms	-9.3%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.18 ms	1.06 ms	-9.8%
✅	add/small_bf16_threads=1-internal/1024	16.44 µs	14.54 µs	-11.6%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.91 ms	1.68 ms	-12.1%
✅	tokenization/decode_tokens_per_second	7.98 ms	6.98 ms	-12.6%
✅	gather/small_f16_threads=1-internal/4096	576.9 ns	500.7 ns	-13.2%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	4.87 ms	4.20 ms	-13.7%
🟢	add/large_bf16_threads=1-internal/4194304	59.09 ms	50.09 ms	-15.2%
🟢	gather/small_f32_threads=1-internal/4096	797.9 ns	663.6 ns	-16.8%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	42.33 µs	34.37 µs	-18.8%
🟢	add/small_f16_threads=1-internal/1024	17.36 µs	13.84 µs	-20.2%
🟢	gather/large_bf16_threads=1-internal/131072	16.53 µs	13.03 µs	-21.2%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	361.32 µs	261.37 µs	-27.7%
🟢	add/large_f16_threads=1-internal/4194304	60.52 ms	43.76 ms	-27.7%
🟢	matmul/small_generic_f16_threads=8/1x256x256	53.93 µs	35.55 µs	-34.1%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.31 3.17 5.34 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)