[justinchuby]

What this fixes

Closes #1266 — the VMM half of the metadata-less mask/KV reservation overflow that #1264 deliberately left for a design decision.

On the VMM native-CUDA decode path (ONNX_GENAI_CUDA_VMM=1), DecodeCudaState::new reserves the full mask island (batch × capacity.max_len × i64) and the full KV extent (full_vmm_kv_allocation_bytes(..., capacity.max_len)) up front. For a model with no max_sequence_length metadata, resolve_cuda_kv_capacity returns capacity.max_len == usize::MAX (the deliberate "unbounded until growth fails" sentinel from #367). Unlike the non-VMM path — where those values are dead (fixed in #1264) — on the VMM path they are live, so the reservation arithmetic overflows and aborts construction with full CUDA mask reservation size overflow.

The decisive fact: the VMM reservation is virtual-only

allocate_device_binding_committed(..., allocation_bytes, committed_ranges) → CudaVmmAllocator::allocate_committed(bytes, align, ranges) only carves bytes out of a fixed 64 GiB virtual arena (RESERVATION_BYTES = 64 << 30, arena.spans.carve). Physical granules are claimed only for committed_ranges. So the up-front reservation costs address space, not memory.

Options weighed

• (a) Derive a concrete bound from device memory — CHOSEN. Resolve the sentinel only for the VMM reservation to device_free / bytes_per_token (the largest sequence the device could ever physically hold). Because the reservation is virtual-only, an over-large-but-finite bound is nearly free (≈device_free of VA + a ~1.5 GB mask, well inside the 64 GiB arena), and a decode that outgrows it still fails at the same physical ceiling the sentinel implied.
• (b) Configurable default ceiling + bucket growth. Rejected: it invents a second policy knob and a magic default when option (a) already has the exactly-right, self-tuning ceiling for free (the reservation is virtual, so there is no reason to pick anything smaller than the physical maximum).
• (c) Refuse VMM for metadata-less models, fall back to eager. Rejected: silently strips VMM's whole benefit (in-place growth, no bind churn, capture stability) from an entire class of models — every model without declared max_sequence_length — for no gain, since (a) makes VMM work.
• (d) Resolve the sentinel at capacity-resolution time. Rejected: cleanest-looking but rewrites capacity.max_len itself, changing the deliberate usize::MAX contract (feat(kv): one KV capacity mechanism for native CUDA and ORT #367) that the non-VMM grow-until-OOM path and cuda_kv_capacity_without_metadata_is_unbounded_until_growth_fails depend on. Not worth touching a deliberate contract when a local resolution suffices.

The fix therefore resolves the bound only locally, only for the VMM reservation; capacity.max_len and the non-VMM path are untouched (the regression test asserts the sentinel survives).

Falsification (RTX 4060, driver 591.55, CUDA 13 runtime)

New CUDA-gated regression test native_cuda_vmm_decoder_constructs_with_unbounded_metadata_less_capacity drives real VMM-path construction with a metadata-less capacity, asserts it is actually on the commit-on-demand path (kv_commits_on_demand()), and asserts capacity.max_len == usize::MAX still holds.

• Fix in place → GREEN.
• Revert the bound resolution (vmm_reserved_max_len = capacity.max_len) → RED with the real failure: Caused by: full CUDA mask reservation size overflow.
• Restore → GREEN.

The VMM path is proven live, not a proxy: the arena log reports device allocations go through a VMM arena over 68719476736 bytes of reserved address space, and the assertion catches any silent fall-back to the eager path. The existing real-model growth test native_cuda_vmm_kv_grows_in_place_and_commits_more_granules (qwen05b-q4, ONNX_GENAI_KV_MIN_BUCKET=8) also passes — it calls ensure_capacity(8193) and asserts committed bytes grow by the exact bucket delta with device_ptrs unchanged (grown in place), confirming this change does not disturb the metadata-having VMM growth path.

Test isolation note

The test deliberately does not set ONNX_GENAI_CUDA_GRAPH: that is a frozen RuntimeConfig knob (#804), so mutating it after any earlier test froze the process-wide config trips the freeze-guard. Construction never captures a graph, so the knob is irrelevant here; vmm_enabled() reads ONNX_GENAI_CUDA_VMM directly from the environment, so setting only VMM is safe regardless of freeze order. Also folds in a rustfmt fix for an overlong line #1264 introduced in the same file.

GPU verification (RTX 4060, --features "cuda,native-backend" -- --test-threads=1)

• Engine lib suite (default): 559 passed, 0 failed, 2 ignored.
• Engine lib suite (ONNX_GENAI_RUN_CUDA_SMOKE=1): 556 passed, 3 failed, 2 ignored. The 3 failures are the known pre-existing model.io.token_input harness ambiguities (native_cuda_capture_replay_..., native_cuda_symbolic_batch_..., native_cuda_symbolic_total_seq_...), unrelated to this change and identical to the set fix(cuda): compute full mask reservation only on the VMM path (fixes red native-CUDA parity test) #1264 documented. Both new metadata-less tests pass; zero new failures.

Known caveat

The derived bound is device_free / bytes_per_token; the summed VA reservation is ≈1.2 × device_free. On this 8 GB card that is 8 GB of the 64 GiB arena. On a very large GPU (≳53 GiB free) the metadata-less VMM reservation could exceed the 64 GiB arena and fail the carve loudly at construction (no silent corruption). That is an acceptable, visible failure mode for a narrow combination (metadata-less and VMM and a huge idle GPU); a follow-up could cap the bound to the arena if it ever bites.

[github-actions]

✅ Benchmarks — No Regression

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	42.15 µs	43.94 µs	+4.3%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	498.39 µs	513.13 µs	+3.0%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	8.61 ms	8.81 ms	+2.3%
✅	sampling_latency/greedy_per_token	3.00 µs	3.06 µs	+2.1%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.60 ms	3.68 ms	+2.0%
✅	sampling_latency/top_k_per_token	48.37 µs	49.34 µs	+2.0%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.28 ms	3.35 ms	+2.0%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	347.60 µs	352.54 µs	+1.4%
✅	grammar_masking/llguidance_compute_mask/32	70.31 µs	71.23 µs	+1.3%
✅	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	517.63 µs	523.36 µs	+1.1%
✅	matmul/small_generic_bf16_threads=1/1x256x256	28.41 µs	28.69 µs	+1.0%
✅	matmul/medium_generic_f16_threads=1/32x512x512	27.72 µs	27.95 µs	+0.8%
✅	sampling_latency/top_p_per_token	355.55 µs	358.38 µs	+0.8%
✅	matmul/small_generic_f32_threads=8/1x256x256	31.90 µs	32.14 µs	+0.8%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.25 ms	1.25 ms	+0.5%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	486.64 µs	487.43 µs	+0.2%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	74.25 µs	74.37 µs	+0.2%
✅	tokenization/encode_tokens_per_second	352.21 µs	352.62 µs	+0.1%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	1.98 ms	1.98 ms	+0.1%
✅	matmul/small_generic_f32_threads=1/1x256x256	35.01 µs	34.89 µs	-0.3%
✅	matmul/medium_generic_f32_threads=8/32x512x512	902.40 µs	899.09 µs	-0.4%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.42 ms	5.37 ms	-0.8%
✅	sampling_latency/min_p_per_token	197.76 µs	195.87 µs	-1.0%
✅	kv_cache/alloc_dealloc_pages	37.52 µs	37.14 µs	-1.0%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.90 ms	1.88 ms	-1.1%
✅	qwen3_sampling_processors/top_k_top_p_fast	625.33 µs	618.61 µs	-1.1%
✅	matmul/small_generic_f16_threads=1/1x256x256	28.76 µs	28.35 µs	-1.4%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.21 ms	2.17 ms	-1.5%
✅	qwen3_sampling_processors/top_k_partial_selection	132.38 µs	130.30 µs	-1.6%
✅	gather/medium_bf16_threads=1-internal/32768	2.27 µs	2.23 µs	-1.8%
✅	matmul/medium_generic_f16_threads=8/32x512x512	28.42 µs	27.79 µs	-2.2%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	80.98 µs	79.02 µs	-2.4%
✅	add/small_bf16_threads=1-internal/1024	422.7 ns	411.0 ns	-2.8%
✅	matmul/small_generic_f16_threads=8/1x256x256	29.63 µs	28.65 µs	-3.3%
✅	gather/small_bf16_threads=1-internal/4096	464.4 ns	445.5 ns	-4.1%
✅	matmul/small_generic_bf16_threads=8/1x256x256	31.16 µs	29.76 µs	-4.5%
✅	add/large_f32_threads=1-internal/4194304	653.67 µs	618.50 µs	-5.4%
✅	gather/large_f16_threads=1-internal/131072	10.33 µs	9.75 µs	-5.6%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	53.54 µs	50.27 µs	-6.1%
✅	add/medium_bf16_threads=1-internal/262144	103.28 µs	95.85 µs	-7.2%
✅	add/large_bf16_threads=1-internal/4194304	1.69 ms	1.57 ms	-7.3%
✅	reduce_mean/medium_f32_threads=1-internal/65536	248.38 µs	229.88 µs	-7.4%
✅	add/large_f16_threads=1-internal/4194304	1.69 ms	1.56 ms	-7.5%
✅	logit_processing/seven_processor_chain_per_step	331.17 µs	305.46 µs	-7.8%
✅	add/small_f16_threads=1-internal/1024	457.9 ns	420.8 ns	-8.1%
✅	gather/large_f32_threads=1-internal/131072	28.09 µs	25.76 µs	-8.3%
✅	add/medium_f16_threads=1-internal/262144	106.69 µs	97.52 µs	-8.6%
✅	reduce_mean/small_f32_threads=1-internal/4096	15.50 µs	14.14 µs	-8.8%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.02 ms	920.34 µs	-10.0%
✅	tokenization/decode_tokens_per_second	6.43 ms	5.77 ms	-10.4%
✅	gather/small_f32_threads=1-internal/4096	696.2 ns	622.9 ns	-10.5%
✅	gather/medium_f32_threads=1-internal/32768	3.93 µs	3.50 µs	-11.0%
✅	add/small_f32_threads=1-internal/1024	204.9 ns	182.3 ns	-11.0%
✅	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	44.68 µs	39.68 µs	-11.2%
✅	add/medium_f32_threads=1-internal/262144	25.63 µs	22.62 µs	-11.8%
✅	gather/small_f16_threads=1-internal/4096	509.7 ns	448.3 ns	-12.0%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	410.70 µs	359.73 µs	-12.4%
✅	gather/large_bf16_threads=1-internal/131072	11.23 µs	9.81 µs	-12.6%
🟢	gather/medium_f16_threads=1-internal/32768	2.72 µs	2.23 µs	-18.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.15 3.16 4.45 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.59%. Comparing base (2ecd8fe) to head (148f608).
⚠️ Report is 49 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##           main    #1281       +/-   ##
=========================================
+ Coverage      0   80.59%   +80.59%     
=========================================
  Files         0      364      +364     
  Lines         0   160710   +160710     
  Branches      0   160710   +160710     
=========================================
+ Hits          0   129532   +129532     
- Misses        0    26521    +26521     
- Partials      0     4657     +4657     

Flag	Coverage Δ
mlas	85.09% <ø> (?)
offline	80.51% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 364 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.