Enforce explicit VRAM limit at load by justinchuby · Pull Request #717 · justinchuby/onnx-genai

justinchuby · 2026-08-06T20:33:48Z

Summary

Closes #712. Context: #704.

Enforces explicit byte VRAM limits in engine load instead of only KV-budget derivation.
Native CUDA now derives a DeviceOffloadPolicy from --vram-limit when weights exceed the limit, passes it into the CUDA EP, and disables graph capture through the same resolved policy.
ONNX_GENAI_WEIGHT_OFFLOAD_DEVICE_BYTES is preserved as an override for the derived residency budget.
Non-offload-capable device providers now fail at load with required-vs-available arithmetic; host-only execution is not rejected by a VRAM limit.

Design decisions

Where enforcement lives: engine load. The scheduler governor only knows how to derive KV budget; the allocator sees allocations too late, especially on WDDM spill. Engine load is the first point that knows model package bytes, selected backend, offload capability, and the engine ledger.

Unset --vram-limit: unchanged. The default remains the existing resource-limit default (0.90 of the current provisional VRAM capacity). I did not turn unset into a new hard real-device-capacity refusal because that would silently change WDDM behavior for users who did not ask for a limit, and this tree still uses provisional capacity providers.

Auto-offload announcement: yes. Native CUDA emits an info line when explicit VRAM causes automatic offload, with model bytes, resolved VRAM bytes, and offload device budget bytes.

Verification

Passed:

cargo fmt --all -- --check
cargo test -p onnx-genai-engine pool_sizing_tests --lib --features native-backend --no-default-features
cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings
cargo clippy --all-targets -p onnx-genai-engine --features cuda,native-backend -- -D warnings
cargo clippy -p onnx-runtime-ep-cuda --lib --features cuda -- -D warnings
cargo build --bin onnx-genai --features native-backend

Manual checks with the 14B artifact using the debug native-backend build:

ONNX_GENAI_EP=cuda --vram-limit 2000000000 -> Error: model weights require 16653143582 bytes ... allows 2000000000 bytes ... cannot automatically offload weights
ONNX_GENAI_EP=cuda --vram-limit 6000000000 -> Error: model weights require 16653143582 bytes ... allows 6000000000 bytes ... cannot automatically offload weights
host-only --vram-limit 6000000000 -> generated "Hello"; host-only execution is intentionally not rejected by a VRAM limit

Sabotage check: I broke device_weight_reservation_for so the offload startup claim did not leave one KV page, then reran the targeted test. It failed as expected:

test engine::load::pool_sizing_tests::offload_startup_reservation_leaves_one_kv_page_unwarned ... FAILED
assertion `left == right` failed: the temporary startup claim must leave the governor a one-page KV floor; the CUDA provider later adopts the full offload budget as the real claim
  left: 6000
 right: 5872

Not completed: the requested CUDA release E2E could not be built on this machine because cudarc generated duplicate cuBLASLt bindings before reaching this change:

error[E0428]: the name `cublasComputeType_t` is defined multiple times
... cudarc-0.19.8\src\cublaslt\sys\mod.rs

Also note: cargo clippy --all-targets -p onnx-runtime-ep-cuda --features cuda -- -D warnings is blocked by pre-existing CUDA test lint errors (unusual byte grouping, clone-on-copy, too-many-arguments, etc.). The touched CUDA EP library target passes clippy.

Derive native CUDA weight offload from explicit VRAM limits and fail non-offload-capable device loads when weights exceed the limit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

Size derived CUDA weight offload from the explicit VRAM limit after reserving native device KV and recurrent state, and fail at load when the remaining weight budget cannot satisfy the largest lazy-weight working set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

justinchuby · 2026-08-06T22:48:33Z

Follow-up after real CUDA E2E correction:

Fixed derived auto-offload budget to subtract required device state instead of using the whole VRAM limit.
For qwen14b-zp at --vram-limit 6000000000, native CUDA now reports:
- required_device_non_weight_bytes=1610612736
- native_kv_bytes=1610612736
- minimum_useful_weight_budget_bytes=437954560
- offload_device_budget_bytes=4389387264
- generation succeeds: Hello! How can
At --vram-limit 1000000000, load fails before generation with arithmetic:
- resolved_vram_bytes=1000000000
- required_device_non_weight_bytes=1610612736
- available_weight_offload_budget_bytes=0
- requested_weight_offload_budget_bytes=1000000000

Validation rerun:

cargo fmt --all -- --check
cargo test -p onnx-genai-engine pool_sizing_tests --lib --features native-backend --no-default-features
cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings
cargo clippy --all-targets -p onnx-genai-engine --features cuda,native-backend -- -D warnings
cargo clippy -p onnx-runtime-ep-cuda --lib --features cuda -- -D warnings
cargo build --release -p onnx-genai-cli --bin onnx-genai --features cuda,native-cuda

codecov · 2026-08-06T23:15:56Z

Codecov Report

❌ Patch coverage is 0% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.12%. Comparing base (0753608) to head (146197d).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...ates/onnx-runtime-session/src/executor/bindings.rs	0.00%	20 Missing ⚠️
crates/onnx-runtime-session/src/lib.rs	0.00%	6 Missing ⚠️
crates/onnx-runtime-ep-api/src/provider.rs	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #717      +/-   ##
==========================================
- Coverage   80.48%   80.12%   -0.37%     
==========================================
  Files         328      331       +3     
  Lines      125517   132533    +7016     
  Branches   125517   132533    +7016     
==========================================
+ Hits       101024   106187    +5163     
- Misses      20318    22107    +1789     
- Partials     4175     4239      +64

Flag	Coverage Δ
cli-ort-linux	`82.43% <ø> (+0.21%)`	⬆️
cli-ort-windows	`81.90% <ø> (?)`
mlas	`80.96% <ø> (?)`
offline	`80.01% <0.00%> (-0.41%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-runtime-ep-api/src/provider.rs	`70.65% <0.00%> (-0.49%)`	⬇️
crates/onnx-runtime-session/src/lib.rs	`64.37% <0.00%> (-0.46%)`	⬇️
...ates/onnx-runtime-session/src/executor/bindings.rs	`47.95% <0.00%> (-2.98%)`	⬇️

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-08-06T23:49:04Z

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	5.65 ms	7.74 ms	+37.2%
🔴	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	41.78 µs	56.30 µs	+34.8%
🔴	`kv_cache/alloc_dealloc_pages`	37.04 µs	48.88 µs	+32.0%
⚠️	`qwen3_sampling_processors/top_k_top_p_fast`	668.92 µs	858.02 µs	+28.3%
⚠️	`grammar_masking/llguidance_compute_mask/32`	76.43 µs	92.52 µs	+21.1%
✅	`logit_processing/seven_processor_chain_per_step`	321.86 µs	351.58 µs	+9.2%
✅	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	39.97 µs	43.59 µs	+9.1%
✅	`qwen3_sampling_processors/top_p_fast_after_top_k`	528.38 µs	575.42 µs	+8.9%
✅	`sampling_latency/top_k_per_token`	50.30 µs	52.81 µs	+5.0%
✅	`qwen3_sampling_processors/top_k_partial_selection`	137.65 µs	144.20 µs	+4.8%
✅	`tokenization/encode_tokens_per_second`	374.72 µs	386.58 µs	+3.2%
✅	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	376.12 µs	385.21 µs	+2.4%
✅	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	3.52 ms	3.59 ms	+1.8%
✅	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	77.17 µs	77.86 µs	+0.9%
✅	`gather/large_f16_threads=1-internal/131072`	12.26 µs	12.33 µs	+0.6%
✅	`matmul/large_generic_f32_threads=1/32x1024x1024`	9.42 ms	9.47 ms	+0.5%
✅	`matmul/small_generic_f32_threads=8/1x256x256`	34.13 µs	34.22 µs	+0.3%
✅	`matmul/small_generic_bf16_threads=1/1x256x256`	32.23 µs	32.31 µs	+0.2%
✅	`sampling_latency/greedy_per_token`	3.22 µs	3.22 µs	+0.2%
✅	`matmul/large_generic_f16_threads=8/32x1024x1024`	85.23 µs	85.30 µs	+0.1%
✅	`matmul/large_generic_bf16_threads=1/32x1024x1024`	2.02 ms	2.02 ms	-0.0%
✅	`matmul/large_generic_f16_threads=1/32x1024x1024`	79.86 µs	79.60 µs	-0.3%
✅	`matmul/small_generic_f16_threads=8/1x256x256`	31.48 µs	31.34 µs	-0.5%
✅	`matmul/medium_generic_bf16_threads=1/32x512x512`	541.68 µs	539.15 µs	-0.5%
✅	`sampling_latency/min_p_per_token`	208.47 µs	206.69 µs	-0.9%
✅	`matmul/medium_generic_f16_threads=1/32x512x512`	30.69 µs	30.34 µs	-1.1%
✅	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	509.27 µs	500.70 µs	-1.7%
✅	`matmul/medium_generic_f32_threads=1/32x512x512`	2.36 ms	2.32 ms	-1.7%
✅	`matmul/large_generic_bf16_threads=8/32x1024x1024`	1.33 ms	1.31 ms	-2.0%
✅	`reduce_mean/small_f32_threads=1-internal/4096`	15.36 µs	15.02 µs	-2.2%
✅	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.16 ms	2.11 ms	-2.3%
✅	`matmul/medium_generic_bf16_threads=8/32x512x512`	380.81 µs	371.60 µs	-2.4%
✅	`add/large_bf16_threads=1-internal/4194304`	43.23 ms	42.17 ms	-2.5%
✅	`matmul/small_generic_f16_threads=1/1x256x256`	32.85 µs	31.93 µs	-2.8%
✅	`add/medium_f32_threads=1-internal/262144`	2.61 ms	2.52 ms	-3.7%
✅	`reduce_mean/medium_f32_threads=1-internal/65536`	256.84 µs	246.91 µs	-3.9%
✅	`gather/small_bf16_threads=1-internal/4096`	487.2 ns	468.0 ns	-3.9%
✅	`sampling_latency/top_p_per_token`	396.02 µs	378.62 µs	-4.4%
✅	`gather/small_f32_threads=1-internal/4096`	687.3 ns	654.4 ns	-4.8%
✅	`add/small_bf16_threads=1-internal/1024`	13.45 µs	12.81 µs	-4.8%
✅	`add/small_f32_threads=1-internal/1024`	212.3 ns	201.8 ns	-4.9%
✅	`add/medium_f16_threads=1-internal/262144`	2.69 ms	2.56 ms	-5.0%
✅	`tokenization/decode_tokens_per_second`	6.61 ms	6.27 ms	-5.2%
✅	`add/large_f16_threads=1-internal/4194304`	43.29 ms	40.99 ms	-5.3%
✅	`gather/small_f16_threads=1-internal/4096`	496.0 ns	468.4 ns	-5.6%
✅	`add/large_f32_threads=1-internal/4194304`	42.99 ms	40.36 ms	-6.1%
✅	`reduce_mean/large_f32_threads=1-internal/262144`	1.06 ms	988.84 µs	-6.4%
✅	`matmul/medium_generic_f32_threads=8/32x512x512`	978.36 µs	912.67 µs	-6.7%
✅	`add/medium_bf16_threads=1-internal/262144`	2.83 ms	2.63 ms	-7.1%
✅	`matmul/large_generic_f32_threads=8/32x1024x1024`	3.97 ms	3.67 ms	-7.6%
✅	`gather/medium_bf16_threads=1-internal/32768`	2.58 µs	2.36 µs	-8.2%
✅	`matmul/medium_generic_f16_threads=8/32x512x512`	32.10 µs	29.26 µs	-8.9%
✅	`matmul/small_generic_bf16_threads=8/1x256x256`	35.39 µs	32.22 µs	-9.0%
✅	`gather/large_bf16_threads=1-internal/131072`	13.69 µs	12.40 µs	-9.4%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	41.19 µs	37.29 µs	-9.5%
✅	`add/small_f16_threads=1-internal/1024`	14.56 µs	12.87 µs	-11.6%
✅	`gather/medium_f32_threads=1-internal/32768`	4.57 µs	3.92 µs	-14.1%
🟢	`gather/large_f32_threads=1-internal/131072`	38.12 µs	27.81 µs	-27.1%
🟢	`gather/medium_f16_threads=1-internal/32768`	5.16 µs	2.35 µs	-54.4%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.31 3.63 5.48 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

justinchuby · 2026-08-07T00:15:02Z

Verified end to end myself on the 14b before merging, on a binary rebuilt from the committed source.

Arm A — --vram-limit 6000000000

enabled CUDA weight offload because model weights exceed the VRAM limit
  model_weight_bytes=16653143582 resolved_vram_bytes=6000000000
  required_device_non_weight_bytes=1610612736 native_kv_bytes=1610612736
  recurrent_state_bytes=0 minimum_useful_weight_budget_bytes=437954560
  offload_device_budget_bytes=4389387264

4,389,387,264 = 6,000,000,000 − 1,610,612,736. The arithmetic is right and the regression is gone. Generation is correct: prompt The capital of France is → Paris., exit 0.

Arm B — --vram-limit 1000000000
Fails at load, naming every term including available_weight_offload_budget_bytes=0 and requested_weight_offload_budget_bytes=1000000000. It now blames the weight budget rather than reporting a KV failure, which was the specific complaint.

One honest gap, carried not closed: activations_bytes=unknown and runtime_overhead_bytes=unknown. The derived budget subtracts KV but not activations or runtime overhead, so a limit can still be exceeded in practice. The engine says so out loud rather than reporting zero — tracked as #514. Worth stating in the docs for this flag.

Merging.

Consolidating today's findings into the document rather than leaving them in issues. ## The new section: a third failure mode `MEMORY_ARCHITECTURE.md` already recorded two ways this area fails — code that is right, tested and never reached; and measurements that are reproducible but support a conclusion they do not license. Today produced a third, and it cost more than either. **An instrument that runs, reports a plausible number, and measures the wrong thing.** `h2d_enqueue_copy_ms` put host-to-device transfer at **1.7%** of decode step time. That figure retired an entire line of work — prefetch depth, pinned versus pageable staging, transfer overlap — and was published on #705 and #718. The arithmetic gives it away: the same step stages **7.33 GiB**, and 7.33 GiB in 30 ms would be ~250 GB/s, roughly ten times this machine's PCIe link. The counter bracketed an asynchronous enqueue *returning*. The first repair renamed it `copy_wait_ms`, pointing it at `compute_wait_fence` — whose own doc comment reads *"a stream-ordered, **non host-blocking** cross-stream wait."* The same non-measurement survived under a name that invited the wrong reading **more** strongly than before. Measured properly with CUDA events, transfer is **18.8%**. Two habits, both cheap: - **Divide the bytes by the time and ask whether the answer is a real bandwidth.** A number near link speed, near memcpy speed, or ten times either tells you what the counter actually bracketed. This is what caught it, and it takes seconds. - **State on every timing counter exactly what lies between start and stop, and whether anything there blocks the host.** Enforced in `weight_paging.rs` since #715. ## The pattern underneath all three Telemetry that never reaches the operator, three times in one subsystem: documented env vars nothing reads (#688), counters computed and never printed (#719), counters printed with nothing writing them (#715). The #719 case is the sharpest. A residency cache paging at a **0% hit rate** printed a weight-offload section in which **every visible row read `0.00`** — and my first reading was that the cache was inert. The opposite was true and provably so: the section printing at all required one of the three *hidden* counters to be non-zero. The instrument reported the exact inverse of what it measured. After a third instance the response should be a guard, not more vigilance. ## Status table rows where today changed the facts - **`--vram-limit`** — was "does not limit VRAM"; now enforced at load, auto-enabling offload with the weight budget derived as `limit − device state`. Caveat carried rather than closed: activations and runtime overhead are still `unknown` and unsubtracted (#514). PR #717 - **Weight residency cache (CUDA)** — new row. It had a **0% hit rate for its entire life**: 6,936 page-ins, 0 hits, at both 3 GB and 6 GB, because `evict_to_fit` is LRU against a cyclic sequential scan. Fixed to **74.18%** against a `B/W ≈ 76%` ceiling, evictions 6,286 → **0**. #720, PR #723 - **Weight offload vs the OS** — the 27x gap now has a named mechanism instead of a mystery, with ~39% of step time placed downstream of that one defect. - **KV page storage** — new row. `onnx-genai-kv` never touched device memory; the contract is now open (`Box<dyn KvPageStore>`), host-only pending stages 2–5. PR #726 - **Captured graphs across a VMM remap** — new row, verified on hardware and sentinel-proven. PR #727 Docs only. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

## Summary - Adds `serve --vram-limit` / `ONNX_GENAI_VRAM_LIMIT` and applies it to `EngineConfig.limits.vram_limit` before any eager model load. - Keeps a single knob and single ledger: no server-private budget; the value flows through the existing engine/resource-governor config that #717 already uses for CUDA auto-offload. - Exposes only process-wide CUDA weight-offload activity metrics (`page_ins_total`, `hits_total`, `evictions_total`, and derived process-wide `hit_rate`). I removed the residency budget/peak gauges because they are not process truth for multi-model servers; canonical VRAM used/limit/headroom remains the engine ledger metrics. I chose the smaller server-side change rather than moving the full CLI `EngineArgs`: `EngineArgs` currently lives in `onnx-genai-cli`, while `ServeArgs` is shared by the standalone server and unified CLI. Moving the full type would be a larger cross-crate refactor; #734's actual divergence is the missing launch-time VRAM limit that feeds load-time offload, and this fixes that without adding another budget or offload-specific flag. Refs #706 Closes #734 ## Acceptance check Built binary verified: ```text C:\Users\justinchu\dev\onnx-genai-706-memory-pressure\target\debug\onnx-genai-server.exe LastWriteTimeUtc 2026-08-07 13:39:19Z ``` Launch: ```powershell .\target\debug\onnx-genai-server.exe --model C:\Users\justinchu\dev\models\qwen14b-zp --addr 127.0.0.1:8134 --native-device cuda --vram-limit 6000000000 ``` Observed expected load-time log: ```text enabled CUDA weight offload because model weights exceed the VRAM limit model_weight_bytes=16653143582 resolved_vram_bytes=6000000000 required_device_non_weight_bytes=1610612736 native_kv_bytes=1610612736 recurrent_state_bytes=0 minimum_useful_weight_budget_bytes=437954560 offload_device_budget_bytes=4389387264 ``` After one request, `/metrics` showed non-zero process-wide offload activity counters: ```text onnx_genai_cuda_weight_offload_page_ins_total 867 onnx_genai_cuda_weight_offload_hits_total 0 onnx_genai_cuda_weight_offload_evictions_total 0 onnx_genai_cuda_weight_offload_hit_rate 0 ``` The non-zero activity counters plus the load log are the offload-engagement proof. The removed residency budget/peak names are intentionally not exported. ## Interleaved sweep Same binary/session, qwen14b-zp, native CUDA, `max_tokens=24`, n=5 per arm, interleaved no-limit vs `--vram-limit 6000000000`. | concurrency | arm | individual total seconds | median | range | offload counters | |---:|---|---:|---:|---:|---| | 1 | no limit | 6.705, 4.841, 4.907, 4.880, 4.906 | 4.906 | 4.841-6.705 | page_ins=0 hits=0 evictions=0 | | 1 | 6GB limit | 39.513, 23.838, 17.515, 18.638, 19.530 | 19.530 | 17.515-39.513 | page_ins=1785 hits=1683 evictions=0 hit_rate=48.53% | | 4 | no limit | 7.824, 6.589, 8.871, 8.607, 10.957 | 8.607 | 6.589-10.957 | page_ins=0 hits=0 evictions=0 | | 4 | 6GB limit | 54.000, 52.079, 71.010, 78.856, 57.951 | 57.951 | 52.079-78.856 | page_ins=5457 hits=8415 evictions=0 hit_rate=60.66% | | 8 | no limit | 11.151, 11.376, 12.463, 73.339, 87.021 | 12.463 | 11.151-87.021 | page_ins=0 hits=0 evictions=0 | | 8 | 6GB limit | 96.150, 96.370, 302.130, 472.220, 471.698 | 302.130 | 96.150-472.220 | page_ins=10353 hits=17391 evictions=0 hit_rate=62.68% | | 16 | no limit | 165.442, 140.933, 138.055, 138.154, 33.480 | 138.154 | 33.480-165.442 | page_ins=0 hits=0 evictions=0 | | 16 | 6GB limit | 805.475, 875.814, 861.425, 916.566, 191.329 | 861.425 | 191.329-916.566 | page_ins=20145 hits=35343 evictions=0 hit_rate=63.69% | The limited/governed arm is slower on this machine. I am not claiming a throughput win; this PR makes the server governed and measurable instead of relying on WDDM demand paging. Ranges remain noisy, and `rejections_total` stayed 0 in completed scrapes, so #706's admission/refusal question remains open. ## Validation - `cargo fmt --all -- --check` - `cargo test --locked -p onnx-genai-server prometheus_does_not_export_non_aggregate_weight_offload_gauges -- --nocapture` - `cargo test --locked -p onnx-genai-server --features "cuda,native-backend" cuda_weight_offload_metrics_are_process_activity_only -- --nocapture` - `cargo test --locked -p onnx-genai-server serve_vram_limit -- --nocapture` - `cargo clippy --locked --all-targets -p onnx-genai-server -- -D warnings` - `cargo clippy --locked --all-targets -p onnx-genai-server --features "cuda,native-backend" -- -D warnings` - `cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings` - `cargo build --locked -p onnx-genai-server --features "cuda,native-backend"` Note: `cargo test --locked -p onnx-genai-server --features cuda ...` without `native-backend` still hits the existing engine cfg issue where CUDA load helpers reference native types behind `native-backend`; the CUDA metrics test passes with the server's actual native CUDA feature set. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

## Summary - authority-owned weak `ReclaimableMappedHolder` registration and transactional RAII `MappedGrowthGrant` - unused-allowance-first transfer, external reclaim callbacks, grant-bound capacity tokens, rollback, FIFO serialization, and atomic reserved-to-mapped attribution - batched native CUDA KV/workspace growth with exact global granule 0→1 charging and 1→0 refund - one canonical mapped allowance per suballocating VMM arena; content lifetime remains separate from physical zone ownership - explicit byte `serve --vram-limit` selects managed no-spill VMM/pool mode; `ONNX_GENAI_DYNAMIC_KV_WEIGHT_LENDING=0` restores compatibility fallback - typed capacity refusals map to pre-header HTTP 429 with `Retry-After` Refs #736 Refs #717 Refs #745 Refs #747 ## Review fixes - managed VMM initialization failure is fatal before model allocation; compatibility mode alone may fall back - managed physical-pool ownership avoids double-charging resident weights when offload is unnecessary - allocator transactions charge only exact global granule transitions, including packed/shared workspace allocations - CUDA provider centrally refunds the canonical arena allowance on full deallocation and partial range decommit, including when an ordinary buffer drops the final shared reference - specialized workspace/session cleanup cannot omit or double-refund attribution - state-only native models load without KV bytes/token when paged geometry is not required ## Validation - `cargo fmt --all -- --check` - memory governor: 36 passed - session: 135 passed - CUDA EP library: 295 passed, 4 ignored - engine CUDA/native library: 434 passed, 1 ignored - hardware VMM GPU: 16 passed on RTX 4060 Laptop GPU - provider canonical full/partial decommit GPU regression: passed - changed-library clippy with `-D warnings`: passed - changed-package all-target clippy with `-D warnings`: passed ## 14B / 6 GiB E2E Model: `C:\Users\justinchu\dev\models\qwen14b-zp`; release native-CUDA binary. - managed mode logged VMM/pool strategy and registered the weight holder - short `max_tokens=24`: HTTP 200, correct `Paris`; weight allowance 4,831,838,208 bytes; owned 5,534,384,128 <= 6 GiB - first physical KV growth transferred/attributed 201,326,592 bytes - 8K refused pre-header with HTTP 429, never late 500/OOM - four queued short requests completed FIFO; server did not demonstrate simultaneous live KV accumulation - opt-out did not auto-enable VMM/pool and retained compatibility behavior Latest commit: `440a1df4`. PR remains open and unmerged. --------- Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>

Copilot AI added 2 commits August 6, 2026 13:32

justinchuby merged commit 820b45b into main Aug 7, 2026
14 of 16 checks passed

justinchuby deleted the enforce-vram-limit branch August 7, 2026 00:15

This was referenced Aug 9, 2026

Use committed granules for VMM weight admission #745

Merged

Dynamic lending between CUDA weights and KV growth #746

Closed

Add transactional mapped growth grants #748

Merged

Navigation Menu

Repository navigation

Enforce explicit VRAM limit at load - #717

Enforce explicit VRAM limit at load#717
justinchuby merged 2 commits into
mainfrom
enforce-vram-limit

justinchuby commented Aug 6, 2026

Uh oh!

justinchuby commented Aug 6, 2026

Uh oh!

codecov Bot commented Aug 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

justinchuby commented Aug 7, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

justinchuby commented Aug 6, 2026

Summary

Design decisions

Verification

Uh oh!

justinchuby commented Aug 6, 2026

Uh oh!

codecov Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Aug 6, 2026

🔴 Benchmark Regression Detected

Uh oh!

justinchuby commented Aug 7, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Aug 6, 2026 •

edited

Loading