[justinchuby]

Summary

Makes the zero-copy hybrid default safety budget platform-conditional instead of a single global 256 MiB constant.

• Windows (WDDM): unchanged at 256 MiB — the Managed offload is 30x SLOWER than WDDM demand paging on Windows — and it is the default: the copy-to-VRAM buys nothing for a single-touch access pattern #864 aperture ceiling is real there.
• Non-Windows (Linux / discrete GPU): raised to 2 GiB.

Why

The 256 MiB default was conservative only because of a WDDM/VidMm memory-manager artifact measured on an RTX 4060 Laptop (#864 / #912): aggregate distinct host-mapped bytes read per decode step above ~0.44–0.65 GB silently return STALE data → wrong tokens, no error (48 cold weights collapsed generation 16 → 3 tokens).

#925 re-measured the hybrid on Linux and proved that ceiling is Windows-specific — absent here:

• H200, driver 580.105.08, CUDA 13.0, kernel 6.6.141.1-1.azl3, Azure Linux 3.0, native VMM decode path.
• Generation stayed byte-identical to the Step-0 baseline with cuda_graph fallbacks=0 up to 6.795 GB of distinct host-mapped weights bound and re-read in place every decode step (704 cuMemHostRegister binds), n=3, all runs byte-identical — ~15× the WDDM ~0.44 GB onset.
• On Linux cuMemHostRegister(READ_ONLY|DEVICEMAP) pins pages in the driver with no VidMm layer above it, so the ceiling that motivates the Windows value does not apply.
• The shipped 256 MiB default left an ~8× decode win unused in the over-budget regime: hybrid @ 8 GiB budget 67.04 tok/s vs managed streaming ~8.5 tok/s median (both byte-identical output).

Full measurement report: #925 (#925 (comment)).

Why 2 GiB (bounded, not unbounded)

Chosen deliberately conservative-but-useful:

• >3× below the Re-measure the #864 zero-copy hybrid on Linux: the #912 negative is WDDM-scoped and does not transfer #925 measured-safe 6.795 GB (margin, since only Hopper/H200 was actually tested).
• >3× above the entire WDDM 0.44–0.65 GB corruption band, so the lever Re-measure the #864 zero-copy hybrid on Linux: the #912 negative is WDDM-scoped and does not transfer #925 proved available is actually unlocked.
• Covers the observed ~0.85 GB per-step cold working set + a realistic deferred fraction with headroom.
• Bounded to avoid over-committing pinned host RAM and to avoid extrapolating past tested hardware (Rule 11 / portability). Operators can still override per-run via ONNX_GENAI_ZERO_COPY_HYBRID_BUDGET_BYTES.

Blast radius

Low. The whole feature is opt-in (defaults OFF behind ONNX_GENAI_ZERO_COPY_HYBRID=1; zero_copy_hybrid_env_defaults_off_and_opts_in kept intact), so this default only affects users who already turned the hybrid on. Windows behaviour is unchanged.

Implementation

• const ZERO_COPY_SAFE_BUDGET_BYTES = if cfg!(target_os = "windows") { WDDM } else { NON_WINDOWS } — both named consts (ZERO_COPY_SAFE_BUDGET_BYTES_WDDM, ZERO_COPY_SAFE_BUDGET_BYTES_NON_WINDOWS) are referenced in every build, so each stays greppable and unit-testable (no dead_code).
• Doc comments cite both Managed offload is 30x SLOWER than WDDM demand paging on Windows — and it is the default: the copy-to-VRAM buys nothing for a single-touch access pattern #864 (the WDDM ceiling → Windows value) and Re-measure the #864 zero-copy hybrid on Linux: the #912 negative is WDDM-scoped and does not transfer #925 (the Linux measurement → non-Windows value, with the 6.795 GB safe-to figure, n=3 byte-identical, fallbacks=0).

Tests / checks (local, green)

• New zero_copy_safe_budget_defaults_are_platform_aware: asserts WDDM=256 MiB (< the measured WDDM-safe 436 MB), non-Windows=2 GiB (< Re-measure the #864 zero-copy hybrid on Linux: the #912 negative is WDDM-scoped and does not transfer #925 6.795 GB and > WDDM band), and that the platform-selected const resolves correctly per cfg.
• python3 .github/scripts/verify_documented_env_vars.py — env var honesty: 70 documented, all accounted for
• cargo fmt --all -- --check — clean
• cargo clippy --locked -p onnx-runtime-ep-cuda --features cuda -- -D warnings — clean
• cargo test -p onnx-runtime-ep-cuda --features cuda weight_paging — pass (incl. the intact opt-in-defaults-OFF test)

Decision drop

.squad/decisions/inbox/holden-zerocopy-linux-default.md (Scribe to merge).

References #925 and #864.

⚠️ Please do not self-merge — a squad member should review.

---

Run by the Squad team on the shared H200 box.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.76%. Comparing base (e331700) to head (4658162).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #936   +/-   ##
=======================================
  Coverage   77.76%   77.76%           
=======================================
  Files         352      352           
  Lines      145983   145983           
  Branches   145983   145983           
=======================================
  Hits       113530   113530           
  Misses      27970    27970           
  Partials     4483     4483           

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	83.01% <ø> (+0.09%)	⬆️
offline	77.55% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	gather/large_f32_threads=1-internal/131072	26.78 µs	38.18 µs	+42.6%
⚠️	matmul/small_generic_bf16_threads=1/1x256x256	29.00 µs	36.42 µs	+25.6%
⚠️	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	75.47 µs	93.96 µs	+24.5%
⚠️	matmul/small_generic_f16_threads=1/1x256x256	29.35 µs	36.27 µs	+23.6%
⚠️	tokenization/encode_tokens_per_second	363.26 µs	438.00 µs	+20.6%
⚠️	matmul/small_generic_f16_threads=8/1x256x256	28.42 µs	34.04 µs	+19.8%
⚠️	matmul/medium_generic_f32_threads=1/32x512x512	2.14 ms	2.57 ms	+19.7%
⚠️	matmul/small_generic_bf16_threads=8/1x256x256	29.31 µs	34.66 µs	+18.3%
⚠️	matmul/medium_generic_f16_threads=1/32x512x512	28.40 µs	33.53 µs	+18.1%
⚠️	matmul/small_generic_f32_threads=8/1x256x256	31.86 µs	37.46 µs	+17.6%
✅	gather/small_bf16_threads=1-internal/4096	458.3 ns	524.6 ns	+14.5%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	502.31 µs	558.99 µs	+11.3%
✅	gather/medium_f32_threads=1-internal/32768	3.75 µs	4.16 µs	+10.9%
✅	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	531.23 µs	584.44 µs	+10.0%
✅	gather/medium_f16_threads=1-internal/32768	2.31 µs	2.50 µs	+8.4%
✅	gather/medium_bf16_threads=1-internal/32768	2.28 µs	2.44 µs	+7.0%
✅	gather/small_f16_threads=1-internal/4096	502.0 ns	536.7 ns	+6.9%
✅	grammar_masking/llguidance_compute_mask/32	71.00 µs	75.34 µs	+6.1%
✅	matmul/medium_generic_f32_threads=8/32x512x512	964.10 µs	1.02 ms	+5.7%
✅	reduce_mean/large_f32_threads=1-internal/262144	979.16 µs	1.02 ms	+4.1%
✅	tokenization/decode_tokens_per_second	6.44 ms	6.67 ms	+3.5%
✅	logit_processing/seven_processor_chain_per_step	303.69 µs	311.34 µs	+2.5%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	76.68 µs	77.54 µs	+1.1%
✅	matmul/small_generic_f32_threads=1/1x256x256	42.38 µs	42.50 µs	+0.3%
✅	sampling_latency/min_p_per_token	210.21 µs	208.94 µs	-0.6%
✅	matmul/medium_generic_f16_threads=8/32x512x512	29.51 µs	29.28 µs	-0.8%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.21 ms	2.19 ms	-1.0%
✅	add/medium_bf16_threads=1-internal/262144	2.94 ms	2.89 ms	-1.5%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.32 ms	1.30 ms	-1.9%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	417.08 µs	408.11 µs	-2.2%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	398.90 µs	387.26 µs	-2.9%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.54 ms	6.25 ms	-4.4%
✅	sampling_latency/top_p_per_token	396.10 µs	377.48 µs	-4.7%
✅	add/medium_f32_threads=1-internal/262144	2.85 ms	2.71 ms	-5.0%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.97 ms	1.87 ms	-5.0%
✅	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	53.74 µs	50.76 µs	-5.5%
✅	kv_cache/alloc_dealloc_pages	39.02 µs	36.81 µs	-5.7%
✅	qwen3_sampling_processors/top_k_top_p_fast	706.44 µs	665.56 µs	-5.8%
✅	sampling_latency/top_k_per_token	54.80 µs	51.57 µs	-5.9%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.99 ms	9.30 ms	-6.9%
✅	gather/small_f32_threads=1-internal/4096	743.3 ns	685.7 ns	-7.8%
✅	sampling_latency/greedy_per_token	3.54 µs	3.26 µs	-7.8%
✅	gather/large_f16_threads=1-internal/131072	14.81 µs	13.64 µs	-7.9%
✅	reduce_mean/medium_f32_threads=1-internal/65536	256.84 µs	232.71 µs	-9.4%
✅	qwen3_sampling_processors/top_k_partial_selection	153.95 µs	139.00 µs	-9.7%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	87.61 µs	79.08 µs	-9.7%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	4.04 ms	3.60 ms	-10.9%
✅	add/small_f16_threads=1-internal/1024	14.36 µs	12.76 µs	-11.1%
✅	add/small_bf16_threads=1-internal/1024	14.72 µs	13.04 µs	-11.4%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.93 ms	3.47 ms	-11.7%
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	56.79 µs	50.05 µs	-11.9%
✅	reduce_mean/small_f32_threads=1-internal/4096	16.00 µs	13.99 µs	-12.5%
✅	add/large_bf16_threads=1-internal/4194304	46.64 ms	40.69 ms	-12.7%
✅	add/large_f32_threads=1-internal/4194304	46.71 ms	40.47 ms	-13.3%
✅	add/large_f16_threads=1-internal/4194304	47.74 ms	41.19 ms	-13.7%
🟢	qwen3_sampling_processors/top_p_fast_after_top_k	601.80 µs	505.21 µs	-16.1%
🟢	add/medium_f16_threads=1-internal/262144	3.01 ms	2.49 ms	-17.3%
🟢	gather/large_bf16_threads=1-internal/131072	17.46 µs	13.11 µs	-24.9%
🟢	add/small_f32_threads=1-internal/1024	264.4 ns	195.0 ns	-26.2%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.00 3.43 5.39 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)