fix(weight-offload): make async page-in opt-in (default sync)

[justinchuby]

What

Flip ONNX_GENAI_WEIGHT_OFFLOAD_ASYNC_PAGEIN from default-ON to opt-IN. Unset/falsey now uses the synchronous device page-in (new default); a truthy value (1/true/yes/on) opts into the asynchronous fence-ordered page-in. Only the default changes — the async path is fully preserved behind the flag.

Why — measured A/B (#544 follow-up)

Async page-in net-regresses in the eviction/thrash regime. qwen3-0.6b-int4, native CUDA, weight-offload engaged, 96 MiB device budget (every admit evicts):

Config	tok/s
async page-in ON	12.16
async page-in OFF (sync)	15.84

Sync is ~1.30x faster. Per-page-in tax breakdown (96 MiB, async): materialize 791 ms + pinned-staging alloc/copy 792 ms co-dominate (~48% each); raw H2D 46 ms (~3%); eviction drain 15 ms; fence wait 7 ms. The transfer async tries to overlap is ~3% of the cost, and when every admit evicts the eviction compute-stream drain re-serializes — so async cannot hide anything and only adds a non-overlappable pinned-staging alloc. Async becomes a net win only once a warm-host materialize cache lands; it stays available via =1.

Correctness (regression-sensitive)

• Byte-exact preserved (weight_paging section 9): offloaded == resident token stream unchanged. Verified on weight_offload_native_cuda_e2e with the NEW sync default — tokens byte-identical to resident baseline, page_ins=12544, evictions=12541 (non-vacuous).
• WAR / eviction-drain safety and fence-ordering primitives untouched. The async fence anti-regression GPU test (async_pagein_fence_orders_weight_page_in_consumer) still passes and still guards the async path.
• No capture / GAP-3 interaction — dynamic page-in is outside any captured region.

Tests

• Unit async_pagein_env_is_opt_in: None -> false, truthy spellings -> true, falsey/garbage -> false (non-vacuous both directions).
• device_policy_defaults_to_disabled extended to assert the default policy is sync.
• e2e asserts the resolved from_env() policy is sync by default AND offloaded == resident on real int4 GPU.
• cargo fmt --all --check clean.

Blast radius

Flag default + tests + docs only. Pager internals, capture (#571), and GAP-3 untouched. Escape hatch: ONNX_GENAI_WEIGHT_OFFLOAD_ASYNC_PAGEIN=1 restores async.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.74%. Comparing base (22a90a7) to head (ae116fd).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #572   +/-   ##
=======================================
  Coverage   80.74%   80.74%           
=======================================
  Files         315      315           
  Lines      123688   123688           
  Branches   123688   123688           
=======================================
  Hits        99876    99876           
  Misses      19745    19745           
  Partials     4067     4067           

Flag	Coverage Δ
cli-ort-linux	86.69% <ø> (ø)
cli-ort-windows	83.84% <ø> (ø)
mlas	77.91% <ø> (ø)
offline	80.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[justinchuby]

VERDICT: APPROVE

Independent review by Harry (detached at PR head ae116fd, GPU CUDA_VISIBLE_DEVICES=6). This is a low-risk, data-backed DEFAULT flip: async fence-ordered device page-in becomes opt-IN; sync is the new default. I independently confirmed it changes only a default, preserves byte-identical offload output, and keeps the async path fully guarded.

1. Diff scope — clean (4 files, +137/-17)

Behavioral change is limited to the async-page-in default:

• weight_paging.rs: new pure helper async_pagein_from_env_value (unset/falsey -> sync, truthy 1/true/yes/on -> async), Default impl async_pagein true->false, CudaWeightResidency::new async_pagein true->false, from_env now routes through the helper. Doc comments updated.
• e2e test + unit test + WEIGHT_OFFLOAD.md + decision inbox note.

No edits to the async transfer machinery: upload/upload_async, admit/eviction compute-stream drain, fence-ordering (compute_wait_fence/record_copy_fence), WAR safety, capture, or GAP-3 are all untouched. provider.rs with_async_pagein wiring is unchanged (transfer path intact, just no longer default). No out-of-scope edits found.

2. Flag helper correct + NON-VACUOUS (mutation-proven)

• weight_paging::tests::async_pagein_env_is_opt_in ... ok (None->false; 1/true/YES/" On "->true; 0/false/""/maybe->false).
• device_policy_defaults_to_disabled ... ok (now asserts !async_pagein).
• Mutation check: I flipped the helper None branch to true. Result: async_pagein_env_is_opt_in FAILED (assertion failed: !async_pagein_from_env_value(None) at weight_paging.rs:700). The default is genuinely asserted, not vacuous. Mutation reverted; worktree clean.

3. Byte-identical offload parity under the NEW sync default (key regression guard) — PASS

Real-GPU e2e offloaded_native_cuda_decode_is_token_identical_and_pages (qwen3-0.6b-int4, native CUDA, 2 MiB device budget):

• The test first asserts DeviceOffloadPolicy::from_env().async_pagein == false with only WEIGHT_OFFLOAD=1 set (locks the new sync default) — passed.
• Parity is FULL-SEQUENCE (assert_eq on the entire 32-token vector, not argmax-only): offloaded token_ids == resident baseline, byte-identical.
• Counters NON-ZERO / non-vacuous: page_ins=12544, evictions=12541, hits=0 (baseline page_ins=0). These match the decision-note figures exactly.
• Observed throughput this run: baseline 1.98 tok/s vs offloaded 2.22 tok/s (unloaded box; absolute numbers differ from the A/B rig but parity + paging are what this test locks).

4. Async path still guarded — PASS

provider::tests::async_pagein_fence_orders_weight_page_in_consumer ... ok, run with ONNX_GENAI_WEIGHT_OFFLOAD_ASYNC_PAGEIN=1. The fence-ordering anti-regression test drives the async primitives directly and remains correct; the async path is preserved and tested, just opt-in.

5. fmt — clean

cargo fmt --all --check: exit 0.

Summary of re-run results

• fmt --all --check: clean
• weight_paging::tests (3): ok
• default mutation: FAILED as required (non-vacuous), reverted
• GPU e2e offload parity: ok, full-sequence byte-identical, page_ins=12544 / evictions=12541
• async fence guard (flag=1): ok

Byte-identical parity is proven and the async guard still runs. Approving.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	matmul/small_generic_f32_threads=8/1x256x256	37.51 µs	55.91 µs	+49.0%
⚠️	gather/large_f32_threads=1-internal/131072	27.55 µs	32.21 µs	+16.9%
✅	gather/large_bf16_threads=1-internal/131072	10.48 µs	12.03 µs	+14.8%
✅	gather/large_f16_threads=1-internal/131072	10.83 µs	12.23 µs	+12.9%
✅	matmul/small_generic_f32_threads=1/1x256x256	36.92 µs	40.30 µs	+9.2%
✅	add/large_f32_threads=1-internal/4194304	42.24 ms	45.55 ms	+7.8%
✅	add/small_bf16_threads=1-internal/1024	14.02 µs	14.86 µs	+6.0%
✅	reduce_mean/small_f32_threads=1-internal/4096	14.97 µs	15.30 µs	+2.2%
✅	gather/medium_f32_threads=1-internal/32768	3.92 µs	4.00 µs	+2.0%
✅	gather/small_bf16_threads=1-internal/4096	477.7 ns	484.4 ns	+1.4%
✅	gather/medium_bf16_threads=1-internal/32768	2.42 µs	2.41 µs	-0.3%
✅	gather/medium_f16_threads=1-internal/32768	2.42 µs	2.40 µs	-0.8%
✅	add/medium_f32_threads=1-internal/262144	2.72 ms	2.69 ms	-1.2%
✅	add/large_bf16_threads=1-internal/4194304	42.81 ms	42.11 ms	-1.6%
✅	add/small_f16_threads=1-internal/1024	14.28 µs	13.86 µs	-2.9%
✅	add/medium_bf16_threads=1-internal/262144	3.00 ms	2.89 ms	-3.8%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.85 ms	9.30 ms	-5.6%
✅	add/large_f16_threads=1-internal/4194304	44.67 ms	41.50 ms	-7.1%
✅	gather/small_f16_threads=1-internal/4096	512.8 ns	467.4 ns	-8.9%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.08 ms	978.69 µs	-9.5%
✅	reduce_mean/medium_f32_threads=1-internal/65536	270.96 µs	244.26 µs	-9.9%
✅	gather/small_f32_threads=1-internal/4096	747.7 ns	665.4 ns	-11.0%
✅	tokenization/encode_tokens_per_second	429.47 µs	379.36 µs	-11.7%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	611.76 µs	540.10 µs	-11.7%
✅	sampling_latency/min_p_per_token	387.13 µs	341.70 µs	-11.7%
✅	kv_cache/alloc_dealloc_pages	41.59 µs	36.64 µs	-11.9%
✅	matmul/small_generic_f16_threads=8/1x256x256	36.98 µs	32.46 µs	-12.2%
✅	logit_processing/seven_processor_chain_per_step	1.34 ms	1.18 ms	-12.2%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	93.67 µs	81.08 µs	-13.4%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.26 ms	1.94 ms	-13.8%
🟢	tokenization/decode_tokens_per_second	7.40 ms	6.17 ms	-16.6%
🟢	grammar_masking/llguidance_compute_mask/32	90.81 µs	74.93 µs	-17.5%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	4.66 ms	3.82 ms	-18.1%
🟢	matmul/large_generic_f16_threads=8/32x1024x1024	103.52 µs	84.63 µs	-18.2%
🟢	sampling_latency/greedy_per_token	3.99 µs	3.22 µs	-19.2%
🟢	matmul/medium_generic_f16_threads=1/32x512x512	38.48 µs	30.85 µs	-19.8%
🟢	matmul/small_generic_f16_threads=1/1x256x256	41.55 µs	32.22 µs	-22.5%
🟢	add/small_f32_threads=1-internal/1024	261.9 ns	201.1 ns	-23.2%
🟢	matmul/large_generic_bf16_threads=8/32x1024x1024	1.84 ms	1.40 ms	-24.0%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	42.31 µs	31.82 µs	-24.8%
🟢	sampling_latency/top_p_per_token	1.34 ms	978.64 µs	-27.1%
🟢	add/medium_f16_threads=1-internal/262144	3.72 ms	2.68 ms	-27.8%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	43.08 µs	29.33 µs	-31.9%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	540.13 µs	367.50 µs	-32.0%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	46.31 µs	31.27 µs	-32.5%
🟢	matmul/medium_generic_f32_threads=1/32x512x512	3.48 ms	2.32 ms	-33.5%
🟢	sampling_latency/top_k_per_token	725.93 µs	467.42 µs	-35.6%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	2.03 ms	1.01 ms	-50.1%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 2.76 3.89 6.89 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)