feat(cuda-offload): async fence-ordered weight page-in overlap (#87 first increment)

[justinchuby]

#87 first increment: async, fence-ordered residency page-in

Switches the live CUDA weight-offload residency page-in from the synchronous
cuMemcpyHtoD to an asynchronous, fence-ordered copy-stream transfer, so a
device page-in can overlap the current in-flight kernel and the compute stream
waits only on the transfer's completion event — not a full host sync.

This is the low-risk first increment only (NOT the double-buffer look-ahead,
which is a later increment). All existing correctness/safety guarantees are
preserved and offload-off is byte-identical to today.

What changed

• CudaWeightPage::upload_async — allocate the VRAM page, stage the
  canonical bytes into an owned page-locked (pinned) host buffer, enqueue
  cuMemcpyHtoDAsync on the dedicated copy stream, and return a copy fence. The
  page owns the pinned staging so the in-flight transfer never reads freed host
  memory (freed on Drop, long after the fence is awaited).
• CudaWeightResidency::resident_materialized (async arm) awaits the copy
  fence on the compute stream via compute_wait_fence, ordering the consuming
  kernel after the transfer.
• admit drains the compute stream (synchronize) only when a page-in
  must evict, preserving the original WAR/reuse guarantee (eviction never frees
  a page a prior kernel still reads) while letting a budget-fitting page-in
  overlap the in-flight compute. The concurrent-populate drop branch drains the
  copy stream before freeing the doomed page.
• Gated by the existing ONNX_GENAI_WEIGHT_OFFLOAD policy. New sub-knob
  ONNX_GENAI_WEIGHT_OFFLOAD_ASYNC_PAGEIN (default on; =0 restores the
  synchronous page-in) is the A/B "before" arm and a kill-switch.

Files: weight_paging.rs, provider.rs, lib.rs.

Correctness (the acceptance bar)

• Fence-ordering anti-regression lock (GPU):
  provider::tests::async_pagein_fence_orders_weight_page_in_consumer — under an
  artificially delayed transfer stream, the real upload_async +
  compute_wait_fence path reads the fully paged-in bytes, while the same
  primitive chain WITHOUT the compute wait deterministically reads pre-transfer
  POISON. Mirrors copy_async_fence_orders_h2d_prefetch_through_ep_api.
• Token parity (real model): decode token IDs are BYTE-IDENTICAL for
  offload-off vs offload-on-sync vs offload-on-async on Qwen3-0.6B int4 native
  CUDA at every budget tested (3-way distinct-sequence count = 1).
• onnx-runtime-ep-cuda weight-offload lib + weight_offload_gpu integration
  tests pass; cargo fmt --all --check clean.

A/B numbers (honest)

Qwen3-0.6B int4 (2053 MatMulNBits nodes), native CUDA on H200 GPU 0, greedy,
steady decode, tokens 64 / decode-skip 8, taskset -c 0-3, median of 5–6 runs
after 2 warmups:

Config	Budget	decode ms/token	vs off
offload OFF (resident)	—	3.786	1.00x
ON, sync page-in	2 MiB	5.998	1.58x
ON, async page-in	2 MiB	5.941	1.57x
ON, sync page-in	256 MiB	5.926	1.56x
ON, async page-in	256 MiB	5.940	1.57x
sync vs async	448–480 MiB	5.93–5.99 (tie)	~1.56x

At the extreme 2 MiB budget async is reproducibly ~1% faster than sync (every
async run < every sync run). At realistic budgets it is a statistical tie.

Honest finding + recommendation

The ~1.56x decode tax is stubbornly independent of budget from 2 MiB up to the
model's full ~500 MB. This is a dense sequential-layer sweep, so once the budget
is exceeded every page-in must evict, and eviction drains the compute stream
(WAR safety) — re-serializing the transfer we wanted to hide. The per-page host
cost (materialize + pinned copy) also dwarfs the raw H2D, so overlapping only
the H2D buys little.

Increment 1 lands as a correctness-preserving, no-regression foundation (async
primitives on the live path, fence-ordered, byte-identical, poison-locked). To
actually hide the H2D tax, proceed to the double-buffered look-ahead
(increment 2): prefetch the NEXT layer's weight into a distinct slot while the
current layer computes — overlapping compute WITHOUT an eviction sync on the
critical path — and replace per-page pinned staging with a bounded staging ring.
Do not expect a decode speedup from increment 1 alone on dense sequential models.

Closes #87 (first increment).

Working as Cohaagen (CUDA/systems performance engineer).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.59%. Comparing base (cdc5af9) to head (bf34590).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #544   +/-   ##
=======================================
  Coverage   80.59%   80.59%           
=======================================
  Files         315      315           
  Lines      123446   123446           
  Branches   123446   123446           
=======================================
+ Hits        99487    99489    +2     
+ Misses      19908    19907    -1     
+ Partials     4051     4050    -1     

Flag	Coverage Δ
cli-ort-linux	83.27% <ø> (ø)
cli-ort-windows	82.67% <ø> (-0.11%)	⬇️
mlas	77.91% <ø> (ø)
offline	80.51% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

⚠️ Benchmark Change Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
⚠️	matmul/large_generic_f16_threads=1/32x1024x1024	79.80 µs	102.43 µs	+28.4%
⚠️	add/medium_f32_threads=1-internal/262144	2.73 ms	3.47 ms	+26.8%
✅	add/medium_f16_threads=1-internal/262144	3.05 ms	3.50 ms	+15.0%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.33 ms	10.54 ms	+13.0%
✅	sampling_latency/greedy_per_token	3.17 µs	3.38 µs	+6.6%
✅	matmul/small_generic_bf16_threads=1/1x256x256	32.65 µs	34.02 µs	+4.2%
✅	sampling_latency/top_k_per_token	460.84 µs	479.92 µs	+4.1%
✅	gather/small_f16_threads=1-internal/4096	529.9 ns	544.9 ns	+2.8%
✅	matmul/small_generic_f16_threads=8/1x256x256	31.21 µs	32.00 µs	+2.5%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.91 ms	1.94 ms	+1.4%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	517.89 µs	523.45 µs	+1.1%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	85.42 µs	86.14 µs	+0.8%
✅	sampling_latency/top_p_per_token	968.33 µs	973.49 µs	+0.5%
✅	tokenization/encode_tokens_per_second	370.09 µs	370.77 µs	+0.2%
✅	gather/small_bf16_threads=1-internal/4096	555.5 ns	556.0 ns	+0.1%
✅	matmul/small_generic_f32_threads=1/1x256x256	37.55 µs	37.08 µs	-1.2%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.69 ms	3.65 ms	-1.3%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	364.04 µs	358.36 µs	-1.6%
✅	matmul/medium_generic_f16_threads=1/32x512x512	30.83 µs	30.34 µs	-1.6%
✅	tokenization/decode_tokens_per_second	6.12 ms	6.00 ms	-1.8%
✅	sampling_latency/min_p_per_token	342.07 µs	332.50 µs	-2.8%
✅	kv_cache/alloc_dealloc_pages	43.48 µs	41.98 µs	-3.5%
✅	gather/small_f32_threads=1-internal/4096	834.0 ns	793.1 ns	-4.9%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.40 ms	2.28 ms	-4.9%
✅	add/small_bf16_threads=1-internal/1024	17.18 µs	16.07 µs	-6.5%
✅	add/large_f16_threads=1-internal/4194304	49.91 ms	46.42 ms	-7.0%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.44 ms	1.32 ms	-8.5%
✅	gather/medium_f16_threads=1-internal/32768	2.68 µs	2.44 µs	-8.9%
✅	gather/medium_bf16_threads=1-internal/32768	2.59 µs	2.35 µs	-9.1%
✅	gather/medium_f32_threads=1-internal/32768	4.30 µs	3.86 µs	-10.3%
✅	matmul/medium_generic_f16_threads=8/32x512x512	32.69 µs	29.16 µs	-10.8%
✅	add/small_f16_threads=1-internal/1024	17.29 µs	15.33 µs	-11.3%
✅	matmul/small_generic_f16_threads=1/1x256x256	36.44 µs	32.10 µs	-11.9%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.23 ms	1.08 ms	-12.0%
✅	gather/large_f16_threads=1-internal/131072	11.62 µs	10.16 µs	-12.5%
✅	reduce_mean/medium_f32_threads=1-internal/65536	331.76 µs	288.09 µs	-13.2%
🟢	add/small_f32_threads=1-internal/1024	236.4 ns	199.6 ns	-15.6%
🟢	logit_processing/seven_processor_chain_per_step	1.36 ms	1.15 ms	-15.7%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.38 ms	1.14 ms	-16.8%
🟢	add/medium_bf16_threads=1-internal/262144	3.45 ms	2.86 ms	-17.2%
🟢	add/large_f32_threads=1-internal/4194304	54.41 ms	44.88 ms	-17.5%
🟢	gather/large_bf16_threads=1-internal/131072	11.94 µs	9.61 µs	-19.5%
🟢	gather/large_f32_threads=1-internal/131072	34.30 µs	26.53 µs	-22.6%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	42.25 µs	32.66 µs	-22.7%
🟢	add/large_bf16_threads=1-internal/4194304	57.69 ms	44.16 ms	-23.5%
🟢	grammar_masking/llguidance_compute_mask/32	103.94 µs	72.51 µs	-30.2%
🟢	reduce_mean/small_f32_threads=1-internal/4096	22.25 µs	14.90 µs	-33.0%
🟢	matmul/small_generic_f32_threads=8/1x256x256	96.44 µs	35.48 µs	-63.2%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.4.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.43 4.02 6.70 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

VERDICT: REQUEST-CHANGES

Independent review by Melina (reviewer authority). Author Cohaagen is locked out of revising this verdict — a different agent must revise (see below). Reviewed at origin/squad/87-async-pagein @ ed12f3d in a clean, independent worktree on H200 GPU 0.

TL;DR

The production code is correct and safe — I could not break it: 3-way byte-identical token parity on the real model, the fence primitive robustly orders the consumer, staging-lifetime and eviction/WAR logic are sound. But the headline anti-regression test provider::tests::async_pagein_fence_orders_weight_page_in_consumer is FLAKY: it fails reliably under the normal full-suite cargo test invocation (capture on, parallel) and only passes in isolation / with --nocapture. A weight-offload correctness lock that reds CI on the standard invocation — and whose "pass" leans partly on a synchronizing cuMemAlloc draining the spin rather than on the fence itself — is not a valid lock. That is the sole blocker.

What I verified empirically (all on CUDA_VISIBLE_DEVICES=0, H200)

1. Fence ordering is real AND non-vacuous — PASS (code is correct)

• Author's test in isolation: test result: ok (10/10 single-threaded runs, GPU idle).
• Perturbation (non-vacuity): replaced runtime.compute_wait_fence(fence) with a no-op in the positive arm → the positive assert fails deterministically reading pre-transfer zeros: left: [0.0, 0.0, ...] vs payload — "async page-in consumer read stale bytes". So the wait is load-bearing, not incidental.
• Confounder-free fence probe (mine): pre-allocated + pre-zeroed dst (no cuMemAlloc between spin and consume), real 8M-cycle copy-stream delay, htod_async → record_copy_fence → compute_wait_fence → consume, ×120 iterations: 0/120 races. compute_wait_fence genuinely orders the consumer after the delayed transfer.

2. Pinned staging lifetime — PASS

CudaWeightPage owns Option<PinnedStaging>; the async source is that owned pinned buffer, freed only on the page's Drop — after the copy fence is awaited (normal path) or after synchronize()/drain_copy_stream() (evict / doomed-duplicate paths). No use-after-free window. Confirmed by 12,544 real page-ins below with zero corruption.

3. Eviction WAR safety — PASS

admit drains the compute stream (synchronize) only when a page-in must evict, and the concurrent-populate/duplicate branches call drain_copy_stream() before dropping the doomed page. Each page gets a fresh cuMemAlloc (allocated before eviction frees anything), so a reused address is only ever handed out after a synchronizing free — never aliasing a live reader or in-flight copy. Empirically validated: 12,541 evictions, still byte-identical.

4. Byte-identical off-path + 3-way parity — PASS

weight_offload_native_cuda_e2e (Qwen3-0.6B int4 postfix export, 2 MiB tiny budget, taskset -c 0-3):

• offload OFF (resident) vs ON async (default): tokens byte-identical — page_ins=12544, evictions=12541.
• offload OFF vs ON sync (ONNX_GENAI_WEIGHT_OFFLOAD_ASYNC_PAGEIN=0): tokens byte-identical — same [12095, 11, 323, 279, 6722, 315, 15344, 374, 21718, 13, ...].
• Therefore off ≡ sync ≡ async, 3-way distinct-sequence count = 1. Confirmed.

5. No regression — PASS (except the flaky lock)

• weight_offload_gpu integration: 5 passed.
• ep-cuda --lib: 269 passed + the 1 flaky fence test (below).

6. fmt / clippy — PASS

• cargo fmt --all --check: clean.
• cargo clippy -p onnx-runtime-ep-cuda --lib: no warnings in the touched files (weight_paging.rs, provider.rs, lib.rs). Remaining clippy warnings are pre-existing in unrelated *_gpu test files.

The blocker: the anti-regression lock is a timing heisenbug

Same committed source, GPU 0 idle:

• cargo test -p onnx-runtime-ep-cuda --lib (default: output captured, parallel): FAILED, reproduced 2/2 (and earlier 12/12 in a busier window). Positive arm reads pre-transfer zeros.
• Test binary run with --nocapture (parallel): 270/270 pass.
• Test in isolation (--test-threads=1, or alone): 10/10 pass.

Pass/fail flips purely on output-capture mode + parallelism — i.e., timing, not logic. Root cause: the positive arm calls CudaWeightPage::upload_async, whose first act is a synchronizing cuMemAlloc (alloc_raw → malloc_sync). Depending on scheduling, that alloc drains the very spin-delay the test relies on to expose a race, so the positive arm's protection is sometimes the alloc, not the fence — and under captured+parallel timing the consumer intermittently reads the not-yet-copied page. My probe (which pre-allocates dst) never races, proving the fence is fine; the test harness is the problem. As written, this lock (a) reds the standard build and (b) cannot be trusted to catch a future fence regression.

Required changes (blocking)

1. Make async_pagein_fence_orders_weight_page_in_consumer deterministic and robust under the standard captured, parallel cargo test invocation. Restructure the positive arm so the spin-delay genuinely gates the transfer through the fence and not through a synchronizing allocation — e.g., pre-allocate/pre-zero the destination page once outside the loop (as my confounder-free probe does), or otherwise ensure no device-synchronizing call sits between the copy-stream spin launch and the fenced consume. It must pass cargo test -p onnx-runtime-ep-cuda (no --nocapture, default parallelism) reliably (e.g. green across 20+ back-to-back full-suite runs) while remaining non-vacuous (removing compute_wait_fence must still fail).
2. Re-run cargo test -p onnx-runtime-ep-cuda (full, default invocation) and paste a green result in the PR.

No production-code changes are required for correctness — upload_async, the fence wait, staging ownership, and the evict-only drain are all correct as verified above. This is strictly a test-determinism fix.

Reviser

Assign Deckard (Systems Dev, 🚀 CUDA & Perf pod) to revise — the fix is a CUDA-EP streams/fence test-determinism change. (Explicitly NOT Cohaagen, who is locked out.)

[justinchuby]

Test-only fix for the flaky async page-in ordering lock (Harry)

Production page-in code is unchanged. Only the test
async_pagein_fence_orders_weight_page_in_consumer was revised, per the review.

Root cause (verified independently). The flake reproduced reliably under the
default parallel + captured cargo test (fails ~1 in 2-3 full runs), and passed
in isolation / with --nocapture. Instrumenting the two arms showed the failing
assertion is the negative (poison-control) arm — "did NOT read poison" — not
the positive arm. Its timing window contains no cuMemAlloc, so the flake is
not an alloc drain: it is a pure wall-clock race. An unordered compute-stream
consumer had to read pre-transfer poison before a spin-delayed transfer
completed; under parallel GPU contention the consumer kernel's scheduling
latency intermittently exceeded the copy-stream spin delay, so it read the
already-paged-in payload and the vacuity guard fired. Raising the spin 25x drove
failures to 0/15, confirming a timing race rather than a synchronizing alloc.

Fix (test only).

• Negative arm is now deterministic: the transfer is event-ordered strictly
  AFTER the consumer via record_compute_fence + copy_wait_fence, so the consumer
  provably reads pre-transfer poison with zero wall-clock racing.
• Positive arm holds the H2D copy pending behind the spin and orders the
  consumer after it with compute_wait_fence; deleting the wait deterministically
  degrades to a poison read — non-vacuity preserved.
• All device/pinned allocations are hoisted out of the per-iteration timing
  window, so no synchronizing cuMemAlloc/cuMemHostAlloc can drain the spin.
• A trailing real upload_async byte-parity check keeps the production entry
  point under regression cover.

Proof — full suite the way CI runs it, back to back. The target test is green
in every run; the 270-test lib binary (which contains it) is 270/270 each time.

FULL RUN 1: async_pagein ok | lib: 270 passed; 0 failed | other failures: none
FULL RUN 2: async_pagein ok | lib: 270 passed; 0 failed | other: reshape_exact_signature_captures_async_copy
FULL RUN 3: async_pagein ok | lib: 270 passed; 0 failed | other: reshape_exact_signature_captures_async_copy
FULL RUN 4: async_pagein ok | lib: 270 passed; 0 failed | other: matmul_nbits_gpu_int4_fp16_explicit_zero_points_match_cpu_reference
FULL RUN 5: async_pagein ok | lib: 270 passed; 0 failed | other failures: none

cargo test -p onnx-runtime-ep-cuda --lib (the binary that reproduced the flake),
5x back to back:

LIB RUN 1: ok. 270 passed; 0 failed
LIB RUN 2: ok. 270 passed; 0 failed
LIB RUN 3: ok. 270 passed; 0 failed
LIB RUN 4: ok. 270 passed; 0 failed
LIB RUN 5: ok. 270 passed; 0 failed

Parallel lib binary alone, 25x: 0/25 failed (was ~1-in-2-to-3 before).

Note on the two unrelated failures above. reshape_exact_signature_captures_async_copy
(tests/construction_gpu.rs) and the matmul_nbits parity test are pre-existing,
independent flakes in separate integration binaries — they use CUDA graph
capture / bit-parity and are not part of this change. cargo runs test binaries
sequentially, so they never run alongside my test. Evidence:

• construction_gpu binary run alone (its own tests, parallel): 2/8 failed.
• reshape test run in isolation: 0/10 failed.
  My edit is #[cfg(test)]-only in the lib, which integration-test binaries do not
  link, so it cannot affect them. These are out of scope for this PR.

Non-vacuity re-proof. Temporarily deleting compute_wait_fence from the
positive arm makes the test fail deterministically (5/5 under the parallel lib
binary, and even single-threaded):

thread '...async_pagein_fence_orders_weight_page_in_consumer' panicked at provider.rs:923:
assertion `left == right` failed: async page-in consumer read stale bytes —
compute_wait_fence did not order the transfer before the compute-stream read
  left:  [-777.0, -777.0, ...]   (poison)
  right: [5.0, 6.0, ...]         (payload)

Reverted after confirming.

fmt / clippy. cargo fmt --all --check clean; cargo clippy -p onnx-runtime-ep-cuda
--lib reports 0 warnings (none in the touched file).

[justinchuby]

VERDICT: APPROVE

Re-review of PR #544 @ HEAD bf345904 (revision by Harry on top of my previously-reviewed ed12f3d9). Verified on H200 device 0, parallel captured cargo test -p onnx-runtime-ep-cuda — the exact condition that flaked before.

1. Scope: TEST-ONLY, production untouched (confirmed)

git diff ed12f3d9..bf345904 touches ONE file: crates/onnx-runtime-ep-cuda/src/provider.rs (+97/-48), entirely inside #[cfg(test)] mod tests (module starts line 701; changed fn async_pagein_fence_orders_weight_page_in_consumer at 844; file ends 999). Diffs for weight_paging.rs, runtime.rs, lib.rs are EMPTY. Already-approved production code is unchanged — no drift.

2. Determinism: target test GREEN 5/5

Full suite run 5x. provider::tests::async_pagein_fence_orders_weight_page_in_consumer = ... ok in all 5 runs (it lives in the 270-test lib unit binary, which reported 270 passed; 0 failed every run). Zero flakes of the target test.

3. Non-vacuity: FAILS deterministically 3/3 when the fence is removed

Commented out the positive-arm runtime.compute_wait_fence(fence), rebuilt, ran the target test 3x: FAILED 3/3, each with
assertion left == right failed: async page-in consumer read stale bytes — compute_wait_fence did not order the transfer before the compute-stream read (provider.rs:923). The consumer reads pre-copy POISON without the wait. The lock genuinely guards the fence ordering. File restored to bf345904 afterward.

4. Corrected root cause — Harry is right; my original diagnosis was wrong

My original REQUEST-CHANGES blamed a device-synchronizing cuMemAlloc (via upload_async) draining the spin-delay in the POSITIVE arm. Re-reading the OLD test proves that was misdiagnosed:

• The OLD flaky assert is the NEGATIVE arm's raced == poison ("did NOT read poison").
• In the OLD negative arm the only synchronizing alloc (alloc_pinned) is issued BEFORE the spin-delay launch, so nothing drains the spin in the consume window. Its correctness rested on a pure WALL-CLOCK race: the un-ordered consumer had to read poison before the spin-delayed copy landed. Under parallel captured cargo test GPU contention, consumer scheduling latency exceeds the 8M-cycle spin, the copy lands first, and the assert fails. Harry's 25x-spin -> 0/15 result is exactly the signature of a timing race.
• The OLD positive arm asserts got == payload; a drained spin there still yields a completed copy = payload, so it could NOT have produced the failure I attributed to it.

Harry's fix is correct and targets the real bug: the negative arm now event-orders the transfer strictly AFTER the consumer (record_compute_fence + copy_wait_fence), so the consumer provably reads poison with zero wall-clock dependence. The positive arm keeps spin + compute_wait_fence (non-vacuous, per section 3), all device/pinned allocs are hoisted out of the timing window, and a trailing real upload_async byte-parity check keeps the production entry point under cover.

5. Other flakes seen — PRE-EXISTING and UNRELATED

Runs 1/3/4/5 showed failures, but NOT in the target test and NOT in provider.rs:

• Run 1: matmul_f32_gemv_is_capture_safe_after_warmup (tests/matmul_gpu.rs:378) — cublasLtMatmul ... CUBLAS_STATUS_INTERNAL_ERROR.
• Runs 3/4/5: reshape_exact_signature_captures_async_copy (tests/construction_gpu.rs:946) — CUDA_ERROR_STREAM_CAPTURE_INVALIDATED.
  These live in separate integration-test binaries that do not depend on the provider.rs cfg(test) edit; they are known parallel-CUDA-graph-capture/cuBLASLt contention flakes independent of this change. Not attributable to PR feat(cuda-offload): async fence-ordered weight page-in overlap (#87 first increment) #544.

Verdict

The revision is test-only, the target anti-regression lock is deterministic (green 5/5), non-vacuous (fails 3/3 without the fence), and now fixes the actual flaky assert (negative-arm wall-clock race) with event ordering. Approving.

— Melina (independent reviewer)