[justinchuby]

Why

This class of defect produced two wrong conclusions this week, both of which cost real work before being caught. Both are the same family: state that is frozen or warmed once per process, making results depend on test order. On a measurement-driven subsystem a wrong number does not just fail — it redirects the design.

• Make native CUDA KV binding commit geometry layout-aware #797 — a GPU residency test failed ~50% of the time on a cold context and looked exactly like a correctness bug in growth-under-capture. Root cause: a readback on the legacy default stream raced a memset on a created non-blocking stream (the two are mutually exempt from implicit synchronization), so the readback returned partial fills with no CUDA error. cuCtxSynchronize did not fix it, which refuted the obvious race theory. It only "passed" because an alphabetically-earlier sibling subtest warmed the context.
• Build KV growth byte geometry under a BSNH fixed stride (seq-major) #801 / captures=0 was cached ONNX_GENAI_CUDA_GRAPH=0, not a model decline #804 — captures=0 was reported as the decoder structurally declining CUDA graph capture, and that attribution reached a merged PR body. Root cause (captures=0 was cached ONNX_GENAI_CUDA_GRAPH=0, not a model decline #804): the capture-OFF phase set ONNX_GENAI_CUDA_GRAPH=0 before the process-wide RuntimeConfig OnceLock was first read, and that config is frozen on first read — so every later phase in the same process inherited it. Enable stride-described seq-major flash prefill #794 and Build KV growth byte geometry under a BSNH fixed stride (seq-major) #801 failed to reproduce an end-to-end measurement and misattributed the reason.

This is the same telemetry-that-misleads lineage as #719 / #715 (documented in MEMORY_ARCHITECTURE.md's "third failure mode"). After the second instance in a week, the response is a guard, not more vigilance.

This PR is test infrastructure + documentation only. No production behavior changes.

What

1. A guard for process-frozen configuration (onnx-genai-runtime-config)

runtime_config() records the environment it froze the RuntimeConfig snapshot from (captured through the same lookup from_env uses, so the set can never drift from the parsed fields). On every later call it re-reads those variables and panics loudly — with the frozen and current values, and a pointer to #804 — if any now differs. A test can no longer silently believe it set a knob that was already frozen.

• Gated on debug_assertions, not cfg(test). The captures=0 was cached ONNX_GENAI_CUDA_GRAPH=0, not a model decline #804 mutation lives in an integration test of a downstream crate, where cfg(test) for this crate is never set. debug_assertions is enabled for dependencies under the dev/test profile, so the guard is live for every cargo test in the workspace, and it is compiled out of --release — the production path is byte-for-byte unchanged and pays nothing (a single OnceLock read).
• The comparison is a pure find_changed() helper with unit tests for the unchanged, changed, newly-set (the captures=0 was cached ONNX_GENAI_CUDA_GRAPH=0, not a model decline #804 shape), and cleared cases.

2. A helper for stream-correct device test code (onnx-runtime-cuda-memory::test_support::TestStream)

A shared, test-only (#[cfg(any(test, feature = "gpu-tests"))]) RAII wrapper around one non-blocking stream. Every device op a test needs — fill, write, read, plus raw() for graph launches/memsets — flows through that stream, so a single cuStreamSynchronize is a total order. The module doc explains the legacy-default-vs-non-blocking exemption at the call site so the next author does not rediscover #797 the hard way. vmm_kv_layout_residency_gpu.rs (the exact #797 site) is converted to it and its four local stream helpers removed; 8/8 reliable in isolation and green in the full single-threaded suite.

3. The pattern, documented where it will be read (docs/MEMORY_ARCHITECTURE.md)

A tight "a fourth failure mode: order-dependent test state" subsection next to the existing third-failure-mode material: both case studies, the two mechanisms (frozen process config; context warmed by an earlier sibling), the two remedies (process isolation for anything that reads process-frozen config; single-stream discipline for device tests), and the one-line discipline that names the defect — run any test you touch in isolation as well as in the full suite. Cites #797/#801/#804/#794/#719/#715 and links rather than duplicating (#791 consolidated this doc to stop drift).

Inventory of other order-dependent tests found (not fixed here)

Per scope discipline, these are listed, not converted. The TestStream helper and the config guard make converting them mechanical follow-ups.

Device tests that mix a created stream with default-stream copies (#797 hazard)

Each creates a non-blocking stream and issues default-stream cuMemcpyHtoD_v2/cuMemcpyDtoH_v2 copies. Where those touch the same buffers as stream-ordered work on a cold context, they are exposed to the exact #797 race. They should migrate to test_support::TestStream:

• crates/onnx-runtime-cuda-memory/tests/vmm_commit_shared_prefix_gpu.rs
• crates/onnx-runtime-cuda-memory/tests/vmm_dummy_page_gpu.rs
• crates/onnx-runtime-cuda-memory/tests/vmm_dummy_write_protect_gpu.rs
• crates/onnx-runtime-cuda-memory/tests/vmm_granularity_gpu.rs
• crates/onnx-runtime-cuda-memory/tests/vmm_graph_remap_gpu.rs
• crates/onnx-runtime-cuda-memory/tests/vmm_kv_contiguous_tail_gpu.rs
• crates/onnx-runtime-cuda-memory/tests/vmm_prefix_share_gpu.rs
• crates/onnx-runtime-cuda-memory/tests/vmm_prefix_share_write_protect_gpu.rs
• crates/onnx-runtime-cuda-memory/tests/vmm_stable_va_weight_slot_gpu.rs

(Non-vmm siblings virtual_memory_gpu.rs and physical_pool_production_gpu.rs also use default-stream copies; they pass but are the same latent shape.)

Tests that mutate a RuntimeConfig-feeding env var within a process

These change a variable that feeds the frozen snapshot (ONNX_GENAI_CUDA_GRAPH, ONNX_GENAI_EP) across phases. The correct pattern is child-process-per-phase; the new guard will now fire loudly on any that mutate in-process and then read runtime_config():

• crates/onnx-genai-engine/tests/mobius_seqmajor_growth_parity_native_cuda.rs — already correct (spawns a child per mode; its own doc comment states ONNX_GENAI_CUDA_GRAPH is parsed once). This is the reference pattern.
• crates/onnx-genai-engine/tests/stable_va_capture_offload_e2e.rs — sets ONNX_GENAI_CUDA_GRAPH per capture phase.
• crates/onnx-genai-engine/tests/gemma4_e2b_native_cuda_pipeline_bench.rs — sets ONNX_GENAI_CUDA_GRAPH to 1 then 0.
• crates/onnx-genai-engine/tests/qwen2_5_1_5b_divergence.rs — switches ONNX_GENAI_EP between cpu and cuda.
• crates/onnx-genai-engine/tests/qwen3_0_6b_capture_step_inputs_decline.rs and gemma3n_native_cuda_capture_realmodel.rs — toggle the native-decoder capture env per phase.

Session-crate tests that toggle env (ONNX_GENAI_DECODE_MEMO, ONNX_GENAI_SCAN_INLINE_SINGLE_TRIP, ONNX_RT_SIBLING_MERGE, the EP-CPU weight-offload knob) read those variables directly at their call sites, not through RuntimeConfig, so they are not affected by the freeze guard and the full onnx-runtime-session suite stays green.

Validation (all local; not waiting for CI)

• cargo fmt -p onnx-genai-runtime-config -p onnx-runtime-cuda-memory -- --check ✅ (--all hits the Windows os error 206 long-command artifact, not a formatting issue)
• cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings ✅
• cargo clippy -p onnx-runtime-cuda-memory --features gpu-tests --all-targets -- -D warnings ✅
• cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings ✅
• cargo test -p onnx-runtime-session ✅ (full suite)
• cargo test -p onnx-runtime-cuda-memory --features gpu-tests -- --test-threads=1 ✅ (all vmm_*/gpu suites, RTX 4060)
• The converted seq_major_fixed_stride_grows_under_captured_replay_without_recapture run 8× in isolation: 8/8 pass — the property this PR is about.

Notes

• Kept strictly to test infra + the docs subsection so the two other active workstreams (GQA pooled QKV staging in group_query_attention.rs; head-major vs seq-major re-capture measurement) merge cleanly — no overlap.
• Does not merge and does not close any issue.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.55%. Comparing base (39c2935) to head (e300b10).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #807       +/-   ##
===========================================
+ Coverage   79.21%   83.55%    +4.33%     
===========================================
  Files         331       12      -319     
  Lines      138203     5192   -133011     
  Branches   138203     5192   -133011     
===========================================
- Hits       109484     4338   -105146     
+ Misses      24372      677    -23695     
+ Partials     4347      177     -4170     

Flag	Coverage Δ
cli-ort-linux	83.50% <ø> (ø)
cli-ort-windows	83.09% <ø> (+0.09%)	⬆️
mlas	?
offline	?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 319 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[justinchuby]

Coordinator pre-merge validation on the merged branch: origin/main merged (no
conflicts); cargo fmt -p onnx-genai-runtime-config -p onnx-runtime-cuda-memory -- --check clean; full cargo test -p onnx-runtime-session: 153 passed;
onnx-genai-runtime-config: 19 passed.

I checked the one thing that could have made this unmergeable — it touches two
production crates, so I verified the guard really is compiled out of release:

#[cfg(debug_assertions)]      { ... freeze_guard::assert_frozen_env_unchanged(); }
#[cfg(not(debug_assertions))] { // identical to the historical behaviour }

Release is byte-for-byte the old path. And the reasoning for debug_assertions
over cfg(test) is correct rather than convenient: the #804 mutation lived in a
downstream crate's integration test, where this crate's cfg(test) is never
set, so cfg(test) would have guarded nothing.

Why this was worth doing at all

This class of defect produced two wrong conclusions this week, and the cost was
not a failing test — it was misdirected design:

• Make native CUDA KV binding commit geometry layout-aware #797 — a GPU test failed ~half the time on a cold context and looked exactly
  like a correctness bug in growth-under-capture. Root cause: a readback on the
  legacy default stream raced a memset on a created non-blocking stream (mutually
  exempt from implicit synchronisation), returning partial fills with no CUDA
  error. cuCtxSynchronize did not fix it.
• Build KV growth byte geometry under a BSNH fixed stride (seq-major) #801/captures=0 was cached ONNX_GENAI_CUDA_GRAPH=0, not a model decline #804 — captures=0 was reported as the decoder structurally declining
  capture, and that attribution reached a merged PR body. Root cause: the
  process-wide RuntimeConfig is frozen on first read, so a phase that set
  ONNX_GENAI_CUDA_GRAPH=0 poisoned every later phase in the same process. Two
  PRs (Enable stride-described seq-major flash prefill #794, Build KV growth byte geometry under a BSNH fixed stride (seq-major) #801) failed to reproduce a measurement and blamed the wrong thing.

Then #805 measured the actual cause of the re-capture question — unconditional
growth invalidation — proving that two of the three intermediate explanations had
been measurement artifacts. On a measurement-driven line, a wrong number does not
merely fail; it redirects the design. Making this class loud is worth a PR.

The caveat is the right call, not a defect

The guard fires on any in-process change to a config-feeding variable,
including five engine e2e tests that already do phase mutation. Those are
#[ignore] without gpu-tests and were outside the required validation set, so
they will trip the guard when run.

That is the intended signal — those tests need child-process isolation, exactly as
the mobius_... reference test already does. Inventorying them (9 vmm_* tests
mixing created and default streams; 5 engine tests mutating config-feeding env
vars) rather than silently converting them is the right scope call: a silent
conversion would have hidden how widespread the pattern is.

I am merging with that inventory as a known follow-up rather than blocking on it.
The guard existing and being loud is what prevents the next two-PR misattribution;
converting the five can follow.

Merging.

[github-actions]

✅ Benchmarks — No Regression

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
✅	add/small_f16_threads=1-internal/1024	13.49 µs	15.47 µs	+14.7%
✅	logit_processing/seven_processor_chain_per_step	329.98 µs	377.59 µs	+14.4%
✅	add/small_f32_threads=1-internal/1024	204.8 ns	232.3 ns	+13.4%
✅	sampling_latency/min_p_per_token	217.67 µs	246.50 µs	+13.2%
✅	sampling_latency/top_p_per_token	432.54 µs	480.14 µs	+11.0%
✅	grammar_masking/llguidance_compute_mask/32	84.20 µs	91.67 µs	+8.9%
✅	gather/medium_f16_threads=1-internal/32768	2.61 µs	2.84 µs	+8.8%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	88.73 µs	95.97 µs	+8.2%
✅	gather/medium_bf16_threads=1-internal/32768	2.62 µs	2.80 µs	+7.0%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	569.19 µs	600.64 µs	+5.5%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.03 ms	2.13 ms	+5.2%
✅	tokenization/encode_tokens_per_second	394.94 µs	414.65 µs	+5.0%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.39 ms	6.58 ms	+3.0%
✅	sampling_latency/top_k_per_token	64.00 µs	64.76 µs	+1.2%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	80.40 µs	79.74 µs	-0.8%
✅	add/medium_f16_threads=1-internal/262144	2.62 ms	2.59 ms	-1.0%
✅	qwen3_sampling_processors/top_k_top_p_fast	767.67 µs	756.61 µs	-1.4%
✅	add/small_bf16_threads=1-internal/1024	14.14 µs	13.83 µs	-2.2%
✅	tokenization/decode_tokens_per_second	7.12 ms	6.96 ms	-2.3%
✅	add/medium_f32_threads=1-internal/262144	2.66 ms	2.59 ms	-2.6%
✅	add/medium_bf16_threads=1-internal/262144	2.71 ms	2.62 ms	-3.3%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.39 ms	2.30 ms	-3.4%
✅	qwen3_sampling_processors/top_k_partial_selection	175.24 µs	169.20 µs	-3.4%
✅	add/large_f32_threads=1-internal/4194304	42.03 ms	40.13 ms	-4.5%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.43 ms	1.34 ms	-6.0%
✅	kv_cache/alloc_dealloc_pages	49.68 µs	45.47 µs	-8.5%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	87.02 µs	77.67 µs	-10.8%
✅	sampling_latency/greedy_per_token	3.83 µs	3.42 µs	-10.8%
✅	matmul/small_generic_bf16_threads=8/1x256x256	34.93 µs	31.15 µs	-10.8%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.76 ms	2.45 ms	-11.5%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.13 ms	997.38 µs	-12.1%
✅	gather/small_f32_threads=1-internal/4096	768.3 ns	669.7 ns	-12.8%
✅	add/large_bf16_threads=1-internal/4194304	48.38 ms	41.95 ms	-13.3%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	11.00 ms	9.42 ms	-14.3%
✅	gather/small_bf16_threads=1-internal/4096	556.6 ns	474.8 ns	-14.7%
🟢	matmul/medium_generic_f16_threads=1/32x512x512	37.04 µs	31.31 µs	-15.4%
🟢	matmul/medium_generic_bf16_threads=1/32x512x512	623.05 µs	526.20 µs	-15.5%
🟢	add/large_f16_threads=1-internal/4194304	49.29 ms	41.35 ms	-16.1%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	4.66 ms	3.88 ms	-16.7%
🟢	matmul/small_generic_f16_threads=1/1x256x256	37.30 µs	30.67 µs	-17.8%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	37.70 µs	30.99 µs	-17.8%
🟢	gather/large_f32_threads=1-internal/131072	35.69 µs	29.13 µs	-18.4%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	39.70 µs	31.41 µs	-20.9%
🟢	gather/small_f16_threads=1-internal/4096	617.7 ns	485.3 ns	-21.4%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	483.94 µs	375.81 µs	-22.3%
🟢	matmul/small_generic_f16_threads=8/1x256x256	40.23 µs	30.75 µs	-23.6%
🟢	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	5.61 ms	4.26 ms	-24.0%
🟢	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	60.59 µs	44.54 µs	-26.5%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	338.67 µs	248.94 µs	-26.5%
🟢	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	740.94 µs	535.70 µs	-27.7%
🟢	reduce_mean/small_f32_threads=1-internal/4096	20.68 µs	14.85 µs	-28.2%
🟢	matmul/small_generic_f32_threads=1/1x256x256	53.78 µs	37.73 µs	-29.8%
🟢	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	576.57 µs	379.58 µs	-34.2%
🟢	matmul/small_generic_f32_threads=8/1x256x256	56.74 µs	35.11 µs	-38.1%
🟢	gather/large_f16_threads=1-internal/131072	20.90 µs	11.98 µs	-42.7%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.59 ms	895.85 µs	-43.7%
🟢	gather/medium_f32_threads=1-internal/32768	6.76 µs	3.74 µs	-44.7%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	82.96 µs	44.92 µs	-45.9%
🟢	gather/large_bf16_threads=1-internal/131072	27.77 µs	12.00 µs	-56.8%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.71 3.72 6.16 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)