[justinchuby]

Summary

C2 follow-up to #728. This closes the capture-contract bug behind its 33 residual LinearAttention seams and audits the rest of the CUDA kernel directory for the same class of violation.

Established mechanism

A direct capture of LinearAttentionKernel::execute reproduced the failure before this change:

CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED: operation not permitted when stream is capturing

The unconditional trailing synchronize() aborted recording. Session quarantine then classified every sibling LinearAttention node as CaptureRecordingFailed.

35B-A3B decode	Segments	LinearAttention declines	Decode
Before (#728 measurement)	34	33 × CaptureRecordingFailed	11.959 ms/tok (83.62 tok/s)
After	Not measurable on this host	Direct kernel capture: 0; capture/replay succeeds	Not measurable on this host

Because #728's complete seam census was exactly those 33 LinearAttention nodes, the deterministic expected result on the same artifact is 34 → 1 segment. I am not presenting that as measured: the required 35B artifacts are absent here and the available GPU is an 8 GB RTX 4060.

Why the synchronization is not load-bearing

LinearAttention launches one kernel on the executor stream and performs no host readback or second-stream work. Native decode deliberately aliases past/present recurrent state, but each thread loads its full state column before overwriting it. The new regression captures and replays four changing decode steps with that aliasing and verifies output plus recurrent state byte-for-byte against eager CUDA. The sync is therefore only eager asynchronous-error surfacing; same-stream ordering supplies visibility during capture.

CUDA kernel audit

Capture-aware trailing synchronization:

• Activations
• ArgReduce
• ConvTranspose
• FlashAttention
• FusedGelu
• GridSample
• LinearAttention
• SpaceToDepth and EyeLike (structural.rs)

Truthful/dynamic admission for allocation or host-staged paths:

• Attention: warmed fused path supported; Phase-2a allocation path unsupported
• CompressedSparseAttention: host-staged fallback explicitly rejected during capture
• NonZero: unsupported because it allocates/uploads coordinate metadata per call
• LpPool: unsupported because it allocates/uploads/frees metadata per call

capture_sync_contract.rs source-audits every kernel synchronization and fails on any new unreviewed unconditional sync reachable by a capture-supported path. Intentional unsupported/fallback sites are explicitly allowlisted. No assertions were added to Drop.

Validation

Passed locally with the requested CUDA wheel paths, including cuBLASLt:

• cargo fmt --all -- --check
• cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings
• cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings
• CUDA EP library suite: 308 passed
• linear_attention_gpu: 7 passed
• attention_gpu: 5 passed, 1 benchmark ignored
• capture synchronization contract test: passed
• session quarantine classification test: passed

Pure-attention qwen2.5-0.5b-int4-onnx-native produced byte-identical 64-token output with capture off/on:

• capture off: 31.932 ms/tok (31.32 tok/s)
• capture on: 7.057 ms/tok (141.70 tok/s)

The re-anchored 35B oracle binaries compiled and were invoked with ONNX_GENAI_CUDA_GRAPH=0 and =1; both real-model cases skipped because /home/justinchu/qwen36-35b-a3b-qmoe-artifacts is unavailable. The same-artifact 34 → 1 segment measurement and 35B oracle run are required before merge.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.36%. Comparing base (675b697) to head (5e773ec).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #757      +/-   ##
==========================================
- Coverage   80.22%   79.36%   -0.86%     
==========================================
  Files         331      331              
  Lines      137462   137462              
  Branches   137462   137462              
==========================================
- Hits       110275   109093    -1182     
- Misses      22844    24029    +1185     
+ Partials     4343     4340       -3     

Flag	Coverage Δ
cli-ort-linux	83.41% <ø> (ø)
cli-ort-windows	83.00% <ø> (+0.09%)	⬆️
mlas	80.69% <ø> (-0.65%)	⬇️
offline	79.17% <ø> (-0.90%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	35.33 µs	114.84 µs	+225.1%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	40.46 µs	95.43 µs	+135.9%
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	518.60 µs	1.15 ms	+122.7%
🔴	matmul/small_generic_f32_threads=8/1x256x256	33.16 µs	62.34 µs	+88.0%
🔴	matmul/medium_generic_bf16_threads=8/32x512x512	370.25 µs	689.13 µs	+86.1%
🔴	matmul/small_generic_f16_threads=1/1x256x256	28.36 µs	49.12 µs	+73.2%
🔴	matmul/medium_generic_f16_threads=8/32x512x512	28.04 µs	47.40 µs	+69.0%
🔴	matmul/small_generic_f16_threads=8/1x256x256	31.55 µs	50.73 µs	+60.8%
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.24 ms	1.95 ms	+57.7%
🔴	gather/large_bf16_threads=1-internal/131072	11.15 µs	17.01 µs	+52.5%
🔴	matmul/small_generic_bf16_threads=1/1x256x256	28.89 µs	43.49 µs	+50.5%
🔴	gather/large_f16_threads=1-internal/131072	10.70 µs	15.67 µs	+46.4%
🔴	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	349.92 µs	511.48 µs	+46.2%
🔴	matmul/large_generic_f32_threads=8/32x1024x1024	3.59 ms	5.06 ms	+41.0%
🔴	matmul/medium_generic_f32_threads=8/32x512x512	899.53 µs	1.23 ms	+36.5%
🔴	matmul/large_generic_f16_threads=8/32x1024x1024	77.46 µs	105.57 µs	+36.3%
🔴	matmul/large_generic_f16_threads=1/32x1024x1024	73.06 µs	98.12 µs	+34.3%
🔴	matmul/small_generic_f32_threads=1/1x256x256	34.00 µs	45.49 µs	+33.8%
🔴	matmul/large_generic_bf16_threads=1/32x1024x1024	1.81 ms	2.35 ms	+30.1%
⚠️	matmul/medium_generic_f16_threads=1/32x512x512	27.94 µs	34.27 µs	+22.6%
⚠️	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	71.51 µs	86.97 µs	+21.6%
⚠️	matmul/medium_generic_bf16_threads=1/32x512x512	480.14 µs	578.67 µs	+20.5%
⚠️	matmul/small_generic_bf16_threads=8/1x256x256	29.79 µs	34.96 µs	+17.3%
⚠️	sampling_latency/greedy_per_token	3.00 µs	3.49 µs	+16.3%
⚠️	tokenization/decode_tokens_per_second	5.76 ms	6.63 ms	+15.2%
✅	grammar_masking/llguidance_compute_mask/32	69.23 µs	79.08 µs	+14.2%
✅	qwen3_sampling_processors/top_k_partial_selection	130.77 µs	148.77 µs	+13.8%
✅	sampling_latency/top_p_per_token	352.81 µs	399.00 µs	+13.1%
✅	tokenization/encode_tokens_per_second	352.17 µs	396.43 µs	+12.6%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.19 ms	2.45 ms	+12.0%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	1.96 ms	2.18 ms	+11.4%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.28 ms	3.65 ms	+11.2%
✅	sampling_latency/top_k_per_token	49.09 µs	54.57 µs	+11.2%
✅	logit_processing/seven_processor_chain_per_step	300.37 µs	333.53 µs	+11.0%
✅	sampling_latency/min_p_per_token	192.36 µs	212.29 µs	+10.4%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	8.82 ms	9.69 ms	+9.9%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	481.36 µs	527.11 µs	+9.5%
✅	qwen3_sampling_processors/top_k_top_p_fast	610.57 µs	666.94 µs	+9.2%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.43 ms	5.77 ms	+6.4%
✅	gather/medium_bf16_threads=1-internal/32768	2.73 µs	2.88 µs	+5.5%
✅	kv_cache/alloc_dealloc_pages	40.87 µs	39.55 µs	-3.2%
✅	gather/small_f16_threads=1-internal/4096	458.3 ns	440.3 ns	-3.9%
✅	reduce_mean/small_f32_threads=1-internal/4096	14.84 µs	14.20 µs	-4.3%
✅	gather/small_f32_threads=1-internal/4096	658.2 ns	629.0 ns	-4.4%
✅	add/large_f16_threads=1-internal/4194304	40.79 ms	38.66 ms	-5.2%
✅	reduce_mean/large_f32_threads=1-internal/262144	964.06 µs	908.03 µs	-5.8%
✅	gather/large_f32_threads=1-internal/131072	38.76 µs	35.98 µs	-7.2%
✅	reduce_mean/medium_f32_threads=1-internal/65536	244.39 µs	226.77 µs	-7.2%
✅	add/small_bf16_threads=1-internal/1024	13.40 µs	11.80 µs	-11.9%
✅	add/large_bf16_threads=1-internal/4194304	44.59 ms	39.24 ms	-12.0%
✅	add/medium_f32_threads=1-internal/262144	2.65 ms	2.31 ms	-12.8%
✅	add/medium_f16_threads=1-internal/262144	2.72 ms	2.36 ms	-13.1%
✅	add/small_f16_threads=1-internal/1024	13.60 µs	11.79 µs	-13.3%
🟢	add/medium_bf16_threads=1-internal/262144	2.91 ms	2.41 ms	-17.2%
🟢	add/large_f32_threads=1-internal/4194304	45.76 ms	37.29 ms	-18.5%
🟢	gather/medium_f16_threads=1-internal/32768	3.02 µs	2.41 µs	-20.3%
🟢	gather/small_bf16_threads=1-internal/4096	571.0 ns	447.0 ns	-21.7%
🟢	add/small_f32_threads=1-internal/1024	255.6 ns	181.5 ns	-29.0%
🟢	gather/medium_f32_threads=1-internal/32768	5.06 µs	3.54 µs	-30.0%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.09 3.40 5.97 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

Handoff to the H200 team

This PR is being handed to the H200 team to finish. Everything below is the
state I can vouch for, and the one gap I cannot close on this box.

What is established

The seam had a concrete, verified cause. LinearAttention declared capture
Supported (it never overrode capture_support, so it took the
CaptureSupport::Supported default) while ending execute with an
unconditional self.runtime.synchronize(). A stream synchronize is illegal
during capture — CudaRuntime::is_capturing exists in
crates/onnx-runtime-ep-cuda/src/runtime.rs precisely to let device-resident
kernels skip the trailing sync while a graph is recording, and nearly every
other kernel in src/kernels/ already had that guard. LinearAttention did not.

Measured on this box, before the change: 34 segments, 33
CaptureRecordingFailed LinearAttention seams, with the failure confirmed as
CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED. That matches the 33 residual seams #728
attributed to "LinearAttention's load-bearing C2 sync" — with the correction
that the sync is not load-bearing: a four-step aliased-state replay is
byte-identical without it.

This PR also audits every CUDA kernel synchronization for the same
contract violation (a kernel that is capture-supported but can reach an
unconditional synchronize()), fixes the ones found, and adds
tests/capture_sync_contract.rs to keep the class of defect from returning.

The gap — please close this

The 35B-A3B end-to-end result is NOT measured. The expected outcome is a
collapse from 34 segments to 1, but the 35B-A3B QMoE hybrid artifacts are
not present on this machine, so that number is a prediction, not a measurement.
Please do not merge on the strength of it.

What still needs to run on hardware with the 35B-A3B artifacts:

1. Segment count before vs after on the 35B-A3B QMoE hybrid decode. This is
   the deterministic primary evidence.
2. The re-anchored oracle test crates/onnx-genai-engine/tests/qwen36_35b_a3b_qmoe_divergence.rs
   must pass with ONNX_GENAI_CUDA_GRAPH=0 and =1. perf(cuda-ep): capture-eligible pointwise ops via build-time growing-symbol classification (C1) + re-anchor 35B QMoE oracle lock on fp32 teacher-forcing (#722) #728 re-anchored that
   lock onto fp32 teacher-forcing specifically so it is capture-independent, so
   it is the right gate for a change that moves work onto the captured stream.
3. Decode ms/tok before vs after, same session, interleaved. Note that
   throughput variance on the dev box this was developed on is extreme
   (identical runs have ranged 3.9–28 tok/s), so segment count should lead and
   wall-clock should be treated as secondary unless your box is quieter.

What has been validated locally

• Pure-attention qwen2.5-0.5b-int4-onnx-native decode output byte-identical,
  capture on and off (KV-length-dependent ops correctly stay eager).
• cargo fmt --all -- --check clean.
• cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings clean
  (the exact CI CUDA gate — note it does not use --all-targets).
• Targeted CUDA/session tests pass, including the new capture-sync contract test.

Blast radius

This touches the trailing-sync behavior of several shared kernels, so a wrong
broadening would silently corrupt models under capture. The mitigations are the
capture-sync contract test, the byte-identical pure-attention validation, and —
critically — the 35B oracle test at both capture settings, which is the item
still outstanding.

Context worth carrying over

Weight offload and CUDA graph capture are currently mutually exclusive
(crates/onnx-genai-engine/src/native_decode/cuda.rs): the pager's
alloc/copy/free ops are capture-illegal, so enabling offload disables capture.
That means segment-collapse wins like this one do not currently reach models
large enough to need offload. #716 is the fix (page weights under a stable
virtual address so page-in remaps physical granules instead of returning a new
pointer) and is in progress separately.