perf(cuda-ep): make Tile capture-eligible via persistent metadata (Lever 2 PR-2)

[justinchuby]

What this unlocks

Lever 2 PR-2. Migrates the CUDA Tile kernel to the PersistentMetadata / launch_persistent_metadata pattern already used by Transpose (#574) and Expand, so a warmed fixed-decode-signature Tile becomes CUDA-graph capture-eligible and drops its per-call host sync.

Post-#574, Tile was the last launch_metadata caller in movement.rs. Per call it did: device alloc_raw + htod of the metadata + kernel launch + synchronize() (to guard the free_raw) + free — plus a blocking dtoh read of the repeats input. On the qwen3.6-27b linear-attention hybrid, Tile runs ~576×/step inside eager Scan bodies, so those syncs are on the decode hot path.

Correctness — byte-exact vs CPU fp32 oracle

ORT-CUDA crashes on the 27b, so the CPU fp32 backend is the oracle.

• 27b stock export native CUDA == CPU fp32 oracle, byte-identical (native_autoderive_io_cuda_e2e, #[ignore] GPU e2e — passed).
• qwen3-0.6b native CUDA == ORT golden, byte-identical (qwen3_0_6b_native_cuda_e2e — passed); capture engagement unchanged (captures=2, replays=34, fallbacks=0).
• New positive test tile_warmed_metadata_captures_and_matches_eager: unwarmed declines capture; warmed reports Supported; eager re-run and capture+replay both byte-identical.
• New negative test tile_rejects_signature_change_during_capture: feeding a changed signature mid-capture returns Err "changed during CUDA graph capture" — locks the guard's teeth.

The repeats input (a blocking dtoh, illegal mid-capture) is now read only on the unwarmed eager path to validate output[i] == input[i] * repeats[i]. Under a stable (dtype, input_shape, output_shape) signature, repeats is mathematically fixed by shape inference, so gating that read behind the warmed signature is byte-exact. Kernel tile_bytes metadata ([output.shape, input.shape, input_strides]) is unchanged; both launch paths pass identical args.

Honest perf

Stock 27b, profile_native --steady --decode-skip 8 --warmups 2 --runs 3 --tokens 64:

	ms/tok	tok/s
Baseline (origin/main @15288d7a)	156.3	6.40
This PR	150.0	6.67

~4.0% faster, tokens byte-identical. This does not beat ORT (17.38 tok/s) on its own — that requires the Inc-1b Scan-body capture lane (awaiting Justin's greenlight) plus removal of the shared eager synchronize() on the ~50 always-sync kernels. This is one more fat op off the eager-sync hot path, in the same reviewable single-op shape as #574.

Blast radius

Tile kernel path + its two tests only. Dead launch_metadata removed (Tile was its sole remaining caller). No change to shared capture machinery, the #443/#543 capture-correctness invariants, weight offload, session-reuse, or GAP-3.

Refs #574 (Lever 2 lineage), #443/#543 (capture invariants).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.46%. Comparing base (15288d7) to head (712d756).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #577      +/-   ##
==========================================
+ Coverage   80.75%   81.46%   +0.71%     
==========================================
  Files         315      315              
  Lines      123737   123737              
  Branches   123737   123737              
==========================================
+ Hits        99920   100803     +883     
+ Misses      19746    18856     -890     
- Partials     4071     4078       +7     

Flag	Coverage Δ
cli-ort-linux	86.69% <ø> (ø)
cli-ort-windows	83.84% <ø> (+0.10%)	⬆️
mlas	77.91% <ø> (ø)
offline	81.29% <ø> (+0.75%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Benchmarks — No Regression

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
✅	gather/large_f16_threads=1-internal/131072	10.32 µs	11.47 µs	+11.1%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.82 ms	4.15 ms	+8.6%
✅	gather/large_f32_threads=1-internal/131072	27.04 µs	28.88 µs	+6.8%
✅	gather/large_bf16_threads=1-internal/131072	11.57 µs	12.24 µs	+5.8%
✅	gather/medium_f32_threads=1-internal/32768	3.71 µs	3.87 µs	+4.4%
✅	sampling_latency/greedy_per_token	3.22 µs	3.31 µs	+2.9%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.95 ms	1.99 ms	+2.1%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	80.47 µs	81.54 µs	+1.3%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	518.95 µs	525.84 µs	+1.3%
✅	matmul/small_generic_f32_threads=1/1x256x256	35.48 µs	35.95 µs	+1.3%
✅	gather/small_f16_threads=1-internal/4096	466.9 ns	473.0 ns	+1.3%
✅	sampling_latency/top_p_per_token	986.14 µs	997.41 µs	+1.1%
✅	sampling_latency/top_k_per_token	472.50 µs	477.13 µs	+1.0%
✅	matmul/small_generic_f16_threads=1/1x256x256	28.79 µs	29.07 µs	+1.0%
✅	matmul/small_generic_f16_threads=8/1x256x256	29.45 µs	29.73 µs	+0.9%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.32 ms	2.34 ms	+0.9%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	76.30 µs	76.93 µs	+0.8%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	376.06 µs	378.94 µs	+0.8%
✅	matmul/medium_generic_f16_threads=8/32x512x512	29.07 µs	29.26 µs	+0.6%
✅	matmul/medium_generic_f16_threads=1/32x512x512	28.42 µs	28.54 µs	+0.4%
✅	matmul/medium_generic_f32_threads=8/32x512x512	954.77 µs	958.50 µs	+0.4%
✅	add/large_f32_threads=1-internal/4194304	40.19 ms	40.28 ms	+0.2%
✅	add/large_f16_threads=1-internal/4194304	41.13 ms	41.21 ms	+0.2%
✅	tokenization/decode_tokens_per_second	6.18 ms	6.19 ms	+0.2%
✅	reduce_mean/medium_f32_threads=1-internal/65536	245.44 µs	244.94 µs	-0.2%
✅	matmul/small_generic_f32_threads=8/1x256x256	34.16 µs	34.04 µs	-0.4%
✅	gather/small_f32_threads=1-internal/4096	669.3 ns	666.5 ns	-0.4%
✅	matmul/small_generic_bf16_threads=8/1x256x256	30.90 µs	30.72 µs	-0.6%
✅	gather/medium_bf16_threads=1-internal/32768	2.45 µs	2.43 µs	-0.7%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.47 ms	9.40 ms	-0.7%
✅	add/medium_f32_threads=1-internal/262144	2.57 ms	2.55 ms	-0.7%
✅	kv_cache/alloc_dealloc_pages	37.91 µs	37.59 µs	-0.8%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.34 ms	1.33 ms	-1.0%
✅	sampling_latency/min_p_per_token	342.67 µs	338.83 µs	-1.1%
✅	gather/medium_f16_threads=1-internal/32768	2.43 µs	2.39 µs	-1.3%
✅	matmul/small_generic_bf16_threads=1/1x256x256	30.48 µs	30.06 µs	-1.4%
✅	reduce_mean/small_f32_threads=1-internal/4096	15.81 µs	15.59 µs	-1.4%
✅	add/small_f32_threads=1-internal/1024	197.6 ns	194.6 ns	-1.5%
✅	add/large_bf16_threads=1-internal/4194304	42.81 ms	42.12 ms	-1.6%
✅	tokenization/encode_tokens_per_second	386.61 µs	379.96 µs	-1.7%
✅	add/medium_f16_threads=1-internal/262144	2.61 ms	2.55 ms	-2.2%
✅	add/medium_bf16_threads=1-internal/262144	2.67 ms	2.61 ms	-2.3%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.03 ms	998.24 µs	-2.7%
✅	grammar_masking/llguidance_compute_mask/32	77.09 µs	74.89 µs	-2.9%
✅	gather/small_bf16_threads=1-internal/4096	495.9 ns	472.7 ns	-4.7%
✅	logit_processing/seven_processor_chain_per_step	1.25 ms	1.17 ms	-6.2%
🟢	add/small_bf16_threads=1-internal/1024	15.07 µs	12.81 µs	-15.0%
🟢	add/small_f16_threads=1-internal/1024	15.55 µs	13.02 µs	-16.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.24 3.43 5.21 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

VERDICT: APPROVE

2026-08-01 · Reviewer: Harry (independent; author was Cohaagen — author-lockout in effect)

PR #577, head 712d7560, base origin/main 15288d7a (post-#574 Transpose merge).
Independently verified in detached worktrees; author claims were re-run, not trusted.

Evidence (all commands run by Harry)

• Blast radius — git diff 15288d7a..712d7560 --stat: only
  crates/onnx-runtime-ep-cuda/src/kernels/movement.rs (Tile path), its two
  construction_gpu.rs tests, and a decisions note. NO shared capture.rs/run.rs/
  segmenter changes. Dead launch_metadata fn removed. ✅
• Clippy — cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings → exit 0. ✅
• GPU suite — construction_gpu --test-threads=1: 22 passed / 0 failed,
  including tile_warmed_metadata_captures_and_matches_eager,
  tile_rejects_signature_change_during_capture, and existing tile_multi_axis_repeats. ✅
• Mutation (non-vacuous, teeth confirmed):
  • Neutered the outer signature guard (if false && capturing && …) →
    tile_rejects_signature_change_during_capture FAILED (the repeats host
    read then trips CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED mid-capture — a real
    second layer of defense; the clean guard is load-bearing for the error message).
  • Corrupted warmed-path metadata (out_dims + 1) →
    tile_warmed_metadata_captures_and_matches_eager FAILED on the byte
    assertion. Tests are not vacuous. Code restored; worktree clean afterward.
• Central correctness — repeats-gating is SOUND (crux):
  Inspected the tile_bytes CUDA kernel: it consumes only out_dims, in_dims,
  in_strides (all derived from input/output shapes) and never reads repeats.
  ONNX Tile fixes output[i] = input[i] * repeats[i], so for a fixed
  (input_shape, output_shape) the value repeats[i] = output[i]/input[i] is
  uniquely pinned; no distinct repeats can yield the same (input_shape, output_shape). A repeats change that MATTERS necessarily changes output_shape,
  hence the signature, hence is caught. Gating the host repeats read/validation
  behind the warmed (dtype,input_shape,output_shape) signature is therefore
  output-preserving. Double layer confirmed: outer execute signature guard
  (Err "…changed during CUDA graph capture…") AND inner PersistentMetadata::prepare
  guard (metadata bytes = [output.shape, input.shape, input_strides]; errors on
  mismatch during capture), plus the host_ints sync itself trips capture. Warmed
  signature is only recorded when NOT capturing; a legitimate later shape change
  outside capture re-reads/re-validates/re-prepares (correct re-warm).
• Byte-exact gates:
  • qwen3_0_6b_native_cuda_e2e (--ignored) → PASS, byte-identical golden tokens. ✅
  • native_autoderive_io_cuda_e2e / stock_export_auto_derives_io_and_matches_cpu_oracle
    (--ignored, 27b vs CPU fp32 oracle) → PASS, cuda tokens == cpu tokens
    byte-identical (oracle wall 5517s; debug build). ✅
• Independent perf re-measure (27b, --steady --decode-skip 8 --warmups 2 --runs 3 --tokens 64, release build, same GPU):
  • Baseline 15288d7a: median 151.90 ms/tok, 6.58 tok/s.
  • PR 712d7560: median 149.01 ms/tok, 6.71 tok/s.
  • Real, non-zero win ≈ +1.9% (2.9 ms/tok), NOT a no-op; generated token ids
    byte-identical between baseline and PR.

Notes / caveats (non-blocking)

• Measured delta (~1.9%) is smaller than the author's claimed ~4.0% (6.2 ms/tok).
  Direction and byte-identity are confirmed; the gap is run-to-run/timing variance,
  not a correctness concern. Worth calibrating future perf claims against a
  same-session baseline.
• capture_support() returns Supported after ANY signature has been warmed
  (mirrors perf(cuda-ep): make Transpose capture-eligible via persistent metadata (Lever 2 PR-1) #574 Transpose/Expand). For genuinely dynamic Tile shapes this yields a
  wasted capture attempt that the double-layer guard aborts with Err → fallback.
  Correctness preserved; matches established precedent.

Conclusion

Structurally identical to merged #574, byte-exact, tightly scoped, guards proven
non-vacuous, correctness crux (repeats-uniqueness) proven sound at the kernel level,
and a real (if modest) decode win with byte-identical tokens on both the 0.6b golden
and 27b oracle. APPROVE.

[justinchuby]

VERDICT: APPROVE (Harry, independent review)

Structurally identical to merged #574; byte-exact; guards mutation-proven non-vacuous; repeats-uniqueness crux proven sound at kernel level (tile_bytes never reads repeats; repeats[i]=output[i]/input[i] pinned by warmed (dtype,input_shape,output_shape) signature). Independent 27b re-measure: 151.90→149.01 ms/tok (+1.9%), token ids byte-identical. 22/22 construction_gpu, clippy -D warnings exit 0, 0.6b golden + 27b fp32 oracle byte-exact. APPROVE.