perf(cuda-ep): make Transpose capture-eligible via persistent metadata (Lever 2 PR-1)

[justinchuby]

Lever 2 PR-1 — make CUDA Transpose capture-eligible via persistent metadata

Part of the Lever-2 non-Scan swarm from cohaagen-27b-decode-perf.md toward closing the native-vs-ORT gap on the qwen3.6-27b (dense-GQA + LinearAttention hybrid). Transpose only — bounded, byte-exact, single-op.

Problem

TransposeKernel used launch_metadata, which on every call did alloc_raw + htod (perm/stride upload) + synchronize() + free_raw. The synchronize() only guards the immediate free_raw, but it serializes the stream on every one of the ~624 Transpose calls per 27b decode step — in both the eager (Scan-child) and captured paths — and left Transpose reporting CaptureSupport::Unsupported, so it could never fold into the parent CUDA graph.

Fix

Migrate Transpose to the in-repo PersistentMetadata + launch_persistent_metadata pattern already blessed for Expand and Slice (#443/#543): cache the perm/stride buffer, upload/sync once at warm-up, then run with no per-call alloc/htod/sync. Once warmed on its exact shape/dtype/perm signature the kernel reports Supported and errors on a mid-capture signature change (identical guard to Expand). Same transpose_bytes kernel, identical metadata bytes → byte-exact.

Blast radius

• ONLY crates/onnx-runtime-ep-cuda/src/kernels/movement.rs (TransposeKernel + factory) + one new EP test.
• Shared capture machinery (segmenter, subgraph_graph_capturable, capture_quarantine_ops, run.rs) untouched — Transpose just starts reporting Supported once warmed.
• Tile deferred (separate PR): shares launch_metadata but also reads repeats device→host via host_ints, needing repeats-warming — not the same trivial mechanism.

Measured yield (stock qwen3.6-27b-int4, CUDA; --steady --decode-skip 8 --warmups 2 --runs 3 --tokens 64)

config	decode ms/tok	tok/s	first 16 token IDs
baseline (origin/main)	165.27	6.05	[11751, 13, 271, 248068, 271, 248069, 271, 4639, 369, 4252, 13, 11751, 369, 279, 6511, 321]
this PR	156.74	6.38	[11751, 13, 271, 248068, 271, 248069, 271, 4639, 369, 4252, 13, 11751, 369, 279, 6511, 321]

-8.5 ms/tok (~5.4%, ~1.05×), token IDs byte-identical.

Honest caveat: this is one op of the Lever-2 swarm and does NOT beat ORT (17.38 tok/s / 57 ms/tok) alone — that is the two-lane result (full Lever-2 swarm + Inc-1b Scan-body capture). Body-internal Transposes get the desync benefit now; their capture-fold benefit needs Inc-1b (the Scan child never captures today).

Correctness (all pass)

• New EP unit test transpose_warmed_metadata_captures_and_matches_eager: unwarmed → declines capture; first eager run warms → Supported; cached-metadata re-run byte-identical; capture+replay byte-identical to eager. Non-vacuous both directions.
• Full construction_gpu EP suite: 19/19 pass — capture surface unregressed.
• 27b oracle gate native_autoderive_io_cuda_e2e::stock_export_auto_derives_io_and_matches_cpu_oracle: PASS — stock 27b native CUDA == CPU fp32 oracle, byte-identical. (ORT-CUDA crashes on this model → CPU fp32 is the oracle, per Large-model (27B) native E2E offload blocked: Unsqueeze rank bug + missing recurrent_state + mobius#432 #384.)
• Small-model gate qwen3_0_6b_native_cuda_e2e: PASS; decode token IDs == pinned golden.
• cargo fmt --all --check clean.

Decision note: .squad/decisions/inbox/cohaagen-lever2-transpose-pr.md.

Do not auto-merge — Harry reviews first (author-lockout).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.36%. Comparing base (7e8a2ec) to head (04db986).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #574      +/-   ##
==========================================
+ Coverage   81.32%   81.36%   +0.03%     
==========================================
  Files         315      315              
  Lines      123737   123733       -4     
  Branches   123737   123733       -4     
==========================================
+ Hits       100629   100675      +46     
+ Misses      19030    18985      -45     
+ Partials     4078     4073       -5     

Flag	Coverage Δ
cli-ort-linux	?
cli-ort-windows	83.84% <ø> (+0.10%)	⬆️
mlas	78.72% <ø> (+0.81%)	⬆️
offline	81.29% <ø> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Benchmarks — No Regression

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
✅	add/large_f32_threads=1-internal/4194304	48.56 ms	52.23 ms	+7.6%
✅	add/medium_bf16_threads=1-internal/262144	3.35 ms	3.45 ms	+2.9%
✅	add/large_f16_threads=1-internal/4194304	50.75 ms	50.03 ms	-1.4%
✅	reduce_mean/small_f32_threads=1-internal/4096	17.09 µs	16.68 µs	-2.4%
✅	grammar_masking/llguidance_compute_mask/32	76.09 µs	74.06 µs	-2.7%
✅	add/medium_f16_threads=1-internal/262144	3.24 ms	3.09 ms	-4.6%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.13 ms	1.07 ms	-5.2%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	108.14 µs	101.29 µs	-6.3%
✅	tokenization/encode_tokens_per_second	378.65 µs	348.97 µs	-7.8%
✅	reduce_mean/medium_f32_threads=1-internal/65536	270.15 µs	248.65 µs	-8.0%
✅	sampling_latency/top_k_per_token	480.77 µs	440.49 µs	-8.4%
✅	sampling_latency/greedy_per_token	3.35 µs	3.03 µs	-9.8%
✅	tokenization/decode_tokens_per_second	6.41 ms	5.76 ms	-10.2%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.77 ms	8.75 ms	-10.4%
✅	sampling_latency/top_p_per_token	1.03 ms	920.19 µs	-10.4%
✅	sampling_latency/min_p_per_token	348.59 µs	312.20 µs	-10.4%
✅	logit_processing/seven_processor_chain_per_step	1.21 ms	1.08 ms	-10.7%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.09 ms	1.85 ms	-11.2%
✅	kv_cache/alloc_dealloc_pages	39.59 µs	35.04 µs	-11.5%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.43 ms	2.14 ms	-11.8%
✅	add/small_bf16_threads=1-internal/1024	15.12 µs	13.18 µs	-12.8%
✅	add/large_bf16_threads=1-internal/4194304	51.88 ms	44.29 ms	-14.6%
🟢	matmul/medium_generic_bf16_threads=1/32x512x512	578.28 µs	480.90 µs	-16.8%
🟢	add/medium_f32_threads=1-internal/262144	2.97 ms	2.44 ms	-17.8%
🟢	matmul/small_generic_f16_threads=8/1x256x256	37.01 µs	28.78 µs	-22.2%
🟢	matmul/large_generic_f16_threads=1/32x1024x1024	95.85 µs	74.07 µs	-22.7%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	38.11 µs	28.81 µs	-24.4%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	39.65 µs	29.84 µs	-24.8%
🟢	matmul/small_generic_f32_threads=1/1x256x256	46.13 µs	33.79 µs	-26.7%
🟢	matmul/small_generic_f16_threads=1/1x256x256	39.10 µs	27.93 µs	-28.6%
🟢	add/small_f16_threads=1-internal/1024	18.06 µs	12.89 µs	-28.7%
🟢	add/small_f32_threads=1-internal/1024	270.2 ns	192.5 ns	-28.7%
🟢	matmul/medium_generic_f16_threads=1/32x512x512	39.38 µs	27.57 µs	-30.0%
🟢	gather/large_f16_threads=1-internal/131072	22.05 µs	15.30 µs	-30.6%
🟢	gather/small_f16_threads=1-internal/4096	730.9 ns	494.3 ns	-32.4%
🟢	gather/small_bf16_threads=1-internal/4096	712.6 ns	479.4 ns	-32.7%
🟢	gather/medium_bf16_threads=1-internal/32768	3.80 µs	2.49 µs	-34.5%
🟢	gather/medium_f16_threads=1-internal/32768	3.76 µs	2.45 µs	-34.9%
🟢	gather/large_bf16_threads=1-internal/131072	19.44 µs	11.91 µs	-38.8%
🟢	matmul/small_generic_f32_threads=8/1x256x256	52.98 µs	31.65 µs	-40.3%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	669.13 µs	377.06 µs	-43.6%
🟢	gather/small_f32_threads=1-internal/4096	1.13 µs	638.4 ns	-43.7%
🟢	gather/medium_f32_threads=1-internal/32768	7.08 µs	3.98 µs	-43.8%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	7.32 ms	3.81 ms	-48.0%
🟢	matmul/large_generic_bf16_threads=8/32x1024x1024	2.40 ms	1.24 ms	-48.5%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	56.84 µs	27.52 µs	-51.6%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.96 ms	902.75 µs	-54.0%
🟢	gather/large_f32_threads=1-internal/131072	63.19 µs	26.24 µs	-58.5%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.12 4.17 7.38 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

VERDICT: APPROVE

PR #574 "perf(cuda-ep): make Transpose capture-eligible via persistent metadata" (Lever 2 PR-1)
Reviewer: Harry (independent, author-lockout). Author: Cohaagen. Head: c532cbe on green main 7e8a2ec.

I independently reproduced every claim on a detached worktree at the PR head. Evidence below.

Blast radius (verified)

• git diff 7e8a2ec..c532cbe touches ONLY: movement.rs (TransposeKernel/Factory), tests/construction_gpu.rs (one new EP test), and the author decision note. Shared capture machinery (capture.rs / run.rs / segmenter) untouched. The noisy "main...HEAD" diff is only because the local main branch is stale; the true base is 7e8a2ec and the PR is a single commit on top of it.

Lane that was red: clippy

• cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings => exit 0 (clean). Lane is green.

EP tests

• construction_gpu suite: 19/19 pass (incl. new transpose_warmed_metadata_captures_and_matches_eager), --test-threads=1.
• NON-VACUOUS proof (mutation testing, done on source):
  • Mutation 1: forced capture_support() to always return Supported => author test FAILS at the pre-warm assert!(matches!(capture_support(), Unsupported)). Test has teeth for the warm->Supported transition.
  • Guard teeth (scratch test I wrote, then removed): warm signature A, begin capture, feed a DIFFERENT input/output shape mid-capture => kernel.execute returns Err ("...changed during CUDA graph capture; warm the exact..."). Guard actively rejects.
  • Mutation 2: disabled the outer signature guard => a shape change mid-capture is STILL caught by the inner PersistentMetadata::prepare() guard (metadata-value change). Two independent layers protect shape/perm.

Byte-exact gates

• qwen3_0_6b_native_cuda_e2e (--include-ignored): PASS. Coherent text; tokens byte-identical to pinned golden.
• native_autoderive_io_cuda_e2e::stock_export_auto_derives_io_and_matches_cpu_oracle: PASS (CPU fp32 oracle ~92 min). CUDA tokens == CPU tokens byte-identical: [11751, 13, 271, 248068, 271, 248069, 271, 4639, 369, 4252, 13, 11751, 369, 279, 6511, 321].

Correctness scrutiny (the specific risks I was asked to attack)

• (a) Stale-metadata wrong output under capture replay: NOT POSSIBLE. During capture the full signature (dtype + input_shape + output_shape + perm) is compared FIRST and any mismatch errors before launch. Shape/perm changes additionally change the metadata bytes, so prepare() would also reject them. A dtype-only change (same shape/perm => same metadata bytes) is caught by the signature guard; and even absent that guard it cannot corrupt output because the metadata buffer is dtype-independent and elem_bytes is recomputed per-execute (captured fresh). No silent stale-reuse path exists.
• (b) Cross-node aliasing/leak: each Transpose node gets its OWN kernel instance from TransposeFactory::create (fresh PersistentMetadata + warmed_signature Mutexes); no sharing between two Transpose nodes. PersistentMetadata::Drop frees the device buffer; re-warm frees the old ptr after sync. No alias, no leak.
• (c) fix(session): expose bound inputs to control flow #443/feat(native-decode): rank-3 mrope native positions unblock qwen3.5 hybrid native-CUDA (#384, #67) #543 capture-invariant regression: none. No mid-capture sync (prepare short-circuits when values match), persistent binding via launch_persistent_metadata exactly like Expand/Slice. Lock order is consistent (warmed_signature then metadata) so no deadlock.

Perf (independently re-measured, apples-to-apples, same debug build + same GPU + same command)

profile_native --model qwen3.6-27b-int4-cuda --ep cuda --steady --decode-skip 8 --warmups 2 --runs 3 --tokens 64

• baseline (movement.rs reverted to 7e8a2ec, incremental rebuild): decode median 245.60 ms/tok (4.07 tok/s)
• PR head: decode median 207.45 ms/tok (4.82 tok/s)
• Delta: -38.2 ms/tok (~15.5%). Larger than the author's release-build -8.5 ms/tok because a debug build amplifies the removed host-side per-call synchronize+alloc+htod+free; direction and mechanism are consistent. Real, non-zero, NOT a no-op.
• Token IDs and generated_text BYTE-IDENTICAL between baseline and PR head on the real 27b: [11, 353, 1044, 4313, 310, 1791, 264, 4145, 5148, 421, 668, 2070, 728, 310, 2222, 279, 1829, 314, ...].

Non-blocking recommendation (not a reject reason)

The shipped test warms only one signature and never exercises the "errors if signature changes mid-capture" branch; that guard's teeth are real (I verified with a scratch test) but are not covered by the committed suite. Suggest adding a small negative case (changed shape/perm mid-capture asserts Err) so the guard is regression-protected. Optional follow-up.

APPROVE: bounded, byte-exact, real host-side win, robust capture guard (double-layered). Ship.