[amd-khushbu]

JIRA ID : AICK-2712

Summary

This is PR 4 of a stack of 9 that brings the rocKE C++ unified-attention engine to parity with Python (see the stack below). It sits on PR 3.

Python's gfx950 production prefill specs use three things the C++ builder rejected or left unset:

• the V double-buffer and staggered-wait schedules, for the single-batch D64 combo cohort;
• the D256 fast path, for bf16 D256 prefill;
• the softmax-MFMA interleave, for the single-batch D128 combo cohort.

C++ now builds all three the way Python does. The tiled-2D gate no longer rejects these cohorts, and the gfx950 spec built from a problem matches Python's on every field.

Changes

All paths are under dnn-providers/hip-kernel-provider/rocke/platform/cpp unless noted.

• Softmax-MFMA interleave. The C++ spec builder sets use_softmax_mfma_interleave (mode 1) for the single-batch D128 combo cohort, with the same predicate that widens it to 4 warps. Before, C++ built the widened kernel without the iglp_opt hint.
• V double-buffer and staggered wait.
  • V_lds gets two slots. V[tile_start] is prefetched before the loop, and each iteration prefetches V[i+1] and K[i+1] into the other slot (K first when staggered).
  • The staggered iteration-start wait drains only K[i].
  • The builder mirrors Python's __post_init__ checks for both flags instead of rejecting them. The deep ring and Q-reread stay rejected (known_gaps.md).
• D256 fast path.
  • The spec struct gains use_kq_lds_pad, kq_lds_pad_halves and use_mask_phase_split. They are defaulted, and gfx942 ignores them.
  • The spec from a problem applies Python's D256 overrides last, as Python does.
  • The builder ports the direct-to-register Q gather, the K-slab LDS pad, interleave mode 2 (alternating MFMA/VALU sched_group_barrier groups) and the mask phase split (a mask-free full-tile loop, then the masked boundary loop).
• Fix: fp8-in-LDS K on the transposed QK. The transposed QK had its own K reader, which read fp8-in-LDS K as bf16. It now reads K through the shared reader, which applies the fp8 dequant as Python does. Config 69 pins this. With the old reader the gate is RED on configs 64, 65, 67, 68 and 69.
• Not ported: the right-bound variant of the mask phase split. The C problem has no right_bound slot yet (known_gaps.md).

Stack

1. feat(hip-kernel-provider): bring the rocKE C++ unified-attention engine to parity with Python #14522 Stopgaps: stop C++ silently misbuilding specs
2. feat(hip-kernel-provider): port rocKE unified-attention selectors, routing and support gates to C++ #14523 Selectors, routing and support gates
3. feat(hip-kernel-provider): assemble rocKE unified-attention specs from a problem in C++ and run selector parity in CTest #14524 Spec from problem (gfx942 2D, 3D), scalar fallback, selector parity in CTest
4. feat(hip-kernel-provider): port the rocKE gfx950 tiled-2D V double-buffer, D256 fast path and softmax interleave to C++ #14525 gfx950 tiled 2D: V double-buffer, D256 fast path, softmax interleave ← this PR
5. feat(hip-kernel-provider): port the rocKE gfx942 tiled-2D mask-limit, sliced-K ring, phase split and HBM-direct K/V to C++ #14526 gfx942 tiled 2D: mask-limit, sliced-K ring, phase split, HBM-direct K/V
6. feat(hip-kernel-provider): port the rocKE gfx942 4-warp GQA attention builder to C++ #14527 gfx942 4-warp GQA builder
7. fix(hip-kernel-provider): rocKE unified-attention C++ parity follow-ups #14528 Follow-ups: gfx942 3D i64 gate, reject messages, emit_lse, CTest labels
8. feat(hip-kernel-provider): close the remaining rocKE unified-attention C++ builder gaps #14758 i64 sliced-K ring, shared arch gate, 3D kv_layout, LDS resolver, gfx950 Q re-read and depth-3 ring, parity coverage
9. feat(hip-kernel-provider): port the rocKE unified-attention mask band and split-KV LSE to C++ #14759 Mask band (top-left / right-bound) and split-KV LSE

Each PR is based on the one before it. Review from 1 upward; each diff shows only its own slice.

Test plan

• gfx950 tiled-2D parity emitter configs:
  • 59–63: the production bf16 and fp16 V double-buffer specs, the staggered wait, and the 16x16, non-transposed 32x32 and early-V paths.
  • 64–68: the production D256 specs with and without invariant hoist, and each lever on its own at D128.
  • 69: fp8-in-LDS K on the transposed path.
• test_attention_selector_cpp_parity.py: the D256-override and softmax-interleave known-gap rows are removed, so the gfx950 spec has no known gaps left
• test_attention_binding_spec_fields.py and test_attention_routing_cpp_parity.py updated for the newly ported fields and cohorts
• CTest, pybind build: 39/39
• Byte-identity gate GREEN at llvm20, llvm22, llvm23 and llvm24
• Golden: gfx950_attention_tiled_2d 59–69 blessed; no other attention entry moves
• pre-commit's black (25.12.0) and clang-format (18) hooks, run by hand on every file this PR changes: clean (pre-commit itself is not installed locally)
• GPU numeric lanes not run for this PR; the byte-identity gate is the parity check

🤖 Generated with Claude Code

[CongMa13]

One actionable correctness issue found in an accepted schedule combination; the inline comment identifies the trigger and failure mechanism.

[CongMa13]

This accepts use_staggered_iter_wait with use_early_v_schedule. On a multi-tile input, the early-V branch issues next-K after QK, while the staggered iter-start wait assumes K was issued before V; the wait can leave current K incomplete when QK reads it. Please reject this combination here and keep the Python spec validation aligned.

[cgmillette]

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14525 — port the rocKE gfx950 tiled-2D V double-buffer, D256 fast path and softmax interleave to C++ (amd-khushbu, AICK-2712) — 🟠 First-round: not merge-ready — rebase needed — conflicts inherited from #14523/#14524; CongMa13's early-V + staggered-wait race is real in both engines

Prior item	Status
CongMa13 (10-28) public_entry_glue.cpp:462: staggered wait accepted with early-V; QK can read an in-flight K	🟠 OPEN, valid — no reply. • The race is real, and it exists in both engines. Python's __post_init__ also admits the pair (gfx950/attention_tiled_2d.py:646). Its comment puts early-V out of scope for the staggered wait, but the code checks only grouped-KV2 and FP8. The glue ports exactly those two checks (:460). • We built three schedules with the Python builder and traced their KV loops (ran: probe). With early-V, each iteration issues V[i+1] before QK and K[i+1] after it. The next iteration's start wait (vmcnt = kv_calls_per_tile) therefore leaves the newest loads, K[i+1], in flight while QK reads them. Without early-V the issue order is K then V, so the wait drains K. Without the stagger, the start wait is a full drain. The C++ port emits the same order: the early-V branch of emit_post_qk_issue (kv_body_qk_softmax.cpp:477) takes precedence over the staggered one, and the start wait is at :628 (read). • Reach: no selector builds this spec (_enable_v_double_buffer is off whenever early-V is on), and no parity config pairs the two flags. But the gfx950 tuning space lists the three flags as independent axes (dispatch/attention/axes.py:65–:67), and its legality rules pass the combination on both the narrow and the transposed path (ran: probe). A full-space sweep can therefore build and time this kernel → reject use_staggered_iter_wait with use_early_v_schedule both in __post_init__ and at glue :460, with one shared message, and add a negative test to each engine
Bots / CI	🟡 • At head 30d8f005, TheRock Multi-Arch CI (10-28) fails only Test hipkernelprovider, on all four shards; an earlier run at this head was cancelled. Linux gfx94X and Windows gfx110X fail two tests. rocke_differential_golden_ll shows, per LLVM flavor, 13 unblessed changes (moe_fused_mega_fp8 ×5, target_intrinsics ×8) plus 1 new config. _canonical shows 4 target_intrinsics changes plus 1 new config. The Windows shards also fail the gfx1100–1103 census entries of TestGfx1151WmmaAttentionDecodePacks, all 9 cases each ("descriptor set ... did not load"). • None of these failures come from this PR. #14524's head fails the identical set with 11 fewer blessed configs per section (1493 vs 1504 at llvm20), so this PR's 11 new gfx950 2D configs (59–69) pass. Both failures are develop breaks carried by the stack base 0e1cfd3c. Develop has since fixed them in #14501 (golden re-bless) and #14553 (census descriptor root), and develop's latest completed run (9d729d32) passes all four shards. A rebase gives a fresh run (read: CI logs). • Coverage: the gfx950 test jobs were skipped, and no GPU lane ran, as the author's checklist says. CI checks only the Python side of the differential golden and never builds the pybind engine. So the C++ side of configs 59–69, and the binding and selector-parity tests, rest on the author's local byte-identity and CTest runs. ASAN, the hipDNN superbuild, clang-tidy and Component CI pass; pre-commit did not run at this head. GitHub reports MERGEABLE (state UNSTABLE) against #14524's branch, not against develop (read)
Our checks	🟠 • git merge-tree with develop d63447c9 (55 behind, 4 ahead: the three lower PRs and this commit) leaves two conflicted files, both inherited. attention_unified_selectors.cpp has 2 hunks, from #14523's clash with #14150. prove_parity_binding.py has 1 hunk: #14524 deletes the problem-driven selector matrix that #14150 extends. #14524's head produces the identical conflict set, and this PR's own selector hunk (spec_gfx950_generic) auto-merges. The golden auto-merges too: the merged llvm_gfx_all.json carries configs 59–69 in all four sections, and develop aliases llvm24 to llvm23 (ran). With no merged tree, the check script skipped the build, byte identity, goldens and tests. • Drift since the stack base: develop's only Python change in this area is #14150 in attention_unified.py, which admits bf16 to the D128 bs16 sliding-window single-batch combo. The gfx950 builder and the spec builder are unchanged. #14150 reaches this PR only through _enable_softmax_mfma_interleave, which calls _enable_single_batch_combo. Resolving #14523's conflict to develop's rule therefore carries it into the C++ interleave predicate, and nothing else needs porting (ran: git; read). • Parity (read): the port follows the Python builder branch for branch. That covers:   – the V double-buffer: the K-then-V prologue into slot 0, the post-QK issue order (K first when staggered), the PV wait that leaves 2×kv_calls_per_tile loads pending, and all three PV v_buf sites;   – the staggered start wait, and early-V on the double buffer;   – the direct-to-register Q gather;   – the K-slab pad: its drop conditions, the padded DMA strides and the read remap;   – the mode-2 group math, and the mask phase split (split point, _b iter-arg names);   – all 20 D256 overrides, applied last, and the interleave predicate;   – the kernel-name tokens and the four new validator blocks, message for message. LDS: both engines lay V_lds out as [2, T, HD] and the padded K_lds as [K slots, T/slab, slab rows × HD + pad]. Neither builder budgets LDS; the tuning rules do (unified_rules.py:273). The transposed-QK fp8 fix matches Python, which reads that K through the shared dequantizing reader. The C and Python parity configs 59–69 match field for field (ran: text probe). • Docs: known_gaps.md §8 keeps the old table after a stray separator row (:303–:307). So it still lists the V double-buffer as selected and rejected, and the staggered wait as unported. The right-bound phase-split gap that the PR body cites is not recorded there (read). • Rules 1–3 clean; no links or tracker IDs added (ran)

#	Sev	file:line	Finding
N1	🟡	gfx950/attention_tiled_2d_public_entry_glue.cpp:833, tests/parity/gfx950_attention_tiled_2d_emit.py:1147, dispatch/attention/unified_rules.py:313	Both validators accept the K-slab pad with Q staged in K_lds, a combination the tuning rules already refuse as wrong output. The staging store indexes the slab-shaped K_lds by (row, col), so in config 67 the 64 Q rows span about four times the [1×16×520] K_lds. The pad's read remap also covers only the transposed QK; in both engines the 16x16 and non-transposed reads index K_lds unpadded. The binding refused the pad before this PR. Now the C++ builds such specs, and the new config 67 (pad without direct-register Q) pins one: Python accepts that spec, and the tuning rule refuses it (ran: probe; C++ read) → in both validators, refuse use_kq_lds_pad when Q would be staged in K_lds, as the tuning rule does; that also rules out the unremapped reads. Give config 67 direct-register Q, like configs 64, 65 and 68

Utility: 🟢 Positive — brings three gfx950 prefill schedules to C++ parity; inherits two validator gaps.

---

Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

First-round review: a local qwen3.6 model made the independent first pass, and the orchestrator (Claude Opus 5.5) verified it. The PR conflicts with develop, so there is no merged tree yet; the build, byte identity, goldens and test runs wait for the rebase. The orchestrator simulated the merge with git merge-tree for this head and for #14524's head, and compared the differential golden at the base, at head, on develop and in the merge result. It read the ported C++ against the Python builder, its validator, the spec builder and the selectors. It used the Python builder to build three V-schedule variants and trace each KV loop's wait and issue order, and it checked the K-slab pad configuration against the tuning rules. It compared the C and Python parity configs, and it read the four failing CI logs alongside #14524's and develop's latest run. The orchestrator owns this merge-worthiness call. This review flags whether a closer look is needed; it is not an approval or a rejection.

[cgmillette]

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14525 — port the rocKE gfx950 tiled-2D V double-buffer, D256 fast path and softmax interleave to C++ (amd-khushbu, AICK-2712) — 🟠 Follow-up: rebase fixed the conflicts; CongMa13's early-V race is still open

Prior item	Status
CongMa13 (10-28) public_entry_glue.cpp:462: staggered wait accepted with early-V; QK can read an in-flight K	🟠 OPEN, valid — unchanged, no reply. The 10-29 push only rebased this commit (same patch-id as 30d8f005), and neither engine refuses the pair. The C++ gate (public_entry_glue.cpp:460) and Python's __post_init__ (gfx950/attention_tiled_2d.py:646, same on develop) still check only the V double-buffer, grouped-KV2 and FP8, although Python's comment already puts early-V out of scope. • On the merge, a D128 spec with all three flags (config 63's shape) builds in both engines and lowers to byte-identical IR. The same gate refuses the stagger without the V double-buffer, so it is live (ran: probe). • Its KV loop runs: start wait vmcnt=8 → V[i+1] DMA → QK reads K → K[i+1] DMA → PV wait vmcnt=16 → PV. The next start wait therefore leaves K[i+1], the newest eight loads, in flight while QK reads it (ran: IR trace). • No selector builds the pair (_enable_v_double_buffer is off under early-V, attention_unified.py:2089). But the gfx950 tuning space has the three flags as independent axes (dispatch/attention/axes.py:65–:67), and its legality rule passes the pair on both the narrow and the transposed path (ran) → reject use_staggered_iter_wait with use_early_v_schedule in __post_init__ and at glue :460, with one shared message, and add a negative test to each engine
Our 10-29 checks 🟠: rebase needed — conflicts inherited from #14523 (attention_unified_selectors.cpp) and #14524 (prove_parity_binding.py)	🟢 FIXED by the 10-29 18:19Z rebase of the stack onto develop 4a3fc208, which contains #14150. This commit's parent 208a0d04 is #14524's current head, so the base is carried unchanged. The merge with develop 6a27c440 (34 behind, 4 ahead) has no conflicts. #14150's bs16 sliding-window rule now sits in the C++ single-batch combo (attention_unified_selectors.cpp:204), and for that cohort the C++ spec builder sets the interleave exactly as Python does, in bf16 and fp16 (ran: git, probe)
Our 10-29 N1 🟡: both validators accept the K-slab pad with Q staged in K_lds (config 67)	🟡 OPEN — unchanged, no reply. On the merge, config 67 (gfx950_attention_tiled_2d_emit.py:1147: pad without direct-register Q) builds in Python and in C++, while _gfx950_2d_legality refuses it as "padded K LDS does not support aliased Q (wrong output)" (unified_rules.py:314) (ran: probe; the first round only read the C++ side). The C++ pad block (public_entry_glue.cpp:833) still drops the pad only for fp8 QK, unsupported geometry or K double-buffering (read) → refuse use_kq_lds_pad with K_lds-staged Q in both validators, and give config 67 direct-register Q
Our 10-29 note: known_gaps.md §8 keeps the old table after a stray separator row; the right-bound phase-split gap is unrecorded	🟡 Unchanged — known_gaps.md:303–:307 still list the V double-buffer as selected and rejected and the staggered wait as unported, and the right-bound gap the PR body cites is still not recorded (read)
Bots / CI	🟡 At head 8c8e4429 (121 check runs: 42 passed, 62 skipped, 10 cancelled, 7 failed): • TheRock Multi-Arch CI fails only on the two Windows gfx110X Test hipkernelprovider shards. Both failed in setup: choco install ninja got HTTP 429 (Too Many Requests) from the Chocolatey feed, so no test ran → re-run both shards (read: job logs). • Linux gfx94X 1/2 and 2/2 pass all 49 ctests, rocke_golden_static and both differential goldens included (0 unblessed changes). The develop breaks that failed all four shards at 30d8f005 are gone on Linux; the new base carries develop's fixes (#14501, #14553). • The other failed checks (a second Multi-Arch summary and the ASAN, clang-tidy and hipDNN Superbuild summaries) belong to duplicate runs cancelled within 20 s of the push. Their twin runs pass, as does Component CI. • Coverage is unchanged: gfx950 and gfx125X built but ran no test shard, the differential goldens run --python-only, and the pybind engine is never built. So the PR's three engine-backed test files and the native routing ctest do not run in CI (read: Linux logs). • GitHub: MERGEABLE (UNSTABLE) into #14524's branch; no approval (CongMa13 commented); a rocKE-lib review is requested; no reply or new review since our first round (read: gh)
Our checks	🟢 • Range-diff vs 30d8f005: this commit is unchanged (same patch-id, 19 files, +1055/−225); only #14522–#14524 changed, and they are re-checked on their own PRs. The first round's branch-for-branch parity read therefore still holds (ran: git). • Merge 96723c16 = head + develop 6a27c440: native build OK; ctest 37/37 (1 skipped), the routing and selector ctests included; byte identity GREEN at llvm20/22/23 for 10 families, gfx950_attention_tiled_2d with all 70 configs (59–69 included); representative golden PASS ×3; differential golden OK at llvm20–24 and canonical, with 0 changed and 0 missing. All 70 gfx950 2D configs are blessed in all four sections, so the NEW (unblessed) configs belong to other families (ran). • Selector parity with the merge's engine: on a 1296-problem gfx950 grid (fp16/bf16 × D64/128/256 × bs16/32/64 × 1 or 4 seqs × q 1/512/2048 × SW 0/256 × no bias/ALiBi/QQ × 2 head ratios), the C++ spec from problem matches Python's on every shared field. The grid takes the V double-buffer 36 times, the interleave 120 times and the D256 fast path 24 times (ran: probe). • Tests with the merge's engine: 162 passed, 1 skipped, 2 failed. Both fail the same way on develop: the graph-replay case needs torch, which our host lacks, and TestKvAddrLimit trips on develop's unclassified gfx950 use_additive_bias (ran). • Compile (ROCm clang 24, -O3): no failures and no new rejections. The one gfx950 2D spill predates this PR: the 59-config merges of #14523 and #14524 report it too (read: our checks on those PRs). gfx942 3D cfg 2 overflows LDS on develop too (ran). • Land order: fourth, after #14522 (approved by CongMa13; needs a rocKE-lib review), #14523 (no approval) and #14524 (CongMa13's approval predates the rebase). • rule_scan clean; no links, tracker IDs or handles added (ran)

Net-new: none.

Utility: 🟢 Positive — C++ now builds gfx950's production prefill specs byte-identically; untested on gfx950 hardware.

---

Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

Follow-up review: a local qwen3.6 model made the independent first pass, and the orchestrator (Claude Opus 5.5) verified it. The checks started with a range-diff of the rebased stack against the head we reviewed (30d8f005). On the merge with develop 6a27c440, they ran the native build and ctest, byte identity at three LLVM flavors, both goldens, the PR's test files with the merge's rocke_engine, and a ROCm clang 24 compile. Probes built CongMa13's flag combination and config 67 in both engines, traced the KV loop's waits and issue order, checked both against the tuning rules, and compared the C++ and Python specs from problem on a 1296-problem grid. The CI logs at head (both Windows shards, both Linux shards and the summaries) and the reviews and threads since our first round were read. The orchestrator owns this merge-worthiness call. This review flags whether a closer look is needed; it is not an approval or a rejection.

[amd-khushbu]

This stack is now combined into one PR, #14522, so it can be reviewed and merged as one change. GitHub shows this PR as merged because its commit is now on #14522's branch; nothing has gone into develop. This PR's change is one commit there, unchanged. Open review comments from here are listed in #14522's description and will be answered there.