Repository navigation
feat(hip-kernel-provider): expand gfx1151 inference SDPA coverage - #12710
Merged
Merged
Conversation
github-actions
Bot
added
documentation
project: hip-kernel-provider
rocke-lib: attention
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
Contributor
PR #12710 — gfx1151 inference SDPA coverage (dpeabody-amd, AICK-1847) — 🟢 Positive; restore deleted guards firstPrior reviews: none human (0 threads). 55 files, +9835/−492.
Carried: pre-existing Utility: 🟢 Positive — broad gfx1151 coverage AOTriton can't match, ~1.36x matched geomean, byte-identical Py/C++ Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good |
Contributor
Author
|
Thanks @cgmillette. Addressed in
Also removed in the same commit: the AOTriton comparison bridge, the remote Slurm launcher, Validation on this head:
|
Pre-commit check failed⛔ pre-commit failed Please run locally:
This repo uses |
yraparti
approved these changes
Oct 5, 2026
dpeabody-amd
force-pushed
the
users/dpeabody/gfx1151-sdpa
branch
from
October 6, 2026 04:17
d7c083b to
57683a7
Compare
BradPepersAMD
approved these changes
Oct 14, 2026
BrianHarrisonAMD
approved these changes
Oct 14, 2026
AnaghaRaoAMD
added a commit
that referenced
this pull request
Oct 14, 2026
Conflicts in mfma_attention.py and sage_attention.py: develop (#12710) already passes the query and key positions to extra_score_transform as positional arguments. Take develop's form for the score hook and keep only epilogue_fn from this branch. The wave32 body now has a Q-length bound, so epilogue_fn receives it as row_valid there. Co-Authored-By: Claude <noreply@anthropic.com>
8 of 9 tasks
dpeabody-amd
added a commit
that referenced
this pull request
Oct 14, 2026
Rebasing onto develop (which now carries #12710 and the widened memory capabilities) left the new gfx12-generic row incomplete: - arch_specs.json: add memory.async_lds_max_dwords (0, as gfx1201); the C++ row used the old 3-field memory initializer, which no longer compiles against the 10-field rocke_memory_caps_t. - intrinsic_arch_domain.llvm{20,22,23}.json: measure the gfx12-generic cells of develop's new maxnum.f64/minnum.f64 rows (ok on roc-7.1.0 clang 20, ROCm 7.2.0 clang 22 and core-7.13 clang 23). - generic_targets is now the single gfx11-generic membership list: target.py reads it instead of its own copy, and gfx1154 (in develop's list) is added to the JSON and the C++ mirror so both engines map it to gfx11-generic. The C++/Python identity probe covers gfx1154/gfx1200. - Known-arch expectations (test_gfx1250_foundation, wgrad_streamk_host) include gfx12-generic.
dpeabody-amd
added a commit
that referenced
this pull request
Oct 15, 2026
Gfx1151WmmaAttentionNative ranks the 32-key single-wave transposed geometry first, which #12710 measured slower than 64-key two-wave for processor-specific gfx1151 objects. The matcher cannot see an object's compile target, so an IngestorGenerator test now requires every gfx1151_wmma_attention kernel to be target: generic, and the score comment names it.
3 tasks done
dpeabody-amd
added a commit
that referenced
this pull request
Oct 16, 2026
…eneric) (#13150) Builds on #12710 (merged as `198636dae1d`); this branch is rebased onto `develop` and contains only its own commits. Turns the gfx1151 SDPA kernels into one kernel set per RDNA generation: `gfx11-generic` code objects for gfx1100, gfx1101, gfx1102, gfx1103, gfx1150, gfx1151, gfx1152 and gfx1153, and `gfx12-generic` code objects for gfx1200 and gfx1201. It then widens what hipDNN can route to them without adding per-shape kernels. ## Does rocKE need to learn generic targets first? Only for gfx12. rocKE IR never names a processor: the LLVM module carries the plain `amdgcn-amd-amdhsa` triple, and only the comgr ISA name selects the processor. `gfx11-generic` was already a catalog row with ISA backends in both engines. What was gfx1151-specific was in the SDPA layer: string gates (`arch == "gfx1151"`, `arch == "gfx1201"`), dispatch arches, CU lookup and packaging. This PR: 1. **rocKE targets** (`feat(rocke): catalog gfx12-generic ...`) - Adds `gfx12-generic` to both catalogs and both backend registries. - Regenerates the llvm20/22/23 intrinsic arch-domain tables. The new column equals gfx1201 cell for cell. The regeneration also records the previously unswept `raw.ptr.buffer.load.f16` row. - Adds `generic_targets` to `arch_specs.json`, plus `generic_arch_from_target_id()` / `rocke_generic_arch_from_target_id()`, mapping device names to generic targets. `base_arch_from_target_id()` keeps processors that have their own catalog row, so `arch="gfx1151"` and `arch="gfx1201"` still build processor-specific code; other members (gfx1100, gfx1200) resolve to their generic row. 2. **SDPA kernels and dispatch** (`feat(rocke): run RDNA WMMA attention as gfx11/gfx12 generic kernel sets`) - The WMMA atom now comes from the target's MMA catalog in both engines: the split-K gfx12 atom when listed, otherwise the gfx11 atom. - Transposed QK relies on the gfx11 operand duplication, so it is accepted on any gfx11 row and rejected on gfx12. Every other feature runs on both generations: dtypes, D16..D256, masks, windows, score features, LSE, additive bias, dense/BHSD/ragged/paged layouts, FP8 KV, output-column tiling, and distinct V widths. - Dispatch admits `gfx11-generic`, `gfx1151`, `gfx12-generic` and `gfx1201`, and reads live CUs from device 0 when it runs the requested object (any family member for a generic request, the named processor otherwise). - The selection policy is retuned for generic objects (see Results). Processor-specific gfx1151 requests keep #12710's wide-launch and D256 tile choices. - GPU tests and the benchmark compile for the device's generic target. Parity configs 136-142 replay representative configs at `gfx11-generic`; their golden hashes equal the replayed gfx1151 configs. 3. **hip-kernel-provider** (`feat(hip-kernel-provider): package RDNA WMMA attention as generic objects`) - rocKE UKDs accept `kernel_source.target: generic`. The packer compiles for the shard's generic processor, and shards stay concrete. - The compile target is recorded in observations and provenance, and the agreement check binds the two. - The IngestorGenerator expresses the target and per-group arch lists. - The catalog now ships to all 10 shards, with transposed-QK variants on gfx11 shards only. The census covers every shard. - The standard variants carry `scheduler_strategy: max-ilp`, the spec public dispatch selects for the same requests, so packaged objects get the generic-object tuning too. - The native score now prefers the 32-key single-wave transposed geometry. 4. **BSHD long-sequence fix** (`perf(rocke): decode heads fastest for head-interleaved dense attention`) - With head-interleaved K/V (BSHD, as from PyTorch's `view(B,S,H,D).transpose(1,2)`), the q-tile-fastest CTA order put one head's rows, a full token apart, in flight. On gfx1151 this lost cache and translation locality at long sequences. - Dense kernels now choose the CTA decode at runtime from the strides: heads fastest when K is head-interleaved, q-tiles fastest otherwise. The grid, ABI and object count are unchanged. 5. **Causal mask plus `window_left`** (`feat(rocke): accept a causal mask with an explicit window_left`): the cuDNN spelling of a causal sliding window now runs as the window `(window_left, 0)` instead of being rejected. 6. **Runtime head dims** (`feat(rocke): runtime head dims for RDNA WMMA attention`) - An opt-in `runtime_head_dims` spec flag lets one object serve every head width up to its compile-time `head_size` (a bucket maximum). Q/K and V/O widths (multiples of 16) and a V head count independent of K's arrive as kernel arguments. - Dense objects read K/V through bounded buffer resources at constant tile offsets, which keeps register pressure at the exact-size kernel's level. Dead Q/K tiles contribute exact zeros; dead V/output columns are never stored. - With the flag off, every existing config emits unchanged IR. The attention ABI is bumped to v6. 7. **Wider hipDNN routing** (`feat(hip-kernel-provider): route more SDPA graphs to RDNA WMMA attention`) - The pack now matches any Q/K and V/O head width that is a multiple of 16 up to 256, and V head counts dividing the query heads. - It also matches an additive FP32 or Q-dtype attention bias (`attn_mask`) with broadcast strides, and every hipDNN scale form: a value, a pass-by-value tensor (constant, or read at launch), or absent = 1.0. - Catalog per dtype: - runtime-head-dims buckets 64/96/128: no-mask, band, band+bias f32/q; - buckets 160/256: no-mask, causal, window, window+bias f32/q; - exact 192 and 256: no-mask, causal, window, kept where the exact kernel is measurably faster; - the 32-key transposed objects on gfx11. - That is **64 objects per gfx11 shard and 56 per gfx12 shard**. Previously, 40 objects covered only D64/96/128/256 with no bias. - Scores prefer transposed, then exact, then the tightest bucket, then mask-specialized objects. The 64-key transposed objects were dropped because the score never selected them. 8. **Tuned registry candidates** (`feat(rocke): register RDNA WMMA attention as tuned registry candidates`) - Each RDNA arch gets an opt-in tuned candidate built with `make_tuned_candidate`, like the gfx950/gfx942 dense candidates: `attention_<arch>_wmma_fwd`, pinned by algorithm `wmma_fmha_fwd` and spec_id `<arch>_wmma_fwd`. Tuning ids and `tuning_knobs` round-trip. - `wmma_rules.py` and `_AXES[(arch, "wmma")]` expose `v_lds_stage`, `causal_tile_skip`, `value_tile_size` and `scheduler_strategy`, plus `block_n` × `num_waves` for transposed QK on gfx11 only. `WmmaFmhaFwdSpec.resolved_scheduler_strategy()` names the unset scheduler so the knob can be projected into metadata. - `AttentionRequest.runtime_head_dims` selects the bucket objects. Only the RDNA WMMA candidates accept it, and it skips the fp16 D64 small-head `v_lds_stage` heuristic so the default spec matches the shipped bucket objects. - The existing `attention_gfx1151_wmma` candidate stays the public auto route and shares the policy and knob space. Public dispatch is unchanged for 29,940 requests, and IR is unchanged for all 1,648 WMMA specs and 163 parity configs. - `KernelId.request_hash` changes for every attention request because `normalized()` now includes `runtime_head_dims` (the same precedent as `emit_lse`); `spec_hash` is unchanged. - With #13593's `registry_enumerator` (`space: slice`, `num_cus` fixed), gfx11-generic and gfx12-generic manifests regenerate exactly the shipped 64 and 56 kernel specs and metadata (names and UUIDs ignored). The full knob space is 238 / 206 instances. The manifests stay out of tree until #13593 lands and the generic-target model is settled with #13129. ## Toolchain, runtime and hardware coverage - **Compile:** LLVM 20 or newer comgr (rocKE's oldest flavor). The LLVM 20 clang of ROCm 7.1 accepts `gfx11-generic`, `gfx12-generic`, `gfx1152` and `gfx1153`. Generic objects never need the member's name, so an older toolchain that cannot name gfx1152/gfx1153 still builds them. - **Load:** ROCm 7.0 or newer. ROCm 7.0.2 (gfx1100), 7.2.0, core 7.13 and core 10.0 load the generic objects. ROCm 6.4.0 on a gfx1151 rejects them (`hipErrorNoBinaryForGpu`), while it loads a code object v6 processor-specific gfx1151 object from the same toolchain. No gfx12 device with a runtime older than 7.2 was available. - **Hardware:** gfx1100, gfx1101, gfx1150, gfx1151, gfx1200 and gfx1201 were run. gfx1102, gfx1103, gfx1152 and gfx1153 are covered only by the LLVM generic-processor definition and were not run. ## Validation - **Device numerics** (rocKE GPU suites, generic objects, current head): | Device | Passed | Skipped | |---|---:|---:| | gfx1100 | 182 | 0 | | gfx1151 | 182 | 0 | | gfx1201 | 174 | 8 (transposed-QK tests) | The suites include the runtime-head-dims tests: - d16-d240 within buckets, dq != dv, and Hv in {1, Hk, Hq}; - bias with masks; - NaN-padded inputs proving no padded column is read into or written from the result; - rows ending exactly at the allocation end. An earlier round also ran gfx1101, gfx1150 and gfx1200 (134/134/126 before the new tests). - **Benchmark correctness:** the 54-case inference benchmark passes 54/54 against its FP64 oracle on every device, in every timed run. - **Code objects:** `gfx11-generic` objects are code object v6, machine 0x054, generic version 1. `gfx12-generic` objects are code object v6, machine 0x059. Both load on every device listed. - **C++/Python parity:** byte-identity is GREEN for all 73 families at llvm20, llvm22 and llvm23; `gfx1151_wmma_fmha_fwd` now has 157 configs (14 runtime-head-dims configs added). The native arch-identity CTest matches Python. - **Provider end-to-end** (gfx1100;gfx1151;gfx1201 shards, current catalog): - Census: 15/15. - `TestGfx1151WmmaAttentionMatchers`: 25/25. - hipDNN `quick_SdpaFwd*` integration bundles: **21 passed** (GPU reference), 20 skipped, 0 failed on each of gfx1100, gfx1151 and gfx1201. #12710's gfx1151-only build gives 17 passed and 24 skipped. - **Python suites:** rocKE CPU suites, IngestorGenerator (862 passed) and descriptor-packaging on a ROCm node (512 passed) all pass. The descriptor-packaging run has one failure, `test_enabled_user_site_runtime_preserved`, which fails identically on #12710's tree in the same cluster Python environment. - pre-commit passes. Golden hashes: `run_diff.py --check-golden` reports unblessed changes in this environment on `develop` itself (638 `ll`, 542 `ir_canonical` at `341b0b606e1`). The WMMA families' entries are re-recorded for the changed configs and report 0 unblessed. ## Results ### Selection policy (generic objects) Scope: relative timing of this selection policy against #12710's selection policy, measured before the BSHD decode and runtime-head-dims commits (both arms then shared the same kernels). The baseline is #12710's gfx1151-tuned profiles applied to the same kernels, with transposed QK limited to gfx11. Workload: the 54-case `benchmarks/gfx1151/attention` inference suite through public dispatch. Method: - Both arms compile for the device's generic target unless noted, and both arms pass the oracle gate in every run. - Two runs per arm, in ABBA order, in one exclusive allocation per device. - Each run times HIP-graph replays of 32 launches and takes the median of 7 batches. - Reported: the geomean of per-case ratios (this / #12710; lower is faster), with the per-case range. One device per processor. | Device | Target | Geomean | Per case | |---|---|---:|---:| | gfx1100 | gfx11-generic | 0.84 | 0.43-1.03 | | gfx1101 | gfx11-generic | 0.82 | 0.37-1.02 | | gfx1150 | gfx11-generic | 0.87 | 0.39-1.08 | | gfx1151 | gfx11-generic | 0.85 | 0.39-1.16 | | gfx1151 | gfx1151 | 0.90 | 0.49-1.03 | | gfx1200 | gfx12-generic | 0.69 | 0.32-1.03 | | gfx1201 | gfx12-generic | 0.63 | 0.29-1.08 | On gfx1151, `gfx11-generic` objects with this policy measured 0.90 (0.48-1.05) against processor-specific gfx1151 objects with #12710's policy, and 0.99 (0.71-1.07) against processor-specific objects with this policy. With #12710's policy alone, generic objects measured 1.06 (0.78-2.26) against processor-specific ones. The gap traced to scheduling under the generic register model, and the `max-ilp` scheduler closes it. Caveats: - Results hold at these measured configurations only, for inference forward attention. - The largest gains are on standard-path cases (tails, packed layouts, score features, decode, and every dense case on gfx12). - On gfx11, aligned dense transposed-QK cases change little apart from wide launches. ### BSHD long sequences B1 H16 bf16 noncausal, S=8192, dense, through public dispatch; median of 5×10 launches. | Device | Kernel | BSHD before | BSHD after | BHSD | |---|---|---:|---:|---:| | gfx1151 | d64 transposed QK | 82-96 ms | 29-42 ms | 15 ms | | gfx1151 | d64 standard | 97 ms | 26 ms | 21 ms | | gfx1151 | d128 transposed QK | 196-201 ms | 151-165 ms | 24 ms | | gfx1100 | d128 transposed QK | 23 ms | 15 ms | 22 ms | BHSD timings and gfx1201 are unchanged within noise. The remaining d128 gap on gfx1151 is not CTA ordering. At d128 with 16 heads each K/V row is exactly 4 KiB, so each 32-row tile touches 32 pages. My inference is that this exhausts the APU's address translation; discrete gfx1100 does not show it. ### Runtime head dims Timing of each runtime-head-dims object against the exact-size object for the same head width (lower is faster), bf16 noncausal, prefill B1 H16 S2048 and decode B8 Hq32 Hkv8 Sq1 Sk4096. Outputs match the exact object bit for bit. | d | Shipped object | gfx1151 prefill | gfx1151 decode | gfx12 prefill | gfx12 decode | |---:|---|---:|---:|---:|---:| | 64 | bucket 64 | 0.94 | 0.93 | 0.99 | 0.96 | | 80 | bucket 96 | 1.12 | 1.02 | 0.98 | 1.06 | | 96 | bucket 96 | 1.02 | 0.97 | 0.93 | 1.01 | | 128 | bucket 128 | 0.80 | 0.85 | 0.91 | 0.99 | | 160 | bucket 160 | 0.84 | 0.81 | 1.16 | 0.96 | | 192 | exact 192 (a 192 bucket would be 1.65 / 2.20 / 0.85 / 0.79) | 1.00 | 1.00 | 1.00 | 1.00 | | 256 | exact 256 (bucket 256: 1.56 / 1.44 / 1.07 / 1.06) | 1.00 | 1.00 | 1.00 | 1.00 | gfx12 columns: gfx1201 up to d128, gfx1200 from d160. These measurements chose the catalog. Up to 128, a bucket is within 2% of the exact kernel, or faster, when d equals the bucket; the one in-bucket width measured, d80 on bucket 96, costs 0.98-1.12×. Bucket 160 is faster than exact on gfx1151 and costs up to 1.16× on gfx12 prefill. Above that, gfx11 kernels sit at the register limit, so exact 192/256 objects stay; they are 1.4-2.2× faster than a runtime bucket on gfx1151. Widths with no exact object (176, 208, 224, 240) run on bucket 256 instead of being declined. Not done here: - A gfx12 transposed-QK variant (the gfx12 standard path already serves those shapes). - Split-KV decode for small batches with long contexts. - The gfx1151 d128 BSHD translation gap above. - Renaming the `gfx1151` module, candidate and pack identifiers (e.g. `hipkernel:Gfx1151WmmaAttention`, which now ships to every RDNA3/3.5/4 shard). - Deriving `TestGfx1151WmmaAttentionMatchers`' standard-catalog selection test from the shipped descriptor instead of its own copy of the catalog. ## Rebase onto develop and third-party review (head `e06957a02bd`) The seven feature commits are replayed onto `develop` (`e581a923f2b`), which already carries #12710 including its review round 2. Conflict resolutions keep develop's #12710 behavior: - The FP8 vector-alignment and byte-extent guards in the gfx1151 binding (`_check_vector_aligned`, `_check_byte_extent_i32`) stay in force with runtime head dims. - `emit_lse` replaces `return_lse` everywhere. - The RDNA census registers each wired arch only when `product` ships `hipkernel:Gfx1151WmmaAttention` for it (`hkp_product_ships_engine_for_arch()`). An independent review of the rebased diff found these issues, now fixed: - `91718c676b9`: the gfx12-generic catalog row didn't match develop's widened memory capabilities. `arch_specs.json` lacked `async_lds_max_dwords`, so `get_target("gfx12-generic")` raised `KeyError`, and the C++ row used the old 3-field initializer, which no longer compiles. Other gaps fixed in the same commit: - The gfx12-generic cells of develop's new `maxnum.f64`/`minnum.f64` arch-domain rows were measured on clang 20 (ROCm 7.1.0), 22 (7.2.0) and 23 (core-7.13); all are `ok`. - `generic_targets` is now the single gfx11-generic membership list. `target.py` reads it, and gfx1154 (already in develop's Python list) is mirrored to C++. - Known-arch expectations include gfx12-generic. - `e430c2da6ba`: a causal mask with `window_left=-1` resolved to an unbounded two-sided window and ran unmasked. The causal diagonal is now kept, with a regression test. - `b5bd11bf9e8` / `e06957a02bd`: the numeric tests use `emit_lse`, and the representative IR goldens are regenerated. Validation on this head: - Windows CPU lanes, rocKE `library/tests` + `platform/tests` (no GPU): the failure set is **identical to develop's** (comgr, `ml_dtypes` and Windows-only environment failures). The full-lane comparison (3,790 passed vs develop's 3,765) ran before the causal-bound fix and the llvm23 arch-domain cells; the dispatch, arch-identity and gfx1250-foundation suites re-run on the final head pass (719 passed, 0 failed). - Arch-domain coverage test passes for the llvm23 column; the llvm20 host-flavor gap for develop's `cluster.*`/prefetch keys exists on develop too. - `run_diff --check-golden`, C engine built from develop and from this head on Alola. The `ll` comparison used `ROCKE_LLVM_FLAVOR=llvm20`, the flavor the golden was blessed with. - No anchor that passes on develop fails here: `ll` passes 253 on develop and 280 here; `ir_canonical` passes 315 on develop and 336 here. - All 157 `gfx1151_wmma_fmha_fwd` configs match their goldens in both modes, and the 6 `gfx1201_wmma_fmha_fwd` anchors that are stale on develop now pass. - The C-vs-Python `ir` DRIFT families are identical on develop. - `pre-commit run --from-ref origin/develop --to-ref HEAD` passes. - Not re-run on this head: GPU numerics, the hipDNN integration bundles, and the 54-case benchmark (the device results above are from the pre-rebase head). This round's code changes are the catalog row, the causal window bound and test wiring. The emitted kernels are unchanged: all 157 `gfx1151_wmma_fmha_fwd` goldens match. ## Independent review, round 2 (head `feb7b7d7ee4`) Fresh review of the full diff against merge base `97fc1832835`, covering the arch catalog, kernel emission (both engines), dispatch, and hip-kernel-provider/IngestorGenerator. Fixed in this PR: - `34f8b95fe6e`: C++ `rocke_generic_arch_alias` kept its own gfx11 member list beside `rocke_ati_generic_targets`. It now reads the table, so `generic_targets` is the only membership list in both engines. Behavior unchanged; the native `rocke_arch_target_identity` binary passes and `test_cpp_target_identity_matches_python` passes against it (43/43, Linux build). - `feb7b7d7ee4`: `_resolve_num_cus` lent a sibling's CU count to a processor-specific request (`arch="gfx1151"` on a gfx1100 host took gfx1100's count for the output-tiling gate). Generic requests still read any family member, but processor requests now require that processor on device 0. A regression subtest fails before the fix and passes after it. The README now says only device 0 is read, and that 64-column D256 noncausal tiles apply only to gfx11-generic objects. Reviewed with no defect found: the gfx12-generic rows (JSON vs C++, field by field; arch-domain column equals gfx1201 in all three tables); runtime-head-dims memory safety (dead Q/K tiles zeroed after load, so NaN padding cannot leak; bounded rsrc extents within the i32 byte-extent guard); BSHD decode bijection; the kernarg order vs the native `kernelSignature()`/`launchWith()`; matcher legality vs the binder; yaml vs checked-in kdp.json (identical expansion). Fresh validation on `0db16e5c758` (new clones and venvs; the code changes since then are host-side selection/arch-alias code that GPU tests don't exercise): | Device | ROCm / LLVM | fmha numeric | dispatch numeric | fp8 decode | benchmark | |---|---|---:|---:|---:|---:| | gfx1100 | 7.2.0 / llvm22 | 41 | 141 (+27 subtests) | 4 | 54/54 | | gfx1100 | 7.0.2 / llvm20 | 41 | 141 (+27) | 4 | 54/54 | | gfx1150 | 7.2.0 / llvm22 | 41 | 141 (+27) | 4 | 54/54 | | gfx1201 | 7.2.0 / llvm22 | 33 (+8 transposed-QK skips) | 141 (+27) | 4 | 54/54 | - No gfx1151 node was usable: one was drained, and the other has ROCm 6.4 only. - Descriptor-packaging on gfx1100: `test_hkp_pack_rocke.py` passes 54/54 (base: 40/40). The full suite gives 618 passed and 1 failed. The failure is `test_enabled_user_site_runtime_preserved`; it fails identically on the base because the cluster system Python contains a nameless `argcomplete` dist-info. - Windows CPU, rocKE `library/tests/dispatch` + `platform/tests/core` + spec/multiarch: the failure set (14) is identical to the base, with 1925 passed vs 1891. - IngestorGenerator: 873 passed. - Golden changes are confined to `gfx1151_wmma_fmha_fwd` (126 per mode) and 6 previously stale `gfx1201_wmma_fmha_fwd` `ll` anchors. - pre-commit passes on the changed files. Left for follow-ups (not this PR): the remaining items under "Not done here"; device-0-only CU lookup (multi-GPU hosts with a mixed device 0); the descriptor-packaging probe's robustness to nameless dist-info. ## Review round 3 (AnaghaRaoAMD), head `601a18c94fa` - `ef7698617c1`: `gfx1200` requests resolve to `gfx12-generic`; before this, no RDNA candidate matched them. Both engines fold any `generic_targets` member that has no row of its own. The processor-specific live-CU check compares base archs. - `289619c0d15`, `601a18c94fa`: the WMMA GPU suites run from `rocke_library_gpu_pytest`, and the CPU spec, selection and tuning tests from `rocke_library_attention_pytest`, which is labelled for CI `quick` selection. - `90660821818`: on-device `causal + window_left=-1` test. - `2934d00bfc4`: an IngestorGenerator test keeps the WMMA catalog `target: generic`, which the native score assumes. - `bc71331224e`: the engine declares graph schema 1.2.0, so runtime pass-by-value scale graphs reach the matcher; before this, every `SdpaFwdRuntimeScale` graph was declined. Override-shape graphs are declined. Validation: - rocKE GPU suites (`2934d00bfc4`, from fresh clones): fmha 41 (33 + 8 transposed-QK skips on gfx12), dispatch 143, fp8 decode 4, benchmark 54/54, on gfx1100 (llvm22 and llvm20), gfx1150, gfx1201 and gfx1200. - hipDNN forward SDPA, all tiers (engine pinned, GPU reference; `bc71331224e` code): 88 passed / 14 skipped / 0 failed on gfx1100, gfx1201 and gfx1200. The skips are FP8 or varlen graphs. - gfx120X Multi-Arch CI dispatch on `601a18c94fa` (https://github.com/ROCm/rocm-libraries/actions/runs/38698822336): `hipkernelprovider` and `hipdnn-integration-tests` pass on Linux gfx1200 and Windows gfx120X. The WMMA suites ran on the gfx1200 runner.
Contributor
TheRock Submodule Bump ActivityNewest first
|
CongMa13
added a commit
that referenced
this pull request
Oct 17, 2026
…12441) ISSUE ID : AICK-2273 # summary ## What this branch is about rocke's emitter had no way to know whether an intrinsic it declares actually exists on the architecture it is lowering for. An unknown `llvm.amdgcn.*` declare is not a parse error, not a verifier error, and not a `clang -S` error — it survives every check rocke ran and fails much later, in the backend, as `Cannot select`. Worse, the same name can be present at one LLVM flavor and absent at another, and present on one arch and absent on its neighbour. #12269 put the measurement on develop: a generator that probes the toolchain, an emitted-IR validity gate that compiles *and links* every corpus case, and three committed columns of answers (one per LLVM flavor). Nothing read them. This branch is their first consumer, plus the fixes that building a consumer forced: 1. **A reader and a warning lane** — the lowerer asks the table whenever a kernel demands an intrinsic, on both backends. 2. **Removal of the implicit `gfx950` target default** everywhere it had spread: both engines, the per-family bindings and the C wrappers, so every lowering names the target it is actually for. This is what lets the lane mean anything. 3. **A corrected table and sturdier gates**: reading the columns as a consumer showed that one of them was wrong, and that the generator and the corpus gates each measured less than they reported. Plus a long series of methodology fixes, all of which draw one line: > **A non-answer must never be recorded as a negative.** > "We could not ask" is not "the answer is no". The lane is warn-only. Making it an error is a separate change (AICK-2274). The commit list includes commits that #12269 already merged by squash; the diff against `develop` is what this PR changes. --- ## 1. The warning lane **Reader** `python/rocke/core/arch/domain.py` — validates the envelope and raises on a malformed or wrong-schema column rather than defaulting every field (which would have presented a corrupt artifact as a complete absence of data and switched the lane off in silence — the one failure the lane exists to prevent). A column that exists but cannot be read raises too; only a *missing* file means "no data". **What it reads** — `python/rocke/core/arch/data/intrinsic_arch_domain.<flavor>.json`, schema `rocke.intrinsic_arch_domain/v1`, **171 keys × 7 wired arches = 1197 cells per column**: | Flavor | `ok` | `name_absent` | `arch_absent` | no-data | |---|---|---|---|---| | llvm20 | 535 | 294 | 194 | 174 | | llvm22 | 669 | 147 | 340 | 41 | | llvm23 | 740 | 21 | 391 | 45 | One column per flavor because **a host can only measure its own flavor** — flavor is a runtime choice (`ROCKE_LLVM_FLAVOR`), so the generator can never produce all three in one run, and a gate must skip the flavors its host cannot measure rather than mark them failing. The lowerer's declaration table has 27 more keys than the columns: the 26 declarations develop added for data prefetch, workgroup clusters and TDM (#12857), and `raw.ptr.buffer.load.f16` (#12710), have not been measured yet. An unmeasured key is no data, so the lane stays silent for those intrinsics until the columns are regenerated. That is a measurement on each of the three toolchains, and it is left to a follow-up. **Status vocabulary** splits answers from non-answers: - Answers: `ok`, `name_absent`, `arch_absent` - No data: `target_unsupported`, `toolchain_crash`, `toolchain_timeout`, `probe_error` Only `arch_absent` is negative (`is_negative`). Everything else is silence. `name_absent` is silent on purpose: it is a flavor-axis fact that holds for every target, and the validity gate already reports a reachable one with the compiler's own diagnostic. **Warning lane** — `_Lowerer._need()` consults the table and emits `ArchDomainWarning`. The default backend is the C++ engine, which returns finished IR without the Python lowerer running, so the same check also runs over its emitted `declare` lines; the declare → key mapping is exact because both engines emit the table's declares verbatim. Controlled by `ROCKE_ARCH_DOMAIN`: `warn` (default) / `off`; any other value **raises**, because every way of getting this variable wrong is somebody trying to switch the lane off, and silently defaulting to `warn` hands them the opposite of what they asked for. The value is read in `lower_kernel_via_backend`, which every lowering passes through (`lower_kernel_to_llvm`, `compile_kernel` and the per-family wrappers), so it raises for every kernel, entry point and backend rather than only for a kernel that happens to demand an intrinsic. The LLVM flavor is likewise resolved once per lowering and handed to each stage. `error` mode is planned (AICK-2274). **Gate** `tools/check_arch_domain.py` — wired into `run_all.py`. It lowers the 194-case corpus at all three committed flavors (pure Python; it needs no toolchain, so it rules on every flavor on any host) and compares the warnings with `EXPECTED_WARNINGS`, which only ever shrinks. The list holds nine entries: three intrinsics (`ds.read.tr16.b64`, `mfma.f32.16x16x32.bf16`, `mfma.f32.16x16x32.f16`) on gfx942 at each flavor. All nine are one defect in the parity harness rather than in a kernel: it builds the gfx950 3D attention kernel for two gfx942 cases. The validity gate lists the same two cases as `KNOWN_BAD`. Fixing the harness is left to the change that makes the lane binding, which needs the list empty. Each entry names the corpus cases allowed to demand it, so the list cannot widen in silence: a case an entry does not name is unexpected even when its intrinsic is already allowlisted, and an entry whose case stops demanding it, or no longer exists, is stale. A case that did not run (`--only`, `--arch`, `--flavor`, or an instance that declines the flavor) is judged neither way. The comparison is a pure function with a test per rule. ## 2. No implicit target default Two commits, in order: first a pure refactor making all 158 implicit lowering call sites name their arch explicitly; then removal of the default itself from both engines. Omitting the target now raises. This is what lets the warning lane mean anything — a kernel lowered at a default arch is a kernel whose arch nobody chose. The default had also spread beyond the lowerer, and the same series removes it from those places: the per-family pybind entry points (one shared `require_arch` replaces their per-family "usual target" fallbacks), the Python wrappers in `core/backend.py` that mirrored them, and the scalar-attention C wrappers below. ## 3. A corrected table and sturdier gates - **Generator 4 → 6, and one column corrected.** The generator built its probes from rocke's hand-written declaration table, which records `immarg` only where an author happened to add it, and passed the unmarked operands as variables where the compiler demands constants. llvm22 and llvm23 reject that in the verifier, so the generator's rescue recorded the true answer; llvm20 has no such check, died at ISel with `Cannot select` — which is also what a genuine negative looks like — and recorded `arch_absent`. Probes are now built from the declare `opt` resolves, so the signature comes from the LLVM being measured. 33 llvm20 cells move: 27 `arch_absent` → `ok` and 6 `arch_absent` → `toolchain_crash` (a crash is no data, so the lane stays silent). llvm22 and llvm23 move no status. Two more ways to bank a non-answer as an answer are closed: a failed name probe (`opt` crashed or timed out) was recorded as `name_absent` for every arch, and a partial `--only`/`--arch` sweep could be filed as a whole column. - **One meaning of "this flavor refuses the instance".** The arch-domain gate counted four gfx1250 instances that refuse an older LLVM as lowering errors — eight failures on every run of a clean tree — while the validity gate already treated them as unmeasurable. Both now import one predicate from `tools/_flavor_gate.py`, and a test asserts they hold the same object. - **Provenance.** The validity gate no longer trusts an `opt` from another install: an older `opt` rejecting a construct its parser predates reads as "rocke emitted invalid IR" when the IR is fine, so that lane is skipped on a mismatch. The generator's identity check covers `opt` as well as `clang`. - **The arch-domain gate lowers through the native lowerer.** Through the dispatching entry point, any host with the C++ engine built would lower every case without consulting the table, harvest nothing, and call every allowlist entry stale. ## 4. What testing this turned up **The scalar-attention C wrappers.** The three public `_lower_to_llvm` wrappers in `attention_unified_entry.cpp` are the only lowering entries whose signature carries no target, and nothing in the tree called them. They passed `NULL`, which the lowerer now rejects, so all three had turned into `ROCKE_ERR_VALUE`. They now name `gfx950` explicitly — a fixed target, not a fallback, and the header says so; a caller who wants another target builds with the `_new` entry and lowers through `rocke_lower_kernel_to_llvm_ex`. **A crash in the C++ scalar reduce builder.** Testing the wrappers found that `rocke_build_unified_attention_reduce_scalar` died with SIGSEGV on every input: its loops read three tensor descriptors that nothing had built. It now builds them, in the order the Python builder does, and emits the kernel the Python builder emits (byte-identical). A new ctest lowers each wrapper and requires text identical to lowering the same kernel through the target-taking entry. **Bindings that ignored their target.** A sweep over every per-family binding that takes a `(spec, arch)` found six that discarded `arch` (layernorm2d, elementwise and the scalar attention family's serialize/verify entry points), two fused-MoE e2e entry points that hard-coded gfx950 into the spec they built, and a ninth that kept a pybind default. All now call `require_arch` first. The sweep is a test now: an extension-gated check reads every `(spec, arch)` binding off the module and requires each to reject an empty target, so a binding added later is held to the same rule. **Bindings that arrived with develop.** Three stream-K wgrad host-plan bindings (`conv_wgrad_is_valid_spec`, `conv_wgrad_streamk_default_pool` and `conv_wgrad_streamk_host`, from #13151) resolved their target with `arch_or_default`, the helper this branch removed, and declared `arch` with a gfx950 default. `rocke_engine.cpp` stopped compiling, so the pybind extension could not be built; CI builds no extension, so nothing turned red. They now take the target like every other binding, and the sweep covers the two that take a spec (pybind prints `arch: str = 'gfx950'` for a defaulted argument, which is why the sweep could not see a default before). One decision to look at: develop's stream-K tests pinned "without an arch both engines default to gfx950" for the host planner. The Python planning helpers keep their default; the C++ binding now refuses an empty target, and the test asserts that. The same thing happened a second time with the gfx950 block-scaled GEMM bindings that arrived with #13450 (`gfx950_scaled_serialize_ir`, `gfx950_scaled_verify` and `gfx950_scaled_lower_llvm`: four uses of `arch_or_default`), and `rocke_engine.cpp` again stopped compiling without anything turning red. They call `require_arch` first now. Because the extension-gated sweep cannot run where no extension is built, a second test reads `cpp/bindings` and fails on either way a binding guesses its target: the removed `arch_or_default` helper, or a gfx target as the default of an `arch` argument. It lists every offending line, and it fails on the previous merge without the fix. It reads the source, so it runs from a source checkout and skips in an installed layout. ## 5. Review Every review thread has a fix, one commit per concern: - **Tracker IDs out of code** (Rule 1): four places cited IDs; each now names the defect. The allowlist's reason string was also wrong about the defect — it blamed two declaration-table items, but the nine warnings come from the parity harness — and now says so. - **Trust boundaries** (Rule 4): a malformed or wrong-schema column, an unreadable column and an unknown `ROCKE_ARCH_DOMAIN` value each raise, instead of reading as "no data" or defaulting to `warn`. - **One flavor per lowering**: with no flavor given, the C++ path and the arch-domain post-pass each resolved their own, and compiler discovery is retryable, so the IR could be checked against the wrong flavor's column. It is now resolved once in the dispatcher, including for both engines under `ROCKE_BACKEND=both`. - **Tests lower at the arch they are about**: three substitutions of an explicit `arch=` had put the wrong arch in, and `test_legacy_equals_gfx950`, which compared a call with itself, is replaced by `test_omitting_the_target_raises`. - **`develop` merged four times** since the first review (111 commits, then 99, 29 and 14). Only the first conflicted textually (four binding translation units and `test_compiler_detection.py`). The trouble in all four was semantic: develop's new code was written against the default target this PR removes, so it called the lowerer, or a binding, without naming one. Each occurrence is fixed here: the compiler-detection tests (three of the calls sit inside a subprocess script string, and they had turned the gfx94X CI shards red) and a binding-selection probe; the flag-attribute test that turned the hipkernelprovider jobs red (twelve subtests), a library gate test that CI does not run, and the FP64 elementwise and reduce tests; and the stream-K and block-scaled GEMM bindings in section 4. A scan of the Python and C++ that the merges brought in finds no other call that names no target. - **Formatting after the merge**: this branch's parenthesized `with` statements make black treat `test_rocke.py` as Python 3.9+, so develop's three multi-line `with a, b, c:` blocks in that file now need reformatting. Each version of the file is black-clean alone; the merge is not, and the pre-commit job checks every file a PR changes. - **The allowlist names its cases** (see the gate in section 1): it was keyed by intrinsic, target and flavor alone, so a second corpus case demanding an allowlisted intrinsic left the gate green, and fixing one of the two cases that demand it could not make the entry stale. - **Wording**: the status vocabulary in `TESTING.md` lists `probe_error`, durations are out of the generator's comments, and the documentation examples name the target. --- ## Verification Run in the three toolchain images (one per LLVM flavor) against an export of the branch tip; no GPU. The pybind extension is built in each image and put on `PYTHONPATH`, so the tests that need it run (CI builds no extension and skips them). | Check | llvm20 | llvm22 | llvm23 | |---|---|---|---| | byte identity, every family | GREEN | GREEN | GREEN | | ctest | 20/20 | 20/20 | 20/20 | | emitted-IR validity (194 cases) | GREEN (2 documented `KNOWN_BAD`, 8 unvalidated) | GREEN (2 documented `KNOWN_BAD`, 4 unvalidated) | GREEN (2 documented `KNOWN_BAD`) | | arch-domain corpus gate (194 cases, 3 flavors) | GREEN (exactly the 9 documented warnings) | same | same | | platform pytest, default and under `ROCKE_BACKEND=both` | 2685 passed, 59 failed (both runs) | 2689 passed, 59 failed (both runs) | 2690 passed, 59 failed (both runs) | | library pytest | 2598 passed, 1 failed | 2599 passed, 1 failed | 2599 passed, 1 failed | Every failure is also a failure of `develop` itself, checked on an export of its tip (59 in the platform suite at every flavor, one in the library suite). The failing set is the same at the three flavors: - 56 tests import `ml_dtypes`, which none of the images has: six in `core/test_gfx1250_fp4_scaled_wmma.py` and fifty in `instances/test_gfx950_mxfp_examples.py` (new with #13450); - two tests in `core/test_arch_domain_artifact.py` (every declaration is covered; regenerating the host's column is a no-op) fail because of the 27 unmeasured declarations in section 1. That file is excluded from the installed test tree, so CI does not run it; - `tools/test_run_all.py::test_existing_engine_archive_is_reused_by_pytest` fails whenever pytest runs under `tests/run_all.py`, which exports the variable the test expects to be unset; - `library/tests/test_kda_hostpack.py::test_runner_geometry_without_gpu` builds a stub kernel without the `cluster_dims` attribute that #12857 made the manifest code read. The key #13450 added to the three columns (`mfma.scale.f32.32x32x64.f8f6f4.v8i32.v8i32`) was re-measured in each image with this branch's generator: every cell, status and evidence, matches the committed one. A note on the llvm20 run. At llvm20, `raw.ptr.buffer.load.async.lds` runs past the 60 s probe budget on six targets (the committed column records `toolchain_timeout` for them), and the generator's timeout kills the `clang` driver but not the `clang -cc1` it started, so each regeneration (including the regeneration test above) leaves twelve runaway compiler processes behind. In a two-CPU container they starve the rest of the suite: an earlier full run timed out `test_binding_compiler_selection.py::test_resolver_failure_releases_binding_builders` (valgrind, a fixed 120 s limit), which passes in about 15 s alone, on this branch and on `develop`. The numbers above are from a run that killed the leftover probes as they appeared. The leak is in the generator already on `develop`, not in this PR, and is left to a follow-up. Not run: a GPU, Windows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
somayeh-rahimipour
added a commit
to somayeh-rahimipour/rocm-libraries-fork
that referenced
this pull request
Oct 18, 2026
…ion_opt Upstream ROCm#12710 (AICK-1847) landed its own gfx1151 attention dispatch module and kept the wmma_fmha_fwd optimization campaign's A/B scaffolding, which an earlier cleanup commit on this branch had deleted. Per the manager's rescope -- limit the cleanup to what Qwen3-8B prefill and decode need -- every one of those files is restored from develop, so this branch is now a strict superset of develop: it adds files and modifies docs, and deletes nothing. Conflict resolutions: * dispatch/attention/gfx1151.py (add/add) -- one module holding both candidate factories. Develop's _make_wmma_fwd_candidate keeps the gfx1151 auto route; _make_swapqk_candidate registers alongside it, opt-in only, under its own SWAPQK_ABI so it does not claim develop's v6 fwd ABI string. register() registers both. dispatch/attention/__init__.py needs no change -- develop already imports and registers this module. * dispatch/attention/bindings.py -- develop's fwd launcher infrastructure and our bind_swapqk_torch are disjoint; both kept. * core/ir.py, core/lower_llvm.py -- same buffer_load_f16_d16 op documented twice. Took develop's PR9710 provenance, kept our mechanism note, took develop's emit formatting, kept our buffer_load_d16_pack / vmcnt0_fence. * helpers/tensor_view.py -- develop added f64 to the load_vec dtype branch, we added temporal_hint; global_load_vN takes the hint dtype-agnostically. * test_builder_signature_contract.py -- develop gave the fwd spec's num_query_heads a default; took that, kept our two entries. * README.md -- kept ours per the user's call. Corrected the three claims the restore invalidated: the campaign scaffolding is present, not deleted; the dispatcher did have a gfx1151 entry; the auto route is attention_gfx1151_wmma, not attention_unified_2d. test_gfx1151_swapqk_wiring.py asserted the auto route was attention_unified_2d. The invariant that matters is that swapqk does not take the auto route without being asked for, which is what it now asserts -- who owns that route is not this module's call.
dpeabody-amd
added a commit
that referenced
this pull request
Oct 18, 2026
…ode kernels (#13895) ISSUE ID : AICK-1847 Continuation of #12710 (gfx1151 WMMA SDPA) and #13150 (generic gfx11/gfx12 SDPA catalog). This PR is the rocKE half; the hipDNN packaging half is draft #13897. ## Summary Adds two rocKE RDNA SDPA forward kernels and routes the rocKE Python attention dispatcher to them. The kernel from #13150 runs one wave per workgroup with no K/V sharing and no split-KV decode; tuning its knobs recovered only 10-30% of the gap to the fastest RDNA attention we could measure, so this change replaces the kernel structure: - **`wmma_fmha_fwd_v2` (prefill):** one query row per lane (S^T orientation), several waves per workgroup sharing K/V tiles through LDS, separate masked and unmasked tile loops, causal tile skipping. - **`wmma_fmha_decode` (decode / speculative decode):** split-KV with GQA packing (the query heads of a K/V head share 16-row tiles), an f32 workspace and a merge kernel (LSE combine). `build_wmma_fmha_decode_module` defines both kernels in one code object. Constraints kept from #13150: one generic object set per family (`gfx11-generic`, `gfx12-generic`), nothing specialised per SKU, and every shape value runtime (batch, sequence lengths, head counts including a separate V head count, strides, window bounds, scales). Both kernels carry every feature of the kernel they replace: additive bias (f32 or Q dtype), LSE, causal/window/bottom-right masks, softcap, attention sinks, ALiBi, QQ-bias, FP8 (OCP e4m3) K/V with k/v scales, ragged (varlen) and paged K/V, runtime head-dim buckets. Each has a C mirror that emits byte-identical LLVM IR, parity pairs and golden anchors. ## What changes - **Kernels and helpers:** `kernels/gfx1151/wmma_fmha_fwd_v2.py`, `kernels/gfx1151/wmma_fmha_decode.py`, `helpers/rdna_attention.py`, their C mirrors (`cpp/instances/gfx1151/`, `cpp/helpers/rdna_attention.cpp`), C host helpers for the decode launch plan, parity pairs, goldens. - **rocKE compile:** `KernelModule` / `compile_module` link several kernels into one code object through comgr (used by the decode module builder). The C builder gains `rocke_b_buffer_load_vN`. - **Python dispatcher:** RDNA attention routes to `attention_rdna_wmma_decode` or `attention_rdna_wmma_prefill` through `rdna_attention_use_decode` (a pure function of the problem, WMMA family and CU count). Opt-in tuned candidates expose the new kernels' knobs. ABI tag moves to v7. - **Kept unchanged:** the previous single-wave `wmma_fmha_fwd` kernel (and its gfx11 transposed-QK body, C mirror, parity pairs, tests, goldens). rocKE dispatch no longer selects it, but the shipped hipDNN catalog `hipkernel:Gfx1151WmmaAttention` still builds from it; the packaging PR moves the catalog and removes it. No hipDNN, hip-kernel-provider native or IngestorGenerator file changes here. - llvm20 arch-domain cells for gfx12-generic FP8 conversions and split waits, measured against the compiler (they were copies of gfx11's). ### Behaviour and API changes reviewers should know about - Dispatcher names: the RDNA routes, tuned candidates, `spec_id`s and `algorithm` pins are renamed (`wmma_fmha_fwd_v2`, `wmma_fmha_decode`); `wmma_fmha_fwd` pins no longer resolve through the dispatcher (the kernel itself remains callable). - `softmax_scale` must be finite and > 0 for the RDNA kernels; the rocKE binding raises `ValueError` otherwise (the new kernels apply the scale after the row max). ## Validation All on the PR content; failures not listed also fail on develop. - Python/C IR identity: `python tools/check_byte_identity.py` on llvm20, llvm22, llvm23: GREEN for every family, including `gfx1151_wmma_fmha_fwd_v2` (170 configs), `gfx1151_wmma_fmha_decode` (128), the kept `gfx1151_wmma_fmha_fwd` and the new `buffer_load_vN` pair. - Binding parity: `python platform/cpp/bindings/prove_parity_binding.py` (llvm20, llvm22): prefill v2 170/170, decode 128/128. - Goldens: `python platform/tests/instances/differential/run_diff.py --check-golden` (ll and canonical IR): every existing family's anchors unchanged; new families fully anchored; unblessed counts equal develop's. - rocKE GPU suites on gfx1100 and gfx1201 (`test_wmma_fmha_fwd_v2_{numeric,features}.py`, `test_wmma_fmha_decode_numeric.py`, `tests/dispatch/attention/test_gfx1151_attention_dispatch_numeric.py`, and the kept `test_wmma_fmha_fwd_numeric.py`): 610 passed, 0 failed (rest are arch-gated skips). Includes -inf leading-tile bias, odd-Sk Q-dtype bias, NaN-poisoned decode workspace with empty splits, 128K context, fully masked rows, every tuned knob value. - Host suites: `ctest -R "rocke_library_attention_pytest|rocke_wmma_fmha_decode_host"`; `platform/tests/core/test_compile_module.py` with real COMGR on Linux 15/15. - hipDNN superbuild with the CI configuration (clang-tidy on, `-DHIPDNN_ENABLE_KERNEL_INGESTOR=ON`, rocKE on): builds cleanly and still packages the catalog from the kept kernel; `TestGfx1151WmmaAttention*` pass. - Multi-arch CI dispatched for the families default PR CI skips (Linux gfx110X/gfx1151/gfx120X, Windows gfx1151/gfx120X): `hipkernelprovider`, `hipdnn`, `hipdnn-integration-tests`, `hipdnn-samples` and `hipdnn_install` passed on every family except one unrelated MIOpen convolution sample timeout on Linux gfx1151. - pre-commit over the range: clean. ## ASIC coverage | Family | Tested on | Not tested | |---|---|---| | gfx11-generic | gfx1100, gfx1101, gfx1150 | gfx1102, gfx1103, gfx1151, gfx1152, gfx1153 | | gfx12-generic | gfx1201 | gfx1200 | The objects are generic per family. The dispatched multi-arch CI jobs listed above also pass on the gfx1151 and gfx120X runners. ## Performance Relative kernel time over 97 real-model attention workloads (LLM prefill and decode, chunked prefill, speculative decode, diffusion/ViT/Whisper attention), fp16 and bf16, 194 cases per GPU, every case correctness-checked. Hot = back-to-back graph replay; cold = caches flushed before every launch. Geometric mean per workload class, measured through the rocKE dispatcher. **Against the kernels on develop (#13150 catalog):** | vs current develop catalog | gfx1100 hot | gfx1100 cold | gfx1101 hot | gfx1101 cold | gfx1150 hot | gfx1150 cold | gfx1201 hot | gfx1201 cold | |---|---:|---:|---:|---:|---:|---:|---:|---:| | LLM prefill | 2.45x faster | 2.48x faster | 2.48x faster | 2.41x faster | 3.11x faster | 3.19x faster | 2.58x faster | 2.62x faster | | chunked prefill | 2.09x faster | 2.10x faster | 2.10x faster | 2.04x faster | 2.92x faster | 3.03x faster | 2.72x faster | 2.84x faster | | non-LLM | 2.28x faster | 1.96x faster | 2.79x faster | 1.96x faster | 2.16x faster | 2.24x faster | 2.62x faster | 2.24x faster | | decode b1 | 29.41x faster | 10.31x faster | 23.81x faster | 6.13x faster | 3.92x faster | 3.82x faster | 15.87x faster | 6.94x faster | | decode batched | 4.15x faster | 2.33x faster | 3.26x faster | 1.90x faster | 2.29x faster | 2.28x faster | 2.17x faster | 1.56x faster | | specdec | 4.05x faster | 2.43x faster | 3.64x faster | 2.02x faster | 2.21x faster | 2.15x faster | 2.39x faster | 1.77x faster | | **all 194 cases** | 3.83x faster | 2.70x faster | 3.52x faster | 2.33x faster | 2.68x faster | 2.71x faster | 2.93x faster | 2.34x faster | **Against the fastest FlyDSL RDNA SDPA engine per case (#13713):** | vs best FlyDSL RDNA engine | gfx1100 hot | gfx1100 cold | gfx1101 hot | gfx1101 cold | gfx1150 hot | gfx1150 cold | gfx1201 hot | gfx1201 cold | |---|---:|---:|---:|---:|---:|---:|---:|---:| | LLM prefill | 1.32x faster | 1.30x faster | 1.29x faster | 1.26x faster | 1.37x faster | 1.31x faster | 1.10x faster | 1.09x faster | | chunked prefill | 1.40x faster | 1.40x faster | 1.32x faster | 1.32x faster | 1.49x faster | 1.38x faster | 1.09x faster | 1.08x faster | | non-LLM | 1.27x faster | 1.19x faster | 1.34x faster | 1.13x faster | 1.27x faster | 1.22x faster | 1.05x faster | 1.02x faster | | decode b1 | 1.06x faster | 1.17x faster | 1.55x faster | 1.09x faster | 1.14x faster | 1.02x faster | 1.12x faster | 1.11x faster | | decode batched | 1.05x faster | 1.02x slower | 1.11x faster | 1.04x faster | parity | parity | 1.01x faster | 1.04x slower | | specdec | 1.04x faster | 1.03x faster | 1.21x faster | 1.12x faster | 1.09x faster | 1.06x faster | 1.10x faster | 1.01x faster | | **all 194 cases** | 1.18x faster | 1.15x faster | 1.25x faster | 1.15x faster | 1.19x faster | 1.14x faster | 1.06x faster | 1.03x faster | Largest remaining losses to FlyDSL: Gemma-3 D256 batched decode on gfx1201 and gfx1150 (about 1.2-1.3x slower), and some batch-1 decode under cold cache (up to about 1.4x slower on gfx1100, 2.4x on one gfx1150 SWA case). ## Follow-ups - hipDNN packaging (catalog on the new kernels, companion merge kernel, removal of the kept kernel): draft #13897, waiting on the UHD runtime PRs and the kernel-slots work (RFC 0021). - Decode D256 batched on gfx12 and cold-cache batch-1 decode are the remaining gaps to FlyDSL.
This was referenced Oct 19, 2026
cgmillette
added a commit
that referenced
this pull request
Oct 20, 2026
The differential golden had gone stale: 627 of its 1240 LLVM IR entries and 537 of its 1206 canonical IR entries no longer matched the tree, because nothing ran the check. Recorded afresh in the new format, with the Python reference emitters, from a tree whose two-engine gate is green for all 82 families: - ll at llvm20, llvm22 and llvm23: 82 families, 1518 configs per flavor; 1615 at llvm23, where gfx1250_lowering has configs that need LLVM 23 and are refused below it. - ir_canonical: 81 families, 1598 configs (one family's emitter has no IR mode). Canonical IR was checked to be identical at llvm20 and llvm23. What the re-bless absorbs: every mismatch that could be traced is an intended code generation change that did not re-bless this file -- the shared LDS pool (#8844, about 425 configs), the ds.swizzle declaration (#9583, about 177), renamed values in the attention transpose reads (#9133, 57), the smaller LDS pool (#10112, about 39), the flat work group size of one-wave wave32 kernels (#12710, about 39), #9220 (6) and the conv changes of #13109 (6). 441 entries were already stale at the start of the history available here and cannot be attributed; no unexplained change was found after it. The two repeated keys (conv_direct_grouped, conv_wgrad_workspace_reduce), which hid their newer copies, are gone. It also takes in the code generation changes merged since then, each traced by recording the file at the commit and at its parent: - the magic division sum emitted without signed-overflow poison (#13802): 215 configs in each of the four sections, in 19 families -- attention_unified 6, attention_unified_stress 32, conv_direct_grouped 16, conv_implicit_gemm 26, conv_implicit_gemm_dgrad 31, conv_implicit_gemm_wgrad 42, deep_fused_conv_pool 2, gfx1201_deep_fused_conv_pool 5, gfx950_deep_fused_conv_pool 5, dynamic_helpers 2, fmha_splitkv_decode 6, img2col 5, moe_fused_mega 5, moe_gemm_fused 6, moe_sorting 6, pooling 4, sage_attention 4, sparse_attention 6, transpose 6; - attention decoding on gfx942 and gfx950 (#13492): 23 changed and 25 new configs per section in attention_strided_kv, gfx942_attention_tiled_3d and gfx950_attention_tiled_3d; - wave_all on wave32 (#13896): 18 gfx1151_wmma_fmha_fwd configs per LLVM flavor, canonical IR unchanged; - new configs: the barrier_fence family (#13575, 10) and one conv_implicit_gemm_dgrad config (#13782). Co-Authored-By: Claude <noreply@anthropic.com>
1 task done
cgmillette
added a commit
that referenced
this pull request
Oct 26, 2026
The differential golden had gone stale, because nothing ran the check: 675 of its 1116 LLVM IR entries (compared at llvm20) and 608 of its 1089 canonical IR entries no longer match the tree. Recorded afresh in the new format, with the Python reference emitters, from a tree whose two-engine gate is green at llvm20, llvm22 and llvm23: - ll at llvm20 and llvm22: 82 families, 1474 configs per flavor. At llvm23: 83 families, 1578 configs, where gfx1250_lowering has configs that need LLVM 23 and are refused below it, and every conv_direct_dgrad_wmma config needs LLVM 23. - ir_canonical: 82 families, 1561 configs (conv_abi's emitter has no IR mode). Canonical IR was checked to be identical at llvm20 and llvm23. What the re-bless absorbs: every mismatch that could be traced is an intended code generation change that did not re-bless this file -- the shared LDS pool (#8844, about 425 configs), the ds.swizzle declaration (#9583, about 177), renamed values in the attention transpose reads (#9133, 57), the smaller LDS pool (#10112, about 39), the flat work group size of one-wave wave32 kernels (#12710, about 39), #9220 (6) and the conv changes of #13109 (6). 441 entries were already stale at the start of the history available here and cannot be attributed; no unexplained change was found after it. The two repeated keys (conv_direct_grouped, conv_wgrad_workspace_reduce), which hid their newer copies, are gone. It also takes in the code generation changes merged since then, each traced by recording the file at the commit and at its parent; the count is per section unless a flavor is named: - the magic division sum emitted without signed-overflow poison (#13802): 215 configs in 19 families -- attention_unified 6, attention_unified_stress 32, conv_direct_grouped 16, conv_implicit_gemm 26, conv_implicit_gemm_dgrad 31, conv_implicit_gemm_wgrad 42, deep_fused_conv_pool 2, gfx1201_deep_fused_conv_pool 5, gfx950_deep_fused_conv_pool 5, dynamic_helpers 2, fmha_splitkv_decode 6, img2col 5, moe_fused_mega 5, moe_gemm_fused 6, moe_sorting 6, pooling 4, sage_attention 4, sparse_attention 6, transpose 6; - attention decoding on gfx942 and gfx950 (#13492): 23 changed and 25 new configs in attention_strided_kv, gfx942_attention_tiled_3d and gfx950_attention_tiled_3d; - the f64 follow-ups (#13887): 3 changed and 1 new elementwise config, 5 changed reduce configs; - MX scale and fp6 decoding (#13885): 6 mx_gemm configs; - the gfx950 conv dgrad speed-up (#13075): 36 new conv_direct_dgrad_ext configs; 22 changed, 15 new and 2 removed conv_implicit_gemm_dgrad configs; 2 changed and 3 new conv_direct_grouped configs; 2 new dynamic_helpers and 3 new target_intrinsics configs; - removing the ds.read.tr16.b128 declaration (#14054): gfx1250_lowering, 100 changed and 2 removed configs at llvm23 and in canonical IR, 5 changed, 2 new and 4 removed at llvm20 and llvm22; - optional LSE output on unified split-KV attention (#13955): 18 gfx942_attention_tiled_3d and 16 gfx950_attention_tiled_3d configs per LLVM flavor, canonical IR unchanged; - the running softmax max in 3D split-KV (#14276): 14 attention_strided_kv, 18 gfx942_attention_tiled_3d and 16 gfx950_attention_tiled_3d configs; - depthwise 7x7 and grouped wgrad on batched MFMA (#14157): 7 changed, 10 new and 1 removed conv_direct_grouped configs; - the retirement of wmma_fmha_fwd (#13897): the gfx1151_wmma_fmha_fwd (157 configs) and gfx1201_wmma_fmha_fwd (12) families are gone; - new configs, every existing hash unchanged: the barrier_fence family (#13575, 10), one conv_implicit_gemm_dgrad config (#13782), 14 conv_direct_grouped configs for the split-C and grouped LDS-halo kernels (#13993), one config each in fmha_splitkv_decode, sage_attention and sparse_attention (#13876), 23 conv_direct_grouped configs for the direct wgrad kernels (#13990), 2 conv_implicit_gemm_dgrad configs for patch-embedding dgrad (#13988), 6 f64_mma configs (#13794), 5 conv_direct_grouped and 7 conv_implicit_gemm configs for depthwise and grouped conv forward (#13987), and the conv_direct_dgrad_wmma family (7 configs, llvm23 and canonical IR only, #14156). #13885, #13955 and #14276 updated the old file's entries for their changes; the new llvm20 and canonical IR sections reproduce every one of those hashes. Co-Authored-By: Claude <noreply@anthropic.com>
cgmillette
added a commit
that referenced
this pull request
Oct 26, 2026
The differential golden had gone stale, because nothing ran the check: 675 of its 1116 LLVM IR entries (compared at llvm20) and 608 of its 1089 canonical IR entries no longer match the tree. Recorded afresh in the new format, with the Python reference emitters, from a tree whose two-engine gate is green at llvm20, llvm22 and llvm23: - ll at llvm20 and llvm22: 82 families, 1474 configs per flavor. At llvm23: 83 families, 1578 configs, where gfx1250_lowering has configs that need LLVM 23 and are refused below it, and every conv_direct_dgrad_wmma config needs LLVM 23. - ir_canonical: 82 families, 1561 configs (conv_abi's emitter has no IR mode). Canonical IR was checked to be identical at llvm20 and llvm23. What the re-bless absorbs: every mismatch that could be traced is an intended code generation change that did not re-bless this file -- the shared LDS pool (#8844, about 425 configs), the ds.swizzle declaration (#9583, about 177), renamed values in the attention transpose reads (#9133, 57), the smaller LDS pool (#10112, about 39), the flat work group size of one-wave wave32 kernels (#12710, about 39), #9220 (6) and the conv changes of #13109 (6). 441 entries were already stale at the start of the history available here and cannot be attributed; no unexplained change was found after it. The two repeated keys (conv_direct_grouped, conv_wgrad_workspace_reduce), which hid their newer copies, are gone. It also takes in the code generation changes merged since then, each traced by recording the file at the commit and at its parent; the count is per section unless a flavor is named: - the magic division sum emitted without signed-overflow poison (#13802): 215 configs in 19 families -- attention_unified 6, attention_unified_stress 32, conv_direct_grouped 16, conv_implicit_gemm 26, conv_implicit_gemm_dgrad 31, conv_implicit_gemm_wgrad 42, deep_fused_conv_pool 2, gfx1201_deep_fused_conv_pool 5, gfx950_deep_fused_conv_pool 5, dynamic_helpers 2, fmha_splitkv_decode 6, img2col 5, moe_fused_mega 5, moe_gemm_fused 6, moe_sorting 6, pooling 4, sage_attention 4, sparse_attention 6, transpose 6; - attention decoding on gfx942 and gfx950 (#13492): 23 changed and 25 new configs in attention_strided_kv, gfx942_attention_tiled_3d and gfx950_attention_tiled_3d; - the f64 follow-ups (#13887): 3 changed and 1 new elementwise config, 5 changed reduce configs; - MX scale and fp6 decoding (#13885): 6 mx_gemm configs; - the gfx950 conv dgrad speed-up (#13075): 36 new conv_direct_dgrad_ext configs; 22 changed, 15 new and 2 removed conv_implicit_gemm_dgrad configs; 2 changed and 3 new conv_direct_grouped configs; 2 new dynamic_helpers and 3 new target_intrinsics configs; - removing the ds.read.tr16.b128 declaration (#14054): gfx1250_lowering, 100 changed and 2 removed configs at llvm23 and in canonical IR, 5 changed, 2 new and 4 removed at llvm20 and llvm22; - optional LSE output on unified split-KV attention (#13955): 18 gfx942_attention_tiled_3d and 16 gfx950_attention_tiled_3d configs per LLVM flavor, canonical IR unchanged; - the running softmax max in 3D split-KV (#14276): 14 attention_strided_kv, 18 gfx942_attention_tiled_3d and 16 gfx950_attention_tiled_3d configs; - depthwise 7x7 and grouped wgrad on batched MFMA (#14157): 7 changed, 10 new and 1 removed conv_direct_grouped configs; - the retirement of wmma_fmha_fwd (#13897): the gfx1151_wmma_fmha_fwd (157 configs) and gfx1201_wmma_fmha_fwd (12) families are gone; - new configs, every existing hash unchanged: the barrier_fence family (#13575, 10), one conv_implicit_gemm_dgrad config (#13782), 14 conv_direct_grouped configs for the split-C and grouped LDS-halo kernels (#13993), one config each in fmha_splitkv_decode, sage_attention and sparse_attention (#13876), 23 conv_direct_grouped configs for the direct wgrad kernels (#13990), 2 conv_implicit_gemm_dgrad configs for patch-embedding dgrad (#13988), 6 f64_mma configs (#13794), 5 conv_direct_grouped and 7 conv_implicit_gemm configs for depthwise and grouped conv forward (#13987), and the conv_direct_dgrad_wmma family (7 configs, llvm23 and canonical IR only, #14156). #13885, #13955 and #14276 updated the old file's entries for their changes; the new llvm20 and canonical IR sections reproduce every one of those hashes. Co-Authored-By: Claude <noreply@anthropic.com>
cgmillette
added a commit
that referenced
this pull request
Oct 27, 2026
…13974) ## Motivation The differential golden (`tests/instances/differential/golden/llvm_gfx_all.json`) is the only tripwire for an output change in about 55 parity families that the representative golden does not cover (norms, reduce, transpose, fmha, wmma GEMMs, flatmm, mx_gemm, ...). The byte-identity gate cannot see a change made the same way in both engines; this file can. But nothing ran its check, and on develop 627 of its 1240 LLVM IR entries and 537 of its 1206 canonical IR entries no longer matched. The checker itself made it easy to go stale: - Entries carried no LLVM flavor, so blesses from hosts of different flavors mixed and no flavor matched the file. - A re-bless limited with `--only` replaced the whole section; one did, and the pasted-back copy left two repeated JSON keys whose older copies silently won. - A check limited with `--only` reported every other family missing. - The check needed the C++ build although it compares Python reference hashes only. ISSUE ID : AICK-2654 ## Technical Details Three commits. 1. **The checker** (`fix`) - New `golden_store.py` owns the file (schema 2): LLVM IR hashes in one section per flavor (`llvm20`, `llvm22`, `llvm23`), canonical IR in one section (produced before lowering; identical across flavors, checked). Loading refuses repeated keys and the old format. A filtered record replaces only the families it ran; a filtered check compares only those. - `run_diff.py` pins the flavor for every golden run (`--flavor`, else `ROCKE_LLVM_FLAVOR`, else the host's) and exports it to both emitters. - `--python-only` records or checks from the Python reference emitters alone: no C++ engine, any flavor on any host, a list of flavors in one run. It produces the same hashes as the two-engine run (all 82 families, checked). - The canonical hash has one definition (`python_batch_worker.canonical_digest`), shared by the batch worker, which now computes it in parallel, and `run_diff.py`. - `run_diff.py` finds the rocke and library packages in an installed test tree too. 2. **The re-bless** (`test`, data only): recorded afresh at llvm20, llvm22, llvm23 and canonical IR from a tree whose two-engine gate is green for all 82 families (re-recorded after rebasing onto develop with #13802, whose magic-division change moved 215 configs in 19 families in every section). Every mismatch that could be traced is an intended code generation change that did not re-bless this file (#8844, #9583, #9133, #10112, #12710, #9220, #13109); 441 entries were already stale at the start of the history available and cannot be attributed; no unexplained change was found after it. The commit message has the breakdown. 3. **CI** (`test`): two quick-tier CTest entries run the Python-only check from the installed test tree, `rocke_differential_golden_ll` (all three flavors) and `rocke_differential_golden_canonical`. The install ships only what that check needs from `tests/instances/differential`; the rest still needs the C++ build and stays out. The golden README and the troubleshooting entry describe the check and re-bless commands. No emitter or IR changes. ## Test Plan - `test_golden_store.py`: repeated keys refused by name; the old format refused with the way out; flavor sections separate; a full record replaces, a filtered record merges; a filtered check ignores other families; new configs alone pass. - The existing differential tests (gate verdict, scaled MFMA batch) still pass. - Python-only vs two-engine record at llvm20: identical hashes for all 82 families; canonical IR identical at llvm20 and llvm23. - The check passes on the re-blessed tree and fails (1058 changed configs) when the lowerer's output is altered. - Installed tree (provider layout, staged install): both CTest entries pass, and fail when the installed lowerer's output is altered. - Byte identity at llvm20, llvm22, llvm23; ctest; full platform suite. ## Test Result - All of the above pass. - Installed CTest time: about 15 s (LLVM IR, three flavors) and 13 s (canonical IR). - Byte identity green at llvm20, llvm22 and llvm23; ctest passes; full platform suite passes with no failures; pre-commit passes. ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
AnaghaRaoAMD
added a commit
that referenced
this pull request
Nov 3, 2026
…13596) ISSUE ID : AICK-2605 ## Summary Adds a rocKE request type that represents every field of a hipDNN SDPA forward node, a resolver for hipDNN's mask attributes, and a numeric reference that kernels can be checked against. `AttentionRequest` cannot carry strides, a scale source, bias, sequence-length tensors, per-tensor dtype, `h_v`, or general band bounds. ## Risk Assessment Low risk. The request type, `from_hipdnn`, `resolve_hipdnn_band`, and the request oracle are new, CPU-only Python with no existing caller. No kernel or helper code changes, so emitted IR is unchanged. ## ASIC Coverage PR CI is sufficient. The dispatch and reference code is ASIC-independent. `test_attention_body_seams.py` builds the real `fmha_mfma` kernel for gfx950, gfx942 and gfx1151 by default; the `--rocke-seam-arches` pytest option (comma-separated) narrows it to the targets a test environment selects. ## Testing Summary - `SdpaRequest` / `from_hipdnn`: features, normalization, frontend-rule rejections, malformed-input rejections (non-integer metadata and bounds, non-boolean flags, invalid ragged offsets, malformed `tuning_knobs`), `getMaskType` precedence, scale exclusivity, and a coverage-table test showing every hipDNN field changes `features()` or `normalized()`. The checked-in `SdpaFwd*` golden-bundle graphs (82) translate and round-trip, including the 9 paged ones (a paged K/V container's block count and block size are not compared with the batch and key length, as in `SdpaFwdNode`); the 3 older `*_group` RuntimeScale bundles, whose `seq_len_*` are `[B + 1]`, are pinned as rejected, as `SdpaFwdNode` now rejects them. `SdpaFwdNode`'s band domain (SEM-005, `left_bound = 0` rejected), page-table, paging-needs-seq-lens, `max_seq_len_kv`-without-paging, `seq_len_*` (INT32 `[B, 1, 1, 1]`) and `sink_token` (FLOAT `[1, H_q, 1, 1]`) rules are mirrored, each with a negative test for graphs and for direct callers. - SDPA entry points: `registered_sdpa_combos` / `sdpa_sweep_space` / `dispatch_sdpa_all` short-circuit on a request error, then delegate to `SDPA_REGISTRY`; a probe candidate registered there is reached through all three, and `dispatch_sdpa_all` returns `DispatchResult`s. - `resolve_hipdnn_band` / `hipdnn_left_to_window`: the resolver matches a transliteration of hipDNN's mask resolvers exhaustively (including SEM-005's rejection of `left_bound = 0`, checked in one place, `check_hipdnn_band`, which `SdpaRequest` also calls); `hipdnn_left_to_window(L)` is `L - 1`, and `keep()` with it keeps exactly hipDNN's `(q + off - L, q + off]` for every `L` and both alignments. - `sdpa_reference_from_request`: matches a scalar fp64 transliteration of hipDNN's forward loop (band from hipDNN's `isMasked`, not the shared `keep()`) on every case-table case (bias broadcast, padding, ragged, GQA with h_k != h_v, d_qk != d_v, FP8 q/k/v descale, strided flat buffers, strided sequence-length and descale buffers), and PyTorch math SDPA (the golden generator's backend) per batch. Pins hipDNN's hand-checked band cases (`L = 2` at query 5 keeps keys {4, 5}; `L = 1` keeps only the diagonal). Every fail-fast check has a negative test; unsupported features raise `NotImplementedError` by name. - Score hook: an identity `extra_score_transform` on the MFMA and WMMA forward bodies (query/key positions from #12710) yields identical IR. - Not checked: the DVC-tracked golden tensors themselves. ## Testing Checklist - [x] SDPA request, reference, and score-hook tests - `pytest library/tests/sdpa/test_sdpa_request.py library/tests/sdpa/test_sdpa_reference.py library/tests/sdpa/test_sdpa_from_hipdnn.py library/tests/sdpa/test_attention_body_seams.py` - Status: Passed (648 with CPU torch 2.15, 1 skipped: the `--rocke-require-hipdnn-assets` guard, which runs only when the option is given; without torch 644 pass and the 4 PyTorch cross-checks also skip) - [x] Whole SDPA directory (the `rocke_library_sdpa_pytest` target) plus library layering - `pytest library/tests/sdpa library/tests/test_library_layering.py -m "not gpu"` - Status: Passed (2460, none skipped, with CPU torch 2.15 and `--rocke-require-hipdnn-assets`; without torch or the option 2453 pass, 7 skip). The installed tree now carries a copy of the hipDNN graph template and the 82 `SdpaFwd*` graphs, and the CTest entry passes `--rocke-require-hipdnn-assets`, so those checks run in CI and fail rather than skip if the copy is missing (verified by replaying the CMake selection: 82 graphs, 192 `test_sdpa_from_hipdnn` tests pass against the copy) - [x] Band resolver and array-oracle platform tests - `pytest platform/tests/test_sdpa_reference.py` - Status: Passed (46) - [x] Platform entry point from the README - `PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=python:../library python tests/test_rocke.py` (from `rocke/platform/`) - Status: Passed (368 run, 132 skipped: no device or C++ engine) - [x] pre-commit over all changed files - `pre-commit run --files <PR files>` - Status: Passed (whitespace, EOF, YAML, black, bandit, cmake-lint) - [x] rocKE pre-merge checks, host stages - `python tools/run_checks.py --steps guard,gate,parity` - Status: Passed (relative-path guard; byte-identity gate on llvm20, llvm22, llvm23 and llvm24; platform instance parity) - [x] rocKE pre-merge checks, pytest stage - `python tools/run_checks.py --steps pytest --op sdpa` - Status: Passed ("rocke checks: ALL PASSED" at the new head; the pinned GPU reference suites skip on this gfx90a host, which no SDPA or CONV cohort enrols) - [x] Platform CI entrypoint - `python tests/run_all.py` (from `rocke/platform/`) - Status: guard, byte-identity, emitted-IR validity, arch-domain and native parity gates passed; pytest 5380 passed, 10 failed, 1523 skipped. The 10 failures are in `core/test_debug_info.py`, `core/test_trap.py` and `core/test_xf32_comgr_compile.py`, which this PR does not touch (identical to develop). They come from the host toolchain: the system `/usr/bin/llc` and `llvm-objdump` lack gfx950 support, and the trap test's subprocess needs `PYTHONPATH`. With `/opt/rocm/llvm/bin` on `PATH` and `PYTHONPATH=python:../library`, all of them pass (9 passed, 6 subtests) - [ ] PR CI - GitHub PR checks - Status: pending on the new head ## Technical Changes - `library/dispatch/sdpa/`: `SdpaRequest` with `SdpaTensor` operands (a ragged operand carries its offset tensor's descriptor) and frozen `SdpaBias` / `SdpaDropout` / `SdpaPaged` / `SdpaFp8` sub-objects. Construction normalizes to one spelling per meaning (an unset bound as -1, scale 1.0 unless baked, bias right-aligned to rank 4, `tuning_knobs` as sorted pairs) and rejects malformed input rather than coercing it, so `normalized()` is a stable cache key. Valid-but-unimplemented fields are features that candidates decline through `Capability.check`. `from_hipdnn()` reads hipDNN flatbuffer JSON. - `library/dispatch/sdpa/__init__.py`: `SDPA_REGISTRY`, an empty `CandidateRegistry`, behind `norm.py`-style `registered_sdpa_combos` / `sdpa_sweep_space` / `dispatch_sdpa_all`. The ABI version arrives with the first candidate's `KernelCandidate.abi_version`. - `platform/.../numeric/sdpa_reference.py`: `resolve_hipdnn_band()` applies hipDNN's mask resolvers (deprecated causal flags, stored bounds, diagonal alignment) and keeps hipDNN's (cuDNN's; SEM-001) left bound: `L` keys ending at the diagonal. `check_hipdnn_band()` is the one copy of hipDNN's band domain (SEM-005), used by the resolver, the conversion and `SdpaRequest`; `as_index()` is the strict integer read both packages share. `hipdnn_left_to_window()` is the one conversion to the FlashAttention window (`L - 1`) that `keep()` and the array oracle `sdpa_reference()` take; both already exist on develop and are unchanged. `L` lowers to `AttentionBand` / `sliding_window` unchanged, since they count keys the same way. - `library/tests/sdpa/request_reference.py`: `sdpa_reference_from_request()` decodes what the array oracle does not model (flat strided buffers, packed ragged buffers, the scale source, FP8 descales, hipDNN's left bound) and delegates the math to `sdpa_reference()`. It sits beside the SDPA tests and gates on `SdpaRequest.features()` rather than re-deriving them. - `library/tests/sdpa/request_cases.py`: the request case table (every feature at least once) and `make_tensors()`. The kernel coverage table, `cases.py`, already exists on develop. - `library/tests/sdpa/test_attention_body_seams.py`: shows that the score hook's query/key positions (#12710) cost no IR. `helpers/mfma_attention.py` is not changed; row epilogues such as LSE use the existing `AttnFwdExt.epilogue_hook`. - `platform/CMakeLists.txt`: a `rocke_library_sdpa_pytest` CTest target for `library/tests/sdpa`, enrolled in the `quick` and `ffm-quick` test categories. When configured from a monorepo checkout it installs the hipDNN SDPA graph template and `SdpaFwd*` golden graphs beside the tests (`tests/sdpa/hipdnn/`) and passes `--rocke-require-hipdnn-assets`. - Overlaps PR #13115, which adds a placeholder `dispatch/sdpa/common.py` and `tests/sdpa/`; whichever lands second rebases. - `KERNEL_AUTHORING.md`: two new Definition of Done change types, one for operators with a hipDNN graph node (lossless request, `from_hipdnn`, reference, case table) and one for new operands or outputs (route only to families that declare the feature, default-off kernel flag, numeric test against the reference). ## Out of Scope (follow-up) No kernel family consumes `SdpaRequest` yet, so this PR changes no routing. `SDPA_REGISTRY` and its entry points are in this PR but empty. Registering candidates in it (each family's mapping from `SdpaRequest` to its candidates: what it admits and how request fields such as bias, scale, and LSE bind to its kernel arguments) and the generated admit/decline table land in follow-up PRs per kernel family. The oracle does not model dropout, paged K/V, block masks, sinks, ALiBi, FP8 S/O scaling or max/sum_exp yet (hipDNN's CPU reference does); it raises `NotImplementedError` for them. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: ThruptiRajLakshmanaGowda <Thruptiraj.LakshmanaGowda@amd.com>
cgmillette
pushed a commit
that referenced
this pull request
Nov 4, 2026
…al tile (#14335) Replaces #13681 with the same commit (1b17bab). GitHub closed #13681 when its branch moved to `users/ysoliman/rocke/` for the rocKE branch-naming rule, and a PR cannot be reopened once its branch is renamed. The review history, including the rocKE checklist reviews, stays on #13681. Carved out of #12928, which is closed. The rest of that PR is already on develop (#12710 landed the nhead_k guard, #12719 replaced the factories this touched and routes `_request_errors` through every dense candidate). ## Motivation The dense `_base_spec` factories set `ragged=True` for any self-attention length that is not a tile multiple, so Sq = Skv = 384 or 1152 at block_m 256 is an ordinary graph taking the on-chip ragged path. The gfx942 predicate then declined with "gfx942 attention_dense: ragged not yet supported" and the shared spec validator with "ragged is not supported with sliding_window". Both read as a varlen problem and name nothing the caller can act on. Found while running the hipDNN kernel-integration runbook on gfx942. ## Technical details - `kernels/common/attention_dense_spec.py`: new `partial_tile_reason(spec)` names the length and the tile, for example "seqlen_q 384 is not a multiple of block_m 256". The sliding-window branch of `__post_init__` now says "sliding_window needs whole tiles: <reason> (the partial-tile path has no window mask)". - `kernels/gfx942/attention_dense.py`: `supports_attention_dense` says "gfx942 attention_dense needs whole tiles: <reason> (the partial-tile path is not yet supported)". - Varlen messages are unchanged. What could break: anything matching the old "ragged not yet supported" or "ragged is not supported with sliding_window" text. A repo grep for those two strings finds only the two lines this PR rewrites, and that was not enough. One test matched on less than the whole string: `tests/dispatch/attention/test_unified_mask_wiring.py` asserted the substring "ragged" in the gfx950 window refusal, and this branch broke it. It is now `test_partial_tile_window_is_declined_not_served` and checks the length, the tile and the window, the same way the gfx942 and gfx950 wiring tests do. Anything outside this repo that keys on the word "ragged" has the same exposure. ## Test plan and result Host only, no GPU, Windows Python 3.12, from `dnn-providers/hip-kernel-provider/rocke/library` with `library` and `platform/python` on PYTHONPATH. - `tests/dispatch/attention` plus `tests/test_attention_dense_gfx942.py` at head `1b17babac3`: 882 passed, 121 skipped, 3296 subtests passed. The merge base, develop at 28af48c, gives 882 passed, 121 skipped, 3293 subtests; the 3 extra subtests are the new 384 / 1152 / 1000 cases. - Negative control: with the two messages reverted, `test_rejects_a_partial_tile_and_names_the_length`, `test_sliding_window_with_partial_tile_rejected_at_dispatch` and the gfx942 `supports` marker all fail. - black 25.12.0 `--check` is clean on the changed files. Refs #12920. ## Submission checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Merged develop 3f51737 and updated the three rocke attention wiring tests to match the new refusal message.
cycheng
pushed a commit
that referenced
this pull request
Nov 6, 2026
…al tile (#14335) Replaces #13681 with the same commit (1b17bab). GitHub closed #13681 when its branch moved to `users/ysoliman/rocke/` for the rocKE branch-naming rule, and a PR cannot be reopened once its branch is renamed. The review history, including the rocKE checklist reviews, stays on #13681. Carved out of #12928, which is closed. The rest of that PR is already on develop (#12710 landed the nhead_k guard, #12719 replaced the factories this touched and routes `_request_errors` through every dense candidate). ## Motivation The dense `_base_spec` factories set `ragged=True` for any self-attention length that is not a tile multiple, so Sq = Skv = 384 or 1152 at block_m 256 is an ordinary graph taking the on-chip ragged path. The gfx942 predicate then declined with "gfx942 attention_dense: ragged not yet supported" and the shared spec validator with "ragged is not supported with sliding_window". Both read as a varlen problem and name nothing the caller can act on. Found while running the hipDNN kernel-integration runbook on gfx942. ## Technical details - `kernels/common/attention_dense_spec.py`: new `partial_tile_reason(spec)` names the length and the tile, for example "seqlen_q 384 is not a multiple of block_m 256". The sliding-window branch of `__post_init__` now says "sliding_window needs whole tiles: <reason> (the partial-tile path has no window mask)". - `kernels/gfx942/attention_dense.py`: `supports_attention_dense` says "gfx942 attention_dense needs whole tiles: <reason> (the partial-tile path is not yet supported)". - Varlen messages are unchanged. What could break: anything matching the old "ragged not yet supported" or "ragged is not supported with sliding_window" text. A repo grep for those two strings finds only the two lines this PR rewrites, and that was not enough. One test matched on less than the whole string: `tests/dispatch/attention/test_unified_mask_wiring.py` asserted the substring "ragged" in the gfx950 window refusal, and this branch broke it. It is now `test_partial_tile_window_is_declined_not_served` and checks the length, the tile and the window, the same way the gfx942 and gfx950 wiring tests do. Anything outside this repo that keys on the word "ragged" has the same exposure. ## Test plan and result Host only, no GPU, Windows Python 3.12, from `dnn-providers/hip-kernel-provider/rocke/library` with `library` and `platform/python` on PYTHONPATH. - `tests/dispatch/attention` plus `tests/test_attention_dense_gfx942.py` at head `1b17babac3`: 882 passed, 121 skipped, 3296 subtests passed. The merge base, develop at 28af48c, gives 882 passed, 121 skipped, 3293 subtests; the 3 extra subtests are the new 384 / 1152 / 1000 cases. - Negative control: with the two messages reverted, `test_rejects_a_partial_tile_and_names_the_length`, `test_sliding_window_with_partial_tile_rejected_at_dispatch` and the gfx942 `supports` marker all fail. - black 25.12.0 `--check` is clean on the changed files. Refs #12920. ## Submission checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Merged develop 3f51737 and updated the three rocke attention wiring tests to match the new refusal message.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ISSUE ID : AICK-1847
Summary
Extend rocKE inference SDPA coverage to
gfx1151through the public attention dispatcher and runtime binding, with matching Python/C++ kernel emission.FP16/BF16, Q/K and V/O head dimensions in multiples of 16 through 256 (including D96), MHA/GQA/MQA, prefill and decode.
Top-left and bottom-right causal masks, runtime two-sided windows, LSE, arbitrary BSHD/BHSD strides, query/KV tails, softcap, sinks, ALiBi, FP32 QQ bias, additive attention bias, packed variable lengths, paged KV, and software OCP E4M3FN KV storage.
Real HIP tensor owners and non-default streams without making torch a mandatory dependency.
A deterministic 54-case correctness/timing benchmark over public rocKE dispatch (
library/benchmarks/gfx1151/attention). The AOTriton comparison below was measured with an out-of-tree harness that is not part of this PR.This remains inference-forward only. It now includes a packaged hipDNN kernel-ingestor engine for gfx1151; backward/training support remains out of scope.
Implementation and review map
Coverage and runtime:
library/kernels/gfx1151/wmma_fmha_fwd.py,library/dispatch/attention/gfx1151.py, andbindings.py. Structural tensor validation, layout-specific metadata, specialized signatures, complete spec/architecture cache identity, and stream/graph-safe ownership.Shared kernel implementation:
platform/python/rocke/helpers/mfma_attention.pyand its C++ peer. Dtype-matched WMMA, strict empty-row normalization, guarded tails, score features, packed/paged addressing, and FP8 decode. Existing shared-helper callers are migrated in the same change.Measured specializations: aligned FP16/BF16 D64/D128 transposed-QK attention using original row-major V; D256 output-column tiling that keeps the full QK dimension; and explicit max-ILP/V-LDS small-head profiles. Unmatched shapes/features retain the existing complete path.
Backend support: corrected wave-sized LLVM launch bounds; real
vector.undefand half-return D16 buffer-load primitives in both engines; explicit scheduler policy in the spec, cache name and emitted attributes. Existing opcode numeric values are preserved.Tests and reproduction: existing/new numerical and selection tests, 136 Python/C++ emitter configurations, scoped golden additions, and the rocKE-only
library/benchmarks/gfx1151/attentionbenchmark. No comparison harness, remote launcher, credentials, host configuration, binaries or raw measurements are committed.The transposed-QK algorithm selectively reuses the design from #9710, pinned at
4ffc69c05a5748692370cca6dcfe61f2405dd0b5. This is not a wholesale cherry-pick: the pre-transposed-V requirement and unused experimental knobs are not carried over, and native C++ support is included here.Native consumers must rebuild against the updated headers/archive. The extended spec/shared parameter structs are not a binary-compatibility shim; string lifetimes and defaults are documented in their headers.
Validation
Recorded implementation commit:
e62c1b21f246f0effc26b5c038de72177fdf0719.Publication formatting commit:
d81c0309efa; no kernel scope expansion.Review follow-up commit:
0f79130bf6a; removes the out-of-tree comparison/launcher files and restores the guard tests (no kernel or emitter change).All 54/54 frozen inference cases pass through public
dispatch_attention(...).bind_torch(...), with zero unsupported rocKE cases, numerical failures or execution errors.57 GPU numerical regressions pass on
gfx1151, including non-default streams, long-query causal windows, fully masked prefixes, packed/paged FP8, and output tiling with all QK signal in the final head dimension.Scoped dispatch/layering CPU lane: 351 tests and 1,980 subtests passed, with 18 GPU/codegen-related cases deselected. This is not a claim that every repository test ran.
Native emission: 119 configurations on each of LLVM 20/22/23, 120 binding cases per flavor, and canonical IR parity. The preceding 99 outputs were unchanged when output tiling was introduced.
Actual final public selections: 54 requests -> 53 unique specs, all 53 byte-identical between Python/C++ on each LLVM flavor, with canonical IR also matching. The final selection-only change reuses the unchanged native archive.
Native C++ output-grid smoke: batch/output-tile mapping, grid overflow, negative batch, and invalid tile rejection.
Local spec/codegen contracts: 23 spec tests and six policy tests passed in the corresponding implementation steps.
Required repository pre-commit hooks pass over the PR range.
All 22 reformatted Python files preserve their syntax trees, excluding docstrings/source locations.
Publication-head GPU smoke: 54/54 correct, 1.3572x matched geomean, 0.8085x worst matched case; both targets still pass.
Publication-head regressions: 57 GPU cases and 351 CPU tests / 1,980 subtests passed.
Review follow-up head (
0f79130bf6a): rocKE benchmark 54/54 correct, zero unsupported/errors ongfx1151; 120 GPU-host tests / 214 subtests passed; restored guard and scoped CPU tests: 491 passed, 57 GPU-skipped, 2,299 subtests.Fresh publication-head native check: 119 configurations on each LLVM 20/22/23, 120 binding cases per flavor, and canonical IR parity. All 119 target LLVM/canonical outputs remain unchanged from the validated implementation; only the documented baseline exception remains.
Relative results: fixed matched inference cohort
On
gfx1151, three independent full runs of the frozen cohort passed both criteria: geomean >= 1.10x and every matched case >= 0.80x versus AOTriton 0.14.2b automatic dispatch.Same 32 matched cases, equally weighted, in every score.
Original three-run geomean range: 1.3585-1.3677x.
Original three-run worst matched-case range: 0.8204-0.8271x.
Publication formatting confirmation (
d81c0309efa): 1.3572x geomean and 0.8085x worst case, with 54/54 correct.These are results for this cohort, not a general superiority claim.
Latest publication-head run; group speedups are geometric means:
| Operation / features | rocKE pass/required | AOTriton pass/required | Speedup |
|---|---:|---:|---:|
| Dense | 14/14 | 14/14 | 1.270x |
| GQA/MQA | 3/3 | 3/3 | 1.789x |
| Decode | 2/2 | 2/2 | 1.135x |
| Bottom-right | 3/3 | 3/3 | 1.410x |
| Sliding window | 2/2 | 2/2 | 1.826x |
| Softcap | 2/2 | 0/2 | N/A |
| Sinks | 4/4 | 0/4 | N/A |
| ALiBi | 2/2 | 0/2 | N/A |
| FP32 QQ bias | 2/2 | 0/2 | N/A |
| FP8 KV | 3/3 | 0/3 | N/A |
| Paged KV | 5/5 | 0/5 | N/A |
| Ragged | 4/4 | 4/4 | 1.126x |
| Boundary cases | 4/4 | 4/4 | 1.537x |
| Feature combinations | 4/4 | 0/4 | N/A |
N/A means the comparator has no semantically matched implementation, not that rocKE lacks coverage.
Method (out-of-tree comparison harness, not committed): original row-major input storage, no free transpose/densification; an independent FP64 oracle reads already-rounded storage. Max-absolute tolerances are 0.02 for FP16 and 0.04 for BF16, with nonfinite/unwritten output rejected. Both arms are checked before and after seven alternating timing batches of 32 HIP-graph replays on one stream. Required AOTriton scheduling-counter resets are included. Compilation, allocation, transfers and the reference calculation are outside timing.
Provenance: frozen suite SHA-256
f6e3fcde2f28c35401cc4684ad50ee98cc717213dcc9b806f6264c3ac5445486; final experiment source snapshot9a5929422bbb8f04d72ca41ed2416533c6e8018f33fc771e776b9d90e05671e2; publication-head snapshot982f05ae057a66ccfd1baecfc8c6a4f84a95c5f2a98eb12022a88a900709917d. AOTriton release 0.14.2b reports runtime ABI 0.14.1, gitc11b5b886a10a00f925f74133d38b3505e67ccd9, forward-parameter version 4. The kernel lane used LLVM 23; Python 3.12.3, NumPy 2.5.1 and ml_dtypes 0.5.4. Raw absolute measurements remain in the retained internal experiment record.Known baseline exceptions and handoff boundaries
The recorded full 70-family native comparison retains the pre-existing
gfx942_attention_tiled_3dconfiguration 3 mismatch, with unchanged hashes on LLVM 20/22/23. The gfx1151 target family passes. This work does not claim a globally clean parity baseline.Historical stored FP16 golden exceptions and unrelated stale/unanchored entries are not rebaselined here. In
llvm_gfx_all.jsonthe wholegfx1151_wmma_fmha_fwdfamily (configs 0..135,llandir_canonical) is re-blessed, plus the 25llanchors of other wave32 families that pass on develop and change with the wave-size launch-bounds fix. Anchors already stale on develop are left untouched.No new gfx950 GPU validation is claimed. The added hipDNN packaged-engine validation is scoped to gfx1151; shared emission/default preservation is covered by the native comparison.
The search stopped at the agreed coverage/performance target. Remaining optional combinatorial sweep points were not exhausted and are not claimed complete.
Review routing follows the repository's CODEOWNERS for rocKE library/platform and hip-kernel-provider. CI on this PR must establish integration status before merge; the hardware results above are the handoff evidence, not a merge-readiness assertion.
Benchmark usage is documented in
library/builders/gfx1151/attention/README.mdunder the rocKE root.Runtime-generic and AOT expansion
This revision also incorporates the three follow-up commits from @yraparti and reconciles their overlapping ABI changes with reusable AOT dispatch:
hipkernel:Gfx1151WmmaAttention: 40 precompiled gfx1151 objects covering FP16/BF16, D64/D96/D128/D256, no-mask/causal/window modes, and aligned fast variants.Validation on the published head
dd1d592d6fd:Review round 2 (head
e61db982cbd)develop(1e094cb6b36).AttentionRequest.return_lseis replaced by develop'semit_lse(samelsefeature, one flag for every candidate); this branch'sVECTOR_UNDEF/TILE_BUFFER_LOAD_F16_D16opcodes are appended after develop'sMEMREF_*opcodes so no develop opcode value shifts.3043a33af31: the gfx1151 binding bounds every operand's addressed byte extent to I32 (matching the native pack'saddressFitsI32). The transposed-QK V gather forms I32 byte offsets against a 2 GiB descriptor, so an fp16 tensor whose element offsets fit I32 but whose bytes do not (e.g. SBHD V 16x32Kx32x128) previously read zeros instead of being rejected.38875bcf541: the gfx1151 census is registered only whenproductshipshipkernel:Gfx1151WmmaAttentionfor gfx1151 (hkp_product_ships_engine_for_arch(), generalized from the gfx950 predicate).4245402700c: goldens re-blessed after the merge.rocke_representative_ir_sha256.jsonregenerated (LLVM 20/22/23; wave32 gemm/conv/moe entries only).llvm_gfx_all.jsonupdated as described above.Validation on this head:
test_rocke_ci_static.py5/5 including IR golden SHA; gfx1250 scaled-WMMA golden 82 passed.run_diff.py --check-golden(C engine built from this head and from develop on Linux): no anchor that passes on develop fails on this branch (ll: develop 639 stale -> branch 633; ir_canonical: 543 -> 537 after the gfx1151 family). Their --canonicalC-vs-Python DRIFT families (conv/deep_fused_conv_pool) are identical on develop.descriptor-packaging/tests/test_hkp_pack_wiring.py: 23/23 (Linux, cmake).gfx1151_wmma_fmha_fwdllandir_canonicalSHAs on this head equal those blessed at the previous PR headc201f5053a7.