Skip to content

fix(rocke): refuse gfx1250 WMMA on LLVM flavors that lack the intrinsics - #14437

Closed
CongMa13 wants to merge 16 commits into
developfrom
users/congma13/rocke/gfx1250-decl-flavor-tables
Closed

CongMa13 wants to merge 16 commits into
developfrom
users/congma13/rocke/gfx1250-decl-flavor-tables

Conversation

@CongMa13

@CongMa13 CongMa13 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

ISSUE ID : AICK-2788


Summary — For human reviewers

The short version

The rocKE lowerer keeps one base table of intrinsic declarations for all LLVM flavors. LLVM 20 does not have the unscaled gfx1250 WMMA intrinsics. The lowerer wrote the call for them on LLVM 20. The module then called an undeclared function, and opt rejected it.

This PR makes the gfx1250 WMMA handlers refuse LLVM 20 with a clear message. It also makes _need refuse a key that no table declares.

How it works

  • A new profile capability, GFX1250_WMMA, marks the flavors that have the intrinsics. LLVM 22, LLVM 23 and LLVM 24 have it. LLVM 20 does not. Python (_LlvmEmissionCapability) and C++ (ROCKE_LL_CAP_GFX1250_WMMA) hold the same capability.
  • The hand-written gfx1250 WMMA handlers check the capability before they demand the intrinsic. A flavor without it raises intrinsic '<key>' requires llvm22, got llvm20. The Python backend (emit_wmma) and the C++ lowerer (_emit_wmma) use the same text.
  • _need refuses a key that no table declares. Python raises KeyError. C++ returns ROCKE_ERR_KEY. A missing key is an emitter defect, because finalize would write a call without a declare.
  • The declaration tables do not change. The emit order and the arch-domain columns stay as on develop. The scaled WMMA path already refuses flavors below LLVM 23 through _supports_llvm23_target_ops.

The refusal uses the existing wording. tools/_flavor_gate.py already matches "requires llvmNN", so the validity gate reports such a case as UNVALIDATED.

What changes

On LLVM 20, every kernel that uses an unscaled gfx1250 WMMA becomes an expected refusal. Those are the gfx1250 gemm, conv, conv_dgrad, conv_wgrad, conv_winograd_fused_1d_wmma, conv_direct_dw_fwd_wmma and depthwise windowed dgrad cases. The LLVM 20 sections of both IR goldens record this. The LLVM 22, LLVM 23 and LLVM 24 sections equal develop.

LLVM 22 and later emit the same IR as before. The fused 1-D Winograd plan declares LLVM 22 and later, so LLVM 20 admits no plan for it.

tools/check_byte_identity.py compares conv_winograd_fused_1d_wmma and conv_direct_dw_fwd_wmma in the default lane from LLVM 22 on. A flavor below LLVM 22 compares them in the LLVM 23 lane only, because LLVM 20 refuses every config of those families.

Known issue, not part of this PR

On an LLVM 23 toolchain, 7 gfx950 fp16 attention cases fail with Operand has incorrect register class. They fail the same way on develop. The cause is a compiler defect.

How this was tested

Runs used the merge of develop at 031d56af855, unless the line names a later merge.

  • Native build and ctest: 46 of 47 pass. rocke_optimization_barrier_hip_pair times out on a node without HIP headers.
  • tools/check_byte_identity.py, full run: GREEN on LLVM 20, LLVM 22, LLVM 23 and LLVM 24.
  • test_gfx1250_scaled_wmma.py, test_matrix_calls.py, test_flavor_decl_tables.py, test_arch_domain_artifact.py (without the regeneration test): 161 pass on each of the four flavors.
  • Winograd dispatch tests: 17 pass on LLVM 20 and LLVM 22. Tiled-attention cache-key tests, gfx1250 cases: 31 pass on LLVM 20 and LLVM 22.
  • Representative golden: the check at LLVM 20 reported drift only in gemm and conv_direct_dw_fwd_wmma. Both shards are re-blessed.
  • Differential golden, Python only: LLVM 22, LLVM 23 and LLVM 24 show no change. LLVM 20 drops the seven conv_direct_dw_fwd_wmma configs.
  • After the merge of develop at 240c8b0dc18, which adds the gfx1250 depthwise dgrad kernels (dw_dgrad_win_wmma_k5, dw_dgrad_win_wmma_k7):
    • Representative golden check at LLVM 20: drift in conv_direct_dgrad only. The shard is re-blessed. The test_rocke_ci_static.py golden tests pass (8).
    • Differential golden: the six conv_direct_dgrad_ext LLVM 20 configs that lower WMMA are re-recorded. The check passes on LLVM 20, LLVM 22, LLVM 23 and LLVM 24.
    • test_lowers_to_wmma_with_lds_staging skips on a flavor without the capability.
    • Two new refusal tests use a K=64 FP8 atom: one in test_gfx1250_scaled_wmma.py and one in the C++ test rocke_future_intrinsic_lowering (92 cases pass).
  • pre-commit passes on the changed files.

What did not run: the full Python suite on LLVM 20 after the latest merge, the byte-identity gate after the latest merge, the IR validity gate, any GPU run and any Windows run.

Reviewing this

The change adds one profile capability, one check in each WMMA emit path, and one check in _need. No lowering handler outside the gfx1250 WMMA path changes.


Summary — For AI reviewers

Change surface

python/rocke/core/llvm_profile.py      GFX1250_WMMA capability; _supports_gfx1250_wmma
python/rocke/core/isa/backend.py       _require_gfx1250_wmma in emit_wmma and _emit_wmma_fp8
python/rocke/core/lower_llvm.py        _require_declared; _need calls it
cpp/core/llvm_profile.cpp              capability on the LLVM 22, 23 and 24 profiles
cpp/include/rocke/llvm_profile_internal.h  ROCKE_LL_CAP_GFX1250_WMMA
cpp/core/lower_llvm/mma.cpp            capability check in _emit_wmma
cpp/core/lower_llvm/core.cpp           rocke_ll_need refuses a key with no declaration
tests/core/test_gfx1250_scaled_wmma.py IntrinsicDeclaredOnFlavorTest (f16 and FP8 K=64 refusal)
tests/core/future_intrinsic_lowering.cpp  C++ refusal and declare cases (f16 and FP8)
tests/golden, tests/instances/differential/golden  LLVM 20 sections re-blessed
tools/check_byte_identity.py           two WMMA families compared from LLVM 22 on
library/dispatch/conv_winograd_plan.py fused 1-D plan declares its flavors
library/tests/test_attention_tiled_cache_key.py  gfx1250 cases skip below LLVM 22
library/tests/test_conv_dgrad_depthwise_windowed.py  WMMA lowering test skips below LLVM 22

Properties to verify

  1. LLVM 22 and later output is unchanged. _decls_for_flavor is the same as on develop. Falsify: any hash change outside the LLVM 20 section of either golden.
  2. Every unscaled gfx1250 WMMA path checks the capability. The Python paths are emit_wmma and _emit_wmma_fp8. The C++ path is _emit_wmma for the shapes WMMA_SHAPE_GFX1250 and WMMA_SHAPE_GFX1250_FP8. Falsify: an unscaled wmma.gfx1250.* call that reaches _need on LLVM 20 by another path.
  3. The Python and C++ refusal texts match. Both read intrinsic '<key>' requires llvm22, got <flavor>. The C++ case in future_intrinsic_lowering.cpp pins the text.
  4. _require_declared accepts every key that finalize declares. _op_vector_smax writes self._decls[intrin] before _need. In C++, rocke_ll_need_dynamic registers the key first. Falsify: a _need call that precedes the registration of a dynamic key.
  5. The matrix-call path is unaffected. MATRIX_CALLS entries derive their declarations per flavor and do not use the hand-written handlers.

Known-weak points

  • Other keys that the arch-domain columns mark name_absent on every arch stay declarable (52 on LLVM 20, 17 on LLVM 22). This PR does not move them. Follow-up AICK-3035 covers them.
  • The Python and C++ capability checks are separate code. The byte-identity gate compares IR only. The C++ refusal case pins the C++ text.

Scope calibration

A finding that this PR does not fix the 7 LLVM 23 attention failures is out of scope. A finding that this PR does not add a CI entry for the validity gate is also out of scope. That work was dropped on purpose.

Test plan

  • Native build and ctest: 46 of 47 (one HIP-header timeout)
  • check_byte_identity.py full run: LLVM 20, 22, 23, 24 GREEN
  • Targeted Python tests on LLVM 20, 22, 23, 24
  • pre-commit on the changed files
  • check_byte_identity.py after the latest merge: not run
  • Full Python suite on LLVM 20 after the latest merge: not run
  • check_ir_validity.py: not run after this change
  • GPU run: not run
  • Windows run: not run

🤖 Generated with Claude Code

CongMa13 and others added 2 commits October 26, 2026 19:30
…that have them

The base intrinsic table is shared by every flavor. It held the gfx1250
WMMA declarations, so an LLVM 20 host emitted calls to intrinsics its
toolchain lacks and `check_ir_validity.py` could not tell that case from
an emitter defect.

Move the `wmma.gfx1250.*` declarations to the LLVM 22 override table and
the `wmma.scale.*` declarations to the LLVM 23 override table, in Python
and in the C++ engine. Prune the arch-domain rows that no longer have a
declaration key.

Make `_Lowerer._need` refuse a key that the flavor cannot declare. A key
that a newer flavor adds raises "requires llvmNN", which the validity
gate already reports as UNVALIDATED. A key in no table raises KeyError.
Without this check the call was emitted with no `declare` and `opt`
rejected the module.

ISSUE ID : AICK-2788

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Python lowerer refuses a `_need` demand for a key that the flavor
table cannot declare. The C++ lowerer recorded such a key with a NULL
declaration and `finalize` skipped it. An LLVM 20 host then emitted a
call with no `declare`.

Make `rocke_ll_need` fail the same way. A key that the LLVM 22 or
LLVM 23 table adds fails with "requires llvmNN", which the validity gate
reports as UNVALIDATED. A key in no table fails with a value error.

ISSUE ID : AICK-2788

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@CongMa13
CongMa13 requested a review from a team as a code owner October 26, 2026 23:44
@therock-pr-bot

therock-pr-bot Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass —
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

therock-pr-bot Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

…ke/gfx1250-decl-flavor-tables

# Conflicts:
#	dnn-providers/hip-kernel-provider/rocke/platform/cpp/core/lower_llvm/core.cpp
#	dnn-providers/hip-kernel-provider/rocke/platform/cpp/core/lower_llvm/data.cpp
#	dnn-providers/hip-kernel-provider/rocke/platform/python/rocke/core/lower_llvm.py
@cgmillette

Copy link
Copy Markdown
Contributor

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14437 — declare gfx1250 WMMA intrinsics only on the LLVM flavors that have them (CongMa13, AICK-2788) — 🔴 First-round: blocking — golden gates red at head; llvm22+ WMMA declares reordered

Prior item Status
Reviewers 🟢 None yet: no reviews or threads, only the policy bot
Bots / CI 🔴
• Policy bot pass. At head d2a11a56 (CI of 10-28, before develop's #14394 and #14415 landed), TheRock Multi-Arch CI fails. All four Test hipkernelprovider shards (Linux gfx94X 1/2 and 2/2, Windows gfx110X 1/2 and 2/2) fail the same three tests: rocke_golden_static, rocke_differential_golden_ll, and rocke_pytest, whose one failure is test_rocke_ci_static.py::…::test_ir_cases_match_golden_sha256 (the representative golden again). Every drifted case is gfx1250 (N1). The other 4,449 (Linux) and 4,321 (Windows) pytest cases pass, including the PR's three IntrinsicDeclaredOnFlavorTest tests.
• Windows shard 2/2 also fails rocke_library_gdn_pytest on a test_gdn_decode_fused.py timeout: the known intermittent (#14633), unrelated.
• Multi-Arch ASAN, both hipDNN Superbuilds, Component CI, clang-tidy and pre-commit pass. gfx950 and gfx125X are built, but their test jobs are skipped (read)
Our checks 🔴
• Clean merge with develop f22bed43 (23 behind, 3 ahead). Native build and 33 ctests OK (1 skipped). C++ vs Python byte identity GREEN for all 85 families at llvm22 and for both gfx1250 families at llvm23. The PR's test file: 18 passed (ran).
• Representative golden FAIL on the merge at all four lanes, gfx1250 cases only. At llvm20, 10 cases move from cases to expected_failures; 10 hashes change at llvm22 and 14 at llvm23 and llvm24. Develop's four new gfx1250 Winograd cases (#14394) account for 2 / 2 / 4 of these, and llvm24 is #14415's new lane; the head alone drifts 8 / 8 / 10 in CI (ran).
• Differential golden ll FAIL (rc=1), gfx1250 configs only: 16 gone at llvm20 (conv_implicit_gemm ×8, its _dgrad and _wgrad ×1 each, gfx1250_lowering ×6), 10 changed at llvm22 and 17 at llvm23/llvm24 (+conv_direct_dgrad_wmma ×7). CI at head shows the same 16 / 10 / 17. Canonical OK; its 121 NEW unblessed configs are develop's (ran).
• Re-bless dry run on the merge, into scratch copies: N1's commands make both gates pass, and the representative golden then differs from develop's in 44 gfx1250 entries and nothing else (ran).
• Both engines refuse gemm/gfx1250/t32x32x32 at llvm20 with the same text, "intrinsic 'wmma.gfx1250.f32.16x16x32.f16' requires llvm22, got llvm20"; through pybind the C++ refusal is a RuntimeError. Both match _flavor_gate.py's "requires llvmNN" (ran).
• Placement vs the committed arch-domain columns: all 8 keys are name_absent on every arch at llvm20, and the 6 wmma.gfx1250.* keys are ok on gfx1250 at llvm22, so both moves are right. The 2 scaled keys were also measured ok on gfx1250 at llvm22 (rows this PR prunes), so llvm23 is a policy choice; it matches the existing gate at isa/backend.py:232, and no kernel changes (ran). The arch-domain artifact test, which the CI install excludes, passes on the merge: 77 passed (ran).
• Rules 1–3 clean; no links or tracker IDs added (ran)
# Sev file:line Finding
N1 🔴 tests/golden/rocke_representative_ir_sha256.json, differential/golden/llvm_gfx_all.json (not updated) CI is red at head: neither IR golden was re-blessed, so the body's "The golden hashes are unchanged" does not hold. The llvm20 part is the intended effect (gfx1250 WMMA cases become expected "requires llvm22" refusals); the llvm22/23/24 part is N2. → Update the branch with develop first (#14394 adds gfx1250 cases to the golden, #14415 the llvm24 lane). Then re-bless from platform/ with PYTHONPATH=python:../library: python tests/instances/rocke_ir_parity_harness.py --write tests/golden/rocke_representative_ir_sha256.json; and from tests/instances/differential: python run_diff.py --mode ll --python-only --record-golden --flavor llvm20,llvm22,llvm23,llvm24 --only conv_implicit_gemm,gfx1250_lowering,conv_direct_dgrad_wmma. With N2 fixed, only llvm20 should move (--flavor llvm20 --only conv_implicit_gemm,gfx1250_lowering) (ran: dry run of these commands on the merge)
N2 🟠 lower_llvm.py:1141, :1206; data.cpp:512; core.cpp:2193 finalize emits declares in table order, and keys that only an override table holds are appended after every base key (dict.update, and the C++ override-only loop). So gfx1250 WMMA kernels that still lower get new IR at llvm22/23/24: on the merge, each of the 24 drifted lowerings differs from develop only in where the WMMA declare sits. That is churn on lanes the PR does not target, and it falsifies the body's "No emitted IR changes for a case that lowers today" → keep each moved key at its old place in the emit order, e.g. leave it in the base table with a minimum flavor that _decls_for_flavor and the C++ table walk filter on; then only llvm20 changes. Otherwise re-bless all four lanes and correct the body (ran: merge vs develop IR)
N3 🟡 lower_llvm.py:433, arch/data/intrinsic_arch_domain.llvm20.json The class is fixed only for WMMA. The llvm20 table still holds 52 keys, and the llvm22 table 17, that their committed columns mark name_absent on every arch (cluster IDs and loads, async LDS copies, prefetch, tensor.load.to.lds, wave reductions). _require_declared passes them because they are in the table, so their safety still rests on each handler's own flavor gate → move them by the same rule (no IR churn with N2's minimum flavor), or add a test that no flavor table holds a key its column marks name_absent on every arch (ran)

Utility: 🟢 Positive — closes the call-without-declare gap in both engines; needs re-bless and order fix.


Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

First-round review: a local qwen3.6 model made the independent first pass. The orchestrator (Claude Opus 5.5) verified it. On a clean merge with today's develop it ran the native build and ctest, engine-wide byte identity, the representative and differential goldens, the PR's test file and the arch-domain artifact test. It lowered every drifted gfx1250 case on the merge and on develop at three lanes, compared the C++ and Python refusals, and dry-ran the re-bless into scratch copies. It also checked develop's changes since the base and read the four hipkernelprovider CI logs at head. The orchestrator owns this merge-worthiness call. It flags whether a closer look is needed; it is not an approval or a rejection.

CongMa13 and others added 4 commits October 29, 2026 16:39
… flavor

Keep the gfx1250 WMMA keys in the base declaration table and record the
lowest flavor that declares each one. `_decls_for_flavor` and the C++
lookup drop a key below its minimum flavor, so a demand is still refused
with "requires llvmNN". Declares on llvm22 and later keep their place in
the emit order, so only llvm20 output changes.

Re-bless the representative and differential goldens for llvm20. Compare
`conv_winograd_fused_1d_wmma` in the llvm23 lane of the byte-identity gate.

ISSUE ID : AICK-2788

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l-flavor-tables' into users/congma13/rocke/gfx1250-decl-flavor-tables
@CongMa13

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Head 56e9d40f2ae addresses N1 and N2. It does not address N3.

N2 (declare order): fixed.

  • The gfx1250 WMMA keys stay in the base table in Python and C++.
  • _INTRINSIC_MIN_FLAVOR (Python) and ROCKE_LL_INTRINSIC_MIN_FLAVOR (C++) give each key a minimum flavor: llvm22 for the six wmma.gfx1250.* keys, llvm23 for the two wmma.scale.* keys.
  • _decls_for_flavor and the C++ lookup drop a key below its minimum flavor. A demand for it still raises "requires llvmNN".
  • Declares on llvm22 and later keep their old position. Only llvm20 output changes.
  • The arch-domain column pruning is unchanged.

N1 (goldens): fixed.

  • Merged develop first.
  • Re-blessed the llvm20 sections of the representative golden and the differential golden (conv_implicit_gemm, gfx1250_lowering).
  • The llvm22 and llvm23 sections of the representative golden are byte-equal to before. --check-golden is OK on llvm20, llvm22, llvm23 and llvm24.
  • The PR body no longer says the goldens are unchanged. It now says only the llvm20 sections move.

Gate change. At llvm20 every config of conv_winograd_fused_1d_wmma is now refused, so check_byte_identity.py reported ALL_REJECTED. The family now joins TARGET_OP_PROFILE_FAMILIES and runs in the llvm23 lane. As a result, the llvm22 gate run also compares it only in the llvm23 lane. Golden and pytest still cover it at llvm22.

Verification, each flavor on its own toolchain image.

  • check_byte_identity.py: GREEN on llvm20, llvm22 and llvm23.
  • Python suite (tests/core, tests/golden, tests/portable_ir): no failures on llvm20, llvm22 and llvm23.
  • Two tests added: emit order is preserved, and every minimum-flavor key is in the base table.
  • After merging the newer develop commit on the remote, I re-ran only the two golden checks. Both pass.

N3: not done in this PR. The same minimum-flavor map can gate the other keys that their columns mark name_absent on every arch. That is a separate sweep. I will open a follow-up unless you want it here.

The test_every_decl_key_is_covered_on_every_wired_arch failure for mfma.scale.f32.16x16x128.f8f6f4 at llvm23 also occurs on develop. This PR does not cause it.

@CongMa13
CongMa13 requested a review from cgmillette October 29, 2026 23:08
@cgmillette

Copy link
Copy Markdown
Contributor

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14437 — declare gfx1250 WMMA intrinsics only on the LLVM flavors that have them (CongMa13, AICK-2788) — 🟠 Follow-up: closer look — N1 and N2 are fixed; a new, unused llvm23 declaration fails the arch-domain test, and TheRock's test shards have not run at this head

Prior item Status
Ours (10-29) N1: neither IR golden was re-blessed, so both gates were red at head 🟢 FIXED c9211931 — the llvm20 sections were re-blessed after merging develop, and nothing else moved.
• Representative golden: we diffed it against develop's. Only llvm20 changes: 11 gfx1250 WMMA cases (gemm ×2, conv ×4, conv_dgrad ×1, conv_wgrad ×2, conv_winograd_fused_1d_wmma ×2) move from cases to expected_failures with "intrinsic 'wmma.gfx1250.f32.16x16x32.f16' requires llvm22, got llvm20". The llvm22 and llvm23 sections (llvm24 aliases llvm23) are byte-equal to develop's (ran).
• Differential golden: only ll/llvm20 changes. The same 16 refused configs as in our first round are dropped (conv_implicit_gemm ×8, its dgrad and wgrad ×1 each, gfx1250_lowering ×6). The scoped re-record also blesses, at llvm20 only, 5 configs that develop never blessed at any lane (conv_implicit_gemm 36–38, gfx1250_lowering 111–112); that is harmless. Canonical IR is unchanged (ran).
• Both gates pass on the merge (see Our checks)
Ours (10-29) N2: override-only keys were declared after every base key, so llvm22+ IR moved 🟢 FIXED c9211931 — the 8 keys are back at their develop positions in both base tables.
• Tables: _INTRINSIC_MIN_FLAVOR (lower_llvm.py:958) and ROCKE_LL_INTRINSIC_MIN_FLAVOR (data.cpp:521) hold the same 8 keys and flavors (6 at llvm22, 2 at llvm23). Both base tables list the same 202 keys in the same order. Against develop, _decls_for_flavor drops only those 8 keys at llvm20 and the 2 scaled keys at llvm22 (it also adds one key at llvm23 and llvm24, see N1 below), and every common key keeps its order at all four flavors (ran: table probe).
• IR: we lowered four kernels in both engines at llvm22 and llvm23: gemm t32x32x32, winograd fused 1x7_4s2_c64_valu, scaled-WMMA fp8 and the gfx950 f8f6f4 atom. The output is byte-identical between the engines and equal to develop's. At llvm20 both engines refuse the WMMA kernels with the same "requires llvm22, got llvm20" text (ran).
• Tests: two new tests pin the base order and the table subset (test_gfx1250_scaled_wmma.py:428, :440).
• PR body: "the arch-domain artifacts are unchanged" is stale. The llvm20 and llvm22 columns still drop those 8 and 2 rows, consistently with the filter (ran)
Ours (10-29) N3: other keys that their columns mark name_absent on every arch are still declarable 🟡 DEFERRED — CongMa13: "That is a separate sweep. I will open a follow-up". That is acceptable for this PR, but no follow-up issue or PR exists yet (ran: gh search).
• Exposed: 52 llvm20 keys and 17 llvm22 keys (ran: table probe). _require_declared passes them, and the arch-domain lane ignores name_absent by design (domain.py:170), so only each handler's own gate and the developer-run validity gate stand in the way.
• Gates: many of these keys sit behind _require_llvm23_target_fact (lower_llvm.py:1963, 20 call sites: cluster loads, async LDS stores, prefetch, s_wait_tensorcnt, barriers). At least seven tile.* handlers have no flavor gate (lower_llvm.py:4918–:4970: av_load/store_b128, s_alloc_vgpr, asyncmark, wait_asyncmark, s_wait_event, s_prefetch_inst), and they demand 11 of the 17 llvm22 keys. A one-op asyncmark kernel for gfx1250 lowers at llvm20 and llvm22 in both engines and declares @llvm.amdgcn.asyncmark(), with no refusal and no warning (ran).
• Live failures: none today. No library kernel uses those ops (ran: grep), and the PR body reports the validity gate green at llvm20 and llvm22 (read) → open the follow-up and link it here
Bots / CI 🟡
• At head 56e9d40f (23:06Z): the policy bot, pre-commit, clang-tidy, Component CI, ASAN and both hipDNN superbuilds (Linux and Windows) pass. A superseded duplicate policy-bot run was cancelled.
• TheRock Multi-Arch CI was still building at 00:18Z. Compiler-runtime and emulation are done, and math-libs for gfx94X, gfx950, gfx125X and Windows gfx110X are in progress. No hipkernelprovider test shard has run at this head, so the goldens that were red in CI at d2a11a56 are green only in our runs so far.
• GitHub: blocked on review; review is requested from cgmillette, and there are no reviews or threads (read: gh)
Our checks 🟢
• Range: the only new code is c9211931. b79ed54c and 8b03bd28 are tree-identical merges of develop e8b95982, and head 56e9d40f has the same tree as c9211931 (ran).
• Merge 05a81114 (head + develop d730d162; 4 behind, 7 ahead, 0 conflicts):
– native build OK and ctest 33/33 (1 skip);
– byte identity GREEN for every family at llvm22 (the llvm23-only families are compared at llvm23);
– representative golden PASS ×3;
– differential golden OK at llvm20–24 and in canonical IR, with 0 changed and 0 missing (the unblessed NEW configs are develop's own, since the PR adds no configs);
– the PR's test file: 20 passed, 38 subtests;
– Rules 1–3 clean, with no links or tracker IDs added (ran).
Develop has since moved to c7e46358 (#14656), which touches no rocKE files (read: gh).
• Land order with #14679 (per-family golden shards): merge-tree with its head 025062e8 gives one conflict, a modify/delete on the monolithic golden. Its 5 affected shards still hold develop's entries, including these 11 cases as llvm20 ok (ran).
– If #14679 lands first, this PR must git rm the old file and re-bless from platform/ with PYTHONPATH=python:../library python tests/instances/rocke_ir_parity_harness.py --write tests/golden/rocke_representative_ir_sha256 --only conv,conv_dgrad,conv_wgrad,conv_winograd_fused_1d_wmma,gemm. Only those 11 llvm20 entries should move.
– If this PR lands first, #14679 must keep the deletion and re-bless the same 5 shards (read: its harness)
# Sev file:line Finding
N1 🟠 lower_llvm.py:1003, data.cpp:507 c9211931 adds an llvm23 override for a new key, mfma.scale.f32.16x16x128.f8f6f4, in both engines. Nothing demands it: both engines use …f8f6f4.v8i32.v8i32, which the llvm23 column marks ok on gfx950, so no IR changes (ran: gfx950 f8f6f4 atom, both engines, equal to develop). But _decls_for_flavor("llvm23") now holds a key that the llvm23 column never measured. With the flavor pinned to llvm23, test_arch_domain_artifact.py:130 fails on the merge ("has no answer for 1 decl key(s): ['mfma.scale.f32.16x16x128.f8f6f4']") and passes on develop (77 passed), so the reply's "also occurs on develop" does not hold. CI will not catch it: CI resolves llvm23 but does not install this source-tree test, and its reproducibility CTest skips on a clang 24 vs llvm23 mismatch (read: CI log at d2a11a56). The comment at lower_llvm.py:999 still says no intrinsic needs a further override → drop the entry from both tables (ran: the test at llvm23 on the merge and on develop)
N2 🟡 check_byte_identity.py:42 c9211931 adds conv_winograd_fused_1d_wmma to TARGET_OP_PROFILE_FAMILIES. The byte-identity gate therefore no longer compares its C++ parity pair at llvm22 as well as llvm20, although the family lowers at llvm22; only llvm20 lacks gfx1250 WMMA. Our probe of one config found both engines byte-identical at llvm22 (ran), but nothing gates that any more → exclude the family only below llvm22, for example with a separate list keyed on the flavor

Utility: 🟢 Positive — closes the call-without-declare gap in both engines; only llvm20 output changes.


Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

Follow-up review: a local qwen3.6 model made the independent first pass, and the orchestrator (Claude Opus 5.5) verified it. It traced the branch since d2a11a56: one new commit and two tree-identical develop merges. On a clean merge with develop, it ran the native build and ctest, byte identity at llvm22, both golden gates and the PR's test file. It also diffed both goldens against develop's and compared the Python and C++ declaration tables. It lowered four gfx1250 and gfx950 kernels in both engines at llvm20, llvm22 and llvm23 and compared the output with develop's. It ran the arch-domain artifact test with the flavor pinned to llvm23 on the merge and on develop, and probed an ungated gfx1250 op at each flavor. It ran merge-tree against #14679 and compared #14679's shards with develop's golden. CI at head and the hipkernelprovider shard logs from the run at d2a11a56 were read. The orchestrator owns this merge-worthiness call. This review flags whether a closer look is needed; it is not an approval or a rejection.

@cgmillette

Copy link
Copy Markdown
Contributor

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14437 — declare gfx1250 WMMA intrinsics only on the LLVM flavors that have them (CongMa13, AICK-2788) — 🔴 Follow-up: CI update — TheRock's gfx94X shards ran at this head, and both fail on a conv dispatch test this change breaks (N3)

Prior item Status
Ours (10-30 00:25Z) Bots / CI: TheRock's hipkernelprovider shards had not run at 56e9d40f 🔴 They have now. Both Linux gfx94X shards (1/2 and 2/2) fail rocke_library_conv_pytest on one test, tests/library/tests/dispatch/test_conv_winograd_dispatch.py::test_unfused_plans_need_llvm23[llvm20] (1 failed, 621 passed in each); the other 48 lanes pass (read: CI logs). See N3
Ours (10-30 00:25Z) N1 🟠, N2 🟡 Unchanged: no push since
# Sev file:line Finding
N3 🔴 library/dispatch/conv_winograd_plan.py:58, :222, :434; library/tests/dispatch/test_conv_winograd_dispatch.py:243–:244 Problem: for gfx1250 (_ARCH, test :30) at llvm20, the grouped-conv dispatcher still picks the fused 1-D Winograd plan, and its kernel step claims every flavor (llvm_flavors is None, :58; the test asserts it at :243). That kernel calls wmma.gfx1250.f32.16x16x32.f16, which this PR no longer declares below llvm22, so lowering it at llvm20 (:244) now raises NotImplementedError: intrinsic 'wmma.gfx1250.f32.16x16x32.f16' requires llvm22, got llvm20 (lower_llvm.py:1894). The test fails on the merge with develop and passes on develop (ran: the test on merge 05a81114 and on develop 587f9942). The PR's own re-bless already lists both conv_winograd_fused_1d_wmma/gfx1250 cases as llvm20 expected failures, so the dispatcher and the golden now disagree.
Consequence: CI is red at this head, and at llvm20 a gfx1250 1-D grouped conv dispatches to a plan that cannot be built.
Fix: give the fused 1-D step the flavors its kernel can lower at (llvm22 and later when it uses the gfx1250 WMMA atom), as the K3 step does at :222, so the flavor check at :434 refuses llvm20 with a reason; then update the test's llvm20 case to expect that refusal for the 1-D request as well, and re-run rocke_library_conv_pytest

Utility: 🟢 Positive — closes the call-without-declare gap in both engines; the conv dispatcher must follow it.


Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

CI update to our follow-up at this head: the orchestrator (Claude Opus 5.5) read both red gfx94X hipkernelprovider shard logs and re-ran the failing test on a clean merge with develop and on develop. The orchestrator owns this merge-worthiness call. This review flags whether a closer look is needed; it is not an approval or a rejection.

CongMa13 and others added 3 commits October 30, 2026 14:27
…ke/gfx1250-decl-flavor-tables

Resolve conflicts in check_byte_identity.py (keep both family lists) and in
the two IR goldens (re-blessed: no hash changes against develop).
The fused kernel issues the gfx1250 f16 WMMA, which llvm20 no longer
declares. Declare its LLVM flavors so the dispatcher admits no plan there,
and update test_unfused_plans_need_llvm23 to expect that on llvm20.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…cale decl key

The llvm23 declaration table gained mfma.scale.f32.16x16x128.f8f6f4 without
a row in intrinsic_arch_domain.llvm23.json, so
test_every_decl_key_is_covered_on_every_wired_arch failed on llvm23.
Regenerated with tools/gen_arch_domain.py on the llvm23 toolchain.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@CongMa13
CongMa13 requested a review from a team as a code owner November 2, 2026 20:44
@cgmillette

Copy link
Copy Markdown
Contributor

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14437 — declare gfx1250 WMMA intrinsics only on the LLVM flavors that have them (CongMa13, AICK-2788) — 🟠 Follow-up: conflicts, test break, second gate

Prior item Status
Reviewers 🟢 None yet
• no reviews or threads; review re-requested 11-02 (read)
• needs: rebase (N4), N1 and N5–N7, CI, approval + code owner
Ours (10-29) N3: other name_absent keys still declarable 🟡 DEFERRED
• CongMa13: "That is a separate sweep. I will open a follow-up"
• fine for this PR; no follow-up filed yet (ran: gh search)
Ours (10-30) N1: unused llvm23 key mfma.scale.f32.16x16x128.f8f6f4 🟠 OPEN
• llvm23 row added 13079488; artifact test passes at head (ran)
• merge: Python's llvm24 inherits the key, the llvm24 column and C++ array don't; test fails at llvm24 (ran)
• fix: drop the key in both engines, revert 13079488
Ours (10-30) N2: byte identity skips fused 1-D at llvm22 🟡 OPEN
• unchanged; the develop merge kept both family lists (read)
Ours (10-30) N3: fused 1-D plan admitted at llvm20 🟢 FIXED 7e2e8c6e
• llvm20 admits no plan; llvm22 lowers (ran)
• dispatch tests: head 17 ✓; merge with develop's AOT tests 31 ✓ (ran)
Ours (10-30) PR body: "arch-domain artifacts are unchanged" 🟡 OPEN
• still says so; rows go at llvm20/llvm22, one added at llvm23 (read)
• no word on the Winograd gate 7e2e8c6e
Bots / CI 🟡 No PR CI since 10-29
• conflicts: pre-commit, Multi-Arch, Component, hipDNN not run at 7e2e8c6e, 13079488
• 5 ✓; policy bot waits on pre-commit
• merge: conflicting (read)
Our checks 🟠 Conflict
• merge vs 56cb51f9: 3 files conflict, so no build or byte identity (ran); N4
• scratch merge: both goldens drift at llvm20 only, as intended (ran)
• tests: head 267 ✓; merge, 27 develop files: 2 cases fail only there, N7 (ran)
• land order: open #14328 conflicts in 8 files (ran); rules 1–3 clean (ran)
Code quality 🟠 Should-fix
• N5: a second flavor gate, in both engines (ran)
• N6: the C++ refusal is untested; its error kind diverges (read)
• fused 1-D flavor list restated by hand (in N5)
# Sev file:line Finding
N4 🟠 lower_llvm/data.cpp:529; both IR goldens Conflicts with develop in 3 files, so no PR CI runs
• impact: 14692's llvm24 array, 14679's shard split, 14386's grouped-conv configs; the PR's differential also blesses 194–275 develop configs per lane
• fix: keep both data.cpp blocks; take develop's goldens; re-record llvm20 only: shards conv,conv_dgrad,conv_wgrad,conv_winograd_fused_1d_wmma,gemm (13 cases), differential conv_implicit_gemm{,_dgrad,_wgrad},gfx1250_lowering (21 configs) (ran: scratch merge)
N5 🟠 lower_llvm.py:974, lower_llvm/data.cpp:529, lower_llvm/core.cpp:451 A per-key minimum-flavor table duplicates the profile gates, in both engines
• impact: the 2 scaled rows change no refusal (ran); the 6 WMMA rows restate develop's measured llvm20 rows, which the PR deletes
• fix: one llvm22+ profile capability, checked where each engine emits gfx1250 WMMA, as the scaled path does; keep _require_declared for unknown keys; restore the rows; derive the fused 1-D list from it
N6 🟠 lower_llvm/core.cpp:512, :524 The C++ refusal has no test, and its unknown-key status differs
• impact: no lane drives it (the differential runs --python-only); ROCKE_ERR_VALUE where Python raises KeyError
• fix: ROCKE_ERR_KEY (PYTHON_STYLE §13.5); a future_intrinsic_lowering.cpp case at llvm20 pinning Python's text
N7 🟠 library/tests/test_attention_tiled_cache_key.py:488 Breaks develop's tiled-attention cache-key test at llvm20
• impact: 2 gfx1250 cases now raise "requires llvm22", pass on develop (ran); no CI lane runs the file (read)
• fix: run gfx1250 only at the flavors that lower it (N5's query)

Utility: 🟢 Positive — still needed: develop declares WMMA intrinsics that LLVM 20 lacks.


Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

Follow-up review: first pass by a local qwen3.6 model, verified by the orchestrator (Claude Opus 5.5). Checks: merge-tree with develop and #14328, scratch merge (both goldens, arch-domain test per flavor, 27 develop test files), llvm20 lowering on develop vs the merge, CI at head, code quality. Flags whether a closer look is needed; not an approval or rejection.

CongMa13 and others added 3 commits November 3, 2026 11:03
- Drop the llvm23 override for mfma.scale.f32.16x16x128.f8f6f4 in both
  engines. No kernel demands it, and the llvm24 column and C++ array did not
  hold it. Revert the artifact regeneration that added its row.
- Compare conv_winograd_fused_1d_wmma in the byte-identity default lane from
  llvm22 on. Only flavors below llvm22 move it to the llvm23 lane.
- Raise ROCKE_ERR_KEY for a key that no table holds, as Python raises
  KeyError. Add a C++ case that pins the llvm20 refusal text of the gfx1250
  f16 WMMA and the llvm22 declare.
- Skip the gfx1250 cases of the tiled-attention cache-key test on flavors
  below llvm22.
- Take the sharded representative golden from develop. Re-bless the llvm20
  sections of the conv, conv_dgrad, conv_wgrad, gemm and
  conv_winograd_fused_1d_wmma shards and the llvm20 differential golden.

ISSUE ID : AICK-2788

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the rocke-lib: attention Touches rocKE library attention/FMHA kernels label Nov 3, 2026
@CongMa13

CongMa13 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@cgmillette Head a1506abb1b5 merges the latest develop and addresses the open items.

  • N4 (conflicts): fixed. The merge keeps both data.cpp blocks. The representative golden is now sharded on develop, so I took develop's shards. I re-blessed only the LLVM 20 sections of conv, conv_dgrad, conv_wgrad, gemm and conv_winograd_fused_1d_wmma, and the LLVM 20 differential entries. LLVM 22 and later sections are unchanged.
  • N1 (unused LLVM 23 key): fixed. I dropped mfma.scale.f32.16x16x128.f8f6f4 from the LLVM 23 override tables in both engines. I reverted the artifact regeneration 13079488.
  • N2 (byte identity at LLVM 22): fixed. conv_winograd_fused_1d_wmma moves to the LLVM 23 lane only for flavors below LLVM 22.
  • N6 (C++ refusal): fixed. An unknown key now raises ROCKE_ERR_KEY. The new case gfx1250_wmma_requires_llvm22 in future_intrinsic_lowering.cpp pins the LLVM 20 refusal text and the LLVM 22 declare.
  • N7 (tiled-attention cache-key test): fixed. The gfx1250 cases skip on flavors below LLVM 22.
  • PR body: updated. It describes the artifact rows, the Winograd gate and the new tests.
  • N5 (one profile capability instead of the per-key table): deferred. It reverses the declare-order fix for your earlier N2 and changes both engines. I will take it as a follow-up unless you want it here.
  • N3 (other name_absent keys): still deferred to a follow-up.

Test results:

  • check_byte_identity.py: GREEN on LLVM 22 and LLVM 23 after the merge. I did not re-run LLVM 20.
  • Python suite: LLVM 22 and LLVM 23 fail 28 tests that need HIP headers or a built standalone importer. LLVM 20 fails 6 timeouts in test_tf32_numerics_validation.py. GPU tests and the arch-domain regeneration tests did not run.
  • Tiled-attention cache-key and Winograd dispatch tests pass on LLVM 20, LLVM 22 and LLVM 23.
  • C++ rocke_future_intrinsic_lowering: 90 cases pass.
  • pre-commit passed on the commit.

@cgmillette

Copy link
Copy Markdown
Contributor

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14437 — declare gfx1250 WMMA intrinsics only on the LLVM flavors that have them (CongMa13, AICK-2788) — 🔴 Follow-up: merge breaks matrix-call tests

Prior item Status
Reviewers 🟢 None yet
• no reviews or threads; cgmillette's review requested (read)
• needs: rebase (N8), N5, CI, approval + code owner
Ours (10-29) N3: other name_absent keys still declarable 🟡 DEFERRED
• CongMa13: "still deferred to a follow-up"; no issue filed since 10-29 (ran: gh search)
• fine for this PR once an issue is linked
Ours (10-30) N1: unused llvm23 key mfma.scale.f32.16x16x128.f8f6f4 🟢 FIXED a1506abb
• dropped in both engines; 13079488 reverted by 107205fd
• probe: no flavor table holds it, at head and on the local merge (ran)
Ours (10-30) N2: byte identity skips fused 1-D at llvm22 🟢 FIXED a1506abb
• llvm22+ compare it in the default lane, llvm20 in the llvm23 lane (ran: local merge)
Ours (10-30) N3: fused 1-D plan admitted at llvm20 🟢 FIXED 7e2e8c6e
• dispatch test passes at head and on the local merge (ran)
Ours (10-30) PR body: "arch-domain artifacts are unchanged" 🟡 OPEN
• fixed: the body lists the dropped column rows; they match the diff (ran)
• test plan still ticks llvm20 byte identity and "0 failures"; its own notes say otherwise (read)
Ours (11-02) N4: conflicts with develop in 3 files 🟢 FIXED b4c28815
• resolved; the llvm20-only re-bless holds as a set: 5 shards, differential ll/llvm20 (ran)
• conflicts again since #14328 landed: N8
Ours (11-02) N5: per-key minimum-flavor table duplicates the profile gates 🟠 OPEN
• deferral reason fails a probe: a handler gate keeps llvm22–24 IR byte-identical (ran)
• the per-key filter is what fails #14328's tests (N8) (ran)
• do it in the rebase, or link an issue; none filed (ran: gh search)
Ours (11-02) N6: C++ refusal untested; unknown-key status differs 🟢 FIXED a1506abb
• new case pins status and text; passes on the local merge build (ran)
• unknown key → ROCKE_ERR_KEY (read)
Ours (11-02) N7: breaks the tiled-attention cache-key test at llvm20 🟢 FIXED a1506abb
• gfx1250 cases skip below llvm22; 6 ✓ at head and on the local merge (ran)
Bots / CI 🟡 No PR CI since 10-29
• conflicting since #14328 landed 11-03: no pre-commit, Multi-Arch, Component, hipDNN at a1506abb
• 3 ✓ · 1 cancelled (PR bot, superseded); policy bot waits on pre-commit
• merge: conflicting (read)
Our checks 🔴 Merge breaks tests
• merge vs 59b6a114: 6 files conflict, all from #14328's 4aa4ab4a (ran)
• local merge, resolved: build ✓; ctest 1 of 46 fails (N8); byte identity GREEN, every family, llvm20–24 (ran)
• goldens: re-record moves llvm20 only (gemm: 2 cases); both gates pass (ran)
• 33 test files: 4 ✗ / 4169 ✓, all 4 #14328's, merge only (ran)
• author's tf32 timeouts and HIP-header failures: host-side, not the PR's (ran); rules 1–3 clean (ran)
Code quality 🟠 Should-fix
• N5: deferral premise fails a probe (ran); llvm22 floor kept in 5 places (read)
• N9: the emit-order test cannot catch N2's regression (ran)
• N6 fixed: C++ refusal CI-tested, ROCKE_ERR_KEY (read)
# Sev file:line Finding
N8 🔴 develop's lower_llvm.py:1064, lower_llvm/core.cpp:484; #14328's tests/core/matrix_call_lowering.cpp:300, test_matrix_calls.py:140, :602, test_flavor_decl_tables.py:62 Conflicts with #14328 in 6 files; resolved, 5 of its CI tests fail
• impact: rocke_matrix_call_lowering aborts ("requires llvm22, got llvm20"); rocke_pytest 4 ✗; CI's rocKE lanes run all 5; they pass on develop (ran: local merge)
• fix: rebase with N5's handler gate, which leaves these tests as they are (read); a kept filter must replace develop's decls = dict(…), outside the markers
N9 🟡 tests/core/test_gfx1250_scaled_wmma.py:428 The emit-order test cannot fail for the regression it is named for
• impact: with our 10-29 N2 regression simulated, it passes while llvm22 IR changes (ran)
• fix: assert the declare order in lowered llvm22 IR, or drop it with the table (N5)

Utility: 🟢 Positive — still needed: develop lowers gfx1250 WMMA at llvm20, which LLVM 20 lacks.


Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

Follow-up review: first pass by a local qwen3.6 model, verified by the orchestrator (Claude Opus 5.5). Checks: merge-tree with develop, local merge (build, ctest, byte identity llvm20–24, both goldens, 33 test files vs develop), saved probes, CI at head, code quality. Flags whether a closer look is needed; not an approval or rejection.

Replace the per-key minimum-flavor declaration table with a
`GFX1250_WMMA` emission-profile capability in both engines. The
hand-written gfx1250 WMMA handlers check it before they demand the
intrinsic. The declaration tables are the same as on develop, so the
emit order and the arch-domain columns do not change.

Keep `_require_declared` (Python) and the matching check in
`rocke_ll_need` (C++) for keys that no table declares.

Re-bless the llvm20 sections of the gemm and conv_direct_dw_fwd_wmma
shards and of the differential golden. Compare both families in the
llvm23 lane of the byte-identity gate below llvm22.

ISSUE ID : AICK-2788

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@CongMa13 CongMa13 changed the title fix(rocke): declare gfx1250 WMMA intrinsics only on the LLVM flavors that have them fix(rocke): refuse gfx1250 WMMA on LLVM flavors that lack the intrinsics Nov 4, 2026
@CongMa13

CongMa13 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

@cgmillette Head ae7f6e640c3 merges the latest develop (031d56af855) and addresses the open items.

  • N8 (conflicts with feat(rocke): backend completeness for every production target and dense MMA instruction #14328): fixed. The per-key minimum-flavor table is gone. The declaration tables now equal develop's, so feat(rocke): backend completeness for every production target and dense MMA instruction #14328's tests run unchanged.
  • N5 (profile capability instead of the per-key table): fixed. A new GFX1250_WMMA capability (LLVM 22 and later) exists in Python (_LlvmEmissionCapability) and C++ (ROCKE_LL_CAP_GFX1250_WMMA). The hand-written gfx1250 WMMA handlers check it before _need: emit_wmma and _emit_wmma_fp8 in Python, _emit_wmma in C++. The refusal text is unchanged: intrinsic '<key>' requires llvm22, got llvm20. The scaled path keeps its existing LLVM 23 check. _require_declared stays, for keys that no table declares only.
  • Arch-domain rows: restored. The columns equal develop's again.
  • N9 (emit-order test): fixed. The table-order and minimum-flavor tests are removed with the table. New tests pin the LLVM 20 refusal and the LLVM 22 to LLVM 24 declare, and the KeyError for an unknown key.
  • N3 (other name_absent keys): follow-up AICK-3035.
  • PR body and title: updated to the new design.

One more change came from the merge. conv_direct_dw_fwd_wmma (new on develop) uses the same WMMA, so LLVM 20 refuses every config. It joins conv_winograd_fused_1d_wmma in the byte-identity gate, which compares both in the LLVM 23 lane below LLVM 22. I re-blessed only the LLVM 20 sections: the gemm and conv_direct_dw_fwd_wmma shards, and the seven conv_direct_dw_fwd_wmma entries of the differential golden. LLVM 22 and later are unchanged.

Test results on the merge:

  • check_byte_identity.py: GREEN on LLVM 20, LLVM 22, LLVM 23 and LLVM 24.
  • Native build and ctest: 46 of 47 pass. rocke_optimization_barrier_hip_pair times out on a node without HIP headers.
  • test_gfx1250_scaled_wmma.py, test_matrix_calls.py, test_flavor_decl_tables.py, test_arch_domain_artifact.py (without the regeneration test): 161 pass on each of the four flavors.
  • Winograd dispatch tests: 17 pass. Tiled-attention cache-key gfx1250 cases: 31 pass. Both on LLVM 20 and LLVM 22.
  • Differential golden check, Python only: LLVM 22 to LLVM 24 show no change.
  • pre-commit passed.

Not run: the full Python suite on LLVM 20 after this merge, the IR validity gate, GPU and Windows.

@cgmillette

Copy link
Copy Markdown
Contributor

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14437 — refuse gfx1250 WMMA on LLVM flavors that lack the intrinsics (CongMa13, AICK-2788) — 🔴 Follow-up: merge fails llvm20 goldens

Prior item Status
Reviewers 🟢 None yet
• no reviews or threads; cgmillette's review requested (read)
• needs: N10, N11, approval + code owner
Ours (10-30, 11-02) N1, N2, N3, N4, N6, N7 🟢 Still FIXED
• N1, N6, N7 probes FIXED on the merge (ran)
• no conflicts; fused 1-D refused at llvm20; byte identity GREEN ×4 (ran)
Ours (10-29) N3: other name_absent keys still declarable 🟡 DEFERRED
• CongMa13: "follow-up AICK-3035"
• AICK-3035 (11-04) lists the same 52 / 17 keys, 7 ungated handlers (read)
• fine for this PR
Ours (10-30) PR body: "arch-domain artifacts are unchanged" 🟡 OPEN
• columns now equal develop's (ran: diff); title updated (read)
• body still describes the removed min-flavor table and column pruning (read)
• fix: rewrite How it works, Change surface, Properties 1, 4, 7, test plan
Ours (11-02) N5: per-key minimum-flavor table duplicates the profile gates 🟢 FIXED ae7f6e64
• table gone; GFX1250_WMMA capability checked before _need, both engines (read)
• drop either engine's check: its new test fails (ran)
• left: floor restated in conv_winograd_fused_1d_wmma.py:62, two tests (read)
Ours (11-04) N8: conflicts with 14328; 5 of its CI tests fail 🟢 FIXED ae7f6e64
• decl tables equal develop's; probe FIXED (ran)
• 14328's ctest, test_matrix_calls.py, test_flavor_decl_tables.py pass on the merge (ran)
Ours (11-04) N9: emit-order test cannot fail for its regression 🟢 FIXED ae7f6e64
• order test removed with the table (read)
• new tests fail on a gate dropped in Python or C++ (ran)
• the fp8 handler has no test: N11
Bots / CI 🟢 Green
• 45 ✓ · 51 skipped · 1 cancelled (PR bot, superseded) (read)
• rocKE shards ✓ ×4; the run predates 14834, so N10 is unseen (read)
• merge: blocked (no approval yet)
Our checks 🔴 Merge fails goldens
• merge vs dc983fd1, then f28ed8f0: clean (ran)
• build + ctest 46 ✓ / 1 skip; byte identity GREEN ×4, every family (ran)
• goldens: llvm20 only, 4 drift lines + 6 gone (N10); canonical ✓ (ran)
• tests: changed 378 ✓, installed 34 ✓; 13 related files 1478 ✓; at llvm20 1 ✗, merge only (N10) (ran)
• compile 30, 0 spills; rules 1–3 clean (ran); cache-key test in no CI lane (read)
Code quality 🟠 Should-fix
• N11: fp8 WMMA refusal untested in both engines (ran: mutants)
• N5 fixed; three hand copies of the floor remain (read)
• N9 moot; N6 still fixed; engines agree at 4 flavors (ran)
# Sev file:line Finding
N10 🔴 …/rocke_representative_ir_sha256/conv_direct_dgrad.json:6, :14; …/differential/golden/llvm_gfx_all.json:2194; library/tests/test_conv_dgrad_depthwise_windowed.py:501 Merge with today's develop fails both llvm20 goldens
• impact: 14834 (landed 11-05) added gfx1250 WMMA dgrad; this PR refuses it at llvm20 (ran)
• impact: rocke_golden_static 4 drift lines, rocke_differential_golden_ll 6 gone (ran); CI runs both on all 4 rocKE shards (read)
• impact: 14834's WMMA lowering test fails at llvm20, merge only (ran)
• fix: merge develop; --write … --only conv_direct_dgrad; run_diff.py … --record-golden --flavor llvm20 --only conv_direct_dgrad_ext; skip that test where _supports_gfx1250_wmma is false (ran: all pass, nothing else moves)
N11 🟠 platform/python/rocke/core/isa/backend.py:637, platform/tests/core/future_intrinsic_lowering.cpp:3792 The fp8 WMMA refusal (4 of 6 keys) has no test in either engine
• impact: drop it in Python or C++: new tests, 26 related files, goldens pass; llvm20 lowers fp8 again (ran)
• fix: one check at the top of emit_wmma (:587), as C++ does; add a K64 fp8 atom to both refusal tests (ran: pass now, fail on each mutant)

Utility: 🟢 Positive — LLVM 20 now refuses gfx1250 WMMA cleanly; re-record and fp8 test left


Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

Follow-up review: first pass by a local qwen3.6 model, verified by the orchestrator (Claude Opus 5.5). Checks: merge with today's develop (build, ctest, byte identity and both goldens at 4 flavors, a re-record dry run), changed and develop's related tests (llvm24, llvm20), mutants of both engines' gate, saved probes, CI at head, code quality. Flags whether a closer look is needed; not an approval or rejection.

CongMa13 and others added 2 commits November 5, 2026 11:13
- Skip the depthwise dgrad WMMA lowering test where the flavor lacks the
  gfx1250 WMMA intrinsics.
- Re-record the llvm20 sections of the conv_direct_dgrad golden and the
  differential golden for conv_direct_dgrad_ext.
- Pin the llvm20 refusal of the fp8 K64 WMMA in both engines.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@CongMa13

CongMa13 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@cgmillette Head 872a02705e4 merges the latest develop and addresses N10 and N11. The PR description now matches the current design.

  • N10 (llvm20 goldens fail after feat(rocke): gfx1250 direct dgrad for small groups at stride 2 and Toeplitz WMMA depthwise dgrad #14834): fixed.
    • The representative golden check at LLVM 20 reported the drift you listed: dw_dgrad_win_wmma_k5 and dw_dgrad_win_wmma_k7 in conv_direct_dgrad. Both cases are now expected refusals. Only the conv_direct_dgrad shard changed.
    • The six conv_direct_dgrad_ext LLVM 20 configs that lower WMMA are re-recorded in the differential golden.
    • test_lowers_to_wmma_with_lds_staging skips on a flavor without the gfx1250 WMMA capability. It failed at LLVM 20 on the merge before this change.
  • N11 (FP8 refusal untested): fixed with tests. The handler is unchanged.
    • _emit_wmma_fp8 already calls _require_gfx1250_wmma, so LLVM 20 refused FP8 before this change. No test covered it.
    • A K=64 FP8 atom now has a refusal test in Python (test_the_fp8_wmma_requires_llvm22) and in C++ (case_gfx1250_wmma_requires_llvm22). Both check the LLVM 20 text and the LLVM 22 declare.
    • I did not move the check to the top of emit_wmma. Each handler keeps its own call. Tell me if you want one check.
    • I did not run a mutant check that drops the gate and expects the new tests to fail.

Test results after the merge:

  • Representative golden tests in test_rocke_ci_static.py: 8 pass.
  • Differential golden check, Python only: passes on LLVM 20, LLVM 22, LLVM 23 and LLVM 24.
  • test_gfx1250_scaled_wmma.py: 18 pass.
  • C++ rocke_future_intrinsic_lowering: 92 cases pass.
  • pre-commit passed on commit.

Not run after this merge: check_byte_identity.py, the full ctest and Python suites, the IR validity gate, GPU and Windows.

@cgmillette

Copy link
Copy Markdown
Contributor

Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max

PR #14437 — refuse gfx1250 WMMA on LLVM flavors that lack the intrinsics (CongMa13, AICK-2788) — 🔴 Follow-up: llvm20 goldens fail after 14832

Prior item Status
Reviewers 🟢 None yet
• no reviews or threads; cgmillette's review requested (read)
• needs: N12 at the develop merge, then 1 approval + code owners (rocke-platform, rocke-lib)
Ours (10-30 – 11-04) N1–N9 🟢 Still FIXED
• N1, N5, N6, N7, N8 probes FIXED at head (ran)
• no engine change since fu5; their tests pass on today's merge (ran)
Ours (10-29) N3: other name_absent keys still declarable 🟡 DEFERRED
• CongMa13 (body): "Follow-up AICK-3035 covers them"
• AICK-3035 still Open (read); fine for this PR
Ours (10-30) PR body: "arch-domain artifacts are unchanged" 🟢 FIXED
• body now describes the capability gate; columns equal develop's (ran: diff)
• after the rebase, add conv_wgrad_two_stage to its llvm20 list
Ours (11-05) N10: merge fails both llvm20 goldens 🟢 FIXED 872a0270
• dgrad cases recorded at llvm20; their WMMA test skips there (ran)
• holds on today's merge; 14832 repeats the pattern: N12 (ran)
Ours (11-05) N11: fp8 WMMA refusal has no test 🟢 FIXED 872a0270
• fp8 K64 case in both engines; each fails without its fp8 gate (ran)
• one check atop emit_wmma not needed now; 2 of 6 keys pinned: N13
Bots / CI 🟢 Green
• 45 ✓ · 52 skipped · 1 cancelled (PR bot, superseded); rocKE shards ✓ ×4 (read)
• Windows attempt 1 timed out in rocke_pytest, 0 failed: replaced Azure runners; attempt 2 ✓ (read)
• run predates 14832; merge: conflicting, so no new CI until develop is merged (read)
Our checks 🔴 Merge fails goldens
• merge vs 41d662e6: conv_wgrad.json conflicts; the checks skipped build and gates (ran)
• own resolution with develop's file: both goldens and 2 tests fail at llvm20 (N12); re-recorded: gates ✓ ×4 (ran)
• 20 test files match develop at llvm24; wgrad byte identity ✓ ×4; rules 1–3 clean (ran)
Code quality 🟡 Minor
• N11 fixed: fp8 refusal pinned in both engines (ran)
• fp8 tests copy the f16 ones; 2 of 6 keys pinned: N13 (ran)
• dgrad skip asks the capability; 3 older copies restate llvm22 (read)
# Sev file:line Finding
N12 🔴 …/rocke_representative_ir_sha256/conv_wgrad.json, conv_wgrad_two_stage.json; …/differential/golden/llvm_gfx_all.json; library/tests/test_conv_wgrad_correctness.py:1260, :1310 Merge with today's develop fails both llvm20 goldens again
• impact: 14832 (11-06 01:21Z) records 6 gfx1250 wgrad cases and 15 differential configs as lowered at llvm20; this PR refuses all of them (ran)
• impact: with develop's conv_wgrad.json, rocke_golden_static drifts on those 6, rocke_differential_golden_ll loses the 15, and 2 of 14832's tests fail at llvm20 (ran)
• fix: take develop's conv_wgrad.json; --write … --only conv_wgrad,conv_wgrad_two_stage; run_diff.py … --record-golden --flavor llvm20 --only conv_implicit_gemm_wgrad; skip both *_dual_engine tests where _supports_gfx1250_wmma is false (ran: gates ✓ ×4, only llvm20 moves)
N13 🟡 platform/tests/core/test_gfx1250_scaled_wmma.py:461, platform/tests/core/future_intrinsic_lowering.cpp:4105 The new fp8 tests copy the f16 ones and pin 2 of 6 keys
• impact: a gate skipped for the 3 bf8 keys passes them (ran: Python mutant); C++ pins the same key (read)
• fix: one test over both key tables × LLVM_FLAVORS, re.escape on the text; one loop in C++ (ran: the Python sweep passes now, fails each mutant)

Utility: 🟢 Positive — llvm20 refuses gfx1250 WMMA in both engines; rebase needs 14832's re-record


Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted

Follow-up review: first pass by a local qwen3.6 model, verified by the orchestrator (Claude Opus 5.5). Checks: merge with today's develop (conflict resolved, both goldens at 4 flavors before and after a re-record, byte identity of the family 14832 changed), 20 changed and related test files on the merge and develop (llvm24, llvm20), mutants of both engines' gate, saved probes, CI at head, code quality. Flags whether a closer look is needed; not an approval or rejection.

@CongMa13

CongMa13 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

This PR has gotten stuck in a review loop. Every time I resolve a conflict and push an update, a new conflict pops up. It's been going on for five days now. Until we come up with a better way to handle these recurring merge conflicts, I'm going to close this PR.

@CongMa13 CongMa13 closed this Nov 6, 2026
@tenpercent

Copy link
Copy Markdown
Contributor

This PR has gotten stuck in a review loop. Every time I resolve a conflict and push an update, a new conflict pops up. It's been going on for five days now. Until we come up with a better way to handle these recurring merge conflicts, I'm going to close this PR.

perhaps running hotspot analysis could help factoring out things that change more frequently than they need

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

organization: ROCm project: hip-kernel-provider rocke: platform Touches rocKE platform rocke-lib: attention Touches rocKE library attention/FMHA kernels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants