Repository navigation
feat(hip-kernel-provider): gfx950 unified SDPA rankers and LSE-on kernels - #15113
ThruptiRajLakshmanaGowda wants to merge 16 commits into
Conversation
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
The gfx950 unified attention engines need LSE-on kernel instances to serve
SDPA graphs that ask for stats. Two gaps blocked that:
- The tiled 2D kernel stored its LSE token-major ([total_q, Hq, 1]), but
hipDNN's stats tensor is head-major [B, Hq, S, 1]. A new kw-only spec
field, lse_layout ("token_major" | "head_major", needs emit_lse), selects
the store index through unified_lse_offset in _lse_store.py, at all three
store sites. The C++ lowerer mirrors it (gfx950 and gfx942 instances).
- A packaged 3D module is built from its segment spec alone, which had no
emit_lse. UnifiedAttention3DTiledSpec gains emit_lse (kw-only, default
off); reduce_spec_for reads it unless the caller passes one, so the
dispatcher, which passes the problem's value, behaves as before. The
segment IR does not change. At decode the reduce's token-major LSE is
also the head-major layout.
Both fields join a kernel name only when on ("lse", "lsehm"), and the
dispatcher leaves them out of config keys and spec hashes at their
defaults, so every existing spec name, key, hash and IR is unchanged: the
default-spec identity sweep matches develop (43,398 specs) and the C++
lowerer's .ll output is byte-identical to Python's.
Co-Authored-By: Claude <noreply@anthropic.com>
An engine manifest could only ship specs the dispatcher's registry selects, so a kernel built with an output the registry never turns on (such as an LSE store) could not be enumerated, and so could not pass the selection's g1 "in registry" gate. An enumerate.also section may now set: - spec_overrides: spec fields replaced on every spec it yields. They are always shipped in the kernel spec, and specs that differed only in an overridden field collapse to one; - kernel_name_template: its own name template, so its kernels are new names beside the other sections'. Its default is the default: true floor of its own (cell, overrides) group, so variant reduction keeps one twin per cell. A metadata field the section's spec omits may now be read off the registry spec when it is at its dataclass default: a builder given a spec without the field builds that default, so metadata still matches the binary. Co-Authored-By: Claude <noreply@anthropic.com>
…atalogs Each cell of both engines gains one LSE-on twin of its default kernel, for SDPA graphs that ask for stats: - Unified2D: 198 `*_lsehm` twins of the baseline-geometry kernel (emit_lse, lse_layout head_major: the hipDNN stats layout [B, Hq, S, 1]). 222 -> 420 kernels. - Unified3D: 116 `*_lse` twins of the 16-segment kernel, whose reduce also writes the LSE (token-major, which at decode is the head-major [B, Hq, 1, 1] layout). 123 -> 239 kernels. The manifests enumerate the twins through an also section with spec_overrides. The selections add one variants group each, and the KMD gains emit_lse (int, default 0). Every existing kernel keeps its name, id, spec and sources; its metadata only gains emit_lse 0. Each twin's spec is its cell's default spec plus the LSE fields. The registry space of the 3D manifest is now 464, over its 400 budget (a WARN); the shipped selection, which the budget gates, is 239. The KDPs are re-added to DVC (not pushed). Matching stats graphs to the twins is the next change. Co-Authored-By: Claude <noreply@anthropic.com>
…SE-on kernels Both gfx950 unified engines declined every graph with generate_stats. A graph with generate_stats and a stats tensor is now served when the stats tensor is FP32, dims [B, Hq, Sq, 1], with head-major strides, the layout the LSE-on kernels write. Unit-extent axes are exempt, so at decode (Sq 1) the token-major spelling, which is the same memory, is accepted too. A lone stats tensor or a lone generate_stats, and the max, sum_exp and rng_dump outputs, still decline. - matchAttributes takes the MatchContext and binds the stats uid (NO_STATS when the graph asks for none). - kernelMatches pairs it with the candidate's new emit_lse metadata in both directions: only an LSE-on kernel serves a stats graph, and it serves no other. A record without emit_lse, or with a value other than 0 or 1, is malformed. - The 2D kernel's and the 3D reduce's signatures append lse_ptr for an LSE-on kernel, at offsets 112 and 40 (read off code objects of the shipped LSE specs). launch passes the stats buffer there. prepare refuses a candidate whose emit_lse disagrees with the binding. The bundle READMEs give the new counts (2D 420, 3D 239). Co-Authored-By: Claude <noreply@anthropic.com>
…SE-on kernels No bundled SdpaFwd graph reached the gfx950 unified LSE-on kernels: of the 56 graphs with a stats output, 54 are BHSD and 2 are paged, which these engines decline. quick/SdpaFwd/BshdStats adds six dense BSHD graphs with a head-major FP32 stats output [B, Hq, Sq, 1]: - four prefill cases for Unified2D: bf16 and fp16, D64 and D128, each mask kind, one in a multi-variant cell (Hq32/Hkv8 bottom-right); - two decode cases for Unified3D: bf16 D128 Hq32/Hkv4 and fp16 D64 Hq8/Hkv1. They carry no golden data: the harness compares against the CPU and GPU reference executors, which both produce the stats output. support.json starts empty; claims are recorded from a device run. The gfx950 attention test categories select the sweep in every tier, and SUPPORT_MATRIX_OVERVIEW.md is regenerated. Co-Authored-By: Claude <noreply@anthropic.com>
Each cell's LSE-on twin has the same geometry and knobs as its default kernel, so a forced-knob case matched two catalog kernels and failed its exactly-one check (12 Unified2D cases on MI355X). The cases' graphs ask for no stats, so kernelMatches declines the twin; the expected-kernel key now says so too. Co-Authored-By: Claude <noreply@anthropic.com>
…SE-on engines Harvested from an MI355X (gfx950, Linux) run of the engine-pinned lanes, where each graph passed against the CPU and the GPU reference: the four prefill cases on hipkernel:Gfx950AttentionUnified2D and the two decode cases on hipkernel:Gfx950AttentionUnified3D. A broken claim now fails instead of skipping. SUPPORT_MATRIX_OVERVIEW.md is regenerated. Co-Authored-By: Claude <noreply@anthropic.com>
…their catalog A trained sort_kernel_catalog model is bound in its UED as "<arch>": [<uhd id>, ...] beside the native default. The generator wrote only the default, so a regeneration dropped the binding. Worse, the loader ties a model to the engine's UED/KMD/UMD ids only, so a catalog edit that keeps them (a kernel added, removed or renamed) leaves the model ranking a catalog it never saw, with no refusal. engine.trained_rankers.<arch> records the bound uhd_ids and the trained_catalog_sha256 of the catalog the models were trained on: - the loader validates the block (gfx arch keys, distinct UUIDs, a 64-hex digest; refused on slotted engines and with heuristic: none); - build_ued emits the arch lists after the native default, as uhd_gen promote leaves them; - codegen/catalog_fingerprint.py hashes each kernel's name, metadata and spec(s), independent of descriptor ids and kernel order, and tools/catalog_fingerprint.py prints it for a KDP; - tests/test_trained_rankers.py covers the loader, the UED and the fingerprint, and gates every shipped config that declares rankers: the shipped KDP must still fingerprint to the recorded value (else retrain), the shipped UED must bind exactly the recorded ids, and each must be installed under heuristics/. The README documents the key and the retraining step. Co-Authored-By: Claude <noreply@anthropic.com>
… attention 2D/3D
hipkernel:Gfx950AttentionUnified2D and ...Unified3D rank their catalogs on
gfx950 with trained sort_kernel_catalog models, one per metric (tflops,
time), bound beside the native scorer, which still ranks every other arch.
heuristic.uhd.json is in git and model.bin in DVC (model.bin.dvc).
- Data: every kernel timed on MI355X over the cells that ship more than
one variant: 810 graphs in 21 Unified2D cells and 250 graphs in 5
Unified3D cells (hipdnn_corpus_gen, hipdnn_bench, uhd_gen).
- Held out, against the native scorer:
- Unified2D: geomean regret 1.0010 vs 1.0178, top-1 97.9% vs 84.5%,
no graph more than 5% slower (worst 1.002x), 97 graphs;
- Unified3D: 1.0015 / 1.0007 vs 1.0771, top-1 92% / 96% vs 64%,
no graph more than 5% slower, 25 graphs.
- Stage V on MI355X: the runtime reads each model, no winner record
decides, and the regret gate passes (validate_l2.py, decide.py l2
--baseline native). The device suites are green with all four bound.
engine.trained_rankers in both configs records the bound ids and the
fingerprint of the catalog they were trained on, so a regeneration keeps
the binding and a catalog change fails tests/test_trained_rankers.py until
retrained. The bundle READMEs say how.
Co-Authored-By: Claude <noreply@anthropic.com>
…cfg71 #15024's later review fix (ca119e0, merged as part of #15024) added its own cfg71 (bf16 D192 one warp) to gfx950_attention_tiled_2d_emit.{c,py} after our LSE phase was authored against an earlier snapshot of that branch. The C switch caught the collision (duplicate `case 71`, a build failure); the Python dict assignment did not -- `_CONFIGS[71] = {...}` silently overwrote cfg71's dict entry with no error. Move our three LSE head-major twins (idx64-66 + head-major) from 71-73 to 72-74, after cfg71. The parity bindings list (platform/cpp/bindings/attention_tiled_parity_configs.py) is positional, not index-keyed, so it already had the correct order (71=cfg71, 72-74=LSE) and needed no change; `check_byte_identity --only attention_tiled` is GREEN across all 6 families, gfx950_attention_tiled_2d now 75/75. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
0537250 to
6a6e38b
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #15113 +/- ##
===========================================
- Coverage 77.76% 77.75% -0.01%
===========================================
Files 2904 2904
Lines 491429 491207 -222
Branches 74499 74462 -37
===========================================
- Hits 382149 381920 -229
- Misses 84278 84291 +13
+ Partials 25002 24996 -6
*This pull request uses carry forward flags. Click here to find out more. 🚀 New features to boost your workflow:
|
#15101 adds engine.heuristics (role -> arch -> UHD ids) to the generator for the same job engine.trained_rankers did here. Adopt its shape, and its loader, emission order, README section and tests verbatim, so whichever PR lands second only drops duplicated hunks. What this PR keeps is the catalog gate, now its own optional key: engine.trained_catalog_sha256 maps each arch bound under heuristics.sort_kernel_catalog to the fingerprint of the catalog its models were trained on. When set, its keys must equal those arch keys. tests/test_trained_rankers.py still fails when a shipped KDP no longer matches, so a catalog edit forces a retrain. The unified 2D/3D configs move to the new keys. Their UEDs regenerate byte-identical. Co-Authored-By: Claude <noreply@anthropic.com>
The first Unified3D models were trained on 250 graphs. A widened corpus (744 more graphs from denser shape grids over the same 5 multi-variant cells, timed on MI355X) brings it to 994. On 100 held-out graphs the retrained models raise top-1 from 86% to 96% (tflops) and 97% (time), and cut the graphs more than 5% slower than native from 7-10 to 1, at geomean regret 1.0029 / 1.0015 vs native 1.0962. The same widening for Unified2D (3,330 graphs) scored the same as the shipped models on 352 held-out graphs, so those stay. Both READMEs now report the wider held-out results, including the one graph per model that is more than 5% slower than native, and the engine.heuristics binding. Co-Authored-By: Claude <noreply@anthropic.com>
|
Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max PR #15113 — feat(hip-kernel-provider): gfx950 unified SDPA rankers and LSE-on kernels (ThruptiRajLakshmanaGowda, AICK-2543, AICK-2889) — 🔴 First-round: CI red; body perf figures
Utility: 🟢 Positive — stats graphs served by LSE-on twins; trained rankers with a catalog gate Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted First-round review: first pass by a local qwen3.6 model, verified by the orchestrator (Claude Opus 5.5). Checks: scratch merge with develop, native build + ctest, byte identity, binding parity, goldens, changed tests, CI logs, code quality. Flags whether a closer look is needed; not an approval or rejection. |
#15101 (engine.heuristics) has merged. This branch carried its hunks verbatim from the draft. The conflicts were where its final form moved lines next to ours: keep develop's engine.heuristics code and add back only this PR's engine.trained_catalog_sha256 (EngineSpec field, the _KNOWN_ENGINE entry, the README schema line). SUPPORT_MATRIX_OVERVIEW.md is regenerated from the merged claims (develop's 12335 graphs plus the 6 BshdStats graphs). Co-Authored-By: Claude <noreply@anthropic.com>
|
Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: max PR #15113 — feat(hip-kernel-provider): gfx950 unified SDPA rankers and LSE-on kernels (ThruptiRajLakshmanaGowda, AICK-2543, AICK-2889) — 🔴 Follow-up: CI red; README, body figures
Utility: 🟢 Positive — LSE-on twins serve stats graphs; trained rankers bind through develop's key Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted Follow-up review: first pass by a local qwen3.6 model, verified by the orchestrator (Claude Opus 5.5). Checks: merge with today's develop, native build + ctest, byte identity, binding parity, goldens, changed tests (source and installed), IngestorGenerator tests, struct layouts, saved probes, CI and superbuild logs, code quality. Flags whether a closer look is needed; not an approval or rejection. |
…,N6,N7,N8)
- hipdnn_assets.sdpa_fwd_golden_graphs: skip any sweep root's
graph.template.json by name instead of filtering on a "Default" folder
name, so the new quick/SdpaFwd/BshdStats sweep root's template is no
longer mistranslated as a golden graph (N1).
- rocke_engine_attention.cpp t3d_build: read "emit_lse" into the 3D segment
spec (it was only read into the derived reduce spec), so the C++ binding's
segment kernel name gains the same "_lse" suffix Python's does. Added a
parity config (idx 20) across the Python/C standalone emitters and the
bindings table that pairs segment+reduce emit_lse, covering the fix (N3).
- catalog fingerprint: moved the trained-ranker catalog identity from
codegen/catalog_fingerprint.py into tools/catalog_fingerprint.py, keyed on
catalog_diff.entry_key instead of a narrower hand-rolled identity, so a
builder, target, arch or priority edit the regeneration gate (g3) catches
also moves the ranker's fingerprint. Added builder/target/arch/priority
cases to test_trained_rankers.py (N4).
- attention_tiled_parity_configs.py: fixed a stale comment (said idx68-70,
the head-major triple is actually at idx72-74) (N6).
- gfx950_attention_unified_{2d,3d} READMEs: dropped absolute per-graph
timings from the trained-ranker sections (ratios only) and named the
architecture (gfx950) instead of the product (N7).
- test_attention_unified_2d_lse_ir.py: dropped the local _IntBuilder in
favor of rocke.helpers.tiling.lds_conflict.NumBuilder, which already
implements the same int-evaluator ops (N8).
Follow-ups needing a GPU/DVC-enabled environment (not done here):
- re-record engine.trained_catalog_sha256 for both Unified2D/3D configs
with the new tools/catalog_fingerprint.py (N4).
- record differential goldens for the new head-major parity configs 72-74
via run_diff.py --record-golden (N6).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Companion to bd776ae (git add missed these paths in one shot: a nonexistent pathspec aborted that `git add` before staging them). Same N1/N3/N4/N6/N7/N8 fixes described there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves conflicts between this branch's spec_overrides/kernel_name_template feature and develop's selector_fields/neutral_fields/cell_keyed_by merging feature in registry_enumerator.py: - keep the stricter spec_overrides manifest validation plus the kernel_name_template check - key enumerate_space's default-floor tracking by both the full (merged) cell and the section's overrides group - drop the now-redundant early spec_overrides application in Registry.space(), since cell_spaces() applies it once via _overridden() - regenerate SUPPORT_MATRIX_OVERVIEW.md rather than hand-merge the generated numbers - keep both independently-added test blocks in test_registry_enumerator.py Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Automated review: Model - qwen3.6:35b-a3b-coding-mtp-q4_K_M, effort: high PR #15113 — feat(hip-kernel-provider): gfx950 unified SDPA rankers and LSE-on kernels (ThruptiRajLakshmanaGowda, AICK-2543, AICK-2889) — 🔴 Follow-up: develop merge broke generator
Utility: 🟢 Positive — LSE-on twins serve stats graphs; trained rankers bind through develop's key Legend: 🔴 blocking · 🟠 should-fix · 🟡 minor nit · 🟢 info/good · ⚪ retracted Follow-up review: first pass by qwen3.6 (local), verified by the orchestrator (Claude Opus 5.5). Checks: merge with today's develop and a trial resolution, native build + ctest, byte identity, binding parity, differential goldens, changed tests, IngestorGenerator tests incl. develop's new binding gate, struct layouts, saved probes, CI logs, code quality. Flags whether a closer look is needed; not an approval or rejection. |
Summary
Trained catalog rankers for the gfx950 unified attention ingestor engines, and LSE-on kernels so
they serve SDPA graphs that ask for stats.
hipkernel:Gfx950AttentionUnified2Dand...Unified3Drank their catalogson gfx950 with trained
sort_kernel_catalogUHD models, one per metric (tflops,time),bound beside the native scorer, which still ranks every other arch. On held-out graphs the
models pick the fastest kernel 95–97% of the time (native: 82% Unified2D, 73% Unified3D) and
beat the native pick by more than 2% on 52–53 of 352 Unified2D and 25–26 of 100 Unified3D
graphs. One held-out graph per model is more than 5% slower than native (details below).
222 → 420 kernels, Unified3D 123 → 239. The engines serve an SdpaFwd graph with
generate_statsand an FP32 head-major[B, Hq, Sq, 1]stats tensor. Only the twins serve sucha graph, and they serve no other.
enumerate.also.spec_overridesenumerates the twins. Models are boundwith
engine.heuristics(feat(hipdnn): IngestorGenerator emits per-arch heuristic bindings (engine.heuristics) #15101, now on develop). This PR addsengine.trained_catalog_sha256, which records the catalog each model was trained on, and a test failswhen the catalog changes under it (forcing a retrain).
Rebased onto develop after #15024 (runtime heads, window and batch for the gfx942/gfx950
tiled attention) merged; this PR carries no copy of it. Follows #13030 for shipping UHD
models (role-map binding,
heuristics/<ued id>/…layout); should be merged only after #13030lands, since this PR's candidate-tree layout and model binding build directly on that convention.
JIRA ID : AICK-2543
JIRA ID : AICK-2889
Risk Assessment
Risk 3 of 5.
trained models replace the native order there. Engine choice between engines does not change.
A model the loader refuses falls back to the native scorer for that metric, not to an error.
spec and code object; its metadata only gains
emit_lse: 0. Graphs without stats never matcha twin (
kernelMatchespairsemit_lsewith the bound stats uid).lse_layoutand 3Demit_lse. Default specs are byte-identical, as are names, cache keys and IR (43,398 defaultspecs swept). The C++ lowerer mirrors them, and its
.lloutput matches Python's.registry_enumeratorgainsspec_overridesand per-section nametemplates. The generator gains
engine.trained_catalog_sha256, beside feat(hipdnn): IngestorGenerator emits per-arch heuristic bindings (engine.heuristics) #15101'sengine.heuristics. All are opt-in; no other engine's output changes.ASIC Coverage
gfx950 only (MI355X measured). The models are bound under the
gfx950arch key; other archs keepthe native
default.Testing Summary
quick/SdpaFwd/BshdStats(6 graphs)test_attention_unified_lse_numeric.py+test_attention_tiled_runtime_shape_numeric.py: 247 passed, 11 skipped (gfx942-only shapes)loaded; no refusal or fallback warning (hipDNN log at info, Unified2D and Unified3D lanes)validate_l2.py,decide.py l2 --baseline native)hipdnn_validate_descriptorson the packed gfx950 treetest_trained_rankers), on develop with #15101 mergedcheck_byte_identity --only attention_tiledRanker results on held-out graphs (never trained on), against the native scorer. Regret is
the chosen kernel's time over the fastest kernel's.
The first version of this PR reported 0 graphs more than 5% slower than native. That was on 97 / 25
held-out graphs. A 4× wider corpus (below) raised the held-out set to 352 / 100 and found these
cases. They persist when comparing minimum times (1.07–1.18×), so they aren't timing noise. They are small in absolute time, and the models still beat
native on every aggregate. Retraining Unified3D on the wider corpus improved it (top-1 86% → 96–97%,
graphs > 5% slower than native 7–10 → 1), so those models were replaced. Retraining Unified2D gave
the same result as the shipped models, which stay.
Ground truth: no kernel-owner timings were supplied (
ground_truth.py compareneeds them). Thetuned baseline above is the exhaustive per-graph autotune: every catalog kernel was timed on
every graph. Whether this satisfies the tickets' AC 2 is an open question on AICK-2543.
Packaging and performance statistics
AOT budget (kernel instances per arch): 420 ≤ 1000 and 239 ≤ 400. Provider load time:
median 0.164 s vs 0.154 s for the same tree without this PR's additions (+10 ms, 5 alternating runs each on MI355X), peak RSS unchanged (~100 MB).
Training data:
valid:
comparison above);
hipdnn_corpus_gen→hipdnn_bench→uhd_gen;Testing Checklist
model.bin);dvc status -c -r ingestorin syncTechnical Changes
kernels/gfx950/attention_tiled_{2d,3d}.py,kernels/common/_lse_store.py, the C++ lowerer mirrors, dispatcher keys):lse_layout(token-major | head-major), applied throughunified_lse_offsetat all three store sites;emit_lseon the segment spec, so a packaged module's reduce writes the LSE;enumerate.also.spec_overrides+kernel_name_template, with a default floor per(cell, overrides) group;
engine.trained_catalog_sha256,codegen/catalog_fingerprint.py,tools/catalog_fingerprint.py,tests/test_trained_rankers.py; README.emit_lse, the KDPs (DVC) and the census for both engines; theLSE-on twins.
UnifiedAttentionGraphMatch.hpp,Gfx950AttentionUnified{2D,3D}Native.cpp):emit_lsepairing inkernelMatches;lse_ptrin the 2D signature at offset 128 and the 3D reduce signature at offset 48, read offcompiled code objects;
prepareconsistency checks;heuristics/…/sort_kernel_catalog/gfx950/{tflops,time}in both bundles,bound in the UEDs; README "Trained ranker" sections.
quick/SdpaFwd/BshdStatswith claims;emit_lse 0;emitsLse-conditional signature check (base counts are perf(rocke): runtime heads, window and batch for gfx942/gfx950 tiled attention #15024's, already fixed there after a flag on that PR; the LSE branch is ours).check_byte_identitycatches this class of collision; the C switch failed to compile, the Python dict assignment did not).🤖 Generated with Claude Code