ingestor: registry-driven selection, generic enumeration and variant reduction for rocKE engines by BrianHarrisonAMD · Pull Request #13593 · ROCm/rocm-libraries

BrianHarrisonAMD · 2026-10-14T00:07:47Z

Stacked on #13024. Retarget to develop after #13024 merges.

Motivation

#13024 sources the hipkernel:Gfx950AttentionDense catalog from the rocKE dispatcher registry. This PR completes the IngestorGenerator side of that change:

Kernel selection is driven by the registry and checked against it.
The registry's variant space can be reduced to the variants that autotuning shows are needed.
The authoring RUNBOOK works as written.

The shipped catalog is unchanged.

Technical Details

Authored selection and gates. configs/gfx950_attention_dense.yaml is the selection the shipped descriptors are generated from. tests/test_registry_regeneration.py checks three things:
- every selected kernel exists in the registry;
- the shipped catalog regenerates from the selection;
- the selection's header matches the engine manifest.
Generic registry enumeration. tools/registry_enumerator.py works for any rocKE operator family through the dispatcher's KernelCandidate/OperatorRequest contract. It can enumerate the full tuning space with each tuning knob projected into the kernel metadata. It also adds:
- --plan-space: per-knob instance counts;
- --shapes: a coverage report for a shape corpus.
Variant reduction. tools/variant_reduce.py reduces a selection using autotuning results from tools/sweep.py. It keeps:
- each cell's default variant;
- a kernel for every benchmarked graph;
- any variant whose measured speedup exceeds the run-to-run noise.
It proposes knob-level reduction rules for the user to apply.
Owner benchmark comparison. tools/ground_truth.py converts a kernel owner's benchmark results into hipDNN graphs and compares them with hipDNN's autotuned results.
Fixes:
- hkp_pack fails the pack when a builder's support predicate cannot read its spec, and resolves each builder's own predicate;
- launch_surface.py checks additional metadata reads and missing source files;
- sweep.py rejects benchmark arguments that would bypass its cache handling.
Docs. The hipdnn-ingestor-engine RUNBOOK and the IngestorGenerator README are rewritten around the registry-driven flow.

Breaking changes

verify_variant_sets.py and coverage_gate.py take --manifest <engine manifest> instead of --profile.
tools/variant_reachability.py is removed.
The engine manifest's enumerate: section uses a new schema (candidates, request, pin, fixed, cells, space). There is a new top-level reduction: section and an optional ground_truth: section.

Test Plan

Run locally; no GPU or build required:

IngestorGenerator: pytest tests in projects/hipdnn/tools/IngestorGenerator.
Descriptor packaging: PYTHONPATH=descriptor-packaging/python pytest descriptor-packaging/tests in dnn-providers/hip-kernel-provider.
rocKE attention: PYTHONPATH=../platform/python:. pytest tests/dispatch/attention in dnn-providers/hip-kernel-provider/rocke/library.
Regenerate the gfx950 attention catalog from the selection and compare it with the shipped one using tools/catalog_diff.py.

Test Result

IngestorGenerator: 883 passed, 19 skipped (they require a build).
Descriptor packaging: 609 passed, 1 skipped.
rocKE attention: 411 passed.
The regenerated catalog matches the shipped catalog: 840 kernels, no differences.

Not yet exercised: a reduction run on a GPU.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

…hot, compare KMD, drop pydantic skip The regeneration gate compared registry-only spec fields to the dataclass default, so a default change moved both sides and passed. Pin the 17 fields to checked-in literals instead. The end-to-end no-op test no longer skips without pydantic (generate.py does not need it) and now compares the KMD. Clarify that --strict-spec sees only what the regenerated catalog carries.

supports_tiled_2d_spec (gfx942) judged a spec by the predicate's default pipeline flags, so valid transposed-x8 / single-K-buffer / sliced-ring variants were rejected at pack time. Forward them. The gfx950 predicate accepts but does not key admission on the flags; the ones the spec carries are forwarded too. Add behaviour tests for the wrappers and correct the stale hkp_pack and wrapper docstrings.

…nch audit to the engine manifest Delete dispatch_parity, knob_sweep and reconcile_applicability with their tests: the rocKE registry plus the engine manifest replace them. The deleted dense profile was the only real launch_surface declaration, so carry it (and the settled knob evidence) into configs/gfx950_attention_dense.engine.yaml and audit the real manifest in test_launch_surface. Rename the synthetic profile fixtures to engine manifests. Rewrite RUNBOOK sections 2, 6 and 7, workloads.md and the README around the enumerator flow and add 'Updating the captured catalog'. Relabel configs/gfx950_attention_dense.yaml as a reference config, not a source of truth.

…ckage

Registry-only spec fields are checked against the spec dataclass defaults again. A new field or a default change needs no edit here; the shipped KDP is refreshed deliberately with the rest of the integration.

…pack predicates, wider launch-surface audit - verify_variant_sets.py and coverage_gate.py take --manifest (the engine manifest): it gates every bundle of engine.name and the vocabulary is specialization.vocabulary. variant_reachability.py is removed. - hkp_pack: a support predicate that raises AttributeError/TypeError could not read the spec, so the pack fails. The reduce and fastkv-register-P aliases, which never validated anything, are removed. A builder's predicate is the one whose spec parameter matches the builder's spec type, so modules with several builders resolve each builder's own predicate. - rocKE: build_gfx942_4warp_gqa is validated through a wrapper that passes use_d256_fast=True, the flag that builder always compiles with. - launch_surface.py also requires fields read through integerMetadata to be declared, and fails on a python_source file that does not exist. - The rocKE wrapper-forwarding and import-resolution tests are tightened.

…ed selection - registry_enumerator.py drives any rocKE family through the KernelCandidate/OperatorRequest contract; the manifest names the candidates, request class, pin, fixed fields and crossed cells. space: slice reproduces the shipped catalog; space: full yields every production-sweep spec per cell and projects each varying performance knob into the spec, a mandatory KMD field, metadata and the name. A knob the registry leaves unset carries the value of a manifest-named resolved accessor. Over-budget counts warn here and in generate.py. --plan-space prints per-knob instance counts; --shapes sorts a shape corpus into outside / declined / covered / GAP. - The generator config accepts a reduction: block and a per-arm default: marker; neither reaches a descriptor. - configs/gfx950_attention_dense.yaml is the authored selection. test_registry_regeneration.py gates it against the registry (G1), the shipped catalog (G2) and the manifest header (G4). catalog_diff --subset checks shipped is a subset of regenerated.

- variant_reduce.py reduces a variants selection that carries a reduction: block, using the autotune winner shards of a tools/sweep.py run: it keeps each cell's default and divisibility antichain, coverage for every ranked graph and measured uplift above a noise-derived epsilon that fits the budget, writes the reduced selection, and proposes knob-level rules (--apply-rules). - ground_truth.py ingests a kernel owner's benchmark rows onto request fields, builds hipDNN graphs from the manifest's graph template, and compares a sweep's served times and correctness against them within a margin. - sweep.py refuses --autotune and --hipdnn-cache-dir in benchmark argv: the driver owns the benchmark cache these tools read.

…iant reduction The RUNBOOK populates a selection from the registry (shape corpus, user checkpoints for corpus, owner data and knob coverage), gates packing on an approved build/sweep plan, reduces it from autotune rankings, and ends with a final correctness sweep. The README update procedure, config surface and tools table describe the selection, gates and new tools.

codecov · 2026-10-14T05:08:43Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files

@@                               Coverage Diff                               @@
##           users/tlakshma/rocke/attention-ingestor-bridge   #13593   +/-   ##
===============================================================================
  Coverage                                           72.14%   72.14%           
===============================================================================
  Files                                                2864     2864           
  Lines                                              473204   473204           
  Branches                                            69902    69902           
===============================================================================
+ Hits                                               341366   341367    +1     
- Misses                                             108080   108081    +1     
+ Partials                                            23758    23756    -2

Flag	Coverage Δ		*Carryforward flag
TensileLite-CPP	`49.03% <ø> (ø)`		Carriedforward from 12de937
TensileLite-Unit	`83.28% <ø> (ø)`		Carriedforward from 12de937
hipBLAS	`90.62% <ø> (ø)`		Carriedforward from 12de937
hipBLASLt	`38.65% <ø> (ø)`		Carriedforward from 12de937
hipCUB	`82.68% <ø> (ø)`		Carriedforward from 12de937
hipDNN	`86.80% <ø> (+<0.01%)`	⬆️
hipFFT	`54.32% <ø> (ø)`		Carriedforward from 12de937
hipRAND	`76.12% <ø> (ø)`		Carriedforward from 12de937
hipSOLVER	`68.96% <ø> (ø)`		Carriedforward from 12de937
hipSPARSE	`86.99% <ø> (ø)`		Carriedforward from 12de937
rocBLAS	`48.25% <ø> (ø)`		Carriedforward from 12de937
rocFFT	`48.78% <ø> (ø)`		Carriedforward from 12de937
rocRAND	`57.45% <ø> (ø)`		Carriedforward from 12de937
rocSOLVER	`76.83% <ø> (ø)`		Carriedforward from 12de937
rocSPARSE	`74.64% <ø> (ø)`		Carriedforward from 12de937
rocThrust	`93.15% <ø> (ø)`		Carriedforward from 12de937

*This pull request uses carry forward flags. Click here to find out more.
see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ngestor-bridge' into users/bharriso/attention-ingestor-gate-hardening # Conflicts: # projects/hipdnn/tools/ai/skills/hipdnn-ingestor-engine/SKILL.md

yraparti · 2026-10-14T18:33:59Z

Gates that cannot fail CI
Why it matters. variant_reduce fails the build for being over budget, but plan_space and the enumerator do not. So the same budget concept has two outcomes depending on which tool you ran. coverage_gate has three rungs: descriptors well-formed, engine loads, engine serves graphs on a GPU. The last, most meaningful rung is never run and its threshold is never enforced. The tool prints "GATE PASSED", then in the next line says that is not evidence anything was served. That is honest, but a green CI job still means less than it sounds.

Impact. Wiring these into CI today gives false assurance. A catalog with coverage holes, over budget, or disagreeing with ground truth would still pass. The fix is a strict mode (--fail-on-gap or similar) on each tool, and making them all use the same exit-code convention.

yraparti

Review notes on the registry-backed ingestor changes. Findings are ordered by importance; inline comments are attached to the most important ones.

Medium

variant_reduce.render_report describes the eps set even under --apply-rules.
Gates that cannot fail (ground_truth compare, GAP/OVER modes, WARN-only budget, --min-served).
Positional JSON pointer in ground_truth.value_map.
Vocabulary check validates against the mapping it was generated from, not the C++ matcher.

Lower

Failure paths outside EnumerationError in the enumerator.
Duplicated "every arch" magic string.
reduction.performance[*].divides is documentation only; variant_reduce heuristics (2x drift, 0.05 epsilon) are unparameterised.
Removed tools (dispatch_parity, knob_sweep, ...): worth a mapping of which of their checks are now covered by launch_surface and test_registry_regeneration.
The rocke_compile predicate fix is independent and could land separately.

Not reviewed in depth: sweep.py, the attention_tiled_* kernel bodies, most test bodies, the full RUNBOOK/rocke-mining.md/workloads.md text.

…eneric) (#13150) Builds on #12710 (merged as `198636dae1d`); this branch is rebased onto `develop` and contains only its own commits. Turns the gfx1151 SDPA kernels into one kernel set per RDNA generation: `gfx11-generic` code objects for gfx1100, gfx1101, gfx1102, gfx1103, gfx1150, gfx1151, gfx1152 and gfx1153, and `gfx12-generic` code objects for gfx1200 and gfx1201. It then widens what hipDNN can route to them without adding per-shape kernels. ## Does rocKE need to learn generic targets first? Only for gfx12. rocKE IR never names a processor: the LLVM module carries the plain `amdgcn-amd-amdhsa` triple, and only the comgr ISA name selects the processor. `gfx11-generic` was already a catalog row with ISA backends in both engines. What was gfx1151-specific was in the SDPA layer: string gates (`arch == "gfx1151"`, `arch == "gfx1201"`), dispatch arches, CU lookup and packaging. This PR: 1. **rocKE targets** (`feat(rocke): catalog gfx12-generic ...`) - Adds `gfx12-generic` to both catalogs and both backend registries. - Regenerates the llvm20/22/23 intrinsic arch-domain tables. The new column equals gfx1201 cell for cell. The regeneration also records the previously unswept `raw.ptr.buffer.load.f16` row. - Adds `generic_targets` to `arch_specs.json`, plus `generic_arch_from_target_id()` / `rocke_generic_arch_from_target_id()`, mapping device names to generic targets. `base_arch_from_target_id()` keeps processors that have their own catalog row, so `arch="gfx1151"` and `arch="gfx1201"` still build processor-specific code; other members (gfx1100, gfx1200) resolve to their generic row. 2. **SDPA kernels and dispatch** (`feat(rocke): run RDNA WMMA attention as gfx11/gfx12 generic kernel sets`) - The WMMA atom now comes from the target's MMA catalog in both engines: the split-K gfx12 atom when listed, otherwise the gfx11 atom. - Transposed QK relies on the gfx11 operand duplication, so it is accepted on any gfx11 row and rejected on gfx12. Every other feature runs on both generations: dtypes, D16..D256, masks, windows, score features, LSE, additive bias, dense/BHSD/ragged/paged layouts, FP8 KV, output-column tiling, and distinct V widths. - Dispatch admits `gfx11-generic`, `gfx1151`, `gfx12-generic` and `gfx1201`, and reads live CUs from device 0 when it runs the requested object (any family member for a generic request, the named processor otherwise). - The selection policy is retuned for generic objects (see Results). Processor-specific gfx1151 requests keep #12710's wide-launch and D256 tile choices. - GPU tests and the benchmark compile for the device's generic target. Parity configs 136-142 replay representative configs at `gfx11-generic`; their golden hashes equal the replayed gfx1151 configs. 3. **hip-kernel-provider** (`feat(hip-kernel-provider): package RDNA WMMA attention as generic objects`) - rocKE UKDs accept `kernel_source.target: generic`. The packer compiles for the shard's generic processor, and shards stay concrete. - The compile target is recorded in observations and provenance, and the agreement check binds the two. - The IngestorGenerator expresses the target and per-group arch lists. - The catalog now ships to all 10 shards, with transposed-QK variants on gfx11 shards only. The census covers every shard. - The standard variants carry `scheduler_strategy: max-ilp`, the spec public dispatch selects for the same requests, so packaged objects get the generic-object tuning too. - The native score now prefers the 32-key single-wave transposed geometry. 4. **BSHD long-sequence fix** (`perf(rocke): decode heads fastest for head-interleaved dense attention`) - With head-interleaved K/V (BSHD, as from PyTorch's `view(B,S,H,D).transpose(1,2)`), the q-tile-fastest CTA order put one head's rows, a full token apart, in flight. On gfx1151 this lost cache and translation locality at long sequences. - Dense kernels now choose the CTA decode at runtime from the strides: heads fastest when K is head-interleaved, q-tiles fastest otherwise. The grid, ABI and object count are unchanged. 5. **Causal mask plus `window_left`** (`feat(rocke): accept a causal mask with an explicit window_left`): the cuDNN spelling of a causal sliding window now runs as the window `(window_left, 0)` instead of being rejected. 6. **Runtime head dims** (`feat(rocke): runtime head dims for RDNA WMMA attention`) - An opt-in `runtime_head_dims` spec flag lets one object serve every head width up to its compile-time `head_size` (a bucket maximum). Q/K and V/O widths (multiples of 16) and a V head count independent of K's arrive as kernel arguments. - Dense objects read K/V through bounded buffer resources at constant tile offsets, which keeps register pressure at the exact-size kernel's level. Dead Q/K tiles contribute exact zeros; dead V/output columns are never stored. - With the flag off, every existing config emits unchanged IR. The attention ABI is bumped to v6. 7. **Wider hipDNN routing** (`feat(hip-kernel-provider): route more SDPA graphs to RDNA WMMA attention`) - The pack now matches any Q/K and V/O head width that is a multiple of 16 up to 256, and V head counts dividing the query heads. - It also matches an additive FP32 or Q-dtype attention bias (`attn_mask`) with broadcast strides, and every hipDNN scale form: a value, a pass-by-value tensor (constant, or read at launch), or absent = 1.0. - Catalog per dtype: - runtime-head-dims buckets 64/96/128: no-mask, band, band+bias f32/q; - buckets 160/256: no-mask, causal, window, window+bias f32/q; - exact 192 and 256: no-mask, causal, window, kept where the exact kernel is measurably faster; - the 32-key transposed objects on gfx11. - That is **64 objects per gfx11 shard and 56 per gfx12 shard**. Previously, 40 objects covered only D64/96/128/256 with no bias. - Scores prefer transposed, then exact, then the tightest bucket, then mask-specialized objects. The 64-key transposed objects were dropped because the score never selected them. 8. **Tuned registry candidates** (`feat(rocke): register RDNA WMMA attention as tuned registry candidates`) - Each RDNA arch gets an opt-in tuned candidate built with `make_tuned_candidate`, like the gfx950/gfx942 dense candidates: `attention_<arch>_wmma_fwd`, pinned by algorithm `wmma_fmha_fwd` and spec_id `<arch>_wmma_fwd`. Tuning ids and `tuning_knobs` round-trip. - `wmma_rules.py` and `_AXES[(arch, "wmma")]` expose `v_lds_stage`, `causal_tile_skip`, `value_tile_size` and `scheduler_strategy`, plus `block_n` × `num_waves` for transposed QK on gfx11 only. `WmmaFmhaFwdSpec.resolved_scheduler_strategy()` names the unset scheduler so the knob can be projected into metadata. - `AttentionRequest.runtime_head_dims` selects the bucket objects. Only the RDNA WMMA candidates accept it, and it skips the fp16 D64 small-head `v_lds_stage` heuristic so the default spec matches the shipped bucket objects. - The existing `attention_gfx1151_wmma` candidate stays the public auto route and shares the policy and knob space. Public dispatch is unchanged for 29,940 requests, and IR is unchanged for all 1,648 WMMA specs and 163 parity configs. - `KernelId.request_hash` changes for every attention request because `normalized()` now includes `runtime_head_dims` (the same precedent as `emit_lse`); `spec_hash` is unchanged. - With #13593's `registry_enumerator` (`space: slice`, `num_cus` fixed), gfx11-generic and gfx12-generic manifests regenerate exactly the shipped 64 and 56 kernel specs and metadata (names and UUIDs ignored). The full knob space is 238 / 206 instances. The manifests stay out of tree until #13593 lands and the generic-target model is settled with #13129. ## Toolchain, runtime and hardware coverage - **Compile:** LLVM 20 or newer comgr (rocKE's oldest flavor). The LLVM 20 clang of ROCm 7.1 accepts `gfx11-generic`, `gfx12-generic`, `gfx1152` and `gfx1153`. Generic objects never need the member's name, so an older toolchain that cannot name gfx1152/gfx1153 still builds them. - **Load:** ROCm 7.0 or newer. ROCm 7.0.2 (gfx1100), 7.2.0, core 7.13 and core 10.0 load the generic objects. ROCm 6.4.0 on a gfx1151 rejects them (`hipErrorNoBinaryForGpu`), while it loads a code object v6 processor-specific gfx1151 object from the same toolchain. No gfx12 device with a runtime older than 7.2 was available. - **Hardware:** gfx1100, gfx1101, gfx1150, gfx1151, gfx1200 and gfx1201 were run. gfx1102, gfx1103, gfx1152 and gfx1153 are covered only by the LLVM generic-processor definition and were not run. ## Validation - **Device numerics** (rocKE GPU suites, generic objects, current head): | Device | Passed | Skipped | |---|---:|---:| | gfx1100 | 182 | 0 | | gfx1151 | 182 | 0 | | gfx1201 | 174 | 8 (transposed-QK tests) | The suites include the runtime-head-dims tests: - d16-d240 within buckets, dq != dv, and Hv in {1, Hk, Hq}; - bias with masks; - NaN-padded inputs proving no padded column is read into or written from the result; - rows ending exactly at the allocation end. An earlier round also ran gfx1101, gfx1150 and gfx1200 (134/134/126 before the new tests). - **Benchmark correctness:** the 54-case inference benchmark passes 54/54 against its FP64 oracle on every device, in every timed run. - **Code objects:** `gfx11-generic` objects are code object v6, machine 0x054, generic version 1. `gfx12-generic` objects are code object v6, machine 0x059. Both load on every device listed. - **C++/Python parity:** byte-identity is GREEN for all 73 families at llvm20, llvm22 and llvm23; `gfx1151_wmma_fmha_fwd` now has 157 configs (14 runtime-head-dims configs added). The native arch-identity CTest matches Python. - **Provider end-to-end** (gfx1100;gfx1151;gfx1201 shards, current catalog): - Census: 15/15. - `TestGfx1151WmmaAttentionMatchers`: 25/25. - hipDNN `quick_SdpaFwd*` integration bundles: **21 passed** (GPU reference), 20 skipped, 0 failed on each of gfx1100, gfx1151 and gfx1201. #12710's gfx1151-only build gives 17 passed and 24 skipped. - **Python suites:** rocKE CPU suites, IngestorGenerator (862 passed) and descriptor-packaging on a ROCm node (512 passed) all pass. The descriptor-packaging run has one failure, `test_enabled_user_site_runtime_preserved`, which fails identically on #12710's tree in the same cluster Python environment. - pre-commit passes. Golden hashes: `run_diff.py --check-golden` reports unblessed changes in this environment on `develop` itself (638 `ll`, 542 `ir_canonical` at `341b0b606e1`). The WMMA families' entries are re-recorded for the changed configs and report 0 unblessed. ## Results ### Selection policy (generic objects) Scope: relative timing of this selection policy against #12710's selection policy, measured before the BSHD decode and runtime-head-dims commits (both arms then shared the same kernels). The baseline is #12710's gfx1151-tuned profiles applied to the same kernels, with transposed QK limited to gfx11. Workload: the 54-case `benchmarks/gfx1151/attention` inference suite through public dispatch. Method: - Both arms compile for the device's generic target unless noted, and both arms pass the oracle gate in every run. - Two runs per arm, in ABBA order, in one exclusive allocation per device. - Each run times HIP-graph replays of 32 launches and takes the median of 7 batches. - Reported: the geomean of per-case ratios (this / #12710; lower is faster), with the per-case range. One device per processor. | Device | Target | Geomean | Per case | |---|---|---:|---:| | gfx1100 | gfx11-generic | 0.84 | 0.43-1.03 | | gfx1101 | gfx11-generic | 0.82 | 0.37-1.02 | | gfx1150 | gfx11-generic | 0.87 | 0.39-1.08 | | gfx1151 | gfx11-generic | 0.85 | 0.39-1.16 | | gfx1151 | gfx1151 | 0.90 | 0.49-1.03 | | gfx1200 | gfx12-generic | 0.69 | 0.32-1.03 | | gfx1201 | gfx12-generic | 0.63 | 0.29-1.08 | On gfx1151, `gfx11-generic` objects with this policy measured 0.90 (0.48-1.05) against processor-specific gfx1151 objects with #12710's policy, and 0.99 (0.71-1.07) against processor-specific objects with this policy. With #12710's policy alone, generic objects measured 1.06 (0.78-2.26) against processor-specific ones. The gap traced to scheduling under the generic register model, and the `max-ilp` scheduler closes it. Caveats: - Results hold at these measured configurations only, for inference forward attention. - The largest gains are on standard-path cases (tails, packed layouts, score features, decode, and every dense case on gfx12). - On gfx11, aligned dense transposed-QK cases change little apart from wide launches. ### BSHD long sequences B1 H16 bf16 noncausal, S=8192, dense, through public dispatch; median of 5×10 launches. | Device | Kernel | BSHD before | BSHD after | BHSD | |---|---|---:|---:|---:| | gfx1151 | d64 transposed QK | 82-96 ms | 29-42 ms | 15 ms | | gfx1151 | d64 standard | 97 ms | 26 ms | 21 ms | | gfx1151 | d128 transposed QK | 196-201 ms | 151-165 ms | 24 ms | | gfx1100 | d128 transposed QK | 23 ms | 15 ms | 22 ms | BHSD timings and gfx1201 are unchanged within noise. The remaining d128 gap on gfx1151 is not CTA ordering. At d128 with 16 heads each K/V row is exactly 4 KiB, so each 32-row tile touches 32 pages. My inference is that this exhausts the APU's address translation; discrete gfx1100 does not show it. ### Runtime head dims Timing of each runtime-head-dims object against the exact-size object for the same head width (lower is faster), bf16 noncausal, prefill B1 H16 S2048 and decode B8 Hq32 Hkv8 Sq1 Sk4096. Outputs match the exact object bit for bit. | d | Shipped object | gfx1151 prefill | gfx1151 decode | gfx12 prefill | gfx12 decode | |---:|---|---:|---:|---:|---:| | 64 | bucket 64 | 0.94 | 0.93 | 0.99 | 0.96 | | 80 | bucket 96 | 1.12 | 1.02 | 0.98 | 1.06 | | 96 | bucket 96 | 1.02 | 0.97 | 0.93 | 1.01 | | 128 | bucket 128 | 0.80 | 0.85 | 0.91 | 0.99 | | 160 | bucket 160 | 0.84 | 0.81 | 1.16 | 0.96 | | 192 | exact 192 (a 192 bucket would be 1.65 / 2.20 / 0.85 / 0.79) | 1.00 | 1.00 | 1.00 | 1.00 | | 256 | exact 256 (bucket 256: 1.56 / 1.44 / 1.07 / 1.06) | 1.00 | 1.00 | 1.00 | 1.00 | gfx12 columns: gfx1201 up to d128, gfx1200 from d160. These measurements chose the catalog. Up to 128, a bucket is within 2% of the exact kernel, or faster, when d equals the bucket; the one in-bucket width measured, d80 on bucket 96, costs 0.98-1.12×. Bucket 160 is faster than exact on gfx1151 and costs up to 1.16× on gfx12 prefill. Above that, gfx11 kernels sit at the register limit, so exact 192/256 objects stay; they are 1.4-2.2× faster than a runtime bucket on gfx1151. Widths with no exact object (176, 208, 224, 240) run on bucket 256 instead of being declined. Not done here: - A gfx12 transposed-QK variant (the gfx12 standard path already serves those shapes). - Split-KV decode for small batches with long contexts. - The gfx1151 d128 BSHD translation gap above. - Renaming the `gfx1151` module, candidate and pack identifiers (e.g. `hipkernel:Gfx1151WmmaAttention`, which now ships to every RDNA3/3.5/4 shard). - Deriving `TestGfx1151WmmaAttentionMatchers`' standard-catalog selection test from the shipped descriptor instead of its own copy of the catalog. ## Rebase onto develop and third-party review (head `e06957a02bd`) The seven feature commits are replayed onto `develop` (`e581a923f2b`), which already carries #12710 including its review round 2. Conflict resolutions keep develop's #12710 behavior: - The FP8 vector-alignment and byte-extent guards in the gfx1151 binding (`_check_vector_aligned`, `_check_byte_extent_i32`) stay in force with runtime head dims. - `emit_lse` replaces `return_lse` everywhere. - The RDNA census registers each wired arch only when `product` ships `hipkernel:Gfx1151WmmaAttention` for it (`hkp_product_ships_engine_for_arch()`). An independent review of the rebased diff found these issues, now fixed: - `91718c676b9`: the gfx12-generic catalog row didn't match develop's widened memory capabilities. `arch_specs.json` lacked `async_lds_max_dwords`, so `get_target("gfx12-generic")` raised `KeyError`, and the C++ row used the old 3-field initializer, which no longer compiles. Other gaps fixed in the same commit: - The gfx12-generic cells of develop's new `maxnum.f64`/`minnum.f64` arch-domain rows were measured on clang 20 (ROCm 7.1.0), 22 (7.2.0) and 23 (core-7.13); all are `ok`. - `generic_targets` is now the single gfx11-generic membership list. `target.py` reads it, and gfx1154 (already in develop's Python list) is mirrored to C++. - Known-arch expectations include gfx12-generic. - `e430c2da6ba`: a causal mask with `window_left=-1` resolved to an unbounded two-sided window and ran unmasked. The causal diagonal is now kept, with a regression test. - `b5bd11bf9e8` / `e06957a02bd`: the numeric tests use `emit_lse`, and the representative IR goldens are regenerated. Validation on this head: - Windows CPU lanes, rocKE `library/tests` + `platform/tests` (no GPU): the failure set is **identical to develop's** (comgr, `ml_dtypes` and Windows-only environment failures). The full-lane comparison (3,790 passed vs develop's 3,765) ran before the causal-bound fix and the llvm23 arch-domain cells; the dispatch, arch-identity and gfx1250-foundation suites re-run on the final head pass (719 passed, 0 failed). - Arch-domain coverage test passes for the llvm23 column; the llvm20 host-flavor gap for develop's `cluster.*`/prefetch keys exists on develop too. - `run_diff --check-golden`, C engine built from develop and from this head on Alola. The `ll` comparison used `ROCKE_LLVM_FLAVOR=llvm20`, the flavor the golden was blessed with. - No anchor that passes on develop fails here: `ll` passes 253 on develop and 280 here; `ir_canonical` passes 315 on develop and 336 here. - All 157 `gfx1151_wmma_fmha_fwd` configs match their goldens in both modes, and the 6 `gfx1201_wmma_fmha_fwd` anchors that are stale on develop now pass. - The C-vs-Python `ir` DRIFT families are identical on develop. - `pre-commit run --from-ref origin/develop --to-ref HEAD` passes. - Not re-run on this head: GPU numerics, the hipDNN integration bundles, and the 54-case benchmark (the device results above are from the pre-rebase head). This round's code changes are the catalog row, the causal window bound and test wiring. The emitted kernels are unchanged: all 157 `gfx1151_wmma_fmha_fwd` goldens match. ## Independent review, round 2 (head `feb7b7d7ee4`) Fresh review of the full diff against merge base `97fc1832835`, covering the arch catalog, kernel emission (both engines), dispatch, and hip-kernel-provider/IngestorGenerator. Fixed in this PR: - `34f8b95fe6e`: C++ `rocke_generic_arch_alias` kept its own gfx11 member list beside `rocke_ati_generic_targets`. It now reads the table, so `generic_targets` is the only membership list in both engines. Behavior unchanged; the native `rocke_arch_target_identity` binary passes and `test_cpp_target_identity_matches_python` passes against it (43/43, Linux build). - `feb7b7d7ee4`: `_resolve_num_cus` lent a sibling's CU count to a processor-specific request (`arch="gfx1151"` on a gfx1100 host took gfx1100's count for the output-tiling gate). Generic requests still read any family member, but processor requests now require that processor on device 0. A regression subtest fails before the fix and passes after it. The README now says only device 0 is read, and that 64-column D256 noncausal tiles apply only to gfx11-generic objects. Reviewed with no defect found: the gfx12-generic rows (JSON vs C++, field by field; arch-domain column equals gfx1201 in all three tables); runtime-head-dims memory safety (dead Q/K tiles zeroed after load, so NaN padding cannot leak; bounded rsrc extents within the i32 byte-extent guard); BSHD decode bijection; the kernarg order vs the native `kernelSignature()`/`launchWith()`; matcher legality vs the binder; yaml vs checked-in kdp.json (identical expansion). Fresh validation on `0db16e5c758` (new clones and venvs; the code changes since then are host-side selection/arch-alias code that GPU tests don't exercise): | Device | ROCm / LLVM | fmha numeric | dispatch numeric | fp8 decode | benchmark | |---|---|---:|---:|---:|---:| | gfx1100 | 7.2.0 / llvm22 | 41 | 141 (+27 subtests) | 4 | 54/54 | | gfx1100 | 7.0.2 / llvm20 | 41 | 141 (+27) | 4 | 54/54 | | gfx1150 | 7.2.0 / llvm22 | 41 | 141 (+27) | 4 | 54/54 | | gfx1201 | 7.2.0 / llvm22 | 33 (+8 transposed-QK skips) | 141 (+27) | 4 | 54/54 | - No gfx1151 node was usable: one was drained, and the other has ROCm 6.4 only. - Descriptor-packaging on gfx1100: `test_hkp_pack_rocke.py` passes 54/54 (base: 40/40). The full suite gives 618 passed and 1 failed. The failure is `test_enabled_user_site_runtime_preserved`; it fails identically on the base because the cluster system Python contains a nameless `argcomplete` dist-info. - Windows CPU, rocKE `library/tests/dispatch` + `platform/tests/core` + spec/multiarch: the failure set (14) is identical to the base, with 1925 passed vs 1891. - IngestorGenerator: 873 passed. - Golden changes are confined to `gfx1151_wmma_fmha_fwd` (126 per mode) and 6 previously stale `gfx1201_wmma_fmha_fwd` `ll` anchors. - pre-commit passes on the changed files. Left for follow-ups (not this PR): the remaining items under "Not done here"; device-0-only CU lookup (multi-GPU hosts with a mixed device 0); the descriptor-packaging probe's robustness to nameless dist-info. ## Review round 3 (AnaghaRaoAMD), head `601a18c94fa` - `ef7698617c1`: `gfx1200` requests resolve to `gfx12-generic`; before this, no RDNA candidate matched them. Both engines fold any `generic_targets` member that has no row of its own. The processor-specific live-CU check compares base archs. - `289619c0d15`, `601a18c94fa`: the WMMA GPU suites run from `rocke_library_gpu_pytest`, and the CPU spec, selection and tuning tests from `rocke_library_attention_pytest`, which is labelled for CI `quick` selection. - `90660821818`: on-device `causal + window_left=-1` test. - `2934d00bfc4`: an IngestorGenerator test keeps the WMMA catalog `target: generic`, which the native score assumes. - `bc71331224e`: the engine declares graph schema 1.2.0, so runtime pass-by-value scale graphs reach the matcher; before this, every `SdpaFwdRuntimeScale` graph was declined. Override-shape graphs are declined. Validation: - rocKE GPU suites (`2934d00bfc4`, from fresh clones): fmha 41 (33 + 8 transposed-QK skips on gfx12), dispatch 143, fp8 decode 4, benchmark 54/54, on gfx1100 (llvm22 and llvm20), gfx1150, gfx1201 and gfx1200. - hipDNN forward SDPA, all tiers (engine pinned, GPU reference; `bc71331224e` code): 88 passed / 14 skipped / 0 failed on gfx1100, gfx1201 and gfx1200. The skips are FP8 or varlen graphs. - gfx120X Multi-Arch CI dispatch on `601a18c94fa` (https://github.com/ROCm/rocm-libraries/actions/runs/38698822336): `hipkernelprovider` and `hipdnn-integration-tests` pass on Linux gfx1200 and Windows gfx120X. The WMMA suites ran on the gfx1200 runner.

… for the gfx950 dense-attention ingestor catalog (#13024) ## Summary Makes the rocKE dispatcher registry the single source of truth for the gfx950 dense-attention ingestor catalog (`hipkernel:Gfx950AttentionDense`, 840 kernels, unchanged), so drift between the registry and the shipped catalog fails a test. The IngestorGenerator gates used to maintain the catalog now refuse input they cannot verify. Builds on #13593. JIRA ID : AICK-2513 ## Risk Assessment Risk 3. Most of the change is developer tooling plus a build-time `hkp_pack` pre-flight; the shipped catalog regenerates with no differences and no engine C++ changes. The rocKE gfx950 dense predicate gains an LDS-capacity gate and a `block_n=256` tile. The branch is merged with current develop, whose gfx950 dense kernel changes (#13125) are packed into this engine, so this PR also carries a gfx950 hardware run (below). ## ASIC Coverage gfx950 only. The predicate and tile changes affect the gfx950 dense catalog; everything else is host-only. Required before merge: PR CI and one gfx950 run of the dense engine (done at `75c5a107cb`; later commits change tests only). Other architectures are covered by PR CI: their catalogs do not change, and CI packs them through the same pre-flight. ## Testing Summary - **Host (Python):** IngestorGenerator, descriptor-packaging and rocKE attention suites; dense catalog regeneration from the selection and from the manifest. - **Host (C++ and CTest):** superbuild with the CI flags (`GPU_TARGETS=gfx950`, ROCKE, kernel ingestor, SDPA, packaging probes ON), provider unit tests with the GPU hidden, and the pack, probe, census and test-name lanes. - **gfx950 (MI355X):** dense `SdpaFwd` integration (`quick`, `standard`, GPU reference) with support claims enforced, gfx950 attention unit tests, census. ## Testing Checklist All at this head unless noted, Linux, host tests with `HIP_VISIBLE_DEVICES=-1`. - [x] Superbuild, CI flags - `ninja` - 0 warnings - Status: Passed - [x] IngestorGenerator - `pytest tests` - 1,150 passed, 9 skipped (flatbuffers headers absent) - Status: Passed - [x] Descriptor packaging - `pytest descriptor-packaging/tests` - 697 passed, 1 skipped; `TestRealArchiveReaders` also passes without PyYAML - Status: Passed - [x] rocKE attention - `pytest tests/test_attention_dense_gfx950_lds.py tests/test_attention_spec_predicate_parity.py tests/dispatch/attention` - 623 passed - Status: Passed - [x] Provider unit tests - `hip_kernel_provider_tests` - 1,087 passed; `*Gfx950Attention*` 167 of 167 - Status: Passed - [x] CTest - `ctest -R 'hkp-pack|hkp-probe-|hkp-census-|_test_name_validation'` - 29 of 29 - Status: Passed - [x] Dense catalog regeneration - `catalog_diff --strict-spec` (selection) and `--subset` (manifest) - no differences - Status: Passed - [x] Installed tree - `rocke_library_attention_cpu_pytest`, `rocke_library_attention_pytest` - 35 passed; 600 passed - Status: Passed - [x] gfx950 hardware - dense `SdpaFwd` `quick` 8, `standard` 45, GPU reference 45 (0 broken, 0 unclaimed), `*Gfx950Attention*` 167, `hip_kernel_provider_tests` 1,148, census 10 of 10 - Devices: MI355X - Status: Passed (at `75c5a107cb`; later commits change tests only) - [ ] PR CI - GitHub PR checks - Status: Pending Not covered: the rocKE gfx950 numeric tests (`rocke_library_attention_gpu_pytest`) need a ROCm torch that no CI image has; the IngestorGenerator suite runs in no CI job yet. ## Technical Changes - **Registry as source of truth.** `configs/gfx950_attention_dense.engine.yaml` (manifest) and `configs/gfx950_attention_dense.yaml` (selection) replace the hand-authored profile. `registry_enumerator.py` expands the manifest through the dispatcher; `test_registry_regeneration.py` fails on drift. - **Gates refuse what they cannot verify.** `catalog_diff`, `variant_reduce`, `ground_truth`, `coverage_gate`, `registry_enumerator` and `generate.py` use exit 0 for pass, 1 for a refusal and 2 for unusable input (`variant_reduce` adds 3 for an incomplete sweep). Malformed input, duplicate keys and unmapped spellings exit 2 instead of passing or tracebacking. - **gfx950 LDS gate in Python.** `supports_attention_dense` rejects tiles over the 160 KiB LDS budget, ported from `Gfx950AttentionDenseGeometry.hpp`; a census test checks that every shipped kernel's tile passes the engine's C++ check. - **`hkp_pack` pre-flight.** Every packed rocKE spec is checked by its builder's support predicate. A builder with no predicate, or a predicate that raises, fails the pack. - **Tests guard contracts, not rocKE internals.** Tests outside rocKE read the shipped catalogs or a synthetic registry instead of rocKE private names, C++ source text, or rocKE kernel counts and messages, so rocKE refactors do not break them. - **Spec-first tiled predicates** (gfx942, gfx950) so `hkp_pack` can validate tiled specs. ### Breaking changes (IngestorGenerator tooling; see its README) - `verify_variant_sets.py` and `coverage_gate.py` take `--manifest` instead of `--profile`; `HIPDNN_INGESTOR_PROFILE*` is renamed `HIPDNN_INGESTOR_MANIFEST*`. - `dispatch_parity.py`, `knob_sweep.py`, `reconcile_applicability.py` and `variant_reachability.py` are removed; the README's *Retired tools* table lists their replacements. - The engine manifest uses a new `enumerate:` schema, with `reduction:` and optional `ground_truth:` sections. Enum values in `enumerate.cells` must be member names. - Gates that passed silently now refuse: see the exit codes above. Ground-truth graph maps written before this change must be regenerated with `ground_truth.py graphs`. - `hkp_pack` fails the pack for a rocKE builder with no resolvable support predicate. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Yaswanth Raparti <yraparti@amd.com> Co-authored-by: BrianHarrisonAMD <169072757+BrianHarrisonAMD@users.noreply.github.com> Co-authored-by: Garby Gilbert <ggilbert@amd.com> Co-authored-by: Brian Harrison <brian.harrison@amd.com>

BrianHarrisonAMD added 5 commits October 13, 2026 11:41

test(rocke): guard that dispatch.attention resolves to the library pa…

d1133a4

…ckage

test(ingestor): drop the registry-only spec field snapshot

ac4d2be

Registry-only spec fields are checked against the spec dataclass defaults again. A new field or a default change needs no edit here; the shipped KDP is refreshed deliberately with the rest of the integration.

github-actions Bot added documentation project: hipdnn project: hip-kernel-provider rocke-lib: attention Touches rocKE library attention/FMHA kernels labels Oct 14, 2026

assistant-librarian Bot added the organization: ROCm label Oct 14, 2026

BrianHarrisonAMD force-pushed the users/bharriso/attention-ingestor-gate-hardening branch 2 times, most recently from 28b8c2d to 3267dba Compare October 14, 2026 01:25

BrianHarrisonAMD added 4 commits October 13, 2026 22:10

BrianHarrisonAMD force-pushed the users/bharriso/attention-ingestor-gate-hardening branch from 3267dba to 12de937 Compare October 14, 2026 02:18

Merge remote-tracking branch 'origin/users/tlakshma/rocke/attention-i…

923e3fd

…ngestor-bridge' into users/bharriso/attention-ingestor-gate-hardening # Conflicts: # projects/hipdnn/tools/ai/skills/hipdnn-ingestor-engine/SKILL.md

yraparti reviewed Oct 14, 2026

View reviewed changes

Comment thread projects/hipdnn/tools/IngestorGenerator/tools/variant_reduce.py Resolved

yraparti reviewed Oct 14, 2026

View reviewed changes

dpeabody-amd mentioned this pull request Oct 14, 2026

feat(rocke): RDNA3/RDNA4 generic SDPA kernels (gfx11-generic, gfx12-generic) #13150

Merged

BrianHarrisonAMD marked this pull request as ready for review October 14, 2026 22:40

BrianHarrisonAMD requested review from a team as code owners October 14, 2026 22:40

BrianHarrisonAMD merged commit 8cba93c into users/tlakshma/rocke/attention-ingestor-bridge Oct 14, 2026
179 of 181 checks passed

BrianHarrisonAMD deleted the users/bharriso/attention-ingestor-gate-hardening branch October 14, 2026 22:40

BrianHarrisonAMD mentioned this pull request Oct 14, 2026

fix(rocke/dispatch): make the dispatcher registry the source of truth for the gfx950 dense-attention ingestor catalog #13024

Merged

9 of 10 tasks

Navigation Menu

Repository navigation

ingestor: registry-driven selection, generic enumeration and variant reduction for rocKE engines - #13593

ingestor: registry-driven selection, generic enumeration and variant reduction for rocKE engines#13593
BrianHarrisonAMD merged 10 commits into
users/tlakshma/rocke/attention-ingestor-bridgefrom
users/bharriso/attention-ingestor-gate-hardening

BrianHarrisonAMD commented Oct 14, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

yraparti commented Oct 14, 2026

Uh oh!

yraparti left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

BrianHarrisonAMD commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Breaking changes

Test Plan

Test Result

Submission Checklist

Uh oh!

codecov Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

yraparti commented Oct 14, 2026

Uh oh!

yraparti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BrianHarrisonAMD commented Oct 14, 2026 •

edited

Loading

codecov Bot commented Oct 14, 2026 •

edited

Loading