[wsantos]

Qualifies the rocm10-rocblas5.6 profile against the build this box actually loads, and then fills the two shape families it was still missing: the decode/MTP grid and the 256/512-token chunk family the live dynamic-prefill controller can put on the wire. 44 → 139 rows/rank.

Why this is not duplicating an existing PR

gh pr list --repo opengfx1030/vllm-rdna --state open --search "tunableop" returns #92 (this), #35, #74, #87 and #99.

• [backlog] perf(rdna): retune the 6 TunableOp rows slower than the heuristic #74 ("retune the 6 TunableOp rows slower than the heuristic") and followup: TunableOp rows for MTP=2 verify-batch GEMM keys (M∈{3,6,12,24}) #35 ("TunableOp rows for MTP=2 verify-batch GEMM keys, M∈{3,6,12,24}") are draft backlog containers with changedFiles: 0 — empty placeholders describing this work, not implementations. They were checked with gh issue view 74 --comments (0 comments) and gh pr list --search. followup: TunableOp rows for MTP=2 verify-batch GEMM keys (M∈{3,6,12,24}) #35's keys and more are covered below; [backlog] perf(rdna): retune the 6 TunableOp rows slower than the heuristic #74's premise is measured below.
• cleanup(tools): delete finished tools/rdna2_028 campaign drivers #87 deletes finished tools/rdna2_028 campaign drivers. This PR uses those drivers; the two do not overlap.
• perf(rdna2): QSA/GDN HIP prefill paths vs Triton/FLA + TunableOp decode/MTP grid (implements #64) #99 bundles the same tunableop/ commits alongside unrelated QSA/GDN HIP prefill work. The tunableop/ trees are now byte-identical (git diff --stat tunableop/ is empty), so merging this PR first makes perf(rdna2): QSA/GDN HIP prefill paths vs Triton/FLA + TunableOp decode/MTP grid (implements #64) #99's tunableop portion collapse out of its diff. This PR is the one to review for the profile.

1. The bug: the profile could not be auto-selected

The profile recorded lib_sha256 = c27e2252cc7a, a retired build. The loaded library is d05939967b5f — and the system /opt/rocm/core-10.0 copy and the venv _rocm_sdk_libraries copy are byte-identical on this box, so both resolve to the same build. With no matching profile, configure_tunableop warned and fell back to the per-user path; the rows were reachable only through the legacy rocblas-<hash>/ directory. Per the storage policy (one named profile per rocBLAS build, nothing archived) that fallback directory is dropped — its contents are not shipped, and it is not committed.

After the fix:

TunableOp profile rocm10-rocblas5.6 auto-selected for rocBLAS build d05939967b5f.
TunableOp lookup enabled for profile rocm10-rocblas5.6 (rocBLAS d05939967b5f; ...); tuning off

The imported rows are deliberately not carried over: their solution IDs came from a different build, and IDs must never be reused across builds even when the version string matches. The import is preserved under imported in provenance.json.

2. Decode / MTP grid (44 → 128 rows)

The 2026-10-12 campaign tuned only the prefill grid (M = 800…8192), so every M ≤ 32 shape fell back to rocBLAS heuristics on this build — while the canonical rocm7.14-rocblas5.5 profile carries the decode (M ≤ 8) and MTP-verify (M = 16/24/32) shapes.

• 84/84 keys produced, 84/84 passed on devices 0-3 against FP32 references.
• Per-shape speedup over the heuristic 1.4x to 3.6x, largest on the K=320 and K=10240 shapes.
• These were raw tuned, not curated — the prefill margins need a same-session comparison, which §3 then ran.

3. The whole grid re-measured, and the 256/512-token chunk family (128 → 139)

tools/rdna2/check_v620_tuning.py reported 7 of the 14 shapes the live cache/batch config needs as missing. The live launcher resolves to --block-size 1024 --max-num-batched-tokens 2048, so the chunks that actually run are M = 1024 and 2,048. Two deliberate mechanisms had kept them missing: tools/rdna2_028/tunableop_rows_pipeline.sh tunes only the shapes present in --shape-source (so the 26 prefill shapes the curation dropped were never re-tested), and curate_tunableop_rows.py folds Default rows into dropped by design.

All 168 shapes (7 dense (N,K) × 12 prefill M + the 84 decode/MTP rows) were re-tuned and then measured heuristic / current / new in one session.

The 7 live-grid holes stay holes. A fresh tune does not beat the heuristic by 3 % on any of them (10240x320, 4096x2560, 2560x1536, 3584x2560 at M=1024; 10240x320, 512x2560, 10240x2560 at M=2048), so a row there buys nothing. An explicit Default row would only silence the coverage warning while pinning the shape against online tuning, so the warning is left as-is — it is telling the truth.

The real gap was one the checker cannot express. The launcher sets VLLM_RDNA_DYNAMIC_PREFILL=1, whose chunk-cap ladder is RUNGS = (256, 512, 1024) with LPT_SHARED = 512; scheduler.py sets num_new_tokens = long_prefill_token_threshold whenever more than one request is eligible, so at ≥4 eligible requests a 512-token chunk reaches the GEMMs (256 at 8). There was no row for M = 256/512 at all, and check_v620_tuning.py takes only --block-size/--max-num-batched-tokens, so it cannot see that family.

shape	M	heuristic	tuned row	delta
10240x2560	512	4.33 ms	1.22 ms	−72 %
336x10240	512	1.05 ms	0.24 ms	−77 %
10240x2560	256	2.32 ms	0.59 ms	−75 %
512x2560	512	0.229 ms	0.102 ms	−55 %
2560x1536	512	0.243 ms	0.207 ms	−15 %
3584x2560	800	0.851 ms	0.728 ms	−15 %

The tuned solver reaches ~22 TFLOPS on the large-K shapes where the heuristic manages ~6. 11 rows added; 2 shipped prefill rows replaced by a different faster solver (10240x2560 M=1024, 2560x1536 M=3200).

Two traps worth recording. (a) Two further "adopt" decisions carried the same solver id as the shipped row — same-session noise, not a better kernel — and were rejected: only adopt when the solver actually changes. (b) 18 decode/MTP rows (M=2..8) measured 10–77 % slower than the heuristic, which looks damning, but that region is launch-bound: identical shape families flip sign between M=1 (row 22–58 % faster) and M=2..8, on ~14 MFLOP GEMMs timed at 80–240 µs. The decode grid is left byte-identical; deleting those rows needs a small-M measurement (no per-call sync, larger reps) first. This is the answer to #74's premise for the decode side: the rows it suspects are real, but the evidence used here cannot separate them from timer noise.

Result and verification

139 rows/rank = 55 prefill + 84 decode/MTP, one validated set on all four ranks.

check	command	result
FP32 numerics, 4 devices	python tools/rdna2/qualify_v620_fp16.py <dir>	139/139 pass on devices 0–3 (relative L2 < 0.002, assert_close atol/rtol 0.01)
lookup hits	tools/rdna2_028/verify_lookup_hit.sh (DEVICE=3)	139/139 hit, 0 miss
coverage tooling	python -m pytest tests/tools/test_v620_tuning_coverage.py -q	4 passed, 2 subtests
auto-selection on a live start	production launcher	auto-selected for rocBLAS build d05939967b5f, lookup enabled, tuning off

Note test_v620_tuning_coverage.py asserts the warning path, so a green test is not a completeness proof — the coverage warning for the 7 live-grid shapes is deliberate and documented.

Files

file	change
tunableop/profiles.json	lib_sha256 → d05939967b5f, rows_per_rank → 139, status, covers
tunableop/README.md	profiles table row; the live grid and the dynamic-prefill chunk ladder documented
tunableop/rocm10-rocblas5.6/README.md	identity, both campaigns, the full-grid re-measure, caveats, regenerate recipe
tunableop/rocm10-rocblas5.6/provenance.json	build sha, decode/MTP campaign, full-grid campaign, import preserved
tunableop/rocm10-rocblas5.6/tunableop_results{0..3}.csv	70 → 44 → 128 → 139 rows/rank

Caveats

• No model-quality (PPL/accuracy) run. The change affects only which GEMM solver executes, and every row is qualified against FP32 on all four cards, but a different solver is not bit-identical to the heuristic — it changes the floating-point reduction order. tunableop/README.md says so explicitly.
• The end-to-end size of the 256/512 win is not measured. It only materialises when the controller is in the shared cell, so a single-request prefill benchmark will not show it; that needs a mixed-load run with ≥4 eligible requests plus a [dyn-prefill] ADJUST line confirming a 512 cap.
• Curation is only valid within one session: the same row set drifts 5–10 % across sessions, which is larger than the 3 % adopt/drop margins.

AI assistance was used to prepare this branch and description. Submitted by a human who has reviewed the change and is accountable for it.