[yhl-amd]

Motivation

On MI455X (gfx1250), DeepSeek-V4.1-Flash served by ATOM with DEP4 plus the MTP draft model (MLPerf Endpoints agentic, c512) runs most of its BF16 GEMMs and part of its 32x32 MXFP8 GEMMs with no gfx1250 tuned row:

• BF16: every BF16 GEMM of the target and draft model misses the table and runs the default Triton path. This covers both lm_heads (N=129280 and N=32320, K=5120; the N=129280 draft lm_head is handled by [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246), the small-N projections (N=1024 with fp32 out, N=512/384/128/32 at K=5120, N=128 at K=512) and the draft N=1 K=5376 (fp32 out) GEMM.
• MX32 (w_scale_block 32x32): the dense (kernelId bmm) and batched GEMMs at the M values DEP4 decode/prefill produce fall back to the FlyDSL heuristic. The existing gfx1250 rows cover other M values.

This PR adds tuned gfx1250 rows for exactly those shapes.

Relation to #7246. This PR complements #7246 (DSv4.1 BF16 OPUS rows by @amd-ruitang3). The draft lm_head (N=129280, K=5120) is intentionally excluded and left to #7246 entirely. Every other key that #7246 adds (N=384 M=4096; N=32320 M=576/768) is also dropped here, so #7246's row is the one used. The two PRs merge cleanly in either order (checked below).

Technical Details

Rows only. No code changes apart from a one-line test expectation (see below). All rows are gfx1250, cu_num 256. No existing row is modified, and no other arch is touched. In dsv41_bf16_tuned_gemm.csv the new rows sit right after the header (not at the end of the file), so they do not textually conflict with #7246, which appends at the end.

File	Change	Rows
aiter/configs/model_configs/dsv41_bf16_tuned_gemm.csv	insert after header	+181 (144 FlyDSL hgemm, 37 OPUS)
aiter/configs/model_configs/dsv41_a8w8_blockscale_mxscale_bpreshuffle_tuned_gemm.csv	append	+181 (32x32, kernelId bmm, FlyDSL flydsl_bmm_mxfp8_32_wmma_*)
aiter/configs/model_configs/dsv41_batched_gemm_a8w8_blockscale_mxscale_bpreshuffle_tuned.csv	new	30 (b=8, n=1024, k=4096, 32x32)
op_tests/test_opus_a16w16_policy_parity.py	+1 line	_EXPECTED_GFX_KEY_FIXES

BF16 shapes covered (N, K, out dtype : number of M values):

• 32320, 5120, bf16 : 18 (target lm_head; M=24..3072, without [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246's M=576/768)
• 1024, 5120, fp32 : 23
• 512, 5120, bf16 : 30
• 384, 5120, bf16 : 31 (router; 27 FlyDSL for M ≤ 5120, OPUS only at M=384/2048/6144/16384; without [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246's M=4096)
• 128, 5120, bf16 : 16
• 32, 5120, bf16 : 30
• 128, 512, bf16 : 13
• 1, 5376, fp32 : 20 (draft)
• not included: 129280, 5120 (draft lm_head) → [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246

MX32 dense shapes (N, K): (1792, 5120), (4608, 5120), (5120, 2304), (5120, 8192), (32768, 1280). Batched: b=8, (1024, 4096).

How the rows were produced:

• The rows come from the in-tree tuners on MI455X (gfx1250, 256 CUs), image rocm/fw-bringup-mlperf:gfx1250-atom-20261030-dsv41. That image's aiter is based on main at 2db75fc. Since then, main changes none of the FlyDSL hgemm, OPUS a16w16, FlyDSL mxfp8_32 BMM kernels or the lookup code these rows use. FlyDSL in the image is 0.4.1.
• BF16 rows keep only FlyDSL/OPUS winners: the tuner benchmarks the Triton candidates as well, and shapes where Triton won (48 of 232) get no row and keep the default path. The tuner log keeps only the winner, so the kernel-level margin over Triton is not recorded per row. Every BF16 row we re-measured against the default path (table below) is faster by far more than 3%, and no measured row was slower, so no row was dropped on performance grounds.
• An MX32 row is kept only if it beat the current production choice (heuristic or existing tuned row) by more than 2% in two independent re-measurements. 211 of 219 candidates passed.
• The row schemas are identical to main's headers.

Merge and duplicate rules:

• None of the new keys (gfx, cu_num + the untuned key columns) exist in configs/ or any model_configs/* sibling of the same op, or in [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246. The runtime merge therefore neither raises nor rewrites files, before or after [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246 lands.
• The gfx1250 OPUS row for (M=16384, N=512, K=5120) has the same M/N/K as an existing gfx950 OPUS row (kid 93). As with the four existing kid-596 entries, this shape is added to _EXPECTED_GFX_KEY_FIXES so test_shipped_tuned_selection_diff_is_exhaustive stays exhaustive.

OPUS kids: the new rows reference gfx1250 a16w16 kids 178, 525–716. The opus module's blob_gen_cmd already globs model_configs/*_bf16_tuned_gemm.csv, so these kids are built with the module. When the CSV was instead passed through AITER_CONFIG_GEMM_BF16 from outside the tree, kid 178 had to be compiled by hand.

Test Plan

No-GPU checks on this branch:

1. Run AITER_CONFIGS.get_config_file for bf16_tuned_gemm, a8w8_blockscale_mxscale_bpreshuffle_tuned_gemm and batched_gemm_a8w8_blockscale_mxscale_bpreshuffle_tuned with no env override. This is the real configs/ + model_configs/ merge with its duplicate check. Then check that every new row appears exactly once in the merged table with its kernelName.
2. FlyDSL hgemm names must match _HGEMM_KERNEL_RE (ktail0, gfx1250, matching out dtype). MX32 names must match _BMM_MX32_KERNEL_NAME_RE.
3. For OPUS rows, get_kernel_instance('gfx1250', 'a16w16', kid, outdtype, cu_num=256).name == kernelName, and resolve_a16w16_tuned_candidate(...) must resolve to the same kid with the stored splitK.
4. pytest op_tests/tuning_tests/test_csv_validation.py op_tests/tuning_tests/test_config_shape_collision.py op_tests/test_opus_a16w16_policy_parity.py.
5. Merge simulation with [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246: git merge of main + [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246 head (d6b8b2b) + this branch, in both orders, then steps 1 and 4 again on the result.

On MI455X:
6. Kernel microbenchmarks against the production default.
7. E2E ATOM A/B on DeepSeek-V4.1-Flash: DEP4 + MTP, MLPerf Endpoints agentic, c512, 10-minute FAST runs, same machine, runs interleaved.
8. A full run with inline accuracy.

Test Result

No-GPU checks:

• Merge (1): all three merges pass. Merged gfx1250 rows: bf16 1203, mxscale dense 807, mxscale batched 50. 181/181, 181/181 and 30/30 new rows are found once each with the same kernelName.
• Kernel names (2, 3): 181/181 BF16 rows are valid (144 FlyDSL names parse; 37 OPUS kid→name match, and the policy resolves the same kid). 211/211 MX32 names parse.
• pytest (4): 68 passed, 50 subtests passed.
• With [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246 (5): both merge orders are conflict-free. dsv41_bf16_tuned_gemm.csv then has 213 gfx1250 rows (181 + 32) and 0 duplicate keys; the bf16 runtime merge passes with 1235 gfx1250 rows; the three pytest files pass (68 passed, 50 subtests passed).

Microbenchmarks (us, default → tuned). Error ratio (fraction of outputs outside tolerance against a reference GEMM) is 0 for every measured row, default and tuned.

GEMM	M	default	tuned
target lm_head N=32320 K=5120	24	74.7 (Triton)	24.5 (OPUS)
target lm_head N=32320 K=5120	384	134.1	56.4 (OPUS)
target lm_head N=32320 K=5120	2304	530.7	339.6 (OPUS)
target lm_head N=32320 K=5120	1536	338.7	217.2 (OPUS)
target lm_head N=32320 K=5120	3072	702.2	413.1 (OPUS)
MX32 dense N=1792 K=5120	48	9.53	5.15
MX32 dense N=5120 K=8192	768	22.40	20.15
MX32 dense N=32768 K=1280	768	21.23	19.55
MX32 batched b=8 N=1024 K=4096	16	5.59	4.89

• Small-N BF16 GEMMs: for N ≤ 1024 and the N=1 draft GEMM (router N=384 at M=6/96/384/768 included), eager timing shows about 70 → 24 us at every measured M. Most of that is Python/launch overhead, and the tuned kernels themselves run in a few microseconds, so this shows the rows are not slower than the default but does not give a kernel-only margin.
• MX32 geomean speedup per (b, N, K) over the kept M values: (1792, 5120) 1.54x; (4608, 5120) 1.16x; (5120, 2304) 1.19x; (5120, 8192) 1.18x; (32768, 1280) 1.09x; batched (8, 1024, 4096) 1.11x.

E2E A/B, same machine, interleaved (DeepSeek-V4.1-Flash, DEP4 + MTP, c512, generated tok/s/GPU over the 600 s issue window):

Config	Runs	tok/s/GPU (mean)	Δ vs base	decode ms/step	prefill ms/step	TTFT mean (ms)	MTP accept %
base (main tables)	10217 / 10412 / 10475	10368	–	24.75	99.4	1566	50.8
+ draft BF16 rows only (incl. draft lm_head)	10549 / 10673	10611	+2.3%
+ BF16 rows	10666 / 10696 / 10741	10701	+3.2%	23.97	93.9	1523	50.9
+ BF16 + MX32 rows	10771 / 10795	10783	+4.0%	23.73	94.0	1509	51.0

• Machine: one MI455X node (e13-03), image rocm/fw-bringup-mlperf:gfx1250-atom-20261030-dsv41.
• Important: all BF16 E2E numbers above were measured with our earlier, full row set, which included the draft lm_head (N=129280) rows now left to [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246. The E2E gain of the remaining rows alone (this PR without the draft lm_head) was not measured separately, so the +3.2% / +4.0% above cannot be attributed to this PR on its own; part of it comes from the draft lm_head rows (now in [Config] [Perf] Tune gfx1250 DSv4.1 BF16 LM-head and router GEMMs #7246; the draft-only row set incl. the draft lm_head gave +2.3%).
• The slowest BF16 run is faster than the fastest base run.
• No preemptions in any run, and the per-run math and tool-call smoke checks all passed.

Accuracy: a full MLPerf Endpoints run with inline accuracy scored 0.5544 against a gate of 0.5236 (PASS), with 27581/27581 requests completed. A reference run without these rows scored 0.5561. That run used a different MI455X node (e15-06) with our earlier full BF16 row set (including the draft lm_head rows) together with an unrelated ATOM scheduling setting (9850 → 10255 tok/s/GPU, +4.1%, on 10-minute runs; 10263 tok/s/GPU on the full run). It did not include the MX32 rows, so MX32 inline accuracy is still pending. The MX32 rows have error ratio 0 in the microbenchmarks and passed the E2E smoke checks above.

🤖 Generated with Claude Code

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355	Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows	Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7247 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

[aiter-bot]

NOT READY FOR HUMAN REVIEW

PR Self-Review

Revision 3d3a4583c968 · mode shadow

Waiting for the PR author. The readiness report will appear after the self-review answers are complete for this revision.

Questions to answer in the PR description

1. What behavior does this PR change, and why is the existing behavior insufficient?
2. What did you validate, and what remains unverified? Include commands, hardware/configuration and results.
3. Which exact workload keys consume these configuration changes, and what evidence shows that unrelated shapes keep their previous selection?
4. For every changed op_tests/ file, what behavior does it cover and why is it not redundant? Write None if no op test changed.

Edit the PR Self-Review section in the PR description. Include the current revision SHA; this comment refreshes automatically.

Missing or invalid answers

• FAIL — missing ### Reviewed revision in the PR description
• FAIL — missing ### Behavior and intent in the PR description
• FAIL — missing ### Validation and unverified scope in the PR description
• FAIL — missing ### PR-specific question in the PR description
• FAIL — missing ### Changed op tests in the PR description
• FAIL — Reviewed revision must contain at least the first 7 characters of the current head SHA
• FAIL — Behavior and intent has no substantive answer
• FAIL — Validation and unverified scope has no substantive answer
• FAIL — PR-specific question has no substantive answer
• FAIL — changed op test not explained in Changed op tests: op_tests/test_opus_a16w16_policy_parity.py
• FAIL — Changed op tests must explain why each changed op test is necessary and non-redundant

_{Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.}

[demonsan]

LGTM