[Perf] Add Kimi-K3 gfx950 A8W8 bpreshuffle tuned configs by yuychang · Pull Request #6410 · ROCm/aiter

yuychang · 2026-10-12T03:38:57Z

Motivation

Kimi-K3 runs FP8 gemm_a8w8_bpreshuffle shapes on gfx950 that have no tuned row, so they fall back to the default kernel. This PR adds tuned rows for them. It also makes gemm_a8w8_bpreshuffle_flydsl fall back to CK for M above the FlyDSL preshuffle limit (65536), which the padded-M lookup would otherwise send to a FlyDSL row.

Technical Details

model_configs/a8w8_bpreshuffle_tuned_gemm_kimik3.csv: 149 new rows (gfx950, 256 CU).
- N=7168 K=3584: 55 rows, covering M=1..256 and M=288..1024 in steps of 32, except M=832, 928 and 992 (see the 8-wave bullet).
- N=6336 K=7168: 34 rows, M=1..256.
- N=2112 K=7168: 16 rows, M=1..128.
- N=6144 K=7168 (24 rows) and N=128 K=7168 (20 rows), at the decode M values of the InferenceX MI355X agentic recipes (1..96) and powers of two up to 8192. N=6144 also has rows at M=544/576/608/640 so padded M 513–640 does not fall back to the slower M=1024 row.
  - With ATOM's online PTPC FP8, these are the KDA in_proj (q|k|v|g) and f_a_proj GEMMs at TP8. in_proj runs in 69 of the 93 layers.
  - Neither shape had a tuned row, so every M used the default CK kernel.
model_configs/a8w8_bpreshuffle_untuned_gemm_kimik3.csv: an untuned row for every new tuned key, so re-tuning keeps them.
model_configs/dsv3_a8w8_bpreshuffle_tuned_gemm.csv: removes 9 rows for N=2112 K=7168 (M=1..128).
- The new rows have the same keys. For M=8/16/32/64/128 the CSV us is higher than the deleted row's, but the two files come from separate tuning runs: M=16 is the same kernel in both, listed at 7.48 vs 6.83 us. Timed back to back with the weight read cold, as in a decode step, the new rows are at parity or faster at every M they serve (table below). With the weight hot in cache, M 65–80 and 113–128 are 0.94–0.98x.
- Keeping both would make the config merge fail with a duplicate-shape error.
No rows use the FlyDSL 8-wave preshuffle kernel. For 6144x7168 at M=2048 and 8192, the tuner's 8-wave picks are replaced by the best non-8-wave candidate, which is about 3% and 10% slower than the 8-wave kernel at those M values.
- The 8-wave kernel (flydsl_bpreshuffle_8w_*) sometimes returns corrupted rows.
- The tuner picked it for N=7168 K=3584 at M=832, 928 and 992. In a stress test, M=832 failed 1 of 100 calls and M=928 failed 4 of 100.
- Those three M values use the padded-M row (M=1024) instead.
- [FlyDSL] [gfx950] Fix 8-wave a8w8 GEMM reading LDS before the last tile lands #6368 has since fixed the 8-wave race on main; this PR still keeps the non-8-wave picks.
FlyDSL 0.4.0 (the version main now pins): every FlyDSL row was re-validated, and 8 N=7168 K=3584 rows (M=1/4/8/12/88/96/320/1024) were re-tuned to faster kernels, 1.04-1.12x over the previous picks.
14 more rows were replaced after review: 13 N=7168 K=3584 CK/cktile rows (M=72, 80, 104, 112, 168, 176, 200, 208, 352, 416, 544, 608, 736) and the 6336x7168 M=1 row now use FlyDSL kernels that are 1.08-1.38x faster across every M each row serves, with identical error.
aiter/ops/gemm_op_a8w8.py: getPaddedM(gl=1) serves every M > 8192 with the M=8192 row when N > 4096, and FlyDSL's preshuffle launcher raises for M > PRESHUFFLE_M_MAX (65536). Such calls now fall back to CK only while CK's output still fits: CK bpreshuffle writes wrong rows, with no error, once its bf16 output (M rounded up to the kernel's M tile) reaches 2 GiB (measured on gfx950: N=6144 is correct at M=174700 and wrong at M=174740, which matches a 128-row tile). The guard rounds M up to 256 rows, the largest CK tile, so it is conservative: it stops falling back after M=174592, and M=174593 to 174720 raise the FlyDSL error although CK would still be correct. Past the guard the FlyDSL error is raised, as on main. FlyDSL rows whose split-K does not divide the runtime K (for example _ks7 on a weight padded to K=8192) also fall back to CK instead of raising, under the same 2 GiB output limit. Checked on gfx950 with real kernels against an fp32 dequant reference: M=65537 at N=6144 K=7168, and the _ks7 rows at N=2112 with the weight padded to K=8192 (M=1/8/16/48), all run on CK with max abs error <= 1.0 (bf16 rounding) against a reference max of up to 520; test_gemm_a8w8_bpreshuffle_flydsl_ck_fallback_matches_reference covers both fallbacks numerically and fails if they are removed. A padded K that is not a multiple of 128 (e.g. 7232) is still rejected by CK's heuristic kernels, so such a call raises (inside CK instead of FlyDSL) rather than computing. op_tests/test_gemm_a8w8_bpreshuffle_pad_k.py covers M=65536 (FlyDSL), M=65537 and M=174592 (CK), and M=174593 plus the 57344x8192 row at M=65537 (FlyDSL error).

Overlap with #6959. #6959 dropped its copy of the 20 N=6144 K=7168 rows (6bd43d0), so this PR owns that shape and the two PRs no longer collide.

Test Plan

python3 -m unittest op_tests.tuning_tests.test_config_shape_collision
pytest op_tests/test_gemm_a8w8_bpreshuffle_pad_k.py -k falls_back_to_ck_above_m_max
A/B against main on one MI355X: the same script on both trees, run back to back, alternating, for 3 rounds.
- Timing uses run_perftest.
- Each output is checked against an fp32 reference built from the same quantized inputs.
- For N=7168 K=3584 M=264..1024, the sweep includes M values between tuned rows, which exercises the padded-M lookup.
The 8-wave picks were stress-tested for correctness: 20 seeds x 5 calls each.
N=2112 K=7168 vs the deleted DSv3 rows: each M is resolved through the padded-M lookup on both sides and the chosen FlyDSL kernels are called directly, 3 alternating rounds. First every M from 1 to 128 except 81–96 (where main uses a CK row) with the weight hot in cache, then 17 M values covering every row with the weight rotated through run_perftest, so each call reads it cold, as in a decode step.
The 6144x7168 and 128x7168 rows: production gemm_a8w8_bpreshuffle A/B, 3 alternating rounds, at every tuned M plus M values between rows (59 points). Each new row was then called 30 times against an fp32 reference (1200 calls).

Test Result

Setup: MI355X (gfx950), ROCm 7.2, torch 2.11.0+rocm7.2, FlyDSL 0.4.0, compared against merge-base a6627a454 (the padded-M check below uses merge-base e0ba6757d).

Duplicate-shape check: OK on the branch (26 tests) and merged with main (30 tests).
GPU kernel time, this PR vs main: 290 points (every new row plus a midpoint in each gap between rows, and M=12288 for N=6144 and N=128), production gemm_a8w8_bpreshuffle through run_perftest, 3 alternating rounds, best of each:

shapes	points	geomean	range
N=7168 K=3584, M 1-256	66	1.54x	1.00x-2.11x
N=7168 K=3584, M 257-1024	42	1.35x	1.07x-1.81x
N=6336 K=7168	66	1.30x	0.99x-1.70x
N=2112 K=7168 (replaced DSv3 rows)	30	1.14x	1.00x-1.73x
N=6144 K=7168 (KDA in_proj)	47	1.39x	0.97x-1.75x
N=128 K=7168 (KDA f_a_proj)	39	2.15x	1.00x-4.85x

The few points below 1.0x (0.97x-1.00x, lowest N=6144 M=832) are at parity within run-to-run noise. With the weight read cold, as in a decode step, N=2112 runs 1.14x geomean over 30 M values (1.00x-1.74x); M=8 and M=16 are at parity (M=16 runs the same kernel as main). The hot-cache direct-kernel sweep in the N=2112 section below is a separate measurement. N=2112 is lower than before because main now has FlyDSL rows for that shape (#6547).

On FlyDSL 0.4.0, all FlyDSL rows (135 after the review changes) match the fp32 reference with the same max abs error as main, and are bit-identical across 10 repeated calls. M=65536 runs FlyDSL correctly; M=65537 falls back to CK while CK's output fits under 2 GiB.
All outputs match the reference. In the 290-point A/B the max abs error equals main at every point except N=6336 M=184 (1.0005 vs 0.9995, one bf16 step). For the 6144x7168 and 128x7168 rows, the max abs error vs fp32 is identical to main at every point, and all 1200 repeated calls are correct and bit-identical. Where measured against fp32 (M=264..1024), the max abs error is the same as main.

N=7168 K=3584, M=264..1024 (original FlyDSL 0.3.4.1 run)

M	main (us)	this PR (us)	speedup
264	27.36	15.92	1.72x
300	27.49	16.30	1.69x
352	27.62	21.47	1.29x
416	27.88	24.29	1.15x
512	23.88	20.41	1.17x
544	31.11	26.55	1.17x
600	31.81	29.32	1.08x
640	28.48	24.27	1.17x
704	33.32	29.72	1.12x
736	34.01	31.43	1.08x
800	34.06	30.26	1.13x
832	34.30	31.93	1.07x
896	33.43	28.77	1.16x
928	35.45	32.28	1.10x
1000	35.56	32.71	1.09x
1024	34.50	32.98	1.05x

N=6144 K=7168 (KDA in_proj): 30 M values, geomean 1.30x (original FlyDSL 0.3.4.1 run)

M	before (us)	after (us)	speedup	kernel after
1	16.36	10.09	1.62x	`flydsl_bpreshuflle_16x32x1024_F8_F8_B16_0x2x0x2_`
2	16.34	10.05	1.63x	`flydsl_bpreshuflle_16x32x1024_F8_F8_B16_0x2x0x1_`
4	16.05	9.76	1.65x	`flydsl_bpreshuflle_16x32x1024_F8_F8_B16_0x1x0x1_`
8	16.18	10.08	1.61x	`flydsl_bpreshuflle_16x32x1024_F8_F8_B16_0x2x0x1_`
12	16.45	10.50	1.57x	`flydsl_bpreshuflle_16x32x1024_F8_F8_B16_0x0x0x1_`
16	15.02	10.55	1.42x	`flydsl_bpreshuflle_16x32x1024_F8_F8_B16_0x0x0x1_`
24	15.50	11.92	1.30x	`flydsl_bpreshuflle_32x32x1024_F8_F8_B16_0x1x0x2_`
32	14.99	12.49	1.20x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_1x2x0x2_d`
40	18.44	14.00	1.32x	`flydsl_bpreshuflle_48x128x256_F8_F8_B16_1x3x4x2_`
48	19.18	14.62	1.31x	`flydsl_bpreshuflle_48x128x256_F8_F8_B16_1x3x4x2_`
56	19.62	15.81	1.24x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_1x4x4x2_d`
64	20.08	15.95	1.26x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_1x0x4x2_d`
72	23.03	16.71	1.38x	`flydsl_bpreshuflle_80x64x256_F8_F8_B16_1x2x0x2_d`
80	23.28	16.91	1.38x	`flydsl_bpreshuflle_80x64x256_F8_F8_B16_1x3x0x2_d`
96	24.31	16.65	1.46x	`flydsl_bpreshuflle_96x64x256_F8_F8_B16_1x1x0x2_d`
100	25.84	19.35	1.34x	`flydsl_bpreshuflle_64x128x256_F8_F8_B16_1x0x4x2_`
128	26.62	20.86	1.28x	`flydsl_bpreshuflle_64x128x256_F8_F8_B16_1x0x4x2_`
200	47.56	24.06	1.98x	`flydsl_bpreshuflle_64x192x256_F8_F8_B16_0x0x4x2_`
256	39.09	25.28	1.55x	`flydsl_bpreshuflle_64x192x256_F8_F8_B16_0x0x4x2_`
384	39.40	32.73	1.20x	`flydsl_bpreshuflle_64x192x256_F8_F8_B16_1x2x4x2_`
512	40.01	33.70	1.19x	`flydsl_bpreshuflle_64x192x256_F8_F8_B16_1x2x4x2_`
768	49.98	49.55	1.01x	`a8w8_bpreshuffle_256x128x192x256_16x16_16x16_16x`
1024	61.69	61.31	1.01x	`a8w8_bpreshuffle_256x128x192x256_16x16_16x16_16x`
1536	86.23	76.37	1.13x	`flydsl_bpreshuflle_128x192x256_F8_F8_B16_1x1x4x1`
2048	105.95	86.32	1.23x	`flydsl_bpreshuflle_128x192x256_F8_F8_B16_1x1x4x1`
3072	146.36	134.61	1.09x	`flydsl_bpreshuflle_128x256x128_F8_F8_B16_1x2x4x2`
4096	191.19	155.97	1.23x	`flydsl_bpreshuflle_128x256x128_F8_F8_B16_1x2x4x2`
6144	261.03	258.87	1.01x	`a8w8_bpreshuffle_256x256x256x128_16x16_16x16_8x3`
8192	355.92	352.68	1.01x	`a8w8_bpreshuffle_256x256x256x128_16x16_16x16_8x3`
12288	511.25	513.23	1.00x	`a8w8_bpreshuffle_256x256x256x128_16x16_16x16_8x3`

N=128 K=7168 (KDA f_a_proj): 29 M values, geomean 2.22x (original FlyDSL 0.3.4.1 run)

M	before (us)	after (us)	speedup	kernel after
1	8.78	4.37	2.01x	`flydsl_bpreshuflle_16x32x512_F8_F8_B16_1x1x0x2_d`
2	8.96	4.39	2.04x	`flydsl_bpreshuflle_16x32x512_F8_F8_B16_0x3x0x2_d`
4	8.93	4.58	1.95x	`flydsl_bpreshuflle_16x32x512_F8_F8_B16_1x3x0x2_d`
8	9.05	4.67	1.94x	`flydsl_bpreshuflle_16x32x512_F8_F8_B16_1x1x0x2_d`
12	9.12	4.73	1.93x	`flydsl_bpreshuflle_16x32x512_F8_F8_B16_1x3x0x2_d`
16	6.60	4.81	1.37x	`flydsl_bpreshuflle_16x32x512_F8_F8_B16_1x3x0x2_d`
24	10.42	4.71	2.21x	`flydsl_bpreshuflle_16x32x512_F8_F8_B16_1x4x0x2_d`
32	7.73	4.75	1.62x	`flydsl_bpreshuflle_16x32x512_F8_F8_B16_1x1x0x2_d`
40	10.33	5.08	2.03x	`flydsl_bpreshuflle_16x64x512_F8_F8_B16_1x3x0x2_d`
48	7.76	5.11	1.52x	`flydsl_bpreshuflle_16x64x512_F8_F8_B16_1x3x0x2_d`
56	10.36	5.14	2.01x	`flydsl_bpreshuflle_16x64x512_F8_F8_B16_1x1x0x2_d`
64	7.75	5.33	1.45x	`flydsl_bpreshuflle_32x32x512_F8_F8_B16_0x4x0x2_d`
72	10.41	5.28	1.97x	`flydsl_bpreshuflle_16x64x512_F8_F8_B16_0x3x0x2_d`
80	7.74	5.21	1.49x	`flydsl_bpreshuflle_16x64x512_F8_F8_B16_1x1x0x2_d`
96	7.79	5.74	1.36x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_0x0x0x2_d`
100	10.42	5.76	1.81x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_1x0x0x2_d`
128	7.77	5.70	1.36x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_1x0x0x2_d`
200	11.33	6.20	1.83x	`flydsl_bpreshuflle_16x64x512_F8_F8_B16_1x2x0x2_d`
256	31.87	6.09	5.24x	`flydsl_bpreshuflle_16x64x512_F8_F8_B16_1x2x0x2_d`
384	32.99	6.98	4.72x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_0x2x0x2_d`
512	32.90	7.00	4.70x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_0x2x0x2_d`
768	34.12	8.31	4.11x	`flydsl_bpreshuflle_16x32x1024_F8_F8_B16_0x0x0x2_`
1024	34.54	8.60	4.01x	`flydsl_bpreshuflle_16x32x1024_F8_F8_B16_0x0x0x2_`
1536	34.09	9.76	3.49x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_1x3x0x2_d`
2048	33.80	10.05	3.36x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_1x3x0x2_d`
3072	34.17	13.39	2.55x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_1x1x0x2_d`
4096	34.52	13.71	2.52x	`flydsl_bpreshuflle_32x64x512_F8_F8_B16_1x1x0x2_d`
6144	35.13	18.97	1.85x	`flydsl_bpreshuflle_64x128x256_F8_F8_B16_1x0x0x2_`
8192	35.69	21.62	1.65x	`flydsl_bpreshuflle_64x128x256_F8_F8_B16_1x0x0x2_`

N=2112 K=7168 vs the deleted DSv3 rows (FlyDSL 0.4.0, cold weights)

Each M uses the row the padded-M lookup picks on that side, so M=65–80 and 97–112 move from main's M=128 row to this PR's new M=80 and M=112 rows.

M	main row	main (us)	this PR row	this PR (us)	speedup
8	8	6.64	8	6.61	1.004x
16	16	6.88	16	6.88	1.000x
24	32	7.64	24	7.31	1.045x
32	32	8.05	32	7.69	1.046x
40	64	9.59	40	8.94	1.072x
48	64	9.71	48	9.18	1.058x
56	64	9.87	56	9.22	1.071x
64	64	10.02	64	9.41	1.065x
65	128	12.17	80	10.74	1.134x
72	128	12.22	80	10.85	1.126x
80	128	12.27	80	11.03	1.112x
97	128	12.35	112	11.05	1.118x
104	128	12.37	112	11.30	1.094x
112	128	12.39	112	11.46	1.081x
113	128	12.35	128	11.80	1.047x
120	128	12.42	128	11.88	1.046x
128	128	12.49	128	11.99	1.042x

All outputs match the fp32 reference. In the hot-cache sweep, M=1–4 run 1.44–1.53x and every other M is at parity or faster except M 65–80 and 113–128 (0.94–0.98x); a decode step reads the weight cold, as above.

Padded-M check. For M without an exact row, the a8w8 bpreshuffle lookup uses M rounded up (16/32/64/128 by size), then the next power of two (8192 once M > 8192 and N > 4096), so each row also serves the uneven M below it. Base-vs-head A/B (base = merge-base e0ba6757d; aiter.gemm_a8w8_bpreshuffle with run_perftest, 3 interleaved rounds, one MI355X) on 2112x7168 (107 points) and 6144x7168 (143 points), covering every row, the row minus 1, the previous row plus 1, the midpoint, and odd M up to 49153. Errors match base at every point.

The first revision had two padding regressions: CK M=8192 (6144x7168) ran 0.36–0.43x for M > 4096 not a multiple of 256, and CK M=112 (2112x7168) ran 0.83–0.86x for M 97–111. Those rows now use FlyDSL 128x256x128_1x2x4x2 and 16x64x512_1x3x4x2.
6144x7168 rows M=544/576/608/640 (FlyDSL 80x192x256_1x1x4x2) were added because the CK M=1024 row ran 0.95–0.97x for M 513–640.
Results vs base: M > 4096 runs 1.11–1.31x, M 513–640 runs 1.15–1.42x, and 2112x7168 M 97–112 runs 1.12–1.15x in this production-path A/B (the cold-weights kernel-direct table above shows 1.08–1.12x for those M). Aligned-M gains are unchanged (e.g. M=8192 1.20x, 16384 1.20x).
In this A/B no point is below 0.97x. The lowest, M 1025–1217 on the M=2048 row, is at parity within base's run-to-run spread (about ±3.5%).
The 8-wave FlyDSL 256x256x128 kernels were faster but gave intermittently wrong results at M ≥ 8193, so they are not used.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

🤖 Generated with Claude Code

github-actions · 2026-10-12T03:39:18Z

🏷️ CI Guide

Runs automatically on every PR:

✅ Pre-checks (submodule verification, code formatting)
✅ Aiter op tests (gfx942 + gfx950)
✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
`ci:gfx1250-ffm-triton`	Run the five-shard gfx1250 FFM Triton test suite
`ci:triton-300x`	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
`ci:triton-355`	Run the full Triton test suite on MI35X, not only the tests the change affects
`ci:multi-gpu`	Aiter multi-GPU tests on the 8-GPU runner
`ci:sglang`	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
`ci:atom`	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
`ci:atom_full`	ATOM accuracy suite for PR and main models from ATOM `models_accuracy.json`
`ci:vllm`	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
`ci:all`	All standard extended tests (excludes `ci:atom_full`)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 6410 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

Copilot

🟡 Changes recommended

Three new 8-wave rows route supported FP16 calls to a BF16-only kernel.

1 open finding

8-wave dispatch ignores output dtype and breaks FP16 calls · New

What changed in this PR

Adds gfx950 Kimi-K3 FP8 bpreshuffle GEMM tuning data.

Changes:

Adds 87 tuned rows and matching untuned keys.
Removes nine slower DSv3 rows to prevent collisions.

File	Description
`dsv3_a8w8_bpreshuffle_tuned_gemm.csv`	Removes superseded rows.
`a8w8_bpreshuffle_untuned_gemm_kimik3.csv`	Adds tuning input shapes.
`a8w8_bpreshuffle_tuned_gemm_kimik3.csv`	Adds tuned gfx950 configurations.

🧠 Review effort: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟢 Approval recommended

The tuned and untuned keys match, merged shapes are unique, and the unsafe configurations are excluded.

0 open findings

1 resolved since last review

8-wave dispatch ignores output dtype and breaks FP16 calls

🧠 Review effort: Balanced

Copilot

🔵 Needs a closer look

Hardware-specific kernel correctness and performance require final confirmation of the reported MI355X validation.

0 open findings

🧠 Review effort: Balanced

Copilot

🟢 Approval recommended

The CSV keys are consistent and collision-free; only the PR description’s row counts need correction.

1 open finding

Update row counts to match the CSV changes · New

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

The new catch-all selection routes M > 65536 to a FlyDSL launcher that rejects those inputs without a fallback.

1 open issue

Gate FlyDSL selection at launcher M limit to preserve fallback · New

1 resolved since last review

Update row counts to match the CSV changes

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

The K-divisibility fallback bypasses the CK output-capacity guard, allowing a bf16 output exceeding the documented unsafe 2 GiB threshold to reach CK on gfx950.

1 open issue

Apply output capacity guard to K-divisibility fallback · New

1 resolved since last review

Padded K=8192 selects kernel with unsupported K divisibility

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The gfx950 tuning and CK 2-GiB fallback boundary require MI355X correctness and latency validation beyond the available static checks and mocked routing tests.

No open issues 🎉

1 resolved since last review

Apply output capacity guard to K-divisibility fallback

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

The CK fallback passes strided activations without contiguous normalization, producing incorrect results for the reported input case and requiring a fix plus a strided-input regression test.

1 open issue

For (M,N,K)=(65537,6144,7168) with XQ = backing[:, ::2], this fallback passes a K-stride-2… · New

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

The fallback passes strided scale views to CK’s fixed-layout raw-pointer reads, producing incorrectly scaled output; both scales need contiguous storage and regression coverage for strided inputs.

1 open issue

For logical (M,N,K)=(2,2112,7168) with the weight padded to K=8192, the new _ks7 row reaches… · New

1 resolved since last review

For (M,N,K)=(65537,6144,7168) with XQ = backing[:, ::2], this fallback passes a K-stride-2…

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

The regression check uses torch.equal instead of the required shared checkAllclose helper, bypassing standard mismatch-count and error diagnostics.

1 open issue

Use checkAllclose for exact numerical comparison · New

1 resolved since last review

For logical (M,N,K)=(2,2112,7168) with the weight padded to K=8192, the new _ks7 row reaches…

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The gfx950 tuning choices and CK 2 GiB fallback boundary require head-matched GPU correctness and base/head performance validation that was unavailable in this review environment.

No open issues 🎉

1 resolved since last review

Use checkAllclose for exact numerical comparison

_{Review effort: Balanced}

- The FlyDSL launcher raises for M > PRESHUFFLE_M_MAX (65536) and for a K the row's split-K or tile_k does not divide. - getPaddedM(gl=1) serves every M > 8192 with the M=8192 row when N > 4096, so a FlyDSL M=8192 row reaches M > 65536. - A K-padded weight runs at K_w, so a logical-K _ksN row (e.g. _ks7 on K=7168 padded to 8192) fails the K check. - gemm_a8w8_bpreshuffle_flydsl now takes its existing CK fallback in both cases instead of raising. - Fall back only while roundup(M, 256) * N * out_bytes < 2 GiB; past that CK writes wrong rows without error. - Measured on gfx950: CK N=6144 is correct at M=174700 and writes wrong rows at M=174740 (128-row M tile). - Past that bound the FlyDSL launcher error is raised, as on main, e.g. main's 57344x8192 row at M=65537. - Pass contiguous XQ, WQ and scales to CK, as the FlyDSL path does: CK reads them through raw contiguous pointers. - Add meta-tensor tests for both sides of the M limit and the 2 GiB bound, and for ks7/ks2/ks4 K divisibility. - The meta tests mock get_gfx to gfx950, use K-strided XQ and row-strided scales, and assert CK gets contiguous ones.

…K3 rows replace - Remove the gfx950 256-CU DSv3 rows for N=2112 K=7168 at M=1, 2, 4, 8, 16, 32, 64, 96 and 128. - The next commit adds Kimi-K3 rows under the same keys; the family-wide config merge raises on duplicate shapes. - Until then those M use the padded-M lookup over the remaining DSv3 N=2112 K=7168 rows (M >= 256).

- Add 149 gfx950 256-CU rows to a8w8_bpreshuffle_tuned_gemm_kimik3.csv and every new key to the untuned file. - N=7168 K=3584: 55 rows, M=1..256 and M=288..1024 in steps of 32, so M above 256 no longer uses the default kernel. - N=6336 K=7168: 34 rows (M=1..256); N=2112 K=7168: 16 rows (M=1..128), incl. the 9 keys the previous commit freed. - KDA in_proj N=6144 K=7168 (24 rows) and f_a_proj N=128 K=7168 (20 rows), quantized online by ATOM (PTPC FP8). - N=6144 M=544/576/608/640 rows stop padded M 513-640 from using the slower M=1024 CK row (0.95-0.97x). - N=6144 M=8192 is FlyDSL 128x256x128_1x2x4x2: a CK row ran 0.36-0.43x for M > 4096 not a multiple of 256. - N=2112 M=112 is FlyDSL 16x64x512_1x3x4x2: a CK row ran 0.83-0.86x for M 97-111. - 8 N=7168 K=3584 rows were re-tuned on FlyDSL 0.4.0 (M=1/4/8/12/88/96/320/1024), 1.04-1.12x over the old picks. - 13 N=7168 K=3584 CK/cktile rows (M=72..736) and 6336x7168 M=1 use FlyDSL kernels 1.08-1.38x faster. - No FlyDSL 8-wave kernels: they intermittently returned corrupted rows; M=832, 928 and 992 use the M=1024 row. - 6144x7168 is 1.30x geomean (1.20x-1.98x up to M=4096); 128x7168 is 2.22x geomean (1.36x-5.24x) in production A/B. - All FlyDSL rows match the fp32 reference with main's max abs error and are bit-identical across repeated calls.

- Run the real CK fallback against the fp32 reference for two Kimi-K3 gfx950 256-CU rows the launcher rejects. - Cover M=65537 > PRESHUFFLE_M_MAX on the 6144x7168 M=8192 row and the _ks7 2112x7168 row on a K=8192 padded weight. - Wrap the CK and FlyDSL launchers to assert only CK ran, and assert the tuned FlyDSL row is still selected. - Compare row-strided scales with contiguous ones through both fallback rows with exact checkAllclose (0 tolerance). - Add both shapes to the script's default -s sweep so the summary table covers them. - Run these pytest checks from __main__ too: CI runs this file with python3, which skips its pytest functions

Copilot

🔵 Needs a closer look

The gfx950 kernel selections and CK fallback’s 2 GiB safety boundary require target-GPU correctness and latency validation that static review and CSV checks could not provide.

No open issues 🎉

_{Review effort: Balanced}

demonsan

LGTM

- Remove the 11 a8w8 bpreshuffle rows at N=6144, K=7168 (and their untuned lines). With M padding they took over M values from ROCm#6410's rows. - Untuned lists now hold exactly the (M, N, K) of the tuned rows this PR adds. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: fall back to CK for FlyDSL bpreshuffle rows its launcher rejects - The FlyDSL launcher raises for M > PRESHUFFLE_M_MAX (65536) and for a K the row's split-K or tile_k does not divide. - getPaddedM(gl=1) serves every M > 8192 with the M=8192 row when N > 4096, so a FlyDSL M=8192 row reaches M > 65536. - A K-padded weight runs at K_w, so a logical-K _ksN row (e.g. _ks7 on K=7168 padded to 8192) fails the K check. - gemm_a8w8_bpreshuffle_flydsl now takes its existing CK fallback in both cases instead of raising. - Fall back only while roundup(M, 256) * N * out_bytes < 2 GiB; past that CK writes wrong rows without error. - Measured on gfx950: CK N=6144 is correct at M=174700 and writes wrong rows at M=174740 (128-row M tile). - Past that bound the FlyDSL launcher error is raised, as on main, e.g. main's 57344x8192 row at M=65537. - Pass contiguous XQ, WQ and scales to CK, as the FlyDSL path does: CK reads them through raw contiguous pointers. - Add meta-tensor tests for both sides of the M limit and the 2 GiB bound, and for ks7/ks2/ks4 K divisibility. - The meta tests mock get_gfx to gfx950, use K-strided XQ and row-strided scales, and assert CK gets contiguous ones. * chore(configs): drop 9 DSv3 N=2112 K=7168 bpreshuffle rows that Kimi-K3 rows replace - Remove the gfx950 256-CU DSv3 rows for N=2112 K=7168 at M=1, 2, 4, 8, 16, 32, 64, 96 and 128. - The next commit adds Kimi-K3 rows under the same keys; the family-wide config merge raises on duplicate shapes. - Until then those M use the padded-M lookup over the remaining DSv3 N=2112 K=7168 rows (M >= 256). * perf(configs): add Kimi-K3 gfx950 A8W8 bpreshuffle tuned configs - Add 149 gfx950 256-CU rows to a8w8_bpreshuffle_tuned_gemm_kimik3.csv and every new key to the untuned file. - N=7168 K=3584: 55 rows, M=1..256 and M=288..1024 in steps of 32, so M above 256 no longer uses the default kernel. - N=6336 K=7168: 34 rows (M=1..256); N=2112 K=7168: 16 rows (M=1..128), incl. the 9 keys the previous commit freed. - KDA in_proj N=6144 K=7168 (24 rows) and f_a_proj N=128 K=7168 (20 rows), quantized online by ATOM (PTPC FP8). - N=6144 M=544/576/608/640 rows stop padded M 513-640 from using the slower M=1024 CK row (0.95-0.97x). - N=6144 M=8192 is FlyDSL 128x256x128_1x2x4x2: a CK row ran 0.36-0.43x for M > 4096 not a multiple of 256. - N=2112 M=112 is FlyDSL 16x64x512_1x3x4x2: a CK row ran 0.83-0.86x for M 97-111. - 8 N=7168 K=3584 rows were re-tuned on FlyDSL 0.4.0 (M=1/4/8/12/88/96/320/1024), 1.04-1.12x over the old picks. - 13 N=7168 K=3584 CK/cktile rows (M=72..736) and 6336x7168 M=1 use FlyDSL kernels 1.08-1.38x faster. - No FlyDSL 8-wave kernels: they intermittently returned corrupted rows; M=832, 928 and 992 use the M=1024 row. - 6144x7168 is 1.30x geomean (1.20x-1.98x up to M=4096); 128x7168 is 2.22x geomean (1.36x-5.24x) in production A/B. - All FlyDSL rows match the fp32 reference with main's max abs error and are bit-identical across repeated calls. * test: check the FlyDSL-to-CK bpreshuffle fallback numerically on gfx950 - Run the real CK fallback against the fp32 reference for two Kimi-K3 gfx950 256-CU rows the launcher rejects. - Cover M=65537 > PRESHUFFLE_M_MAX on the 6144x7168 M=8192 row and the _ks7 2112x7168 row on a K=8192 padded weight. - Wrap the CK and FlyDSL launchers to assert only CK ran, and assert the tuned FlyDSL row is still selected. - Compare row-strided scales with contiguous ones through both fallback rows with exact checkAllclose (0 tolerance). - Add both shapes to the script's default -s sweep so the summary table covers them. - Run these pytest checks from __main__ too: CI runs this file with python3, which skips its pytest functions

yuychang requested review from a team and a balanced review from Copilot October 12, 2026 03:38

github-actions Bot added the Config label Oct 12, 2026

Copilot started reviewing on behalf of yuychang October 12, 2026 03:39

View session

yuychang mentioned this pull request Oct 12, 2026

[Triton/Gluon] Kimi-K3 GEMM tuning, caller-owned outputs and large-M bpreshuffle reuse #6262

Closed

1 task done

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread aiter/configs/model_configs/a8w8_bpreshuffle_tuned_gemm_kimik3.csv Resolved

Copilot AI balanced review requested due to automatic review settings October 12, 2026 06:23

Copilot started reviewing on behalf of yuychang October 12, 2026 06:24

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

yuychang mentioned this pull request Oct 12, 2026

[FlyDSL] perf: Reuse GEMM outputs and tuned A8W8 configurations above the largest tuned M #6205

Closed

1 task done

Copilot AI balanced review requested due to automatic review settings October 12, 2026 17:50

Copilot started reviewing on behalf of yuychang October 12, 2026 17:51

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

zufayu requested a review from demonsan October 13, 2026 01:42

zufayu assigned demonsan Oct 13, 2026

Copilot AI balanced review requested due to automatic review settings October 14, 2026 12:55

Copilot started reviewing on behalf of yuychang October 14, 2026 12:56

View session

Copilot AI reviewed Oct 14, 2026

View reviewed changes

Comment thread aiter/configs/model_configs/a8w8_bpreshuffle_tuned_gemm_kimik3.csv Resolved

yuychang added a commit to yuychang/aiter that referenced this pull request Oct 14, 2026

Merge ROCm#6410

f564d70

yuychang added a commit to yuychang/aiter that referenced this pull request Oct 14, 2026

Merge ROCm#6410

e990cf6

yuychang added a commit to yuychang/aiter that referenced this pull request Oct 14, 2026

Merge ROCm#6410

64b0502

This was referenced Oct 17, 2026

[Perf] [AMD] kimi-k3: Run the fused KDA decode with a BF16 state and Quark weights sgl-project/sglang#41794

Open

[Perf] [AMD] kimi-k3: Fold decode activation FP8 quant into its producers sgl-project/sglang#43125

Closed

yuychang force-pushed the k3/a8w8-bpreshuffle-configs branch from 71872e7 to 76acb56 Compare October 17, 2026 02:41

Copilot AI balanced review requested due to automatic review settings October 17, 2026 02:41

Copilot started reviewing on behalf of yuychang October 17, 2026 02:42

View session

Copilot AI reviewed Oct 17, 2026

View reviewed changes

Comment thread aiter/configs/model_configs/a8w8_bpreshuffle_tuned_gemm_kimik3.csv Resolved

Copilot AI balanced review requested due to automatic review settings October 17, 2026 04:02

github-actions Bot changed the title ~~[Config] Add Kimi-K3 gfx950 A8W8 bpreshuffle tuned configs~~ Add Kimi-K3 gfx950 A8W8 bpreshuffle tuned configs Oct 17, 2026

Copilot AI reviewed Oct 28, 2026

View reviewed changes

Comment thread aiter/ops/gemm_op_a8w8.py Resolved

Copilot AI balanced review requested due to automatic review settings October 28, 2026 09:07

Copilot started reviewing on behalf of yuychang October 28, 2026 09:08

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 28, 2026 14:32

Copilot started reviewing on behalf of yuychang October 28, 2026 14:32

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

Comment thread aiter/ops/gemm_op_a8w8.py Resolved

Copilot AI balanced review requested due to automatic review settings October 28, 2026 14:45

Copilot started reviewing on behalf of yuychang October 28, 2026 14:45

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

Comment thread aiter/ops/gemm_op_a8w8.py Resolved

Copilot AI balanced review requested due to automatic review settings October 28, 2026 15:13

Copilot started reviewing on behalf of yuychang October 28, 2026 15:14

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

Comment thread op_tests/test_gemm_a8w8_bpreshuffle_pad_k.py Resolved

Copilot AI balanced review requested due to automatic review settings October 28, 2026 15:28

Copilot started reviewing on behalf of yuychang October 28, 2026 15:31

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

yuychang changed the title ~~Add Kimi-K3 gfx950 A8W8 bpreshuffle tuned configs~~ [Perf] Add Kimi-K3 gfx950 A8W8 bpreshuffle tuned configs Oct 28, 2026

yuychang added 3 commits October 29, 2026 01:48

yuychang force-pushed the k3/a8w8-bpreshuffle-configs branch from b0c9fc6 to 9689a39 Compare October 29, 2026 02:10

yuychang force-pushed the k3/a8w8-bpreshuffle-configs branch from 9689a39 to a424cd8 Compare October 29, 2026 02:11

Copilot AI balanced review requested due to automatic review settings October 29, 2026 02:11

Copilot started reviewing on behalf of yuychang October 29, 2026 02:12

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

demonsan approved these changes Oct 29, 2026

View reviewed changes

demonsan merged commit ac32ca7 into ROCm:main Oct 29, 2026
55 checks passed

Conversation

yuychang commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

github-actions Bot commented Oct 12, 2026

🏷️ CI Guide

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

demonsan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yuychang commented Oct 12, 2026 •

edited

Loading