Skip to content

[Config] [tune] DSv4 a8w8 blockscale: add gfx950 configs for three MI355X shapes - #4664

Merged
zufayu merged 3 commits into
ROCm:mainfrom
jiacao-amd:dsv4-a8w8-blockscale-tuning
Sep 11, 2026
Merged

zufayu merged 3 commits into
ROCm:mainfrom
jiacao-amd:dsv4-a8w8-blockscale-tuning

Conversation

@jiacao-amd

@jiacao-amd jiacao-amd commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds 106 tuned rows to dsv4_a8w8_blockscale_tuned_gemm.csv for gfx950, covering four DeepSeek-V4 shape families:

N K rows M coverage
6144 7168 15 1 .. 16384
65536 1536 15 1 .. 16384
7168 3072 15 1 .. 16384
7168 7168 61 observed DSv4 values from 4 .. 65536

These shapes previously fell back to an untuned default.

The M=8192, N=65536, K=1536 entry uses the peer-provided CKTile config 11. On a local MI355X A/B run, it reduced latency from 1303.55 us to 947.08 us (27.35% lower, 1.376x speedup), with errRatio=0.0.

Why merge into the existing file

These rows are merged into the existing dsv4_a8w8_blockscale_tuned_gemm.csv, because AITER_CONFIG.get_config_file() globs model_configs/*<tuned_file_name>*.csv and merges every match. A second file would be redundant at load time, and a8w8_..._dsv4 would also be inconsistent with the existing dsv4_a8w8_blockscale_{tuned,untuned}_gemm.csv naming.

Details

  • Config-only change; no operator code path changes.
  • Final diff against main: 106 insertions, 0 deletions.
  • No duplicate (gfx, cu_num, M, N, K) keys or malformed rows.
  • All selected kernel names are present in the compiled kernel registry.

Test Plan

Correctness test passed on MI355X (gfx950, 256 CUs): 106/106 OK, 0 mismatches, 0 errors.

python3 csrc/ck_gemm_a8w8_blockscale/gemm_a8w8_blockscale_tune.py \
    -i aiter/configs/a8w8_blockscale_untuned_gemm.csv \
    --run_config .pr4664_added_shapes_plus_peer.csv \
    --warmup 2 --iters 5

Adds 45 tuned rows to dsv4_a8w8_blockscale_tuned_gemm.csv for
gfx950, covering N,K = 6144x7168, 65536x1536 and 7168x3072 with
an M sweep of 1..256 each. None of these shapes had an entry, so
they previously fell back to an untuned default.

Rows are merged into the existing DSv4 file rather than added as
a new CSV: get_config_file() globs model_configs/*<name>*.csv and
merges every match, so a separate file would be redundant and the
naming would be inconsistent with dsv4_a8w8_blockscale_untuned_gemm.csv.

Append-only: no existing rows are modified or removed.
Signed-off-by: jiacao-amd <jiahui.cao@amd.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4664 --add-label <label>

jiacao-amd and others added 2 commits August 12, 2026 00:31
Adds 61 gfx950 rows for the (N=7168, K=7168) non-preshuffle blockscale
GEMM -- the most frequently hit untuned shape in DSv4 MI355X runs. It
appears in every DSv4 server log we sampled, across 61 distinct M values
(4 .. 65536), and every one of them was falling back to the heuristic
default kernel.

The bpreshuffle variant of this shape is already tuned (75 gfx950 rows in
dsv4_a8w8_blockscale_bpreshuffle_tuned_gemm.csv); only the non-preshuffle
path was missing.

Tuned on MI355X (gfx950, cu_num=256) with
csrc/ck_gemm_a8w8_blockscale/gemm_a8w8_blockscale_tune.py --mp 8
--shape_grouped. 61/61 shapes tuned successfully, errRatio 0.0 on all.

Measured with --run_config against the production operator (heuristic
default vs tuned), same harness for both sides:

    M range      speedup
    4 - 32       1.21 - 1.23x
    40 - 64      1.01 - 1.05x
    72 - 128     1.46 - 1.73x
    136 - 144    1.10x
    152 - 256    1.48 - 1.96x
    288 - 512    1.62 - 2.55x
    544 - 1024   2.20 - 3.24x
    8192 - 65536 5.06 - 5.59x

    total        35998 us -> 8244 us  = 4.37x (77.1% saved)

No shape regresses. Weighting each M by how often it appears in the
sampled logs gives the same picture: 4.40x.

All 10 selected kernel names (43 ck rows + 18 cktile rows) are already in
the compiled kernel registry of module_gemm_a8w8_blockscale.so /
module_gemm_a8w8_blockscale_cktile.so, so this is a pure config change --
no codegen and no JIT rebuild. Verified 0 key collisions against the
existing non-preshuffle tuned set.
Signed-off-by: jiacao-amd <jiahui.cao@amd.com>
@jiacao-amd
jiacao-amd marked this pull request as ready for review September 10, 2026 21:59
@jiacao-amd
jiacao-amd requested a review from a team September 10, 2026 21:59
@github-actions github-actions Bot changed the title [tune] DSv4 a8w8 blockscale: add gfx950 configs for three MI355X shapes [Config] [tune] DSv4 a8w8 blockscale: add gfx950 configs for three MI355X shapes Sep 10, 2026
@zufayu
zufayu requested a review from yzhou103 September 11, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants