Skip to content

[GLM5.2] Add GLM5.2-FP8 PTPC GEMM & MoE tuned configs - #4236

Merged
yifehuan merged 2 commits into
mainfrom
mi308_tune
Jul 16, 2026
Merged

[GLM5.2] Add GLM5.2-FP8 PTPC GEMM & MoE tuned configs#4236
yifehuan merged 2 commits into
mainfrom
mi308_tune

Conversation

@qichu-yun

@qichu-yun qichu-yun commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation

Add tuned configs for the GLM-5.2-FP8 PTPC online-quant TP8 scenario on gfx942, so the runtime can use tuned MoE and GEMM configs instead of falling back to defaults.

Technical Details

Adds GLM-5.2 model-specific tuned/untuned configs under aiter/configs/model_configs/:

  • PTPC per-token FP8 MoE configs for gfx942 / 80 CU TP8 shapes.
  • A8W8 bpre-shuffle GEMM configs for GLM-5.2-FP8 PTPC shapes.
  • A8W8 blockscale bpre-shuffle GEMM configs.
  • BF16 GEMM configs.
    The MoE rows cover model_dim=6144, inter_dim=256, expert=257, topk=9, ActivationType.Silu, and QuantType.per_Token FP8 A/W.

Test Plan

Test Result

Submission Checklist

@qichu-yun
qichu-yun requested a review from a team July 14, 2026 11:19
@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4236 --add-label <label>

@zufayu
zufayu requested a review from yifehuan July 15, 2026 02:20
@qichu-yun
qichu-yun force-pushed the mi308_tune branch 2 times, most recently from 21d811b to c7eb45e Compare July 15, 2026 03:18
@qichu-yun qichu-yun changed the title [GLM5.2] Add GLM5.2-FP8 PTPC MoE tuned configs [GLM5.2] Add GLM5.2-FP8 PTPC GEMM & MoE tuned configs Jul 15, 2026
yifehuan
yifehuan reviewed Jul 15, 2026
View reviewed changes
Comment thread aiter/configs/model_configs/glm5_a8w8_blockscale_bpreshuffle_tuned_gemm.csv Outdated
@yijuc @qichu-yun
[MI308] Add GLM-5.2-FP8 gfx942 GEMM tuned configs and de-duplicate sh…
bee41e7 
…ared shapes

Add GLM-5.2-FP8 GEMM tuning for MI308 (gfx942, 80 CU): a new ptpc config
(a8w8_bpreshuffle) plus gfx942 entries added to the existing GLM-5.2
block-scale (a8w8_blockscale_bpreshuffle) and bf16 model configs under
aiter/configs/model_configs/.

At load time aiter merges every model_configs/*<op>_tuned_gemm*.csv with the
base config and de-duplicates on (gfx, cu_num, M, N, K[, q_dtype_w]); a shape
key present in more than one file makes the loader rewrite the source CSVs and
raise "please re-run". The new gfx942/cu80 entries shared keys with three
existing files, all resolved so each key lives in exactly one file:

  - qwen3.5_397b (a8w8_blockscale_bpreshuffle, N=4096/K=2048, 16 M): kept the
    existing entry for the 8 shapes with an identical kernel; for the other 8,
    kept the GLM-5.2 entry and dropped it from qwen3.5_397b. Measured on this
    MI308 the GLM-5.2 kernel is faster for all 8 (e.g. M=128 13.7us vs 19.3us,
    M=2048 123us vs 146us). The qwen3.5_397b gfx942/cu80 us values there imply
    throughput above the MI308X (80 CU) FP8 ceiling, so they are not
    representative of an 80-CU part.

  - dsv3 (a8w8_bpreshuffle, N=6144/K=1536, 19 M): kept dsv3 for the 5 shapes
    with an identical kernel; for the other 14, kept the GLM-5.2 entry. Measured
    on this MI308 the GLM-5.2 kernel is much faster at large M (e.g. M=4096
    198us vs 322us, M=32768 1468us vs 2428us) and within noise at small M.

  - dsv4 (a8w8_blockscale_bpreshuffle, N=4096/K=2048, 4 M): kept dsv4 for the
    2 shapes with an identical kernel (M=48, 768); for M=384 and M=1536 kept
    the GLM-5.2 (MI308-tuned) entry and dropped it from dsv4.

gfx950 rows in the existing GLM-5.2 files are left unchanged. No duplicate
shape keys remain on gfx942/cu80 after this change.

Signed-off-by: Chen Yi-Jun (Eveline) <yijuchen@amd.com>
yifehuan
yifehuan approved these changes Jul 16, 2026
View reviewed changes

@yifehuan yifehuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yifehuan
yifehuan merged commit 6a272f8 into main Jul 16, 2026
66 of 67 checks passed
@yifehuan
yifehuan deleted the mi308_tune branch July 16, 2026 02:03
@yifehuan yifehuan mentioned this pull request Jul 20, 2026
Merged
1 task
Raiden-Makoto pushed a commit to Raiden-Makoto/aiter that referenced this pull request Jul 20, 2026
Raiden-Makoto
feat(gemm_a8w8_blockscale_bpreshuffle): add GLM-5.2 o_proj tuned conf…
6aa9329 
…ig (gfx950)

Adds tuned a8w8_blockscale_bpreshuffle configs for GLM-5.2's o_proj (N=6144,
K=4096) on gfx950 — the one 128-aligned MLA dense-projection shape missing from
the GLM5 block-scale configs merged in ROCm#4236 (which covers q_b_proj 4096,2048 and
other GLM5 shapes but not o_proj 6144,4096). Config coverage only, no code changes;
merges via the existing *a8w8_blockscale_bpreshuffle_tuned_gemm*.csv glob.

60 rows, gfx950/cu_num=256, M = prefill (1024-65536) + decode/cuda-graph (1-512).
@Raiden-Makoto Raiden-Makoto mentioned this pull request Jul 20, 2026
Merged
3 tasks
yifehuan pushed a commit that referenced this pull request Jul 21, 2026
@Raiden-Makoto Raiden-Makoto
feat(gemm_a8w8_blockscale_bpreshuffle): add GLM-5.2 o_proj tuned conf…
9f0a99b 
…ig (gfx950) (#4243)

Adds tuned a8w8_blockscale_bpreshuffle configs for GLM-5.2's o_proj (N=6144,
K=4096) on gfx950 — the one 128-aligned MLA dense-projection shape missing from
the GLM5 block-scale configs merged in #4236 (which covers q_b_proj 4096,2048 and
other GLM5 shapes but not o_proj 6144,4096). Config coverage only, no code changes;
merges via the existing *a8w8_blockscale_bpreshuffle_tuned_gemm*.csv glob.

60 rows, gfx950/cu_num=256, M = prefill (1024-65536) + decode/cuda-graph (1-512).

Co-authored-by: Raiden-Makoto <raidenmakoto916@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yijuc yijuc yijuc left review comments
@yifehuan yifehuan yifehuan approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@qichu-yun @yijuc @yifehuan