Skip to content

[Config] configs: GLM-5.3 fused shared MoE rows for gfx950 - #5735

Merged
yifehuan merged 1 commit into
ROCm:mainfrom
Raiden-Makoto:RM/glm53-shared-fmoe-tuned-config
Sep 23, 2026
Merged

yifehuan merged 1 commit into
ROCm:mainfrom
Raiden-Makoto:RM/glm53-shared-fmoe-tuned-config

Conversation

@Raiden-Makoto

@Raiden-Makoto Raiden-Makoto commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Why

GLM-5.3 shared-expert fusion changes the routed fused-MoE dispatch from E=288, topk=8 to E=289, topk=9. The ninth expert is always selected with weight 1.0, so one AITER launch performs the routed and shared gate/up GEMM, SiLU(gate) * up, down GEMM, and routed-weight accumulation.

The two TP-local signatures are:

  • TP4: model/intermediate (4096,512), E=289, topk=9
  • TP8: model/intermediate (4096,256), E=289, topk=9

The existing model-config set does not cover either signature. Without exact rows, both use fallback kernels.

What this adds

Sixteen gfx950/cu256 rows are added in a dedicated GLM-5.3 fused-shared table:

  • TP4: M={4,8,1024,2048,4096,8192,16384}
  • TP8: M={8,32,128,256,512,2048,4096,8192,16384}

The matching untuned table records the complete 15-shape power-of-two ladder for both TP4 and TP8 (30 source rows).

Both signatures were tuned with the standard fused-MoE tuner over the compatible one-stage ASM block-FP8 family. Only rows improving AITER's production operator by at least 3% were retained. The remaining buckets are intentionally absent and keep fallback dispatch.

Validation

  • All 16 retained rows have err1=0.0%, run_1stage=1, kernelName2 empty, and us2=0.
  • Production-operator execution passed for every retained row: 7 TP4 and 9 TP8, all status OK.
  • TP4 retained rows improve production-operator latency by 10.98-48.12%.
  • TP8 retained rows improve production-operator latency by 4.61-45.11%.
  • Direct fused-versus-separate tensor comparison passed at TP4 and TP8 from M=1 through M=16384: all outputs finite, minimum cosine 0.999822, maximum mean absolute error 7.45e-6.
  • test_csv_validation.py and test_config_shape_collision.py: 32 passed, 37 subtests passed.
  • Every new dispatch key occurs exactly once across the complete current-main fused-MoE config set.
  • No open AITER PR contains any new dispatch key.

Hardware: MI355X (gfx950, 256 CUs).
Image: rocm/sgl-dev:v0.5.20-rocm724-mi35x-20260921.
Tuning branch base: upstream ROCm/aiter main at commit 7d2f6a51a6364e0aad4d2e81dda5cc28c82bfdd9.

Submission Checklist

  • Looked over the ROCm contributing guidelines.
  • Targeting the repository default branch (main).
  • Included successful config-validation and production-operator results.
  • Commit includes the required DCO sign-off.
  • CI green for the updated head.

Signed-off-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
@Raiden-Makoto
Raiden-Makoto requested a review from a team September 21, 2026 16:55
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
multigpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5735 --add-label <label>

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to opt this PR out.

@zufayu
zufayu requested a review from yifehuan September 22, 2026 01:51

@yifehuan yifehuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yifehuan
yifehuan merged commit 86f61d3 into ROCm:main Sep 23, 2026
46 of 48 checks passed
jin-amd added a commit to jin-amd/vllm that referenced this pull request Oct 2, 2026
AITER has fused-MoE configs tuned for GLM-5.3-Flash's fused shared-expert
shape (E=289, topk=9) only on gfx950 (ROCm/aiter#5735, in AITER v0.1.23).
The gfx942 rows are still in review (ROCm/aiter#5953), so on gfx942 the
fused MoE would run untuned fallback kernels.

Keep the shared experts as a separate MLP on every other GPU, and warn
once when VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS is set there. gfx942
follows once AITER ships its rows.

Signed-off-by: Jin Tao <jin.tao@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
jin-amd added a commit to jin-amd/vllm that referenced this pull request Oct 5, 2026
AITER has fused-MoE configs tuned for GLM-5.3-Flash's fused shared-expert
shape (E=289, topk=9) only on gfx950 (ROCm/aiter#5735, in AITER v0.1.23).
The gfx942 rows are still in review (ROCm/aiter#5953), so on gfx942 the
fused MoE would run untuned fallback kernels.

Keep the shared experts as a separate MLP on every other GPU, and warn
once when VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS is set there. gfx942
follows once AITER ships its rows.

Signed-off-by: Jin Tao <jin.tao@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
jin-amd added a commit to jin-amd/vllm that referenced this pull request Oct 5, 2026
…etups

The fused shared-expert MoE runs E=289, topk=9, and AITER's tuned rows
for that shape (ROCm/aiter#5735) exist only on gfx950 at TP4 and TP8,
with all 289 experts on every rank. Data and prefill context parallelism
fold into the MoE's tensor-parallel split, and expert parallelism splits
the experts across ranks, so each of them moves the per-rank shape off
those rows.

Check the parallel layout next to the GPU, the way DeepSeek-V4's
heterogeneous shared-expert gate does: fusion needs gfx950,
tensor_parallel_size 4 or 8, data_parallel_size 1,
prefill_context_parallel_size 1 and no expert parallelism (EPLB already
requires it). Otherwise VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS logs a
one-time warning that lists every reason, and the shared experts stay a
separate MLP. The MTP draft model's layers see the same parallel config,
so its loader follows.

Signed-off-by: Jin Tao <jin.tao@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants