Repository navigation
[Config] configs: GLM-5.3 fused shared MoE rows for gfx950 - #5735
Merged
yifehuan merged 1 commit intoSep 23, 2026
Merged
Conversation
Signed-off-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Contributor
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
PR title tags & labels: |
1 task done
This was referenced Sep 24, 2026
This was referenced Sep 29, 2026
Merged
jin-amd
added a commit
to jin-amd/vllm
that referenced
this pull request
Oct 2, 2026
AITER has fused-MoE configs tuned for GLM-5.3-Flash's fused shared-expert shape (E=289, topk=9) only on gfx950 (ROCm/aiter#5735, in AITER v0.1.23). The gfx942 rows are still in review (ROCm/aiter#5953), so on gfx942 the fused MoE would run untuned fallback kernels. Keep the shared experts as a separate MLP on every other GPU, and warn once when VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS is set there. gfx942 follows once AITER ships its rows. Signed-off-by: Jin Tao <jin.tao@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
jin-amd
added a commit
to jin-amd/vllm
that referenced
this pull request
Oct 5, 2026
AITER has fused-MoE configs tuned for GLM-5.3-Flash's fused shared-expert shape (E=289, topk=9) only on gfx950 (ROCm/aiter#5735, in AITER v0.1.23). The gfx942 rows are still in review (ROCm/aiter#5953), so on gfx942 the fused MoE would run untuned fallback kernels. Keep the shared experts as a separate MLP on every other GPU, and warn once when VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS is set there. gfx942 follows once AITER ships its rows. Signed-off-by: Jin Tao <jin.tao@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
jin-amd
added a commit
to jin-amd/vllm
that referenced
this pull request
Oct 5, 2026
…etups The fused shared-expert MoE runs E=289, topk=9, and AITER's tuned rows for that shape (ROCm/aiter#5735) exist only on gfx950 at TP4 and TP8, with all 289 experts on every rank. Data and prefill context parallelism fold into the MoE's tensor-parallel split, and expert parallelism splits the experts across ranks, so each of them moves the per-rank shape off those rows. Check the parallel layout next to the GPU, the way DeepSeek-V4's heterogeneous shared-expert gate does: fusion needs gfx950, tensor_parallel_size 4 or 8, data_parallel_size 1, prefill_context_parallel_size 1 and no expert parallelism (EPLB already requires it). Otherwise VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS logs a one-time warning that lists every reason, and the shared experts stay a separate MLP. The MTP draft model's layers see the same parallel config, so its loader follows. Signed-off-by: Jin Tao <jin.tao@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Open
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
GLM-5.3 shared-expert fusion changes the routed fused-MoE dispatch from
E=288, topk=8toE=289, topk=9. The ninth expert is always selected with weight 1.0, so one AITER launch performs the routed and shared gate/up GEMM,SiLU(gate) * up, down GEMM, and routed-weight accumulation.The two TP-local signatures are:
(4096,512),E=289,topk=9(4096,256),E=289,topk=9The existing model-config set does not cover either signature. Without exact rows, both use fallback kernels.
What this adds
Sixteen gfx950/cu256 rows are added in a dedicated GLM-5.3 fused-shared table:
M={4,8,1024,2048,4096,8192,16384}M={8,32,128,256,512,2048,4096,8192,16384}The matching untuned table records the complete 15-shape power-of-two ladder for both TP4 and TP8 (30 source rows).
Both signatures were tuned with the standard fused-MoE tuner over the compatible one-stage ASM block-FP8 family. Only rows improving AITER's production operator by at least 3% were retained. The remaining buckets are intentionally absent and keep fallback dispatch.
Validation
err1=0.0%,run_1stage=1,kernelName2empty, andus2=0.OK.test_csv_validation.pyandtest_config_shape_collision.py: 32 passed, 37 subtests passed.Hardware: MI355X (
gfx950, 256 CUs).Image:
rocm/sgl-dev:v0.5.20-rocm724-mi35x-20260921.Tuning branch base: upstream
ROCm/aitermainat commit7d2f6a51a6364e0aad4d2e81dda5cc28c82bfdd9.Submission Checklist
main).