[CK] Add dsv41 some MoE tuning configs by kkHuang-amd · Pull Request #5967 · ROCm/aiter

kkHuang-amd · 2026-09-30T00:53:09Z

Motivation

DeepSeek-V4.1-Flash on gfx950 (TP2, SGLang with --enforce-shared-experts-fusion) runs FlyDSL a8w4 MoE shapes (fp8 activation x fp4 weight, E=385 topk=7 and E=129 topk=4) that have no tuned entries, so they fall back to the untuned heuristic.

Technical Details

aiter/configs/model_configs/dsv41_fp4_tuned_fmoe.csv: add 23 tuned rows (gfx950, cu_num 256, model_dim 5120, inter_dim 1152; E385/topk7 for 13 token sizes, E129/topk4 for 10 token sizes). No key overlaps the existing fp4/fp4 rows.
aiter/configs/model_configs/dsv41_fp4_untuned_fmoe.csv: add the same 23 shapes so they can be re-tuned.
csrc/ck_gemm_moe_2stages_codegen/gemm_moe_tune.py: generate the v2 stage1 test data on the target GPU (with torch.device(device)). Packing the E x 2I x H fp4 weights on the CPU took tens of minutes per shape at E=385.

Test Plan

Tuned with gemm_moe_tune.py on MI355X (gfx950).
Checked the merged CSVs: no duplicate keys, and every untuned shape has a tuned row.
Served DeepSeek-V4.1-Flash TP2 with SGLang and ran GSM8K.

Test Result

Server startup shows no untuned-shape warnings for the E385/E129 shapes.
GSM8K: 0.895-0.908 (1319 questions). This was measured on the PR [Triton/Gluon] [JIT] [Feature] Add tuned native group32 A8W8 Triton GEMM for gfx950 #5750 based env that carries the same tuned rows.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

…che, MoE tuner GPU datagen, torch.Stream, tuned FMoE CSV) - flydsl mixed MoE stage1: full vmcnt drain + sched_barrier before the LDS-DMA barrier (partial waits let LDS-DMA loads cross it; ROCm#5561) - fp4 prefill mqa logits: unbounded kernel cache (lru_cache(32) evicted and recompiled per shape) - MoE tuner: generate the E x 2I x H test weights on the GPU (CPU fp4 packing took tens of minutes per shape at E=385) - cpp_itfs torch_utils: accept any torch.Stream (derive the HIP handle when cuda_stream is missing) - add DSV4.1 TP2 shared-experts-fusion fp8/fp4 tuned FMoE config

…moe.csv - move the 23 shared-experts-fusion rows (E385 topk7, E129 topk4, fp8 activation x fp4 weight) into dsv41_fp4_tuned_fmoe.csv and their shapes into dsv41_fp4_untuned_fmoe.csv instead of a separate file - drop the torch_utils, fp4 prefill cache and flydsl stage1 wait changes

github-actions · 2026-09-30T00:53:27Z

🏷️ CI Guide

Runs automatically on every PR:

✅ Pre-checks (submodule verification, code formatting)
✅ Aiter op tests (gfx942 + gfx950)
✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
`ci:gfx1250-ffm-triton`	Run the five-shard gfx1250 FFM Triton test suite
`ci:triton-300x`	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
`ci:triton-355`	Run the full Triton test suite on MI35X, not only the tests the change affects
`multigpu`	Aiter multi-GPU tests on the 8-GPU runner
`ci:sglang`	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
`ci:atom`	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
`ci:atom_full`	ATOM accuracy suite for PR and main models from ATOM `models_accuracy.json`
`ci:vllm`	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
`ci:all`	All standard extended tests (excludes `ci:atom_full`)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5967 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

Copilot

Copilot review overview

🟡 Changes recommended

The new rows measure a different stage-1 path from serving, and input preparation can mix CUDA devices.

Review effort: Balanced
Findings: 1 · 2

Open (3)

Avoid cross-device buffers when tuning on non-default CUDA device · New
Retune expert groups with serving-compatible scaled FP8 inputs · New
Generate stage-2 FP4 inputs on GPU to avoid CPU packing delays · New

What changed in this PR

This PR adds DeepSeek V4.1 FP4 MoE tuning configurations and moves stage-1 tuning input generation to the GPU to reduce preparation time.

Changes:

Adds untuned and tuned configurations for 385- and 129-expert shapes.
Generates v2 stage-1 inputs on the GPU.

File	Description
`csrc/ck_gemm_moe_2stages_codegen/gemm_moe_tune.py`	Moves stage-1 input generation to the GPU.
`aiter/configs/model_configs/dsv41_fp4_untuned_fmoe.csv`	Adds shapes to the tuning inputs.
`aiter/configs/model_configs/dsv41_fp4_tuned_fmoe.csv`	Adds measured kernel selections.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        with torch.device(device):
+            d = _v2_gen(
+                token,
+                model_dim,
+                inter_dim,
+                expert,
+                topk,
+                blockM,
+                adtype=adtype,
+                b_dtype=b_dtype,
+                activation=act_type,
+                situ_beta=DEFAULT_SITUV2_BETA,
+                situ_linear_beta=DEFAULT_SITUV2_LINEAR_BETA,
+            )
        v = _v2_build_inputs(d, token, model_dim, inter_dim, expert, topk, blockM)


 gfx950,256,8192,5120,1152,384,6,ActivationType.Silu,torch.bfloat16,torch.float4_e2m1fn_x2,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,64,0,1178.4115,flydsl_mxmoe_g1_a4w4_64x256x256_xcd2,1.2%,0,flydsl_moe2_layout_afp4_wfp4_bf16_t64x256x128_atomic_sbm64,1.2%,1178.4115,0,0,0,1476.11,5872.82
 gfx950,256,16384,5120,1152,384,6,ActivationType.Silu,torch.bfloat16,torch.float4_e2m1fn_x2,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,128,0,1847.0189,flydsl_mxmoe_g1_a4w4_128x256x256_xcd4,1.2%,0,flydsl_moe2_layout_afp4_wfp4_bf16_t128x128x128_atomic_sbm128,1.2%,1847.0189,0,0,0,1883.53,3815.03
 gfx950,256,32768,5120,1152,384,6,ActivationType.Silu,torch.bfloat16,torch.float4_e2m1fn_x2,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,128,0,3253.4182,flydsl_mxmoe_g1_a4w4_128x256x256_xcd2,1.2%,0,flydsl_moe2_layout_afp4_wfp4_bf16_t128x128x128_atomic_sbm128,1.2%,3253.4182,0,0,0,2138.63,2243.21
+gfx950,256,1,5120,1152,385,7,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,32,0,12.5991,flydsl_moe1_afp8_wfp4_bf16_t32x64x256_w2_gui_kw4_fp8,0.0%,9.4207,flydsl_moe2_layout_afp8_wfp4_bf16_t32x256x128_atomic_nt_sbm32,0.0%,22.0198,0,0,0,11.25,309379.86


-        )
+        # Generate on the GPU: the CPU fp4 packing of E x 2I x H weights takes
+        # tens of minutes per shape at E=385.
+        with torch.device(device):


Copilot

Copilot review overview

🔵 Needs a closer look

Two recorded kernel selections differ from their runtime launches, and the new FP8 dispatch path lacks a focused correctness check.

Review effort: Balanced
Findings: 1 · 4

Open (5)

Avoid cross-device buffers when tuning on non-default CUDA device
Add FP8 interleaved dispatch and correctness coverage · New
Exclude invalid stage-1 tiles and retune E=385 rows · New
Generate stage-2 FP4 inputs on GPU to avoid CPU packing delays
Retune expert groups with serving-compatible scaled FP8 inputs

 gfx950,256,8192,5120,1152,384,6,ActivationType.Silu,torch.bfloat16,torch.float4_e2m1fn_x2,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,64,0,1178.4115,flydsl_mxmoe_g1_a4w4_64x256x256_xcd2,1.2%,0,flydsl_moe2_layout_afp4_wfp4_bf16_t64x256x128_atomic_sbm64,1.2%,1178.4115,0,0,0,1476.11,5872.82
 gfx950,256,16384,5120,1152,384,6,ActivationType.Silu,torch.bfloat16,torch.float4_e2m1fn_x2,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,128,0,1847.0189,flydsl_mxmoe_g1_a4w4_128x256x256_xcd4,1.2%,0,flydsl_moe2_layout_afp4_wfp4_bf16_t128x128x128_atomic_sbm128,1.2%,1847.0189,0,0,0,1883.53,3815.03
 gfx950,256,32768,5120,1152,384,6,ActivationType.Silu,torch.bfloat16,torch.float4_e2m1fn_x2,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,128,0,3253.4182,flydsl_mxmoe_g1_a4w4_128x256x256_xcd2,1.2%,0,flydsl_moe2_layout_afp4_wfp4_bf16_t128x128x128_atomic_sbm128,1.2%,3253.4182,0,0,0,2138.63,2243.21
+gfx950,256,1,5120,1152,385,7,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,32,0,12.5991,flydsl_moe1_afp8_wfp4_bf16_t32x64x256_w2_gui_kw4_fp8,0.0%,9.4207,flydsl_moe2_layout_afp8_wfp4_bf16_t32x256x128_atomic_nt_sbm32,0.0%,22.0198,0,0,0,11.25,309379.86


+gfx950,256,256,5120,1152,385,7,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,32,0,363.7643,flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8,0.0%,201.7408,flydsl_moe2_layout_afp8_wfp4_bf16_t32x256x128_atomic_nt_sbm32,0.0%,565.5051,0,0,0,112.14,12053.65
+gfx950,256,512,5120,1152,385,7,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,32,0,367.7689,flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w3_gui_fp8,0.0%,191.9912,flydsl_moe2_layout_afp8_wfp4_bf16_t32x256x128_atomic_nt_sbm32,0.0%,559.7601,0,0,0,226.59,12184.38
+gfx950,256,1024,5120,1152,385,7,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,32,0,378.2528,flydsl_moe1_afp8_wfp4_bf16_t32x128x256_w4_gui_fp8,0.0%,225.2484,flydsl_moe2_layout_afp8_wfp4_bf16_t32x256x128_atomic_nt_sbm32,0.0%,603.5012,0,0,0,420.33,11314.3
+gfx950,256,2048,5120,1152,385,7,ActivationType.Silu,torch.bfloat16,torch.float8_e4m3fn,torch.float4_e2m1fn_x2,QuantType.per_1x32,1,0,64,0,433.2996,flydsl_moe1_afp8_wfp4_bf16_t64x256x256_w3_gui_fp8,0.0%,238.7783,flydsl_moe2_layout_afp8_wfp4_bf16_t64x256x128_atomic_nt_sbm64,0.0%,672.0779,0,0,0,754.89,10183.23


wunhuang added 2 commits September 30, 2026 00:41

kkHuang-amd requested review from a team and a balanced review from Copilot September 30, 2026 00:53

github-actions Bot changed the title ~~Add dsv41 some MoE tuning configs~~ [CK] Add dsv41 some MoE tuning configs Sep 30, 2026

github-actions Bot added the CK label Sep 30, 2026

Copilot started reviewing on behalf of kkHuang-amd September 30, 2026 00:53 View session

kkHuang-amd mentioned this pull request Sep 30, 2026

[AITER-Upgrade] PR readiness sgl-project/sglang#21302

Open

Copilot AI reviewed Sep 30, 2026

View reviewed changes

kkHuang-amd changed the title ~~[CK] Add dsv41 some MoE tuning configs~~ Add dsv41 some MoE tuning configs Sep 30, 2026

Merge branch 'main' into dsv41-atomport-main

2bd1472

Copilot AI balanced review requested due to automatic review settings September 30, 2026 04:50

github-actions Bot changed the title ~~Add dsv41 some MoE tuning configs~~ [CK] Add dsv41 some MoE tuning configs Sep 30, 2026

Copilot started reviewing on behalf of kkHuang-amd September 30, 2026 04:51 View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

zufayu self-requested a review October 1, 2026 00:36

zufayu self-assigned this Oct 1, 2026

michaelzhang-ai mentioned this pull request Oct 2, 2026

[AMD] Skip the AITER #6042/#5967 patches on gfx1250 sgl-project/sglang#42219

Merged

Fangzhou-Ai mentioned this pull request Oct 5, 2026

[FlyDSL] Add full-expert DeepSeek V4.1 Flash MegaMoE tuning for gfx950 #6154

Draft

valarLip approved these changes Oct 6, 2026

View reviewed changes

valarLip merged commit 356c53d into ROCm:main Oct 6, 2026
63 checks passed

Navigation Menu

Repository navigation

[CK] Add dsv41 some MoE tuning configs - #5967

[CK] Add dsv41 some MoE tuning configs#5967
valarLip merged 3 commits into
ROCm:mainfrom
kkHuang-amd:dsv41-atomport-main

kkHuang-amd commented Sep 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

kkHuang-amd commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

github-actions Bot commented Sep 30, 2026

🏷️ CI Guide

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kkHuang-amd commented Sep 30, 2026 •

edited

Loading