[Perf] [AMD] Qwen3.8: AITER FlyDSL hc_mix for Qwen3.8 hyper-connections on gfx950 by amd-nprotaso · Pull Request #41880 · sgl-project/sglang

amd-nprotaso · 2026-09-30T10:53:29Z

Depends on ROCm/aiter. Without that AITER build this PR changes nothing: the path is gated on aiter.ops.flydsl.hc_mix being importable.

Motivation

GatedResidual's input mix is a low-rank gate over the [M, 4·2560] hyper-state. It runs in every Qwen3.8 decode step, and at decode sizes (M ≤ 16) it is bound by reading the weights.

On gfx950, AITER's FlyDSL hc_mix is 1.7–2.4x faster than the persistent Triton fused_hc_mix SGLang uses today.

Modifications

python/sglang/srt/layers/hc_mix_flydsl.py (new)

flydsl_hc_mix_supported(...) accepts a call only when all of these hold:

HIP, gfx950, BF16, contiguous inputs;
1 ≤ M ≤ 16, hc = 4;
the shape constraints of the AITER kernel;
AITER and FlyDSL are importable.

The checks are ordered so NVIDIA and disabled paths never import FlyDSL. weight_cache_key(...) gives the packed-weight cache a key from the weights' data pointer and version.

python/sglang/srt/layers/hyperconnection.py

GatedResidual tries FlyDSL hc_mix before fused_hc_mix.
Weights are packed once with pack_hc_weights, cached on the layer, and repacked only if the weight key changes.

Benchmark and tests

benchmark/kernels/bench_hc_mix_flydsl.py compares FlyDSL, persistent Triton and torch.compile. It uses CUDA-graph replay with rotating weights, so the weights are cold in L2.
test/registered/kernels/ops/hyperconnection/test_hc_mix_flydsl.py is registered in the AMD stage-b-test-1-gpu-small-amd suite. It checks:
- the gate rejects inputs when disabled or given unsupported shapes;
- GatedResidual dispatches to FlyDSL and matches the reference (atol=4e-3);
- the packed-weight cache reuses packed weights and repacks after a weight change;
- inference-tensor weights get a key with no version.

Accuracy Tests

The FlyDSL output matches the float64 reference within atol=4e-3, rtol=1e-2, in both the AITER op test and this PR's dispatch test.

Speed Tests and Profiling

Command: python benchmark/kernels/bench_hc_mix_flydsl.py --rows 1 2 4 8 16 --weights 64

Setup: MI355X (gfx950), torch 2.11.0+rocm7.2. Time is the per-call median of CUDA-graph replays.

M	FlyDSL hc_mix (µs)	persistent Triton `fused_hc_mix` (µs)	torch.compile (µs)	vs Triton
1	18.1	36.3	34.3	2.00x
2	18.8	44.9	38.1	2.38x
4	20.3	44.9	38.1	2.21x
8	22.1	45.7	38.2	2.07x
16	26.3	45.5	38.6	1.73x

CI States

Latest PR Test (Base): ✅ Run #40173502184
Latest PR Test (Extra): ❌ Run #40173502156
Latest PR Test (AMD ROCm 10): ❌ Run #40173502206

The lint job runs `pre-commit run --all-files`; isort and ruff-format failed on the files added/touched by this PR. Formatting only, no behavior change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Split hc_combine is a gate + apply pair: it gives small-M rows more CTAs than the one-CTA-per-row fused kernel, but costs a second dispatch. On gfx950 Qwen decode both halves sit at the ~4.2us launch floor, so the pair spends 8.3us on work the fused kernel does in one launch. Keep the split on elsewhere, where launches are cheaper and the split was tuned. Co-Authored-By: Claude <noreply@anthropic.com>

amd-nprotaso · 2026-10-21T10:50:07Z

ROCm/aiter#5993 - was merged

…hc-mix-flydsl # Conflicts: # python/sglang/srt/layers/hyperconnection.py # test/registered/kernels/ops/gemm/test_hc_mix_flydsl.py

…lang into pr/hc-mix-flydsl # Conflicts: # python/sglang/srt/layers/hyperconnection.py

1am9trash · 2026-10-22T01:56:30Z

The lint check is failed.
Also the aiter commit is not in the sglang aiter version yet. I think we can leave a msg in issue#21302 for tracking, and wait the next aiter version bump.

amd-nprotaso · 2026-10-22T08:09:53Z

The lint check is failed.

Also the aiter commit is not in the sglang aiter version yet. I think we can leave a msg in issue#21302 for tracking, and wait the next aiter version bump.

Fixed, lint(just update branch), my code have guards to check, aiter kernel.
Poste message to issue.

1am9trash · 2026-10-23T03:06:33Z

Remove amd-aiter-not-ready.
The PR has guard on it. It's okay to merge it w/o aiter version bump.

1am9trash · 2026-10-23T03:13:21Z

Hi, @amd-nprotaso

I am a bit confused because the description does not seem to match the diff.
It mentions SGLANG_AITER_HC_MIX and SGLANG_HC_COMBINE_SPLIT in environ.py, and an M ≤ 16 limit. I do not see those part in the change.

Could we update the description to match the diff? Thanks.

amd-nprotaso · 2026-10-23T10:08:46Z

Hi, @amd-nprotaso

I am a bit confused because the description does not seem to match the diff. It mentions SGLANG_AITER_HC_MIX and SGLANG_HC_COMBINE_SPLIT in environ.py, and an M ≤ 16 limit. I do not see those part in the change.

Could we update the description to match the diff? Thanks.

Yes. I removed ENV variables, to not increase number of parameters of sglang run, will modify PR description

1am9trash

LGTM. The comments resolved.
Only impact amd code path.

1am9trash · 2026-10-27T01:33:18Z

PR-test-base passed.
Link: https://github.com/sgl-project/sglang/actions/runs/40173502184/job/121109327703?pr=41880

The seam is validated on MI355X (gfx950) only, and the default AMD CI suite runs on MI300. Gate it with is_gfx95_supported(), as sgl-project#41880 gates its FlyDSL path on gfx950. Move the test to the MI35x suite and skip it on other GPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ns on gfx950 (sgl-project#41880) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Thomas Wang <thomawan@amd.com>

amd-nprotaso and others added 4 commits September 30, 2026 08:08

add hc mix kernel

bb4f7d7

Apply isort and ruff-format to new FlyDSL hc_mix files

d1758f2

The lint job runs `pre-commit run --all-files`; isort and ruff-format failed on the files added/touched by this PR. Formatting only, no behavior change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

add env varable to use spllit combine hc kernel

d1e676e

amd-nprotaso requested review from BBuf, Edwardf0t1, Fridge003, HaiShaw, Ying1123, ch-wan, ispobock and merrymercy as code owners September 30, 2026 10:53

ChangLiu0709 mentioned this pull request Oct 14, 2026

[ROCm][AMD] Qwen3.8-Flash-Next gfx950 / MI355X Performance Optimization #44304

Open

fix hc kernels

7f54a2f

This comment has been minimized.

Sign in to view

amd-nprotaso changed the title ~~[AMD] AITER FlyDSL hc_mix for Qwen3.8 hyper-connections on gfx950~~ [Perf][AMD] Qwen3.8: AITER FlyDSL hc_mix for Qwen3.8 hyper-connections on gfx950 Oct 21, 2026

This comment has been minimized.

Sign in to view

amd-nprotaso changed the title ~~[Perf][AMD] Qwen3.8: AITER FlyDSL hc_mix for Qwen3.8 hyper-connections on gfx950~~ [Perf] [AMD] Qwen3.8: AITER FlyDSL hc_mix for Qwen3.8 hyper-connections on gfx950 Oct 21, 2026

amd-nprotaso added 3 commits October 21, 2026 11:19

remove env flags guard

d98023d

remove kernel benchmark

c24fb69

Merge branch 'main' into pr/hc-mix-flydsl

0bc1153

This comment has been minimized.

Sign in to view

amd-nprotaso added 2 commits October 21, 2026 12:08

Merge branch 'main' of https://github.com/sgl-project/sglang into pr/…

8d8a645

…hc-mix-flydsl # Conflicts: # python/sglang/srt/layers/hyperconnection.py # test/registered/kernels/ops/gemm/test_hc_mix_flydsl.py

Merge branch 'pr/hc-mix-flydsl' of https://github.com/amd-nprotaso/sg…

1c57400

…lang into pr/hc-mix-flydsl # Conflicts: # python/sglang/srt/layers/hyperconnection.py

This comment has been minimized.

Sign in to view

1am9trash added amd amd-aiter-not-ready labels Oct 22, 2026

Merge branch 'main' into pr/hc-mix-flydsl

b1cda98

amd-nprotaso mentioned this pull request Oct 22, 2026

[AITER-Upgrade] PR readiness #21302

Open

1am9trash added the run-ci CI: run the baseline test suite on this PR label Oct 22, 2026

Merge branch 'main' into pr/hc-mix-flydsl

25ac33a

1am9trash removed the amd-aiter-not-ready label Oct 23, 2026

1am9trash reviewed Oct 23, 2026

View reviewed changes

Comment thread python/sglang/srt/layers/hyperconnection.py Resolved

stay previous logic of hc_combine_split call

80fd440

This comment has been minimized.

Sign in to view

amd-nprotaso closed this Oct 23, 2026

amd-nprotaso reopened this Oct 23, 2026

This comment has been minimized.

Sign in to view

Merge branch 'main' into pr/hc-mix-flydsl

3e7a24d

This comment has been minimized.

Sign in to view

fix lint

c06b2f5

amd-nprotaso requested a review from 1am9trash October 26, 2026 08:40

ChangLiu0709 mentioned this pull request Oct 26, 2026

[AMD][MI35X] Qwen3.8-Flash-Next FP4 SGLang agentic MTP: 20261031 image, monokernel at c1-c8 / [AMD][MI35X] Qwen3.8-Flash-Next：20261031 镜像，c1-c8 启用 monokernel SemiAnalysisAI/InferenceX#4542

Merged

6 of 11 tasks

1am9trash approved these changes Oct 27, 2026

View reviewed changes

eppaneamd mentioned this pull request Oct 27, 2026

[Perf] [AMD] qwen4_exp: Fuse hyper-connection combine, norm and mix #46881

Draft

4 of 5 tasks

sogalin added the amd-aiter-not-ready label Oct 28, 2026

HaiShaw merged commit fcecdad into sgl-project:main Oct 30, 2026
212 of 224 checks passed

amd-cgilli mentioned this pull request Nov 5, 2026

[Perf] [AMD] Qwen4-Exp HC: fuse decode-sized hyper-connection norm and inject into FlyDSL mix #48648

Open

4 of 5 tasks

Navigation Menu

Repository navigation

[Perf] [AMD] Qwen3.8: AITER FlyDSL hc_mix for Qwen3.8 hyper-connections on gfx950 - #41880

[Perf] [AMD] Qwen3.8: AITER FlyDSL hc_mix for Qwen3.8 hyper-connections on gfx950#41880
HaiShaw merged 15 commits into
sgl-project:mainfrom
amd-nprotaso:pr/hc-mix-flydsl

amd-nprotaso commented Sep 30, 2026 •

edited by github-actions Bot

Loading

Uh oh!

This comment has been minimized.

amd-nprotaso commented Oct 21, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

1am9trash commented Oct 22, 2026 •

edited

Loading

Uh oh!

amd-nprotaso commented Oct 22, 2026

Uh oh!

1am9trash commented Oct 23, 2026

Uh oh!

1am9trash commented Oct 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

amd-nprotaso commented Oct 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

1am9trash left a comment

Uh oh!

1am9trash commented Oct 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

amd-nprotaso commented Sep 30, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

CI States

Uh oh!

This comment has been minimized.

amd-nprotaso commented Oct 21, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

1am9trash commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amd-nprotaso commented Oct 22, 2026

Uh oh!

1am9trash commented Oct 23, 2026

Uh oh!

1am9trash commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

amd-nprotaso commented Oct 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

1am9trash left a comment

Choose a reason for hiding this comment

Uh oh!

1am9trash commented Oct 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amd-nprotaso commented Sep 30, 2026 •

edited by github-actions Bot

Loading

1am9trash commented Oct 22, 2026 •

edited

Loading

1am9trash commented Oct 23, 2026 •

edited

Loading