Repository navigation
[Perf] [AMD] Qwen3-Next: fused QK Gemma RMSNorm + partial RoPE on gfx950 - #45739
Merged
HaiShaw merged 2 commits intoOct 27, 2026
Merged
Conversation
karverma-amd
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
October 22, 2026 04:59
This comment has been minimized.
This comment has been minimized.
karverma-amd
force-pushed
the
karverma/qwen3-next-fused-prologue
branch
from
October 22, 2026 18:41
5aff2db to
9062146
Compare
This comment has been minimized.
This comment has been minimized.
karverma-amd
changed the title
karverma-amd
force-pushed
the
karverma/qwen3-next-fused-prologue
branch
from
October 22, 2026 18:54
9062146 to
bb16153
Compare
Contributor
Author
|
Aiter PR - ROCm/aiter#6835 |
Rebased onto current main to resolve the merge conflict. The stacked smallm_fp8_blockscale_gfx950 import fix (sgl-project#45738) is now redundant -- main already imports the renamed hipcc helpers -- so this drops to the fused prologue wiring in qwen3_next.py alone. Verified get_bool_env_var / is_gfx95_supported and the model attrs the wiring uses still exist on main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
karverma-amd
force-pushed
the
karverma/qwen3-next-fused-prologue
branch
from
October 26, 2026 17:29
bb16153 to
8f7bae0
Compare
HaiShaw
approved these changes
Oct 26, 2026
HaiShaw
left a comment
HaiShaw
left a comment
Collaborator
There was a problem hiding this comment.
HIP/AITER specific optimization
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Route the Qwen3-Next full-attention prologue — the per-head
[Q||gate]split, Q/K Gemma RMSNorm (x*(1+w)), and partial NeoX RoPE (rotary_dim 64) — through the fused aiter opfused_attention_qk_norm_ropeon gfx950, replacing three small launches with one.Gating
SGLANG_USE_AITER+is_hip()— no bespoke per-kernel flag.fused_attention_qk_norm_rope_supported), not asserted, so any uncovered shape (dtype/width/arch) falls back to the native split + GemmaRMSNorm + RoPE. The attention backend still writes the KV cache.Dependencies
fused_attention_qk_norm_ropein ROCm/aiter — https://github.com/ROCm/aiter/compare/main...karverma-amd:aiter:karverma/qwen3-next-fused-prologue?expand=1smallm_fp8_blockscale_gfx950import fix), which Qwen3-Next FP8 needs to serve on gfx950. The first commit in this PR is that fix; the second commit is the wiring under review here.Test results
E2E on Qwen3-Next-80B-A3B-Instruct-FP8, MI355X 2×TP4, latest nightly (
v0.5.22-rocm724-mi35x-20261021) with Triton 3.8.0:CI States
Latest PR Test (Base): ✅ Run #40387799828
Latest PR Test (Extra): ❌ Run #40387799898
Latest PR Test (AMD ROCm 10): ⏳ Run #40387799746