[ROCm] Gate AITER MoE GateMode.INTERLEAVE on the gu-interleaved weight layout - #46664
Closed
Rohan138 wants to merge 1 commit into
Closed
[ROCm] Gate AITER MoE GateMode.INTERLEAVE on the gu-interleaved weight layout#46664Rohan138 wants to merge 1 commit into
Rohan138 wants to merge 1 commit into
Conversation
vllm-project#44893 passes GateMode.INTERLEAVE for all use_mxfp4_w4a16 models, but only gpt-oss shuffles weights into the gate-up interleaved layout. DeepSeek-V4 keeps the separated layout, so INTERLEAVE routed its small-M decode to an a16w4 path with no kernel ("Unsupported kernel config for moe heuristic dispatch"). Mark the separated path and gate INTERLEAVE on it so DeepSeek-V4 uses SEPARATED while gpt-oss is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Collaborator
Author
|
Closed in favor of #46122 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#44893 passes
GateMode.INTERLEAVEfor everyuse_mxfp4_w4a16model, but only gpt-oss shuffles its weights into the gate-up interleaved layout. DeepSeek-V4 keeps the separated layout, so INTERLEAVE routes small-M decode to an a16w4 path with no kernel →RuntimeError: Unsupported kernel config for moe heuristic dispatch. This marks the separated path and gates INTERLEAVE on it; gpt-oss is unchanged.Not a duplicate: #46122 takes the opposite approach (move DSV4 onto gu-interleave +
AITER_BF16_FP8_MOE_BOUND=0); this keeps DSV4 on the working SEPARATED path with no env-var workaround.Test:
pre-commit run --files <changed>passes. The resulting SEPARATED routing serves + decodes DeepSeek-V4-Pro (TP8, gfx950) — gsm8k 5-shot 0.945 — whereas the default INTERLEAVE crashes at startup with the dispatch error above.AI assistance (Claude) was used.