[Deepseek V4] Keep fp32 routing weights in the mxfp4 trtllm MoE - #33608
Merged
Merged
Conversation
b8zhong
force-pushed
the
mxfp4-unpacked-routing
branch
from
August 4, 2026 22:30
7544842 to
187cacd
Compare
b8zhong
force-pushed
the
mxfp4-unpacked-routing
branch
from
August 4, 2026 22:30
187cacd to
9610835
Compare
b8zhong
changed the title
This was referenced Aug 5, 2026
b8zhong
marked this pull request as ready for review
September 4, 2026 08:20
b8zhong
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
HaiShaw,
OrangeRedeng,
ch-wan and
mmangkad
as code owners
September 4, 2026 08:20
mmangkad
approved these changes
Sep 5, 2026
b8zhong
enabled auto-merge (squash)
September 5, 2026 12:02
hnyls2002
added a commit
that referenced
this pull request
Sep 13, 2026
The unpacked routing input introduced by #33608 supplies genuine FP32 weights, but the branch-specific deferred finalize adapter reinterprets FP32 buffers as BF16. CUDA graph decoding then produces repeated BOS tokens. Restore the packed routed-MoE input so deferred finalize receives BF16 weights with the expected layout. Validated on 4 GB300 GPUs with TP4, EP4, CUDA graph enabled, and no DSpark. GPQA Diamond: 178/198 correct, 2 truncated, 0 request errors. Chat and thinking probes both terminate normally.
mqhc2020
pushed a commit
to mqhc2020/sglang
that referenced
this pull request
Sep 15, 2026
…project#33608) Co-authored-by: Brayden Zhong <brayden@radixark.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The packed
topk_idsformat is(expert_id << 16) | bf16(weight), so it truncates the fp32 routing weights thatmoe_fused_gateproduces. Pass the ids and weights unpacked instead, the kernel then keeps them in fp32, and thePackTopkIdslaunch goes away.The reference DeepSeek-V4 implementation keeps the routing weights in fp32 (
inference/model.py,Gate.forward).Requires flashinfer 0.6.18 (flashinfer-ai/flashinfer#3763)
For BS = 1 under MTP, 771.97 -> 780.27 TPS
As the topK IDs can be consumed without packing the expert IDs and topK expert weights, this kernel disappears. The MoE finalize use
finalizeKernel<KernelParams<bfloat16_t, bfloat16_t, 2, true>>->finalizeKernel<KernelParams<bfloat16_t, float, 2, true>>Before
After
CI States
Latest PR Test (Base): ✅ Run #34046732801
Latest PR Test (Extra): ❌ Run #34046732525
Latest PR Test (AMD ROCm 7.2): ❌ Run #34046732635