Repository navigation
moe: mask padded rows before capturing routed experts for R3 - #36404
Merged
yueming-yuan merged 1 commit intoAug 26, 2026
Merged
Conversation
The routed-experts capture runs at the top of _post_process_topk_ids, ahead of every padded-region mask in that function. Rows at or past num_token_non_padded therefore reach the capturer holding whatever the router left in them, and the capturer writes buffer[:batch] without clearing, so those rows carry a stale routing row from an earlier forward. Replayed into training, a padded region comes back as one repeated row of real expert ids, which the R3 replay check counts as zero-overlap mismatches -- 104 of them on an 8192-token batch against a threshold of 82, all from a contiguous block at the chunked-prefill boundary. Masking after the fact is not enough: the CUDA path masks to -1 only further down, and the ROCm path masks to 0, a valid expert id that replay cannot tell from a real selection. Mask a copy at the capture site instead, so the recorded rows are -1 and the replay check skips them, leaving the tensor the MoE kernels consume untouched. The NPU call site is left alone: _mask_topk_ids_padded_region is a no-op there, so passing the count would only add a clone.
yueming-yuan
requested review from
BBuf,
Edwardf0t1,
Fridge003,
HaiShaw,
Ying1123,
ch-wan,
ispobock and
merrymercy
as code owners
August 26, 2026 01:19
Kh4L
pushed a commit
to Kh4L/sglang
that referenced
this pull request
Sep 8, 2026
…ject#36404) (cherry picked from commit 1189271)
8 of 33 tasks
Zhichenzzz
pushed a commit
that referenced
this pull request
Sep 19, 2026
HJSang
pushed a commit
to HJSang/sglang
that referenced
this pull request
Sep 21, 2026
…ject#36404) (cherry picked from commit 1189271)
ziang-and
pushed a commit
to zianglih/sglang
that referenced
this pull request
Oct 2, 2026
…l-project#18642, sgl-project#36404) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
nv-hwoo
pushed a commit
to nv-hwoo/sglang
that referenced
this pull request
Oct 20, 2026
Rockdu
pushed a commit
that referenced
this pull request
Oct 30, 2026
nanjiangwill
added a commit
to modal-projects/sglang
that referenced
this pull request
Oct 30, 2026
…l-project#18642, sgl-project#36404) (cherry picked from commit d9f62b9) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VDwTQn4gGFgVhzGXFs4aJx
Rockdu
pushed a commit
that referenced
this pull request
Oct 30, 2026
Rockdu
pushed a commit
that referenced
this pull request
Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The R3 routed-experts capture runs at the top of
_post_process_topk_ids, aheadof every padded-region mask in that function. Rows at or past
num_token_non_paddedtherefore reach the capturer holding whatever the routerleft in them, and the capturer writes
buffer[:batch]without clearing, so thoserows carry a stale routing row from an earlier forward.
Replayed into training, a padded region comes back as one repeated row of real
expert ids. The R3 replay check counts each as a zero-overlap mismatch.
Evidence
From a miles CI run (DeepSeek-V3.2-5layer, 8192-token batch,
chunked_prefill_size=4096):the recomputed routing varied per token
Same check on the pre-bump build: 171 replay rows, all 171 unique, zero
repeats, and at most 9 mismatches per check.
Why not just move the capture after the mask
The masks are per-branch and disagree on fill value: the CUDA path fills -1, the
ROCm path fills 0 -- a valid expert id that replay cannot tell from a real
selection. Masking a copy at the capture site gives replay -1 on both, and leaves
the tensor the MoE kernels consume untouched.
The NPU call site is left alone:
_mask_topk_ids_padded_regionis a no-op there,so passing the count would only add a clone.
CI States
Latest PR Test (Base): ❌ Run #32918538859
Latest PR Test (Extra): ❌ Run #32918538639
Latest PR Test (AMD ROCm 7.2): ❌ Run #32918538816