Skip to content

moe: mask padded rows before capturing routed experts for R3 - #36404

Merged
yueming-yuan merged 1 commit into
sglang-miles-v0.5.18from
yueming/r3-capture-masks-padded-rows
Aug 26, 2026
Merged

yueming-yuan merged 1 commit into
sglang-miles-v0.5.18from
yueming/r3-capture-masks-padded-rows

Conversation

@yueming-yuan

@yueming-yuan yueming-yuan commented Aug 26, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The R3 routed-experts capture runs at the top of _post_process_topk_ids, ahead
of every padded-region mask in that function. Rows at or past
num_token_non_padded therefore reach the capturer holding whatever the router
left in them, and the capturer writes buffer[:batch] without clearing, so those
rows carry a stale routing row from an earlier forward.

Replayed into training, a padded region comes back as one repeated row of real
expert ids. The R3 replay check counts each as a zero-overlap mismatch.

Evidence

From a miles CI run (DeepSeek-V3.2-5layer, 8192-token batch,
chunked_prefill_size=4096):

  • one replay row repeated 67 times across contiguous tokens 4039-4053+, while
    the recomputed routing varied per token
  • 104 mismatches against a threshold of 82
  • the block starts at the chunked-prefill boundary

Same check on the pre-bump build: 171 replay rows, all 171 unique, zero
repeats, and at most 9 mismatches per check.

Why not just move the capture after the mask

The masks are per-branch and disagree on fill value: the CUDA path fills -1, the
ROCm path fills 0 -- a valid expert id that replay cannot tell from a real
selection. Masking a copy at the capture site gives replay -1 on both, and leaves
the tensor the MoE kernels consume untouched.

The NPU call site is left alone: _mask_topk_ids_padded_region is a no-op there,
so passing the count would only add a clone.


CI States

Latest PR Test (Base): ❌ Run #32918538859
Latest PR Test (Extra): ❌ Run #32918538639
Latest PR Test (AMD ROCm 7.2): ❌ Run #32918538816

The routed-experts capture runs at the top of _post_process_topk_ids, ahead of
every padded-region mask in that function. Rows at or past num_token_non_padded
therefore reach the capturer holding whatever the router left in them, and the
capturer writes buffer[:batch] without clearing, so those rows carry a stale
routing row from an earlier forward. Replayed into training, a padded region
comes back as one repeated row of real expert ids, which the R3 replay check
counts as zero-overlap mismatches -- 104 of them on an 8192-token batch against
a threshold of 82, all from a contiguous block at the chunked-prefill boundary.

Masking after the fact is not enough: the CUDA path masks to -1 only further
down, and the ROCm path masks to 0, a valid expert id that replay cannot tell
from a real selection. Mask a copy at the capture site instead, so the recorded
rows are -1 and the replay check skips them, leaving the tensor the MoE kernels
consume untouched.

The NPU call site is left alone: _mask_topk_ids_padded_region is a no-op there,
so passing the count would only add a clone.
@yueming-yuan
yueming-yuan merged commit 1189271 into sglang-miles-v0.5.18 Aug 26, 2026
81 of 90 checks passed
@yueming-yuan
yueming-yuan deleted the yueming/r3-capture-masks-padded-rows branch August 26, 2026 01:49
Kh4L pushed a commit to Kh4L/sglang that referenced this pull request Sep 8, 2026
Zhichenzzz pushed a commit that referenced this pull request Sep 19, 2026
HJSang pushed a commit to HJSang/sglang that referenced this pull request Sep 21, 2026
ziang-and pushed a commit to zianglih/sglang that referenced this pull request Oct 2, 2026
nanjiangwill added a commit to modal-projects/sglang that referenced this pull request Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant