Conversation
ep_scatter_from_psum calls the Triton kernel _fwd_kernel_ep_scatter_2 without the two positional arguments expert_start and num_experts, so Triton's dynamic_func() raises "missing 2 required positional arguments" on the DeepEP v2 contiguous/extend (prefill) path. Pass expert_start=0 (the psum recv layout already carries local expert ids) and num_experts (already computed locally), mirroring the ep_scatter call.
cynton503
requested review from
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
September 14, 2026 09:22
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #39402:ep_scatter_from_psum calls the Triton kernel _fwd_kernel_ep_scatter_2 without the two positional arguments expert_start and num_experts, so Triton's dynamic_func() raises "missing 2 required positional arguments" on the DeepEP v2 contiguous/extend (prefill) path. Pass expert_start=0 (the psum recv layout already carries local expert ids) and num_experts (already computed locally), mirroring the ep_scatter call.
Motivation
ep_scatter_from_psumcalls the Triton kernel_fwd_kernel_ep_scatter_2without the two positional argumentsexpert_startandnum_experts. The kernel signature declares them as positional params (immediately beforetopk_num: tl.constexpr):The other caller,
ep_scatter, passes both. Butep_scatter_from_psumjumps straight fromoutput_index.stride(1)to thetopk_num=keyword, so Triton'sdynamic_func()fails at argument binding with:This path is reached on the DeepEP v2 contiguous/extend (prefill) route via
pre_permute_deepep_v2_to_deep_gemm(layers/moe/moe_runner/deep_gemm.py), and is required whenever--moe-a2a-backend deepep_v2runs an FP8 blockwise-quantized model (the psum path). The masked/decode path goes throughexpand_to_masked_slabinstead,so it never hits this call — which is why the bug only surfaces on the first extend/prefill forward.
Modifications
In
ep_scatter_from_psum, add the two missing positional args to the kernel call, mirroringep_scatter:expert_start=0: the psum input layout is already organized by local expert id, so there is no global-to-local id shift (the kernel computesexpert_id = global_expert_id - expert_start). This matchesep_scatter's defaultexpert_start=0.num_experts: reuse the local variable already computed at the top of the function (num_experts = psum_num_recv_tokens_per_expert.shape[0]), no new computation needed.Accuracy Tests
Pure call-site fix — restores the missing kernel args so the kernel runs at all; it does not change the kernel's math. No accuracy regression expected. (Verified manually that
health_generatereturns HTTP 200 on a DeepSeek-R1 FP8 +deepep_v2TP16/EP16 run after the fix; before the fix every rank raised the aboveTypeErroron the first extend forward.)No unit test is added: this is a pure Triton kernel call-site fix, and the only CPU-side observable is the kernel-launch argument binding, which cannot run on CPU (a mock-based assertion would be meaningless).
Speed Tests and Profiling
No speed impact: this only unbricks a previously crashing code path. No benchmark change expected.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #34827597020
Latest PR Test (Extra): ❌ Run #34827596722
Latest PR Test (AMD ROCm 10): ❌ Run #34827596916