Repository navigation
Conversation
elvircrn
requested review from
mgoin,
pavanimajety and
zyongye
as code owners
September 3, 2026 07:20
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
mgoin
reviewed
Sep 3, 2026
| max_expanded_rows = ( | ||
| self.moe_config.max_num_tokens | ||
| * self.moe_config.dp_size | ||
| * self.moe_config.ep_size |
Member
There was a problem hiding this comment.
what about for TP with DP? maybe it should be max(DP, EP)?
Contributor
Author
There was a problem hiding this comment.
ep_size should still be correct, we just care about the total number of ranks participating in MoE.
Collaborator
There was a problem hiding this comment.
I'm ok with this change but I'm still wondering why it only applies to DeepEPV2 and not other dispatchers?
elvircrn
requested review from
AndreasKaratzas,
WoosukKwon,
tlrmchlsmth and
yewentao256
as code owners
September 13, 2026 18:13
elvircrn
force-pushed
the
humming-buffer-sizing
branch
2 times, most recently
from
September 13, 2026 20:22
d8582e0 to
bced643
Compare
Collaborator
|
Why is this only needed for DeepEPV2? |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Size DeepEP v2 buffers from sequence-parallel token capacity and use the same per-rank capacity for Humming permute scratch. Preserve the current shared scratch manager and allocate activation scratch for its runtime dtype. Upstream already handles grouped SITU activation through process_input, so keep that path while rebasing the capacity fix. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
elvircrn
force-pushed
the
humming-buffer-sizing
branch
from
October 7, 2026 14:32
bced643 to
4a1d672
Compare
6 tasks done
5 of 6 tasks
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix the Humming MoE scratch-buffer assertion with DeepEP v2, expert parallelism, and tensor sequence parallelism.
FusedMoEConfig.max_num_tokensis the scheduler capacity for one DP rank. DeepEP v2 previously treated that unsharded value as the capacity of every physical EP input rank. With TP sequence parallelism, each input rank instead owns at mostceil(max_num_tokens / sp_size)rows. The oversized DeepEP CUDA-graph receive tensor then exceeded Humming's DP-sized scratch buffer.This change establishes one CUDA-graph-stable capacity for both sides:
DeepEP v2 uses the per-rank value for handle initialization and runtime decode buckets. Humming uses the corresponding post-dispatch value. For DP=4, TP/SP=8, EP=32, and
max_num_tokens=128, both now agree on 512 rows instead of allocating 4096 rows in DeepEP and 512 in Humming.This is separate from #56079: that PR invalidates sequence-parallel padding during routing, but does not change DeepEP's receive allocation or Humming's scratch-capacity validation.
Duplicate-work check
This updates the existing PR for this assertion. A search for open DeepEP v2/Humming scratch-buffer fixes found no duplicate PR.
Test plan and results
No model evaluation was run because this changes buffer capacity, not routing results or model numerics.
AI assistance was used to analyze the capacity mismatch, implement the change, and add tests. The submitter must review every changed line and the test output before merge.
Signed-off-by: Elvir Crncevic elvircrn@gmail.com