Repository navigation
[DeepSeek V4] Fix fused MoE expert distribution - #57465
tlrmchlsmth merged 4 commits into
Conversation
DeepSeek V4's fused MoE path incorrectly used the TP size and rank to calculate expert distribution and the number of physical experts on each GPU. These values are determined by the EP group. Use the EP size and rank instead, matching the MegaMoE path and other MoE models. Signed-off-by: Itay Alroy <ialroy@nvidia.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
/ci run |
|
✅ Triggered Buildkite CI #89757 for commit |
|
✅ @itayalroy, CI is now available for this PR.
|
| assert self.n_physical_experts % ep_size == 0, ( | ||
| f"n_physical_experts={self.n_physical_experts} must be divisible by " | ||
| f"tp_size={self.tp_size}. Adjust num_redundant_experts." | ||
| f"ep_size={ep_size}. Adjust num_redundant_experts." |
There was a problem hiding this comment.
this is actually too strict - we don't require divisibility for ep_size... at least not in all cases
There was a problem hiding this comment.
I checked a few other FusedMoE models (including DSV2/V3). They also calculate the number of local physical experts as num_physical_experts // ep_size, they just do not assert divisibility. I aligned DeepSeek V4 with them. Currently this is only read by EPLB, but I wonder if we should make a broader change to account for uneven distribution across all models.
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Head branch was pushed to by a user without write access
|
/ci run |
|
✅ Triggered Buildkite CI #89852 for commit |
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #89852. |
2 participants
DeepSeek V4's fused MoE path incorrectly used the TP size and rank to calculate expert distribution and the number of physical experts on each GPU. These values should be determined by the EP group.
Use the EP size and rank instead, matching the MegaMoE path and other MoE models.
cc @zyongye @tlrmchlsmth