Skip to content

[DeepEP] Reduce routed scaling overhead #5277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 13, 2025
Merged

[DeepEP] Reduce routed scaling overhead #5277

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2a4fc7e
[DeepEP] Reduce routed scaling overhead
yuleil Apr 10, 2025
ca00065
Revert "[DeepEP] Reduce routed scaling overhead"
yuleil Apr 13, 2025
8991697
[DeepEP] Reduce routed scaling overhead
yuleil Apr 13, 2025
3dcd9fc
Merge branch 'main' into masked_scale
ch-wan Apr 13, 2025
757443e
use inplace mul
ch-wan Apr 13, 2025
f791c81
Merge branch 'main' into masked_scale
ch-wan Apr 13, 2025
2cc8994
Merge branch 'main' into masked_scale
zhyncs Apr 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 9 additions & 10 deletions python/sglang/srt/models/deepseek_v2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,16 +337,13 @@ def forward_deepep(
topk_weights,
forward_mode=forward_mode,
)
final_hidden_states = (
self.experts(
hidden_states=hidden_states,
reorder_topk_ids=reorder_topk_ids,
seg_indptr=seg_indptr,
masked_m=masked_m,
expected_m=expected_m,
forward_mode=forward_mode,
)
* self.routed_scaling_factor
final_hidden_states = self.experts(
hidden_states=hidden_states,
reorder_topk_ids=reorder_topk_ids,
seg_indptr=seg_indptr,
masked_m=masked_m,
expected_m=expected_m,
forward_mode=forward_mode,
)
if self.ep_size > 1:
final_hidden_states = self.deepep_dispatcher.combine(
Expand All @@ -355,6 +352,8 @@ def forward_deepep(
topk_weights,
forward_mode,
)
final_hidden_states *= self.routed_scaling_factor

if shared_output is not None:
final_hidden_states = final_hidden_states + shared_output

Expand Down
Loading