[DeepSeek V2] Reorder dual-stream MoE to main-first to avoid CUDA graph stream explosion by kpham-sgl · Pull Request #30460 · sgl-project/sglang

kpham-sgl · 2026-07-08T02:30:14Z

Motivation

Profiling Kimi-K2.5-NVFP4 (TP8, EAGLE3 spec decode, flashinfer_trtllm MoE) showed the target-verify decode CUDA graph fanning out across ~61 streams — one per model layer — instead of the intended 2 (main + alt).

Root cause is in DeepseekV2MoE.forward_normal_dual_stream: the shared-expert branch was enqueued on the alt stream before the main (routed) branch. During CUDA graph capture this alt-first, per-layer ordering makes cudaGraphInstantiate allocate a fresh side stream for every layer — the same "stream explosion" mechanism fixed for the DSA indexer in #30025.

Alt-first was originally required by #29463: the routed deep_gemm pre-permute calls dispose_tensor(hidden_states), which set_()s the storage to empty. A shared-expert kernel reading hidden_states afterward would capture data_ptr() == 0 into the decode graph and replay from null.

Change

Issue the main (routed) branch first, then the shared expert on the alt stream. To keep deep_gemm correct, take a storage alias shared_in = hidden_states[:] before the routed call runs dispose_tensor, so the shared expert reads live storage. This satisfies all three constraints simultaneously:

No stream explosion — main issued before the alt block, so capture reuses a single alt stream.
PDL overlap preserved — routed is the last main-stream kernel and still fuses with the residual add.
deep_gemm dispose_tensor hazard avoided — shared reads shared_in, a reference taken pre-dispose.

The alias keeps the buffer alive past dispose_tensor, but decode/verify hidden_states is tiny (bs * num_draft_tokens rows vs thousands in prefill), so the lost early free is negligible.

Since shared_output is now computed after the deferred-finalize decision, that decision is gated on a precomputed has_shared_output flag instead of shared_output is not None.

Results

Re-profiled with the same TP8 Kimi-K2.5 config + client. Per full TARGET_VERIFY forward, TP-0:

	before (alt-first)	after (main-first + alias)
streams per verify forward	61-62	2-3
stream-count histogram	`{2:78, 3:4462, 4:140, 61:63, 62:15}`	`{2:124, 3:32}`
total distinct kernel streams in trace	~61	3

Perf/correctness unchanged (within noise), confirming the alias preserves shared-expert input:

metric	before	after
Accept length	2.42	2.40
Mean TPOT	3.33 ms	3.30 ms
Benchmark duration	42.10 s	41.53 s

Test plan

Profile Kimi-K2.5-NVFP4 TP8 (EAGLE3) — verify stream count collapses 61 -> 2-3, accept length unchanged.
deep_gemm EP gsm8k (test_moe_ep_extra.py) to validate the alias against the [DeepSeek V3] Reland: run routed experts on main stream in dual-stream MoE #29463 path.

Refs #29463, #30025.

Made with Cursor

Profile

Before

After

CI States

Latest PR Test (Base): 🚫 Run #29049292993
Latest PR Test (Extra): ❌ Run #29049292883

gemini-code-assist · 2026-07-08T02:30:17Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

kpham-sgl · 2026-07-08T02:34:36Z

/rerun-test test/registered/models_e2e/test_deepseek_v3_fp4.py test/registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py test/registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py test_moe_ep_extra.py

github-actions · 2026-07-08T02:34:56Z

Results for /rerun-test test/registered/models_e2e/test_deepseek_v3_fp4.py test/registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py test/registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py test_moe_ep_extra.py:

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v3_fp4.py
cd test/ && python3 registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/moe/test_moe_ep_extra.py

⛔ test/registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py: File not found: test/registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py

kpham-sgl · 2026-07-08T02:52:51Z

/rerun-test test/registered/quant/test_kimi_k25_nvfp4_eagle.py

github-actions · 2026-07-08T02:53:12Z

Results for /rerun-test test/registered/quant/test_kimi_k25_nvfp4_eagle.py:

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/quant/test_kimi_k25_nvfp4_eagle.py

…ph stream explosion DeepseekV2MoE.forward_normal_dual_stream enqueued the shared-expert branch on the alt stream *before* the main (routed) branch. During decode/target- verify CUDA graph capture this alt-first ordering makes cudaGraphInstantiate allocate a fresh side stream per layer, exploding to ~61 streams for Kimi K2.5 (one per layer) -- the same mechanism fixed for the DSA indexer in #30025. Alt-first was originally required (#29463) because the routed deep_gemm pre-permute calls dispose_tensor(hidden_states), which set_()s the storage to empty; a later shared-expert kernel reading hidden_states would then capture data_ptr()==0 into the graph and replay from null. Fix: issue the routed branch first, then the shared expert on the alt stream, and take a storage alias `shared_in = hidden_states[:]` before the routed call so the shared expert reads live storage past dispose_tensor. This satisfies all three constraints at once: no stream explosion (capture reuses one alt stream), PDL overlap preserved (routed is the last main-stream kernel, fuses with the residual add), and the deep_gemm dispose_tensor hazard is avoided. Since shared_output is now computed after the deferred-finalize decision, gate that decision on a precomputed has_shared_output flag instead. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Follow-up refactor of #30460. Instead of keeping hidden_states alive past the routed deep_gemm's dispose_tensor set_() via a `shared_in = hidden_states[:]` alias, add CaptureFlags.disable_dispose_tensor: model_capture_mode() sets it for the duration of decode/spec graph capture, and dispose_tensor() is a no-op while set. So the routed deep_gemm no longer frees hidden_states mid-capture and the dual-stream shared expert reads it directly. Same effect, no aliasing trick. Keeps #30460's main-first ordering (no CUDA-graph stream explosion). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

kpham-sgl · 2026-07-09T04:16:43Z

/rerun-test test/registered/models_e2e/test_deepseek_v3_fp4.py test/registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py test_moe_ep_extra.py test/registered/quant/test_kimi_k25_nvfp4_eagle.py

github-actions · 2026-07-09T04:17:06Z

Results for /rerun-test test/registered/models_e2e/test_deepseek_v3_fp4.py test/registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py test_moe_ep_extra.py test/registered/quant/test_kimi_k25_nvfp4_eagle.py:

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v3_fp4.py
cd test/ && python3 registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py
cd test/ && python3 registered/quant/test_kimi_k25_nvfp4_eagle.py

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/moe/test_moe_ep_extra.py

kpham-sgl · 2026-07-09T05:11:44Z

/rerun-failed-ci

…first-dual-stream # Conflicts: # python/sglang/srt/models/deepseek_v2.py

…E to main-first to avoid CUDA graph stream explosion (#30460) (#30714) Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Jiminator <jimmysh341@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…E to main-first to avoid CUDA graph stream explosion (sgl-project#30460) (sgl-project#30714) Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Jiminator <jimmysh341@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ph stream explosion (sgl-project#30460) Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Jiminator <jimmysh341@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

kpham-sgl requested review from Fridge003, ch-wan, fzyzcjy, ispobock and merrymercy as code owners July 8, 2026 02:30

github-actions Bot added the deepseek label Jul 8, 2026

kpham-sgl and others added 2 commits July 9, 2026 02:40

Condense shared_in alias comment

0f4d6ea

Co-authored-by: Cursor <cursoragent@cursor.com>

Jiminator force-pushed the kp/deepseek-moe-main-first-dual-stream branch from b765c9e to bf43480 Compare July 9, 2026 03:42

Jiminator force-pushed the kp/deepseek-moe-main-first-dual-stream branch from bf43480 to 4aa6ec3 Compare July 9, 2026 03:57

Jiminator requested review from Ying1123 and hnyls2002 as code owners July 9, 2026 03:57

Jiminator force-pushed the kp/deepseek-moe-main-first-dual-stream branch from 4aa6ec3 to 727d67f Compare July 9, 2026 03:59

Jiminator force-pushed the kp/deepseek-moe-main-first-dual-stream branch from 727d67f to 29007eb Compare July 9, 2026 04:01

ch-wan approved these changes Jul 9, 2026

View reviewed changes

kpham-sgl added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) labels Jul 9, 2026

kpham-sgl added the bypass-fastfail label Jul 9, 2026

Merge remote-tracking branch 'origin/main' into kp/deepseek-moe-main-…

7af0291

…first-dual-stream # Conflicts: # python/sglang/srt/models/deepseek_v2.py

kpham-sgl merged commit 87992ee into main Jul 9, 2026
130 of 167 checks passed

kpham-sgl deleted the kp/deepseek-moe-main-first-dual-stream branch July 9, 2026 23:59

kpham-sgl mentioned this pull request Jul 10, 2026

[Cherry-pick to release/v0.5.15] [DeepSeek V2] Reorder dual-stream MoE to main-first to avoid CUDA graph stream explosion (#30460) #30714

Merged

Navigation Menu

[DeepSeek V2] Reorder dual-stream MoE to main-first to avoid CUDA graph stream explosion - #30460

[DeepSeek V2] Reorder dual-stream MoE to main-first to avoid CUDA graph stream explosion#30460
kpham-sgl merged 4 commits into
mainfrom
kp/deepseek-moe-main-first-dual-stream

kpham-sgl commented Jul 8, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot commented Jul 8, 2026

Uh oh!

kpham-sgl commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

kpham-sgl commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

kpham-sgl commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

kpham-sgl commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kpham-sgl commented Jul 8, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Change

Results

Test plan

Profile

CI States

Uh oh!

gemini-code-assist Bot commented Jul 8, 2026

Uh oh!

kpham-sgl commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kpham-sgl commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kpham-sgl commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kpham-sgl commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kpham-sgl commented Jul 8, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading