Skip to content

[Spec] Enable draft extend cuda graph for DeepSeek-V4 attention backend - #30853

Merged
hnyls2002 merged 1 commit into
mainfrom
lsyin/eagle-dsv4-draft-extend-graph
Jul 11, 2026
Merged

hnyls2002 merged 1 commit into
mainfrom
lsyin/eagle-dsv4-draft-extend-graph

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

DeepseekV4AttnBackend already implements the DRAFT_EXTEND graph bucket (capture + replay); register it in the EAGLE draft worker's supported list so draft extend runs under cuda graph instead of eager.

Verification: test_deepseek_v4_flash_fp8_h200.py (TP=4 + EAGLE) exercises this path -- graph capture happens at server startup, so the test passing covers capture and replay.


CI States

Latest PR Test (Base): ❌ Run #29147214733
Latest PR Test (Extra): ❌ Run #29147214595

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_deepseek_v4_flash_fp8_h200.py

@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_deepseek_v4_flash_fp8_h200.py:

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp8_h200.py

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates eagle_worker_v2.py to import and add DeepseekV4AttnBackend to the list of supported backend types for CUDA graph capture. I have no feedback to provide as there are no review comments and the changes are straightforward.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_deepseek_v4_flash_fp4_b200.py test_deepseek_v4_flash_fp4_h200.py test_deepseek_v4_flash_fp4_megamoe_b200.py

@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_deepseek_v4_flash_fp4_b200.py test_deepseek_v4_flash_fp4_h200.py test_deepseek_v4_flash_fp4_megamoe_b200.py:

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py
cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py

@hnyls2002
hnyls2002 merged commit 7bac9c8 into main Jul 11, 2026
109 of 125 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/eagle-dsv4-draft-extend-graph branch July 11, 2026 09:29
@merrymercy

Copy link
Copy Markdown
Contributor

FYI, this change appears to make test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py fail on B200 because the newly enabled DeepSeek-V4 draft-extend CUDA graph consumes substantially more memory.

Comparison:

The effective server args for the comparable DeepSeek-V4 + DeepEP + CP server are essentially the same. Both runs also disable prefill CUDA graph for DeepSeek-V4, so this is not from prefill BCG:

Breakable CUDA graph is incompatible with DeepSeek-V4 ... disabling prefill CUDA graph.

The key log difference is that the failing run, which includes this commit, captures draft-extend CUDA graph:

Capture draft extend CUDA graph begin ... avail mem=8.84 GB
Capture draft extend CUDA graph end ... mem usage=4.50 GB, avail mem=4.34 GB
max_total_num_tokens=11328768 ... available_gpu_mem=4.34 GB

The passing comparison run does not show Capture draft extend CUDA graph for the comparable DeepEP server and starts serving with about available_gpu_mem=8.50 GB.

The failing run then OOMs during EAGLE draft extend in deepseek_v4_nextn.py / hc_head, trying to allocate around 956 MiB with only around 775-789 MiB free on some ranks. The OOM report also shows about 8.04 GiB in private CUDA graph pools.

Based on the code diff, the direct trigger is adding DeepseekV4AttnBackend to graph_supported_backend_types in eagle_worker_v2.py, which makes this test enter the existing EAGLEDraftExtendCudaGraphRunner capture path. Could you take a look at either reducing this graph memory footprint or disabling draft-extend graph for this DeepSeek-V4 + DeepEP + attn-CP configuration?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants