add NVSHMEM_DISABLE_NCCL to avoid cuda graph hang - #1374
Conversation
DeepEP initializes NVSHMEM, whose built-in NCCL path opens a second NCCL communicator that collides with our own NCCL and hangs during SGLang CUDA-graph replay. Set NVSHMEM_DISABLE_NCCL=1 centrally on the rollout (server_group) and training (actor_group) actor env_vars so it covers all scripts; overridable via the env var.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request configures the environment variable NVSHMEM_DISABLE_NCCL to default to "1" for both training actors and rollout engines. This prevents DeepEP/NVSHMEM's internal NCCL from conflicting with the main NCCL communicator, which can cause hangs during SGLang CUDA-graph replay. Documentation explaining this change and how to opt out has also been added to docs/platforms/nvidia.md. No review comments were provided, so there is no feedback to address.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
… cuda-graph hang) Migrated from radixark#1374 (THUDM/slime#2093). DeepEP/NVSHMEM's internal NCCL opens a second NCCL communicator that collides with miles' NCCL and hangs during SGLang CUDA-graph replay. Applied to test whether it also resolves the GLM-5.2 dp-attention EVAL-ON deadlock (skeptical: that hang is in CPU/gloo collectives + GLM-5 LoRA uses triton MoE not DeepEP, so NVSHMEM may be inert here -- but empirical test pending a devbox). If it does not fix it, fall back to the miles-side eval grouped-submission fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MuN5ow5hHFxPze36uhgZC4
…of PR radixark#1374) PR radixark#1374 only touches nvidia.md + actor_group.py + server_group.py. The windowing changes to miles/rollout/sglang_rollout.py + miles/utils/arguments.py got staged into 10a26f9 by an earlier 'git revert -n d4f0431' and are NOT part of the cherry-pick. Windowing was empirically NOT the fix (it still hung); the real fix is NVSHMEM_DISABLE_NCCL=1 (kept). Restore these 2 files to their pre-windowing state (d4f0431) so the branch carries only the actual fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MuN5ow5hHFxPze36uhgZC4
2 participants
Migrated from THUDM/slime#2093.
Thanks @zhuzilin!