Skip to content

[WideEP] Update NCCL to 2.30.7 to enable DeepEPv2 in the vllm/vllm-openai image - #45321

Merged
vllm-bot merged 49 commits into
mainfrom
fix/deepep-commit-hash
Jul 24, 2026
Merged

[WideEP] Update NCCL to 2.30.7 to enable DeepEPv2 in the vllm/vllm-openai image#45321
vllm-bot merged 49 commits into
mainfrom
fix/deepep-commit-hash

Conversation

@tlrmchlsmth

@tlrmchlsmth tlrmchlsmth commented Jun 11, 2026

Copy link
Copy Markdown
Member

Update DeepEP to deepseek-ai/DeepEP@d4f41e4 so that we can pick up DeepEPv2.

Missed this in #41183

Ideally we'd have a single source of truth for the commit across both the install script and the Dockerfile but I don't see a sane way to do it at the moment

PyTorch ships an earlier version of NCCL. Per feedback from the NCCL team, I think this should be safe as

libnccl.so is designed to be backward compatible. This means you can compile your program with an older version (e.g., v1.28) and run it with a newer version at runtime (e.g., v1.30). However, the actual runtime environment should only use one version of NCCL at a time.

For the pypi wheel, no change is needed since we don't ship DeepEP in the wheel.

@mergify mergify Bot added the ci/build label Jun 11, 2026
@tlrmchlsmth tlrmchlsmth added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 11, 2026
@tlrmchlsmth
tlrmchlsmth marked this pull request as draft June 11, 2026 21:42
@tlrmchlsmth

tlrmchlsmth commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

Actually an issue here -- we need to update NCCL to > 2.30.4 during DeepEP compilation as well, not just rely on installing it afterwards like I thought we could

@tlrmchlsmth
tlrmchlsmth force-pushed the fix/deepep-commit-hash branch 2 times, most recently from a8cb6bd to 49fbfce Compare June 12, 2026 18:30
@tlrmchlsmth
tlrmchlsmth marked this pull request as ready for review June 12, 2026 19:03
@tlrmchlsmth
tlrmchlsmth marked this pull request as draft June 16, 2026 19:49
DeepEPv2 (ElasticBuffer API) requires NCCL >= 2.30.4 at both compile
and runtime. NCCL backward compatibility is guaranteed by NVIDIA, so
installing a newer NCCL after PyTorch is safe.

- Pin NCCL 2.30.4-1+cuda13.2 in the base build stage so the
  extensions-build stage inherits it for DeepEP compilation
- Pin the same version in the vllm-base runtime stage, replacing the
  dynamic apt-cache madison lookup
- Update DEEPEP_COMMIT_HASH from 73b6ea4 to d4f41e4e93 to pick up
  DeepEPv2

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth
tlrmchlsmth force-pushed the fix/deepep-commit-hash branch from 2de2b77 to 71604cb Compare June 25, 2026 20:23
@tlrmchlsmth
tlrmchlsmth marked this pull request as ready for review June 25, 2026 20:26

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

tlrmchlsmth and others added 7 commits June 25, 2026 16:40
The CUDA base image holds NCCL packages, so installing them in the
base stage without --allow-change-held-packages fails. Move NCCL
installation to the extensions-build stage where it's actually needed
for DeepEP compilation, with the correct apt flag.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
The apt libnccl-dev package only ships host API headers (nccl.h), not
the GIN device headers (ncclGinRequest_t etc.) that DeepEPv2 compiles
against. Install nvidia-nccl-cu13 via pip instead, which includes the
full header set and is auto-detected by DeepEP's build system via
EP_NCCL_ROOT_DIR.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
- Pin NCCL_VERSION to exactly 2.30.7
- Guard pip install (extensions-build) and apt install (vllm-base)
  behind CUDA major >= 13 checks
- CUDA 12 builds fall back to whatever NCCL the base image ships

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
The previous approach failed because:
1. install_python_libraries.sh runs `uv pip install torch` which
   re-pins nvidia-nccl-cu13==2.28.9, overwriting our 2.30.7
2. apt libnccl-dev only has host API headers, not GIN device headers

Fix both by:
- Adding NCCL re-install to install_python_libraries.sh after torch,
  gated on NCCL_VERSION env var and CUDA >= 13
- Replacing apt NCCL install in vllm-base with pip install after all
  wheels are installed
- Removing separate pre-build NCCL RUN from extensions-build

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
The test stage installs requirements/dev.txt which includes
requirements/test/cuda.txt pinning nvidia-nccl-cu13==2.28.9,
overwriting the 2.30.7 upgrade from vllm-base. Add another
NCCL upgrade after all test dependencies are installed.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Replace the three separate `uv pip install nvidia-nccl-cu13==2.30.7
--no-deps` commands (in install script, vllm-base, and test stage) with
a single UV_OVERRIDE env var that forces uv to resolve nvidia-nccl-cu13
to our pinned version whenever it appears as a transitive dependency.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Add instructions for pinning NCCL >= 2.30.4 when installing vLLM via
uv/pip rather than the Docker image, covering both UV_OVERRIDE and
pip --no-deps approaches.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@mergify

mergify Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--45321.org.readthedocs.build/en/45321/

@mergify mergify Bot added the documentation Improvements or additions to documentation label Jun 26, 2026
Comment thread vllm/distributed/device_communicators/pynccl_allocator.py
@mergify

mergify Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Hi @tlrmchlsmth, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

The NCCL symmetric memory allocator JIT-compiles a C++ extension with
`-lnccl`, but the system linker can't find `libnccl.so` when NCCL is
installed only via pip (in site-packages/nvidia/nccl/lib/). Add
`find_nccl_library_paths()` to discover the pip package's lib directory
and pass `-L<path>` to the linker.

This was a pre-existing issue masked by the apt `libnccl-dev` install;
removing apt NCCL in favor of pip-only exposed it.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth
tlrmchlsmth force-pushed the fix/deepep-commit-hash branch from 78c38b5 to 11feadc Compare June 26, 2026 21:45
tlrmchlsmth and others added 6 commits July 10, 2026 17:44
FusedMoEKernel no longer accepts inplace after the fused experts mechanism was removed. Update both DeepEPv2 test construction sites.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Use the established FP8 MoE tolerance for quantized cases and provide the forward context required by the CUDA-graph dispatch path.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Use shared static activation scales so the reference and distributed Triton paths execute the same FP8 quantization algorithm. Restore the original strict tolerance while retaining the CUDA-graph forward context fix.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@mergify

mergify Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Hi @tlrmchlsmth, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>

@LucasWilkinson LucasWilkinson left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mergify

mergify Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Hi @tlrmchlsmth, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

@tlrmchlsmth
tlrmchlsmth enabled auto-merge (squash) July 23, 2026 19:37
@AndreasKaratzas
AndreasKaratzas disabled auto-merge July 23, 2026 20:59
@AndreasKaratzas

AndreasKaratzas commented Jul 23, 2026

Copy link
Copy Markdown
Member

@tlrmchlsmth would it be alright if our team takes an hour or two to check if test groups are good on rocm too? I promise I ll have my eye here to merge it ASAP

UPDATE: Done :)

@AndreasKaratzas AndreasKaratzas added the rocm Related to AMD ROCm label Jul 23, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Jul 23, 2026
@AndreasKaratzas
AndreasKaratzas enabled auto-merge (squash) July 23, 2026 21:23

@mgoin mgoin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tyler is confident :)

@vllm-bot
vllm-bot merged commit 5d8e90a into main Jul 24, 2026
245 of 252 checks passed
@vllm-bot
vllm-bot deleted the fix/deepep-commit-hash branch July 24, 2026 20:00
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build documentation Improvements or additions to documentation ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

8 participants