[WideEP] Update NCCL to 2.30.7 to enable DeepEPv2 in the vllm/vllm-openai image - #45321
Conversation
|
Actually an issue here -- we need to update NCCL to > 2.30.4 during DeepEP compilation as well, not just rely on installing it afterwards like I thought we could |
a8cb6bd to
49fbfce
Compare
DeepEPv2 (ElasticBuffer API) requires NCCL >= 2.30.4 at both compile and runtime. NCCL backward compatibility is guaranteed by NVIDIA, so installing a newer NCCL after PyTorch is safe. - Pin NCCL 2.30.4-1+cuda13.2 in the base build stage so the extensions-build stage inherits it for DeepEP compilation - Pin the same version in the vllm-base runtime stage, replacing the dynamic apt-cache madison lookup - Update DEEPEP_COMMIT_HASH from 73b6ea4 to d4f41e4e93 to pick up DeepEPv2 Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
2de2b77 to
71604cb
Compare
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
The CUDA base image holds NCCL packages, so installing them in the base stage without --allow-change-held-packages fails. Move NCCL installation to the extensions-build stage where it's actually needed for DeepEP compilation, with the correct apt flag. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
The apt libnccl-dev package only ships host API headers (nccl.h), not the GIN device headers (ncclGinRequest_t etc.) that DeepEPv2 compiles against. Install nvidia-nccl-cu13 via pip instead, which includes the full header set and is auto-detected by DeepEP's build system via EP_NCCL_ROOT_DIR. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
- Pin NCCL_VERSION to exactly 2.30.7 - Guard pip install (extensions-build) and apt install (vllm-base) behind CUDA major >= 13 checks - CUDA 12 builds fall back to whatever NCCL the base image ships Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
The previous approach failed because: 1. install_python_libraries.sh runs `uv pip install torch` which re-pins nvidia-nccl-cu13==2.28.9, overwriting our 2.30.7 2. apt libnccl-dev only has host API headers, not GIN device headers Fix both by: - Adding NCCL re-install to install_python_libraries.sh after torch, gated on NCCL_VERSION env var and CUDA >= 13 - Replacing apt NCCL install in vllm-base with pip install after all wheels are installed - Removing separate pre-build NCCL RUN from extensions-build Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
The test stage installs requirements/dev.txt which includes requirements/test/cuda.txt pinning nvidia-nccl-cu13==2.28.9, overwriting the 2.30.7 upgrade from vllm-base. Add another NCCL upgrade after all test dependencies are installed. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Replace the three separate `uv pip install nvidia-nccl-cu13==2.30.7 --no-deps` commands (in install script, vllm-base, and test stage) with a single UV_OVERRIDE env var that forces uv to resolve nvidia-nccl-cu13 to our pinned version whenever it appears as a transitive dependency. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Add instructions for pinning NCCL >= 2.30.4 when installing vLLM via uv/pip rather than the Docker image, covering both UV_OVERRIDE and pip --no-deps approaches. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
|
Documentation preview: https://vllm--45321.org.readthedocs.build/en/45321/ |
|
Hi @tlrmchlsmth, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
The NCCL symmetric memory allocator JIT-compiles a C++ extension with `-lnccl`, but the system linker can't find `libnccl.so` when NCCL is installed only via pip (in site-packages/nvidia/nccl/lib/). Add `find_nccl_library_paths()` to discover the pip package's lib directory and pass `-L<path>` to the linker. This was a pre-existing issue masked by the apt `libnccl-dev` install; removing apt NCCL in favor of pip-only exposed it. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
78c38b5 to
11feadc
Compare
FusedMoEKernel no longer accepts inplace after the fused experts mechanism was removed. Update both DeepEPv2 test construction sites. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Use the established FP8 MoE tolerance for quantized cases and provide the forward context required by the CUDA-graph dispatch path. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Use shared static activation scales so the reference and distributed Triton paths execute the same FP8 quantization algorithm. Restore the original strict tolerance while retaining the CUDA-graph forward context fix. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
|
Hi @tlrmchlsmth, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
|
Hi @tlrmchlsmth, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
|
@tlrmchlsmth would it be alright if our team takes an hour or two to check if test groups are good on rocm too? I promise I ll have my eye here to merge it ASAP UPDATE: Done :) |
Update DeepEP to deepseek-ai/DeepEP@d4f41e4 so that we can pick up DeepEPv2.
Missed this in #41183
Ideally we'd have a single source of truth for the commit across both the install script and the Dockerfile but I don't see a sane way to do it at the moment
PyTorch ships an earlier version of NCCL. Per feedback from the NCCL team, I think this should be safe as
For the pypi wheel, no change is needed since we don't ship DeepEP in the wheel.