Skip to content
Merged
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
71604cb
[WideEP] Update DeepEP version and pin NCCL >= 2.30.4
tlrmchlsmth Jun 25, 2026
1152444
Move NCCL install from base stage to extensions-build stage
tlrmchlsmth Jun 25, 2026
2763c72
Use pip-installed NCCL for DeepEPv2 GIN device headers
tlrmchlsmth Jun 25, 2026
1783ac7
Pin NCCL to 2.30.7 and guard against CUDA 12
tlrmchlsmth Jun 25, 2026
72ab9d7
Use pip for all NCCL installs, drop apt libnccl-dev
tlrmchlsmth Jun 25, 2026
2d16eab
Re-upgrade NCCL in test stage after dev requirements
tlrmchlsmth Jun 26, 2026
54f604a
Use UV_OVERRIDE to pin NCCL instead of separate re-upgrade lines
tlrmchlsmth Jun 26, 2026
9892bce
Document NCCL override for non-Docker DeepEPv2 installs
tlrmchlsmth Jun 26, 2026
11feadc
Fix NCCL allocator link failure with pip-only NCCL
tlrmchlsmth Jun 26, 2026
58859ae
Guard against DeepEPv2 SIGSEGV when NCCL GIN is unavailable
tlrmchlsmth Jun 27, 2026
1e9e837
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jun 29, 2026
7d6c8bc
Fix GIN skip crashing in spawned subprocess
tlrmchlsmth Jun 29, 2026
a8fa73f
Add NCCL GIN diagnostics to B200 fused MOE CI step
tlrmchlsmth Jun 29, 2026
4bc4ecf
Fix ruff formatting for GINNotAvailableError raise
tlrmchlsmth Jun 29, 2026
519c554
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jun 30, 2026
cc6712f
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jun 30, 2026
40e191a
Fix GIN check: treat query failure as GIN-unavailable
tlrmchlsmth Jun 30, 2026
97dbca7
Apply ruff formatting fixes
tlrmchlsmth Jun 30, 2026
5d469f4
Create libnccl.so linker symlink from pip-installed NCCL
tlrmchlsmth Jun 30, 2026
441a522
Link nccl_allocator against libnccl.so.2 directly
tlrmchlsmth Jun 30, 2026
27121dd
Skip test_deepep_v2_manager_lifecycle when GIN is unavailable
tlrmchlsmth Jun 30, 2026
20b07f6
Address review comments
tlrmchlsmth Jul 1, 2026
f975515
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 1, 2026
a6ab383
Initialize ncclCommProperties with required header fields
tlrmchlsmth Jul 1, 2026
0f698a0
Fix GIN check: defer when NCCL comm not yet initialized
tlrmchlsmth Jul 2, 2026
b289f00
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 2, 2026
d7012ac
Fix NCCL downgrade from bare pip install in CI
tlrmchlsmth Jul 2, 2026
99e5ea6
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 6, 2026
54bcda0
[CI] Use pip --no-deps for tensorizer to fix AMD CI
tlrmchlsmth Jul 6, 2026
4f04dd6
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 6, 2026
e565ca9
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 7, 2026
b9e57d5
[Test] Skip DeepEP MoE layer tests without P2P access
tlrmchlsmth Jul 7, 2026
af75196
[Test] Add async TP device diagnostics
tlrmchlsmth Jul 7, 2026
583cb55
Remove async TP debug diagnostics
tlrmchlsmth Jul 8, 2026
cfc0c1d
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 8, 2026
c67d81f
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 9, 2026
be54bf0
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 9, 2026
6c0391f
Initialize NCCL communicator before GIN check
tlrmchlsmth Jul 10, 2026
0070be8
Restore async TP test module spacing
tlrmchlsmth Jul 10, 2026
09647bb
Merge remote-tracking branch 'origin/main' into fix/deepep-commit-hash
tlrmchlsmth Jul 10, 2026
30dc611
Remove obsolete DeepEPv2 MoE inplace argument
tlrmchlsmth Jul 10, 2026
f420c20
Fix DeepEPv2 MoE test expectations
tlrmchlsmth Jul 13, 2026
a411679
Keep strict DeepEPv2 FP8 accuracy checks
tlrmchlsmth Jul 13, 2026
1184890
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 14, 2026
fe4d994
Fix DeepEPv2 MoE test references
tlrmchlsmth Jul 16, 2026
1cbaab6
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 16, 2026
f8a25c8
Fix DeepEP v2 test formatting
tlrmchlsmth Jul 17, 2026
be27f79
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 23, 2026
aec70ce
Merge branch 'main' into fix/deepep-commit-hash
tlrmchlsmth Jul 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .buildkite/test_areas/misc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ steps:
- vllm/multimodal
- examples/
commands:
- pip install tensorizer # for tensorizer test
- pip install --no-deps tensorizer # for tensorizer test
# for basic
- python3 basic/offline_inference/chat.py
- python3 basic/offline_inference/generate.py --model facebook/opt-125m
Expand Down
2 changes: 1 addition & 1 deletion .buildkite/test_areas/model_runner_v2.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ steps:
commands:
- set -x
- export VLLM_USE_V2_MODEL_RUNNER=1
- pip install tensorizer # for tensorizer test
- pip install --no-deps tensorizer # for tensorizer test
- python3 basic/offline_inference/chat.py # for basic
- python3 basic/offline_inference/generate.py --model facebook/opt-125m
#- python3 basic/offline_inference/generate.py --model meta-llama/Llama-2-13b-chat-hf --cpu-offload-gb 10 # TODO
Expand Down
33 changes: 25 additions & 8 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@
ARG CUDA_VERSION=13.0.2
ARG PYTHON_VERSION=3.12
ARG UBUNTU_VERSION=22.04
# DeepEPv2 requires NCCL >= 2.30.4 (GIN backend).
# This version is only used for CUDA 13+ builds; CUDA 12 falls back to
# the default NCCL version shipped with the base image.
ARG NCCL_VERSION=2.30.7

# By parameterizing the base images, we allow third-party to use their own
# base images. One use case is hermetic builds with base images stored in
Expand Down Expand Up @@ -477,10 +481,17 @@ WORKDIR /workspace
# Build DeepEP wheels
COPY tools/ep_kernels/install_python_libraries.sh /tmp/install_python_libraries.sh
# Defaults moved here from tools/ep_kernels/install_python_libraries.sh for centralized version management
ARG DEEPEP_COMMIT_HASH=73b6ea4
ARG DEEPEP_COMMIT_HASH=d4f41e4e93
ARG NVSHMEM_VER
ARG NCCL_VERSION
RUN --mount=type=cache,target=/opt/uv/cache \
mkdir -p /tmp/ep_kernels_workspace/dist && \
CUDA_MAJOR=$(echo $CUDA_VERSION | cut -d. -f1) && \
if [ "$CUDA_MAJOR" -ge 13 ] && [ -n "$NCCL_VERSION" ]; then \
echo "nvidia-nccl-cu${CUDA_MAJOR}==${NCCL_VERSION}" \
> /tmp/nccl-override.txt && \
export UV_OVERRIDE=/tmp/nccl-override.txt; \
fi && \
export TORCH_CUDA_ARCH_LIST='9.0a 10.0a' && \
/tmp/install_python_libraries.sh \
--workspace /tmp/ep_kernels_workspace \
Expand Down Expand Up @@ -644,6 +655,7 @@ FROM ${FINAL_BASE_IMAGE} AS vllm-base

ARG CUDA_VERSION
ARG PYTHON_VERSION
ARG NCCL_VERSION
ARG DEADSNAKES_MIRROR_URL
ARG DEADSNAKES_GPGKEY_URL
ARG GET_PIP_URL
Expand Down Expand Up @@ -696,7 +708,6 @@ RUN apt-get update -y \
# Install CUDA development tools for runtime JIT compilation
# (FlashInfer, DeepGEMM, EP kernels all require compilation at runtime)
RUN CUDA_VERSION_DASH=$(echo $CUDA_VERSION | cut -d. -f1,2 | tr '.' '-') && \
CUDA_VERSION_SHORT=$(echo $CUDA_VERSION | cut -d. -f1,2) && \
apt-get update -y && \
apt-get install -y --no-install-recommends --allow-change-held-packages \
cuda-nvcc-${CUDA_VERSION_DASH} \
Expand All @@ -709,12 +720,6 @@ RUN CUDA_VERSION_DASH=$(echo $CUDA_VERSION | cut -d. -f1,2 | tr '.' '-') && \
libnuma-dev \
# numactl CLI for NUMA binding at runtime
numactl && \
# Fixes nccl_allocator requiring nccl.h at runtime
# https://github.com/vllm-project/vllm/blob/1336a1ea244fa8bfd7e72751cabbdb5b68a0c11a/vllm/distributed/device_communicators/pynccl_allocator.py#L22
# NCCL packages don't use the cuda-MAJOR-MINOR naming convention,
# so we pin the version to match our CUDA version
NCCL_VER=$(apt-cache madison libnccl-dev | grep "+cuda${CUDA_VERSION_SHORT}" | head -1 | awk -F'|' '{gsub(/^ +| +$/, "", $2); print $2}') && \
apt-get install -y --no-install-recommends --allow-change-held-packages libnccl-dev=${NCCL_VER} libnccl2=${NCCL_VER} && \
Comment thread
LucasWilkinson marked this conversation as resolved.
Resolved
rm -rf /var/lib/apt/lists/*

# Install uv for faster pip installs
Expand All @@ -734,6 +739,18 @@ RUN mkdir -p "${UV_PYTHON_INSTALL_DIR}" "${UV_CACHE_DIR}" \
&& chgrp -R 0 /opt/uv \
&& chmod -R g+rwX,a+rX /opt/uv

# DeepEPv2 GIN requires NCCL >= 2.30.4 at both compile and runtime. torch pins
# an older version as a transitive dep; this override forces uv to use our
# pinned version whenever nvidia-nccl-cu* is resolved. Empty on CUDA 12 (no-op).
RUN CUDA_MAJOR=$(echo $CUDA_VERSION | cut -d. -f1) && \
if [ "$CUDA_MAJOR" -ge 13 ]; then \
echo "nvidia-nccl-cu${CUDA_MAJOR}==${NCCL_VERSION}" \
> /etc/uv-overrides.txt; \
else \
touch /etc/uv-overrides.txt; \
fi
ENV UV_OVERRIDE=/etc/uv-overrides.txt

# ----------------------------------------------------------------------
# Non-root support (opt-in)
# ----------------------------------------------------------------------
Expand Down
5 changes: 4 additions & 1 deletion docker/versions.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@
"UBUNTU_VERSION": {
"default": "22.04"
},
"NCCL_VERSION": {
"default": "2.30.7"
},
"BUILD_BASE_IMAGE": {
"default": "nvidia/cuda:13.0.2-devel-ubuntu22.04"
},
Expand Down Expand Up @@ -56,7 +59,7 @@
"default": "cuda"
},
"DEEPEP_COMMIT_HASH": {
"default": "73b6ea4"
"default": "d4f41e4e93"
},
"GIT_REPO_CHECK": {
"default": "0"
Expand Down
5 changes: 5 additions & 0 deletions docs/serving/expert_parallel_deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ Before using EP, you need to install the necessary dependencies. We are actively
2. **Install DeepGEMM library**: Follow the [official instructions](https://github.com/deepseek-ai/DeepGEMM#installation).
3. **For disaggregated serving**: Install `gdrcopy` by running the [`install_gdrcopy.sh`](../../tools/install_gdrcopy.sh) script (e.g., `install_gdrcopy.sh "${GDRCOPY_OS_VERSION}" "12.8" "x64"`). You can find available OS versions [here](https://developer.download.nvidia.com/compute/redist/gdrcopy/CUDA%2012.8/).

!!! note "NCCL version (CUDA 13+)"
The `deepep_v2` backend requires NCCL >= 2.30.4. PyTorch ships an older
NCCL, so you must upgrade it before building or running DeepEP. See the
[EP kernels guide](../../tools/ep_kernels) for instructions.

### Backend Selection Guide

vLLM provides multiple communication backends for EP. Use `--all2all-backend` to select one:
Expand Down
12 changes: 9 additions & 3 deletions tests/distributed/test_mnnvl_alltoall.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,10 @@ def _spawn_workers(worker_fn, world_size, *, dp_size=None):
err_queue.close()
err_queue.join_thread()
if errors:
pytest.fail("Worker(s) failed:\n" + "\n---\n".join(errors))
combined = "\n---\n".join(errors)
if "NCCL GIN" in combined:
pytest.skip("NCCL GIN not available on this system")
pytest.fail("Worker(s) failed:\n" + combined)


def _run_worker(rank, world_size, port, worker_fn, dp_size, dp_port, err_queue):
Expand Down Expand Up @@ -916,8 +919,11 @@ def _deepep_v2_lifecycle_worker(rank, world_size):
DeepEPV2All2AllManager,
)

cpu_group = get_ep_group().cpu_group
manager = DeepEPV2All2AllManager(cpu_group)
ep_group = get_ep_group()
manager = DeepEPV2All2AllManager(
ep_group.cpu_group,
device_group=ep_group.device_group,
)

assert manager.rank == rank
assert manager.world_size == world_size
Expand Down
48 changes: 36 additions & 12 deletions tests/kernels/moe/parallel_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@
P = ParamSpec("P")


class GINNotAvailableError(RuntimeError):
pass


@dataclasses.dataclass
class ProcessGroupInfo:
world_size: int
Expand Down Expand Up @@ -96,19 +100,27 @@ def parallel_launch(
**kwargs: P.kwargs,
) -> None:
assert not kwargs
spawn(
_worker_parallel_launch,
args=(
world_size,
world_size,
0,
f"tcp://{os.getenv('LOCALHOST', 'localhost')}:{get_open_port()}",
worker,
try:
spawn(
_worker_parallel_launch,
args=(
world_size,
world_size,
0,
f"tcp://{os.getenv('LOCALHOST', 'localhost')}:{get_open_port()}",
worker,
)
+ args,
nprocs=world_size,
join=True,
)
+ args,
nprocs=world_size,
join=True,
)
except Exception as exc:
# pytest.skip cannot propagate directly through torch.multiprocessing.
if "GINNotAvailableError" in str(exc):
import pytest

pytest.skip("NCCL GIN not available (no IBGDA-capable hardware)")
raise


## DeepEP specific utils
Expand Down Expand Up @@ -225,6 +237,18 @@ def make_deepep_v2_a2a(
):
import deep_ep

from vllm.utils.nccl import query_nccl_gin_type

# ElasticBuffer can segfault when GIN is unavailable. Initialize the
# lazy communicator and reject unsupported systems before entering DeepEP.
probe = torch.zeros(1, device=pgi.device)
torch.distributed.all_reduce(probe, group=pg)
gin_type = query_nccl_gin_type(pg)
if gin_type is None:
raise RuntimeError("Failed to determine NCCL GIN support")
if gin_type == 0:
raise GINNotAvailableError("NCCL GIN not available")

buffer = deep_ep.ElasticBuffer(
group=pg,
num_max_tokens_per_rank=v2_args.max_tokens_per_rank,
Expand Down
Loading
Loading