Skip to content

[docker] add vllm-openai-efa target - #58768

Merged
khluu merged 2 commits into
vllm-project:mainfrom
snadampal:efa_install
Oct 26, 2026
Merged

khluu merged 2 commits into
vllm-project:mainfrom
snadampal:efa_install

Conversation

@snadampal

@snadampal snadampal commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

This PR is to improve vLLM out-of-box experience on AWS GPU clusters.

  1. Added a new build target vllm-openai-efa to build image with EFA user space libraries and mooncake-transfer-engine-efa.
  2. Extended CI workflow scripts to build and publish the newly added target images.
    NOTE:
    The PR adds a new target vllm-openai-efa instead of updating the existing vllm-openai because the rdma-core version from EFA installer is different from the version comes with distro. Though the latest rdma-core version from the EFA installer should work for non-EFA platforms as well, we didn't test that configuration. And also not to interfere with the existing build configurations, we took a new target approach.

Purpose

To improve vLLM out-of-box experience on AWS GPU clusters.

Test Plan

  1. P-D disaggregated inference on AWS p5en.48xlarge 2 node cluster with NIXL LIBFABRIC (efa provider) backend - to be specific, I ran vllm multi-turn conversation benchmark with Qwen3-32B model.

  2. TBD - test Mooncake TransferEngine with EFA trasport

Test Result

Passed

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify

mergify Bot commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--58768.org.readthedocs.build/en/58768/

@mergify mergify Bot added documentation Improvements or additions to documentation ci/build labels Sep 25, 2026
@robertgshaw2-redhat

Copy link
Copy Markdown
Collaborator

nice! can we also ensure that nixl and deepepv2 work properly in these images?

@functionstackx functionstackx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can u also test if Mooncake p2p store works and not just mooncake transfers?

here is an example here SemiAnalysisAI/InferenceX#3405

@snadampal

Copy link
Copy Markdown
Contributor Author

Thanks for the prompt response, @robertgshaw2-redhat ! I've already tested Nixl with P/D disagg with this image.
mooncake TE testing is in progress, will plan deepepv2 as well.

@functionstackx , yeah, will use mooncake store instead of p/d for mooncake TE coverage here.

@erezzarum

Copy link
Copy Markdown
Contributor

nice! can we also ensure that nixl and deepepv2 work properly in these images?

NIXL is included in the base vLLM image, the Python wheels from pypi include the libfabric plugin.

@snadampal

snadampal commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor Author

Here are the steps to build and run image from this PR on a AWS GPU cluster:

Build
pre-requisites: docker installed, vllm cloned and cherrypicked this PR

cd ~/vllm

DOCKER_BUILDKIT=1 docker build . \
    --target vllm-openai-efa \
    --tag vllm/vllm-openai:efa_pr_58768 \
    --file docker/Dockerfile \
    --build-arg INSTALL_KV_CONNECTORS=true \
    --build-arg max_jobs=$(nproc) \
    --build-arg nvcc_threads=2

Testing P/D disagg with NIXL+ libfabric/EFA backend
note: Qwen3-32B model is used here on a AWS p5en.48xlarge 2 node cluster with 1P-1D topology

1. Launch prefill server 

docker run --gpus all \
    --network host \
    --ipc=host \
    $(for d in /dev/infiniband/uverbs*; do echo -n "--device $d "; done) \
    --ulimit memlock=-1 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    --env "VLLM_NIXL_SIDE_CHANNEL_HOST=<PREFILL_NODE_IP>" \
    --env "VLLM_NIXL_SIDE_CHANNEL_PORT=5600" \
    vllm/vllm-openai:efa_sn \
    --model Qwen/Qwen3-32B --served-model-name Qwen --dtype bfloat16 \
    --max-num-seqs 8 --port 8100 --max-model-len 32768 --tensor-parallel-size 8 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_buffer_device":"cuda","kv_role":"kv_producer”,”engine_id":"prefill-engine-001","kv_rank":0,"kv_parallel_size":2,"kv_connector_extra_config":{"backends":["LIBFABRIC"]}}'


2. Launch decode server 

docker run --gpus all \
    --network host \
    --ipc=host \
    $(for d in /dev/infiniband/uverbs*; do echo -n "--device $d "; done) \
    --ulimit memlock=-1 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    --env "VLLM_NIXL_SIDE_CHANNEL_HOST=<DECODE_NODE_IP>" \
    --env "VLLM_NIXL_SIDE_CHANNEL_PORT=5600" \
    vllm/vllm-openai:efa_sn \
    --model Qwen/Qwen3-32B --served-model-name Qwen --dtype bfloat16 \
    --max-num-seqs 8 --port 8200 --max-model-len 32768 --tensor-parallel-size 8 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_buffer_device":"cuda","kv_role":"kv_consumer”,”engine_id”:”decode-engine-001","kv_rank":0,"kv_parallel_size":2,"kv_connector_extra_config":{"backends":["LIBFABRIC"]}}'

3. Launch router on any one of the above two nodes or launch on a different node

python3 ~/vllm/examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py \
  --model Qwen \
  --port 8000 \
  --prefill <PREFILL_NODE_IP>:8100 \
  --decode <DECODE_NODE_IP>:8200

4. Issue requests. Here i'm issuing requests from the router node, hence use localhost.

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
        "model": "Qwen",
        "messages": [{"role": "user", "content": "Explain KV cache in detail."}],
        "max_tokens": 128,
        "temperature": 0
      }' | python3 -m json.tool


@erezzarum

erezzarum commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

The current cuda base image doesn't have an /etc/shinit_v2 so it doesn't set NCCL_NET_PLUGIN=ofi and NCCL_TUNER_PLUGIN=ofi.
I recommend we also set this already.
Applies to sgl-project/sglang#41006 as well

@snadampal

Copy link
Copy Markdown
Contributor Author

Mooncake Store has been tested, it's working fine with EFA using mooncake-transfer-engine-efa.

I0925 23:37:34.597007  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@cb2b074418dd6ceed530eea5481e47d44b0dcf06b8544fa9c8c3be391de4b2e6 completed successfully
I0925 23:37:34.597009  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@5cea8a86068a10d86096b23d6e4993d65061099640f7e0a15008e7b4cb06e364 completed successfully
I0925 23:37:34.597011  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@3b7ddfcb110fca38cf3428134e36edeb72fa287d9cb30989f3593358e4a082ed completed successfully
I0925 23:37:34.597013  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@e55671e4bf4257146571d3097d2da2214efc349f227341b4f5e0681b3a532efb completed successfully
I0925 23:37:34.597015  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@4bcfbdaff3c119ae2556b423afaa3a1c5aad203491109448f90b67ecd9bc0c14 completed successfully
I0925 23:37:34.597018  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@14794eb235ffae2974db897ff3748e85db6d9c5f1661a9e334c261badf4cb2b4 completed successfully
I0925 23:37:34.597019  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@c2418628aae9d9f0923b2f67396a58ac5ec40cde86194fb8f1e11167c13d00f2 completed successfully
I0925 23:37:34.597021  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@f83c45baea0fa3a1c664ed169dff33459a07c751d098bdfb38094d6cccecb9e7 completed successfully
I0925 23:37:34.597023  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@b9ed6654d086cbbec6b4c918ef7a4971b7122d647c2185418888e49e966bd463 completed successfully
I0925 23:37:34.597025  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@78eb55a4be7671bcd438f179b79dd23215d5078f62633d905c95538b283ec8c2 completed successfully
I0925 23:37:34.597028  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@38fd1da8c39337c0e6e2067dd9493aacdcfcdc039cc1b439a10e10b9d7f79321 completed successfully
I0925 23:37:34.597029  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@609e9a189156ec02bddc6c1386eee3be6c94ee87a7960c6510b843b309507dbe completed successfully
I0925 23:37:34.597031  3163 client_service.cpp:3092] Operation for key efa-store-test@Qwen3-32B@tp_rank:4@pcp0@dcp0@pp_rank:0@group:0@5506761908d64db02fef596a711505d4a33d5f59999c40482e78e07a10c2f91d completed successfully
(APIServer pid=1) INFO:     127.0.0.1:59702 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:59708 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:59710 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:59726 - "GET /metrics HTTP/1.1" 200 OK

and

EfaEndPoint[10.0.60.212:12379@rdmap110s0 <-> 10.0.50.128:13196@rdmap110s0]
EfaEndPoint[10.0.60.212:12379@rdmap110s0 <-> 10.0.50.128:13196@rdmap136s0]
EfaEndPoint[10.0.60.212:12379@rdmap110s0 <-> 10.0.50.128:13196@rdmap137s0]
EfaEndPoint[10.0.60.212:12379@rdmap110s0 <-> 10.0.50.128:13196@rdmap138s0]
EfaEndPoint[10.0.60.212:12379@rdmap110s0 <-> 10.0.50.128:13196@rdmap160s0]

@functionstackx

Copy link
Copy Markdown
Contributor

hi @snadampal

since the new proposed approach is to have an separate efa specific build for vllm/sglang, we might not need to wait for nccl-EP to get well integrated into  sglang/vllm to get multi node WideEP (>=EP16) working out of the box for EFA since  you can reinstall the efa fork'ed DeepEP in the dockerfile i.e.

@functionstackx

Copy link
Copy Markdown
Contributor

@erezzarum

erezzarum commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

seems like here is the AWS file for building deepepv2 https://github.com/dmvevents/awsome-distributed-ai/blob/ca5327caa8b263c0d308d3b992e86161d9db8ef7/examples/inference/vllm/deepep-v2-efa/setup_deepep_v2_efa.s

We have a better way to integrate this into vLLM and SGLang containers, i'll wait for @snadampal to come online tomorrow and we'll work on it.

We are already rolling this into our DLCs, you can see an example: https://github.com/aws/deep-learning-containers/blob/main/docker/vllm/Dockerfile.amzn2023#L163

Comment thread docker/Dockerfile Outdated
ARG EFA_INSTALLER_SHA256=fa6dff8593d866866c13cb4640d9059835cd4efa427971f100ab40c97bef2841
# Mooncake version the vllm-openai-efa target pins its EFA wheel to. Keep in
# sync with requirements/kv_connectors*.txt, which install the non-EFA build.
ARG MOONCAKE_VERSION=0.3.13.post1

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please make a comment that this is temporary --- also, can we not just figure this out below?

e.g. we start by uninstalling, then re-install. Instead of manually keeping this in sync

export MY_PKG_VERSION=$(uv pip show mooncake | grep -i '^Version:' | awk '{print $2}')

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it's possible to check the installed wheel's version instead of passing the version here.

Comment thread docker/Dockerfile Outdated
ARG DEEPEP_REPO="https://github.com/amazon-contributing/DeepEP.git"
ARG DEEPEP_REF="874779c9ccd2294b56304bd6cc5f138f1f71d097"
# DeepEPv2 GDA over EFA requires NCCL 2.31.x or later.
ARG EFA_NCCL_VERSION=2.31.2

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just change the nccl version globally for vllm to 2.31.2, we already have a custom version

cc @tlrmchlsmth - can you confirm this is okay?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed. Should be ok to bump

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great! it will make things more cleaner. @tlrmchlsmth , do you prefer to make this change outside, or ok to make it part of this PR?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm OK to make it part of the pr

@robertgshaw2-redhat

Copy link
Copy Markdown
Collaborator

cc @tlrmchlsmth - what do you think about updating nccl to 2.31?

@erezzarum

erezzarum commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

I think we should add this # syntax=docker/dockerfile:1.10.0-labs to the top of the Dockerfile, to make sure compatibility and choose the correct buildkit frontend.

@snadampal

Copy link
Copy Markdown
Contributor Author

thanks for aligning on the base image nccl version, I will update the PR

@xinli-sw

xinli-sw commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Hi @snadampal could you update the PR? I can try to test it out

@snadampal

Copy link
Copy Markdown
Contributor Author

Hi @xinli-sw, I have updated the PR, please give it a try and let me know if you see any issues. thanks.

@mergify

mergify Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @snadampal.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Oct 7, 2026
@robertgshaw2-redhat
robertgshaw2-redhat enabled auto-merge (squash) October 22, 2026 00:15
@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #96518 for commit 287930dcf93b.

@snadampal

Copy link
Copy Markdown
Contributor Author

The CI is all green now, all checks have passed!

@functionstackx functionstackx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you @snadampal , Erez, Kevin, robert, Tyler, Xin for this PR!

Comment thread docker/Dockerfile.efa Outdated
# conditional, so this is "tcp if TARGETARCH is not amd64, else efa".
ARG ARCH_IF_NOT_AMD64=${TARGETARCH#amd64}
ARG PROTOCOL_IF_NOT_AMD64=${ARCH_IF_NOT_AMD64:+tcp}
ENV MOONCAKE_PROTOCOL=${PROTOCOL_IF_NOT_AMD64:-efa}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if vllm code is reading this env var

Comment thread .buildkite/release-pipeline.yaml Outdated
- "cuda-13-0-ubuntu-24-04"
# AWS EFA variants; no-ops when the opt-in EFA image was not built.
- "cuda-13-0-efa"
- "cuda-13-0-ubuntu-24-04-efa"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default ubuntu is 24.04 now so we can remove 24-04 variant

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but i still see this: "cuda-13-0-ubuntu-24-04"
are you planning to delete that on mainline?

# No cu134 entry: Rubin CUDA 13.4 is nightly-only and is not promoted to
# DockerHub for the base variant either.

target_enabled cuda-13-0-efa && publish_image_family "-efa" || true

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we have || true here?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's best effort but can we fail loudly here, just in case 1 platform pass but not the other and the failure is silenced

@khluu

khluu commented Oct 23, 2026

Copy link
Copy Markdown
Member

Can we also make the publish release image step depends on the EFA build, but with allow_failure: true so it doesn't block publishing other images?

@snadampal

Copy link
Copy Markdown
Contributor Author

thanks for the feedback, @khluu , I will update once you confirm the cuda-13-0-ubuntu-24-04 state on mainline.

@mergify

mergify Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @snadampal.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Oct 23, 2026
Add docker/Dockerfile.efa, a layer on top of a vllm-openai image
(ARG BASE_IMAGE, built with INSTALL_KV_CONNECTORS=true). It adds:
- The AWS EFA userspace stack (libfabric, aws-ofi-nccl), with
  build-time checks that NIXL's LIBFABRIC plugin resolves to the AWS
  libfabric. Also sets NCCL_NET_PLUGIN=ofi and NCCL_TUNER_PLUGIN=ofi.
- On x86_64, the EFA Mooncake wheel at the base image's Mooncake version,
  with MOONCAKE_PROTOCOL=efa (tcp on other architectures).
- DeepEP rebuilt from the amazon-contributing fork for DeepEPv2 GIN over
  EFA.

Bump NCCL_VERSION to 2.31.2 in docker/Dockerfile, which DeepEPv2 GDA over
EFA requires.

Release pipeline: build EFA variants of the CUDA 13.0, and nightly
Rubin CUDA 13.4 images by layering Dockerfile.efa on each staging
image, and create their multi-arch manifests. publish-release-images.sh
adds a shared publish_image_family helper and publishes the -efa families
when present.

Co-authored-by: Erez Zarum <erezz@amazon.com>
Co-authored-by: Sean Westfall <sean.westfall@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
auto-merge was automatically disabled October 24, 2026 02:04

Head branch was pushed to by a user without write access

@mergify mergify Bot removed the needs-rebase label Oct 24, 2026
@snadampal

Copy link
Copy Markdown
Contributor Author

Hi @khluu , I've updated the PR, please check and run CI if it looks good.

@khluu

khluu commented Oct 24, 2026

Copy link
Copy Markdown
Member

(Claude Code here, on @khluu's behalf.) Kicking off PR CI for the updated head, and a release-pipeline test build (shadow pipeline) to check that Dockerfile.efa builds on both x86_64 and aarch64.

@khluu

khluu commented Oct 24, 2026

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #97029 for commit bda1f6f42684.

ℹ️ This PR is 10 commits behind upstream main. Merged upstream main into this branch as bda1f6f42684 before starting CI. Pull the branch before pushing more changes.

@wzhao18

wzhao18 commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Queued 2 failed job(s) for retry in Buildkite CI #97029.

@wzhao18

wzhao18 commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Seems the CI failures are irrelevant, and appear in other builds as well

@khluu
khluu merged commit ffcabb8 into vllm-project:main Oct 26, 2026
485 of 488 checks passed
@snadampal

Copy link
Copy Markdown
Contributor Author

Thanks for merging the PR, @khluu !

AndreasKaratzas added a commit to AndreasKaratzas/vllm that referenced this pull request Oct 26, 2026
Conflict in publish-release-images.sh: keep main's new cuda-13-0-efa target
(vllm-project#58768) and this branch's single ROCm target (no rocm72).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
Micador pushed a commit to Micador/vllm that referenced this pull request Oct 27, 2026
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Co-authored-by: Erez Zarum <erezz@amazon.com>
Co-authored-by: Sean Westfall <sean.westfall@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Mackenzie <christopher.mackenzie.software@gmail.com>
Micador pushed a commit to Micador/vllm that referenced this pull request Oct 27, 2026
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Co-authored-by: Erez Zarum <erezz@amazon.com>
Co-authored-by: Sean Westfall <sean.westfall@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Mackenzie <christopher.mackenzie.software@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build documentation Improvements or additions to documentation ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants