Skip to content

fix(container): use released NIXL for SGLang EFA #12106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nv-nmailhot merged 2 commits into from
Jul 24, 2026
Merged

fix(container): use released NIXL for SGLang EFA #12106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f87c876
feat(container/AWS): Update to EFA Installer 1.49.0 (#12010)
erezzarum Jul 24, 2026
c7ae914
fix(container): install NIXL 1.3.2 in SGLang EFA runtime
jh-nv Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 5 additions & 22 deletions container/compliance/native_packages.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,40 +34,23 @@

packages:
# --- EFA (Elastic Fabric Adapter) components ---
# Installed by container/templates/aws.Dockerfile via the AWS-published
# `efa_installer.sh` (efa_version 1.47.0). The installer drops binaries
# directly without registering with apt, so dpkg/python/rust/go
# generators can't see them. This YAML overlay attributes them.
#
# Image filter:
# - vllm-runtime-efa: baseline (nvcr.io/nvidia/cuda:X.Y-runtime) has
# no libfabric, so EFA legitimately adds both libs.
# - trtllm-runtime-efa: baseline (cuda-dl-base) already ships
# libfabric-aws@2.1.0amzn5.0, but the EFA installer may install a
# different version; explicit attribution closes the version-skew
# gap defensively.
# - (sglang-runtime-efa intentionally omitted: sglang's upstream
# image already ships libfabric1@1.17.0-3build2; the make_efa
# build is redundant and appears to be dead config.)
- name: libfabric
# AWS fork: shipped by EFA installer 1.47.0 as libfabric1-aws_2.4.0amzn1.0
# (Ubuntu 24.04 DEB). The `amzn1.0` suffix indicates an AWS patch set on
# top of upstream libfabric 2.4.0; preserved here for OSRB traceability.
version: 2.4.0amzn1.0
version: 2.4.0amzn5.0
license: BSD-2-Clause
source: https://github.com/ofiwg/libfabric
source: https://github.com/aws/libfabric
images:
- vllm-runtime-efa
- trtllm-runtime-efa
- sglang-runtime-efa

- name: aws-ofi-nccl
# Shipped by EFA installer 1.47.0 as libnccl-ofi_1.18.0-1 (Ubuntu 24.04 DEB).
version: 1.18.0
version: 1.20.0
license: Apache-2.0
source: https://github.com/aws/aws-ofi-nccl
images:
- vllm-runtime-efa
- trtllm-runtime-efa
- sglang-runtime-efa
Comment on lines 37 to +53

@devin-ai-integration devin-ai-integration Bot Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Compliance version attributions rely on audit job for verification

The native package attributions were updated for EFA 1.49.0: libfabric bumped to 2.4.0amzn5.0 with source changed to github.com/aws/libfabric, and aws-ofi-nccl to 1.20.0 (container/compliance/native_packages.yaml:38-47), plus sglang-runtime-efa added to both image filters. These exact upstream version strings could not be verified from within the repo; per the file's own note the authoritative check is the <framework>-compliance-audit syft scan. If the EFA 1.49.0 installer actually ships different versions, the NOTICES attribution would be inaccurate. Recommend confirming against the 1.49.0 installer's bundled DEB versions.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


# --- From-source binaries built in wheel_builder / dynamo_base and copied
# into the runtime images (no apt/pip/cargo metadata, so the python/rust/dpkg
Expand Down
4 changes: 2 additions & 2 deletions container/context.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ dynamo:
nixl_gdrcopy_ref: v2.5.2
nixl_ucx_efa_ref: 9d2b88a1f67faf9876f267658bd077b379b8bb76
nixl_libfabric_repo: https://github.com/ofiwg/libfabric.git
nixl_libfabric_ref: v2.5.1
nixl_libfabric_ref: v2.4.0
# Keep in sync with upstream NIXL's contrib/Dockerfile.manylinux.
hwloc_version: 2.12.2
enable_kvbm: "true"
Expand All @@ -48,7 +48,7 @@ dynamo:
nv_codec_headers_ref: "n13.0.19.0"
libvpx_ref: "v1.14.1"
sccache_version: "v0.14.0"
efa_version: 1.47.0
efa_version: 1.49.0

vllm:
cuda13.0:
Expand Down
30 changes: 1 addition & 29 deletions container/templates/aws.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,45 +27,17 @@ USER root
# --no-verify: Skip GPG verification (optional, can be removed if verification is needed)
# Cache apt downloads; sharing=locked avoids apt/dpkg races with concurrent builds.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
mkdir -p /tmp/efa && \
cd /tmp/efa && \
curl --retry 3 --retry-delay 2 -fsSL -o aws-efa-installer-${EFA_VERSION}.tar.gz \
https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_VERSION}.tar.gz && \
tar -xf aws-efa-installer-${EFA_VERSION}.tar.gz && \
cd aws-efa-installer && \
apt-get update && \
./efa_installer.sh -y --skip-kmod --skip-limit-conf --no-verify && \
rm -rf /tmp/efa && \
rm -rf /opt/amazon/aws-ofi-nccl /etc/ld.so.conf.d/aws-ofi-nccl.conf && \
cd .. && rm -rf aws-efa-installer* && \
ldconfig

ENV EFA_VERSION="${EFA_VERSION}"

Comment on lines 39 to 40

@devin-ai-integration devin-ai-integration Bot Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 aws.Dockerfile libfabric overlay removal changes EFA behavior for all frameworks

This PR (via the underlying #12113 cherry-pick) removes the wheel_builder libfabric overlay logic that conditionally overwrote the EFA-bundled libfabric when older than NIXL_LIBFABRIC_REF (container/templates/aws.Dockerfile:44-67 deleted), and stops purging /opt/amazon/aws-ofi-nccl. This means vllm-runtime-efa and trtllm-runtime-efa now ship the EFA-bundled libfabric/aws-ofi-nccl unconditionally rather than the wheel_builder-built version. nixl_libfabric_ref was also downgraded v2.5.1 -> v2.4.0 in context.yaml but is still consumed by wheel_builder (container/templates/wheel_builder.Dockerfile:437). Behavior change is presumably intentional with the newer EFA installer, but the runtime libfabric version now shipped by vllm/trtllm EFA images differs from before — worth confirming those images still function with the EFA-stock libfabric.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

ARG NIXL_LIBFABRIC_REF

# Copy the wheel_builder-built libfabric and register it with the dynamic linker
# ONLY if the EFA-bundled libfabric is older than NIXL_LIBFABRIC_REF.
# When a future EFA installer ships libfabric >= the version we build, the
# version comparison evaluates to false and this becomes a no-op automatically.
RUN --mount=from=wheel_builder,source=/usr/local/libfabric,target=/tmp/libfabric_build \
EFA_PC=$(find /opt/amazon/efa -path '*/pkgconfig/libfabric.pc' 2>/dev/null | head -n1) && \
EFA_LIBFABRIC_RAW=$(cat "$EFA_PC" 2>/dev/null | grep '^Version:' | awk '{print $2}') && \
EFA_LIBFABRIC_VER=$(echo "$EFA_LIBFABRIC_RAW" | grep -oE '^[0-9]+\.[0-9]+(\.[0-9]+)?') && \
REF_VER=$(echo "${NIXL_LIBFABRIC_REF}" | sed 's/^v//') && \
if [ -n "$EFA_LIBFABRIC_VER" ] && [ -n "$REF_VER" ] && \
[ "$(printf '%s\n' "$EFA_LIBFABRIC_VER" "$REF_VER" | sort -V | head -n1)" = "$EFA_LIBFABRIC_VER" ] && \
[ "$EFA_LIBFABRIC_VER" != "$REF_VER" ]; then \
rm -rf /opt/amazon/efa && \
cp -Pfr /tmp/libfabric_build /opt/amazon/efa && \
sed -i 's|^prefix=.*|prefix=/opt/amazon/efa|' /opt/amazon/efa/lib/pkgconfig/libfabric.pc && \
echo "/opt/amazon/efa/lib" > /etc/ld.so.conf.d/000_efa.conf && \
rm -f /etc/ld.so.conf.d/efa.conf && \
ldconfig && \
echo "[aws] libfabric overlay: ${REF_VER} (overwrites EFA stock ${EFA_LIBFABRIC_RAW})"; \
else \
echo "[aws] libfabric overlay: skipped (EFA stock ${EFA_LIBFABRIC_RAW:-unknown} >= ${REF_VER})"; \
fi

{% if framework == "trtllm" %}
# After the upstream mesonpy refactor, libplugin_LIBFABRIC.so lands under the
# Dynamo venv while the rest of the NIXL plugin set (GDS/UCX/POSIX) remains at
Expand Down
24 changes: 24 additions & 0 deletions container/templates/sglang_runtime.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,30 @@ RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \
export PIP_CACHE_DIR=/root/.cache/pip && \
pip install --break-system-packages --no-deps "distro==1.9.0"

{% if make_efa == true and device == "cuda" and target == "runtime" %}

@erezzarum erezzarum Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for this, we should just introduce ability to upgrade NIXL wheels, it should not be specific to EFA or SGLang.
I'm also not sure why install_nixl_from_wheel is a case just for EFA? i see this in the vLLM container and it's not related to SGLang nor EFA, can you explain more about this?

@jh-nv jh-nv Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m trying to keep this PR narrowly scoped to the know broken image, which is the SGLang EFA image to reduce impact radius.

Your suggestion makes sense for long term in main branch.

# Release/1.3 SGLang EFA runtime needs a released NIXL wheel with the LIBFABRIC
# KV transfer fix. Keep this configurable so the release image can move forward
# without changing the Dockerfile after upstream framework images catch up.
ARG NIXL_WHEEL_VERSION=1.3.2
COPY --chmod=755 container/deps/vllm/install_nixl_from_wheel.sh /usr/local/bin/install_nixl_from_wheel
RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \
set -eux; \
export PIP_CACHE_DIR=/root/.cache/pip; \
pip install --break-system-packages --force-reinstall --no-deps --only-binary=:all: \

@erezzarum erezzarum Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be introduced as a general argument to force update wheel files to a nixl version.
It should not be related just to EFA and should be applied on the vLLM and SGLang container runtime.
When building for EFA we override this argument with the updated nixl version.
Why? hard coding this will create issues when the upstream framework container images will catch up on NIXL versions, we should make it dynamic and configurable.

@jh-nv jh-nv Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to make the PR scope narrow for this specific release.

"nixl==${NIXL_WHEEL_VERSION}" \
"nixl-cu13==${NIXL_WHEEL_VERSION}"; \
site_packages=$(python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'); \
install_nixl_from_wheel \
--wheel-lib-dir "${site_packages}/.nixl_cu13.mesonpy.libs" \
--prefix /opt/nvidia/nvda_nixl/lib64 \
--skip-headers; \
ln -sfn /opt/nvidia/nvda_nixl/lib64/plugins /opt/nvidia/nvda_nixl/plugins
ENV NIXL_PREFIX=/opt/nvidia/nvda_nixl
ENV NIXL_LIB_DIR=/opt/nvidia/nvda_nixl/lib64
ENV NIXL_PLUGIN_DIR=/opt/nvidia/nvda_nixl/lib64/plugins
ENV LD_LIBRARY_PATH=/opt/nvidia/nvda_nixl/lib64:/opt/nvidia/nvda_nixl/lib64/plugins:${LD_LIBRARY_PATH:-}
{% endif %}

# Install gpu_memory_service wheel if enabled (all targets)
ARG ENABLE_GPU_MEMORY_SERVICE
RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \
Expand Down
Loading