feat(container): configurable libfabric repo + v2.5.1 overlay for EFA - #9727
Merged
yifjiang merged 1 commit intoMay 26, 2026
Merged
Conversation
Contributor
|
👋 Hi yifjiang! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
github-actions
Bot
added
feat
external-contribution
yifjiang
force-pushed
the
yifjiang/patched-libfabric-in-aws-stage
branch
from
May 19, 2026 04:27
a68bfc2 to
6ea40ec
Compare
yifjiang
force-pushed
the
yifjiang/patched-libfabric-in-aws-stage
branch
from
May 19, 2026 04:45
6ea40ec to
de43246
Compare
3 tasks
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 19, 2026
Bumps `nixl_ref: 0.10.1 → v1.1.0` in all four occurrences in container/context.yaml (one in `dynamo:` common section, one each in the `vllm:`, `sglang:`, `trtllm:` framework sections), and adds the Abseil source-build prereq that NIXL >= 1.0.0 requires. ### Why bump NIXL NIXL v1.1.0 brings four fixes that matter on AWS p6e-gb200 EFA fabric: ai-dynamo#1461 NUMA-aware EFA rail selection — without this, the LIBFABRIC backend assigns all initiator GPUs to a single rail and caps aggregate bandwidth at ~1.79 GB/s on p6e-gb200 instead of the expected ~190 GB/s with 4 GPUs + 4 EFA NICs per pod. ai-dynamo#1510 Active rail tracking for multi-rail concurrent transfers. ai-dynamo#1506 Multi-GPU memory-region fix (relevant for TP > 1 workers registering large VRAM blocks across GPUs). ai-dynamo#1433 Transfer handle repost notification fix. These gains were validated by yutwu (NVIDIA teammate) during the GLM-5.1 v7→v7.5 image bump on AWS p6e-gb200. For dynamo-trtllm disagg over EFA, NIXL 0.10.1's rail policy is the difference between "looks correct, caps at 1.79 GB/s" and "190 GB/s aggregate". ### The Abseil prereq NIXL >= 1.0.0 uses VLOG(1)/DVLOG(2) in nixl_log.h, which require Abseil >= 20240116. NIXL's meson.build first searches for system Abseil via pkg-config: absl_base_dep = dependency('absl_base', required: false) absl_log_dep = dependency('absl_log', required: false) ... if absl_base_dep.found() and not absl_log_dep.found() error('Your Abseil version is too old: found absl_base but missing support for absl_log. Cannot fallback to subproject because that would result in a mix of Abseil versions at runtime.') If pkg-config finds an old Abseil (absl_base present but no absl_log), NIXL HARD-ERRORS at configure time. Subproject fallback only triggers when NO system Abseil is found. This trips two ways for dynamo: 1. **wheel_builder (AlmaLinux 8 / manylinux_2_28)** typically lacks system Abseil, so subproject fallback would work — BUT the subproject builds shared libs with SONAMEs like libabsl_*.so.20250814. 2. **runtime images (cuda-dl-base Ubuntu 24.04)** ship stock libabsl-dev 20220623 (libabsl_*.so.20220623 SONAMEs). NIXL's libs built against subproject 20250814 can't dlopen against the runtime image's 20220623 — SONAME mismatch. The clean fix is to pre-install Abseil consistently at /usr/local in wheel_builder so meson uses it deterministically, then propagate the .so files to runtime stages. This matches yutwu's validated approach in the GLM-5.1 EFA-patch script (`install_abseil_from_source`). ### What changes 1. `container/context.yaml`: - Bump `nixl_ref: 0.10.1 → v1.1.0` in all 4 sections. - Add `dynamo.abseil_ref: 20240722.0` (the Abseil LTS yutwu validated). 2. `container/templates/args.Dockerfile`: - Declare `ARG ABSEIL_REF` (gated by `{% if device == "cuda" %}`, same as the other NIXL-related ARGs). 3. `container/templates/wheel_builder.Dockerfile`: - New RUN block BEFORE the NIXL clone+build that source-builds Abseil ${ABSEIL_REF} to /usr/local with `BUILD_SHARED_LIBS=ON` + `ABSL_ENABLE_INSTALL=ON`. - Gated by `pkg-config --exists absl_log` as a no-op if a future base image already ships a recent Abseil. - Validates `pkg-config --modversion absl_log` succeeds after install (build fails if not). 4. `container/templates/dynamo_runtime.Dockerfile`: - New COPY line bringing /usr/local/lib/libabsl_*.so* from wheel_builder so libnixl can resolve its Abseil deps at dlopen. 5. `container/templates/trtllm_runtime.Dockerfile`: - Same Abseil .so COPY. trtllm_runtime overrides the dynamo runtime stage, so it needs its own COPY independently. vllm and sglang frameworks are unaffected at the runtime level — they use upstream image NIXL packages (nixl-cu12 from vllm-openai, sglang's bundled NIXL), not dynamo's wheel_builder NIXL. Their wheel_builder stage still builds Abseil (gated by `device == "cuda"`), which is consistent with the existing pattern of building UCX/NIXL/etc. in wheel_builder regardless of whether the framework runtime uses them. ### Why source-build (and not apt-install a newer libabsl-dev) I evaluated `apt-install libabsl-dev` as an option. Ubuntu 24.04's main archive ships `libabsl-dev 20220623.1-1build1`, which is exactly the version NIXL hard-errors on. Backports / -updates / -proposed don't have a newer version. yutwu hit the same investigation and arrived at source-build as the only working path. If a maintainer knows of a repo (NVIDIA apt, PPA, etc.) with a newer libabsl-dev that works on Ubuntu 24.04 / cuda-dl-base, I'm happy to swap source-build for apt-install — the source-build adds ~2-3 min of build time per arch. ### Risk MEDIUM. This is a major version bump (0.x → 1.x) plus a new build dependency. Specifically: - The C++ libnixl.so ABI is the primary compat concern. Subproject fallback would have produced a 20250814 Abseil SONAME mismatch at runtime; this PR avoids that by pinning to 20240722.0 consistently across build + runtime stages. - Python NIXL bindings may have changed across 0.10.1 → v1.1.0. Dynamo's serving code that imports `nixl` needs verification. Worker init + a basic KV transfer smoke is sufficient. - Image size: +~30-40 MB for the Abseil shared libs in runtime stages (libabsl_log, libabsl_base, libabsl_strings, libabsl_status, libabsl_synchronization, libabsl_time, libabsl_flat_hash_map, etc. plus their dependency closure). - Build time: +~2-3 min in wheel_builder for the Abseil compile. ### Tag-format note `ai-dynamo/nixl` mixes tag conventions (older releases lack `v` prefix, newer ones use it): - Older releases: `0.1.1`, `0.10.0`, `0.10.1` (no `v` prefix) - Newer releases: `v1.1.0` (with `v` prefix) This bump uses `v1.1.0` to match the upstream tag. wheel_builder uses `git checkout ${NIXL_REF}` so the value must be exactly the tag name. ### What's needed before this can merge - [ ] `trtllm-pipeline` / `vllm-pipeline` / `sglang-pipeline` / `dynamo-pipeline` CI pass with both Abseil source-build and NIXL v1.1.0. - [ ] Manual smoke: rebuild a trtllm-runtime image, deploy a 1P1D Qwen3-8B disagg, confirm KV transfer works end-to-end. - [ ] (Stretch) `nixlbench` from inside the rebuilt image hits the expected ~190 GB/s on full-node p6e-gb200 allocation. ### References - NIXL v1.1.0 release: https://github.com/ai-dynamo/nixl/releases/tag/v1.1.0 - NIXL meson Abseil logic: https://github.com/ai-dynamo/nixl/blob/v1.1.0/meson.build (the hard-error block we're working around) Companion PRs: - ai-dynamo#9703 — `fix(container)`: ofi-nccl rm path + clearer HAS_TRTLLM_CONTEXT error - ai-dynamo#9704 — `feat(container)`: render.py --has-trtllm-context flag - ai-dynamo#9705 — `build(container)`: nixl_gdrcopy_ref v2.5.1 → v2.5.2 - this PR — `build(container)`: nixl_ref v1.1.0 + Abseil prereq - ai-dynamo#9727 — `feat(container)`: patched libfabric in aws stage Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 19, 2026
Bumps `nixl_ref: 0.10.1 → v1.1.0` in all four occurrences in container/context.yaml (one in `dynamo:` common section, one each in the `vllm:`, `sglang:`, `trtllm:` framework sections), and adds the Abseil source-build prereq that NIXL >= 1.0.0 requires. ### Why bump NIXL NIXL v1.1.0 brings four fixes that matter on AWS p6e-gb200 EFA fabric: ai-dynamo#1461 NUMA-aware EFA rail selection — without this, the LIBFABRIC backend assigns all initiator GPUs to a single rail and caps aggregate bandwidth at ~1.79 GB/s on p6e-gb200 instead of the expected ~190 GB/s with 4 GPUs + 4 EFA NICs per pod. ai-dynamo#1510 Active rail tracking for multi-rail concurrent transfers. ai-dynamo#1506 Multi-GPU memory-region fix (relevant for TP > 1 workers registering large VRAM blocks across GPUs). ai-dynamo#1433 Transfer handle repost notification fix. These gains were validated by yutwu (NVIDIA teammate) during the GLM-5.1 v7→v7.5 image bump on AWS p6e-gb200. For dynamo-trtllm disagg over EFA, NIXL 0.10.1's rail policy is the difference between "looks correct, caps at 1.79 GB/s" and "190 GB/s aggregate". ### The Abseil prereq NIXL >= 1.0.0 uses VLOG(1)/DVLOG(2) in nixl_log.h, which require Abseil >= 20240116. NIXL's meson.build first searches for system Abseil via pkg-config: absl_base_dep = dependency('absl_base', required: false) absl_log_dep = dependency('absl_log', required: false) ... if absl_base_dep.found() and not absl_log_dep.found() error('Your Abseil version is too old: found absl_base but missing support for absl_log. Cannot fallback to subproject because that would result in a mix of Abseil versions at runtime.') If pkg-config finds an old Abseil (absl_base present but no absl_log), NIXL HARD-ERRORS at configure time. Subproject fallback only triggers when NO system Abseil is found. This trips two ways for dynamo: 1. **wheel_builder (AlmaLinux 8 / manylinux_2_28)** typically lacks system Abseil, so subproject fallback would work — BUT the subproject builds shared libs with SONAMEs like libabsl_*.so.20250814. 2. **runtime images (cuda-dl-base Ubuntu 24.04)** ship stock libabsl-dev 20220623 (libabsl_*.so.20220623 SONAMEs). NIXL's libs built against subproject 20250814 can't dlopen against the runtime image's 20220623 — SONAME mismatch. The clean fix is to pre-install Abseil consistently at /usr/local in wheel_builder so meson uses it deterministically, then propagate the .so files to runtime stages. This matches yutwu's validated approach in the GLM-5.1 EFA-patch script (`install_abseil_from_source`). ### What changes 1. `container/context.yaml`: - Bump `nixl_ref: 0.10.1 → v1.1.0` in all 4 sections. - Add `dynamo.abseil_ref: 20240722.0` (the Abseil LTS yutwu validated). 2. `container/templates/args.Dockerfile`: - Declare `ARG ABSEIL_REF` (gated by `{% if device == "cuda" %}`, same as the other NIXL-related ARGs). 3. `container/templates/wheel_builder.Dockerfile`: - New RUN block BEFORE the NIXL clone+build that source-builds Abseil ${ABSEIL_REF} to /usr/local with `BUILD_SHARED_LIBS=ON` + `ABSL_ENABLE_INSTALL=ON`. - Gated by `pkg-config --exists absl_log` as a no-op if a future base image already ships a recent Abseil. - Validates `pkg-config --modversion absl_log` succeeds after install (build fails if not). 4. `container/templates/dynamo_runtime.Dockerfile`: - New COPY line bringing /usr/local/lib/libabsl_*.so* from wheel_builder so libnixl can resolve its Abseil deps at dlopen. 5. `container/templates/trtllm_runtime.Dockerfile`: - Same Abseil .so COPY. trtllm_runtime overrides the dynamo runtime stage, so it needs its own COPY independently. vllm and sglang frameworks are unaffected at the runtime level — they use upstream image NIXL packages (nixl-cu12 from vllm-openai, sglang's bundled NIXL), not dynamo's wheel_builder NIXL. Their wheel_builder stage still builds Abseil (gated by `device == "cuda"`), which is consistent with the existing pattern of building UCX/NIXL/etc. in wheel_builder regardless of whether the framework runtime uses them. ### Why source-build (and not apt-install a newer libabsl-dev) I evaluated `apt-install libabsl-dev` as an option. Ubuntu 24.04's main archive ships `libabsl-dev 20220623.1-1build1`, which is exactly the version NIXL hard-errors on. Backports / -updates / -proposed don't have a newer version. yutwu hit the same investigation and arrived at source-build as the only working path. If a maintainer knows of a repo (NVIDIA apt, PPA, etc.) with a newer libabsl-dev that works on Ubuntu 24.04 / cuda-dl-base, I'm happy to swap source-build for apt-install — the source-build adds ~2-3 min of build time per arch. ### Risk MEDIUM. This is a major version bump (0.x → 1.x) plus a new build dependency. Specifically: - The C++ libnixl.so ABI is the primary compat concern. Subproject fallback would have produced a 20250814 Abseil SONAME mismatch at runtime; this PR avoids that by pinning to 20240722.0 consistently across build + runtime stages. - Python NIXL bindings may have changed across 0.10.1 → v1.1.0. Dynamo's serving code that imports `nixl` needs verification. Worker init + a basic KV transfer smoke is sufficient. - Image size: +~30-40 MB for the Abseil shared libs in runtime stages (libabsl_log, libabsl_base, libabsl_strings, libabsl_status, libabsl_synchronization, libabsl_time, libabsl_flat_hash_map, etc. plus their dependency closure). - Build time: +~2-3 min in wheel_builder for the Abseil compile. ### Tag-format note `ai-dynamo/nixl` mixes tag conventions (older releases lack `v` prefix, newer ones use it): - Older releases: `0.1.1`, `0.10.0`, `0.10.1` (no `v` prefix) - Newer releases: `v1.1.0` (with `v` prefix) This bump uses `v1.1.0` to match the upstream tag. wheel_builder uses `git checkout ${NIXL_REF}` so the value must be exactly the tag name. ### What's needed before this can merge - [ ] `trtllm-pipeline` / `vllm-pipeline` / `sglang-pipeline` / `dynamo-pipeline` CI pass with both Abseil source-build and NIXL v1.1.0. - [ ] Manual smoke: rebuild a trtllm-runtime image, deploy a 1P1D Qwen3-8B disagg, confirm KV transfer works end-to-end. - [ ] (Stretch) `nixlbench` from inside the rebuilt image hits the expected ~190 GB/s on full-node p6e-gb200 allocation. ### References - NIXL v1.1.0 release: https://github.com/ai-dynamo/nixl/releases/tag/v1.1.0 - NIXL meson Abseil logic: https://github.com/ai-dynamo/nixl/blob/v1.1.0/meson.build (the hard-error block we're working around) Companion PRs: - ai-dynamo#9703 — `fix(container)`: ofi-nccl rm path + clearer HAS_TRTLLM_CONTEXT error - ai-dynamo#9704 — `feat(container)`: render.py --has-trtllm-context flag - ai-dynamo#9705 — `build(container)`: nixl_gdrcopy_ref v2.5.1 → v2.5.2 - this PR — `build(container)`: nixl_ref v1.1.0 + Abseil prereq - ai-dynamo#9727 — `feat(container)`: patched libfabric in aws stage Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 19, 2026
Bumps `nixl_ref: 0.10.1 → v1.1.0` in all four occurrences in container/context.yaml (one in `dynamo:` common section, one each in the `vllm:`, `sglang:`, `trtllm:` framework sections), and adds the Abseil source-build prereq that NIXL >= 1.0.0 requires. ### Why bump NIXL NIXL v1.1.0 brings four fixes that matter on AWS p6e-gb200 EFA fabric: ai-dynamo#1461 NUMA-aware EFA rail selection — without this, the LIBFABRIC backend assigns all initiator GPUs to a single rail and caps aggregate bandwidth at ~1.79 GB/s on p6e-gb200 instead of the expected ~190 GB/s with 4 GPUs + 4 EFA NICs per pod. ai-dynamo#1510 Active rail tracking for multi-rail concurrent transfers. ai-dynamo#1506 Multi-GPU memory-region fix (relevant for TP > 1 workers registering large VRAM blocks across GPUs). ai-dynamo#1433 Transfer handle repost notification fix. These gains were validated by yutwu (NVIDIA teammate) during the GLM-5.1 v7→v7.5 image bump on AWS p6e-gb200. For dynamo-trtllm disagg over EFA, NIXL 0.10.1's rail policy is the difference between "looks correct, caps at 1.79 GB/s" and "190 GB/s aggregate". ### The Abseil prereq NIXL >= 1.0.0 uses VLOG(1)/DVLOG(2) in nixl_log.h, which require Abseil >= 20240116. NIXL's meson.build first searches for system Abseil via pkg-config: absl_base_dep = dependency('absl_base', required: false) absl_log_dep = dependency('absl_log', required: false) ... if absl_base_dep.found() and not absl_log_dep.found() error('Your Abseil version is too old: found absl_base but missing support for absl_log. Cannot fallback to subproject because that would result in a mix of Abseil versions at runtime.') If pkg-config finds an old Abseil (absl_base present but no absl_log), NIXL HARD-ERRORS at configure time. Subproject fallback only triggers when NO system Abseil is found. This trips two ways for dynamo: 1. **wheel_builder (AlmaLinux 8 / manylinux_2_28)** typically lacks system Abseil, so subproject fallback would work — BUT the subproject builds shared libs with SONAMEs like libabsl_*.so.20250814. 2. **runtime images (cuda-dl-base Ubuntu 24.04)** ship stock libabsl-dev 20220623 (libabsl_*.so.20220623 SONAMEs). NIXL's libs built against subproject 20250814 can't dlopen against the runtime image's 20220623 — SONAME mismatch. The clean fix is to pre-install Abseil consistently at /usr/local in wheel_builder so meson uses it deterministically, then propagate the .so files to runtime stages. This matches yutwu's validated approach in the GLM-5.1 EFA-patch script (`install_abseil_from_source`). ### What changes 1. `container/context.yaml`: - Bump `nixl_ref: 0.10.1 → v1.1.0` in all 4 sections. - Add `dynamo.abseil_ref: 20240722.0` (the Abseil LTS yutwu validated). 2. `container/templates/args.Dockerfile`: - Declare `ARG ABSEIL_REF` (gated by `{% if device == "cuda" %}`, same as the other NIXL-related ARGs). 3. `container/templates/wheel_builder.Dockerfile`: - New RUN block BEFORE the NIXL clone+build that source-builds Abseil ${ABSEIL_REF} to /usr/local with `BUILD_SHARED_LIBS=ON` + `ABSL_ENABLE_INSTALL=ON`. - Gated by `pkg-config --exists absl_log` as a no-op if a future base image already ships a recent Abseil. - Validates `pkg-config --modversion absl_log` succeeds after install (build fails if not). 4. `container/templates/dynamo_runtime.Dockerfile`: - New COPY line bringing /usr/local/lib/libabsl_*.so* from wheel_builder so libnixl can resolve its Abseil deps at dlopen. 5. `container/templates/trtllm_runtime.Dockerfile`: - Same Abseil .so COPY. trtllm_runtime overrides the dynamo runtime stage, so it needs its own COPY independently. vllm and sglang frameworks are unaffected at the runtime level — they use upstream image NIXL packages (nixl-cu12 from vllm-openai, sglang's bundled NIXL), not dynamo's wheel_builder NIXL. Their wheel_builder stage still builds Abseil (gated by `device == "cuda"`), which is consistent with the existing pattern of building UCX/NIXL/etc. in wheel_builder regardless of whether the framework runtime uses them. ### Why source-build (and not apt-install a newer libabsl-dev) I evaluated `apt-install libabsl-dev` as an option. Ubuntu 24.04's main archive ships `libabsl-dev 20220623.1-1build1`, which is exactly the version NIXL hard-errors on. Backports / -updates / -proposed don't have a newer version. yutwu hit the same investigation and arrived at source-build as the only working path. If a maintainer knows of a repo (NVIDIA apt, PPA, etc.) with a newer libabsl-dev that works on Ubuntu 24.04 / cuda-dl-base, I'm happy to swap source-build for apt-install — the source-build adds ~2-3 min of build time per arch. ### Risk MEDIUM. This is a major version bump (0.x → 1.x) plus a new build dependency. Specifically: - The C++ libnixl.so ABI is the primary compat concern. Subproject fallback would have produced a 20250814 Abseil SONAME mismatch at runtime; this PR avoids that by pinning to 20240722.0 consistently across build + runtime stages. - Python NIXL bindings may have changed across 0.10.1 → v1.1.0. Dynamo's serving code that imports `nixl` needs verification. Worker init + a basic KV transfer smoke is sufficient. - Image size: +~30-40 MB for the Abseil shared libs in runtime stages (libabsl_log, libabsl_base, libabsl_strings, libabsl_status, libabsl_synchronization, libabsl_time, libabsl_flat_hash_map, etc. plus their dependency closure). - Build time: +~2-3 min in wheel_builder for the Abseil compile. ### Tag-format note `ai-dynamo/nixl` mixes tag conventions (older releases lack `v` prefix, newer ones use it): - Older releases: `0.1.1`, `0.10.0`, `0.10.1` (no `v` prefix) - Newer releases: `v1.1.0` (with `v` prefix) This bump uses `v1.1.0` to match the upstream tag. wheel_builder uses `git checkout ${NIXL_REF}` so the value must be exactly the tag name. ### What's needed before this can merge - [ ] `trtllm-pipeline` / `vllm-pipeline` / `sglang-pipeline` / `dynamo-pipeline` CI pass with both Abseil source-build and NIXL v1.1.0. - [ ] Manual smoke: rebuild a trtllm-runtime image, deploy a 1P1D Qwen3-8B disagg, confirm KV transfer works end-to-end. - [ ] (Stretch) `nixlbench` from inside the rebuilt image hits the expected ~190 GB/s on full-node p6e-gb200 allocation. ### References - NIXL v1.1.0 release: https://github.com/ai-dynamo/nixl/releases/tag/v1.1.0 - NIXL meson Abseil logic: https://github.com/ai-dynamo/nixl/blob/v1.1.0/meson.build (the hard-error block we're working around) Companion PRs: - ai-dynamo#9703 — `fix(container)`: ofi-nccl rm path + clearer HAS_TRTLLM_CONTEXT error - ai-dynamo#9704 — `feat(container)`: render.py --has-trtllm-context flag - ai-dynamo#9705 — `build(container)`: nixl_gdrcopy_ref v2.5.1 → v2.5.2 - this PR — `build(container)`: nixl_ref v1.1.0 + Abseil prereq - ai-dynamo#9727 — `feat(container)`: patched libfabric in aws stage Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
sara4dev
reviewed
May 19, 2026
yifjiang
force-pushed
the
yifjiang/patched-libfabric-in-aws-stage
branch
from
May 20, 2026 18:27
de43246 to
714af60
Compare
3 tasks
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 21, 2026
… GDRCopy known issues Three additions to docs/kubernetes/cloud-providers/eks/efa.md, all inside the existing Known Issues block: 1. Issue 1 (libfabric CUDA dmabuf): the existing workaround was missing three defensive steps that cause it to silently produce a broken image on GB200 — SONAME symlink force (make install does NOT overwrite the EFA installer's libfabric.so.1 → stock 1.30.x lookup wins), stock binary cleanup (defends against hardcoded RPATHs), and build-time fi_info --version validation (fails the build instead of failing in production). Added all three inline, with a Tracking PR pointer to ai-dynamo#9727 which upstreams the same fix. 2. Issue 2 (NEW): TRT-LLM rc14's libtensorrt_llm_nixl_wrapper.so is compiled against NIXL 0.9.x and references types dropped in NIXL ≥ 1.0 (nixlDescList<nixlBlobDesc>, <nixlBasicDesc>). Bumping dynamo's nixl_ref to v1.1.0 CrashLoopBackOffs every TRT-LLM disagg pod at executor init. Workaround: keep nixl_ref at 0.10.1; don't merge ai-dynamo#9706 until TRT-LLM upgrades its wrapper. 3. Issue 3 (NEW): GDRCopy v2.5.1 source RPM fails to compile against host kernel ≥ 6.15 (vm_flags_set redefinition). /dev/gdrdrv missing → GPU-Direct RDMA falls back to slower paths. Workaround: bump nixl_gdrcopy_ref to v2.5.2 (tracked in ai-dynamo#9705). Also added a summary table at the top of Known Issues for at-a-glance triage, and two new rows to the Common Failure Modes table mapping the new symptom signatures to Issues 2 and 3. Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 21, 2026
…el-6.15 known issue Two additions to docs/kubernetes/cloud-providers/eks/efa.md, both inside the existing Known Issues block: 1. Issue 1 (libfabric CUDA dmabuf): the existing workaround was missing three defensive steps that cause it to silently produce a broken image on GB200 — SONAME symlink force (make install does NOT overwrite the EFA installer's libfabric.so.1 → stock 1.30.x lookup wins), stock binary cleanup (defends against hardcoded RPATHs), and build-time fi_info --version validation (fails the build instead of failing in production). Added all three inline, with a Tracking PR pointer to ai-dynamo#9727 which upstreams the same fix. 2. Issue 2 (NEW): GDRCopy v2.5.1 source RPM fails to compile against host kernel ≥ 6.15 (vm_flags_set redefinition). /dev/gdrdrv missing → GPU-Direct RDMA falls back to slower paths. Workaround: bump nixl_gdrcopy_ref to v2.5.2 (tracked in ai-dynamo#9705). Also added a summary table at the top of Known Issues for at-a-glance triage, and one new row to the Common Failure Modes table mapping the new symptom signature to Issue 2. Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
Contributor
Author
|
/ok to test 714af60 |
yifjiang
force-pushed
the
yifjiang/patched-libfabric-in-aws-stage
branch
from
May 21, 2026 18:55
714af60 to
fa6c658
Compare
Contributor
Author
|
/ok to test fa6c658 |
yifjiang
force-pushed
the
yifjiang/patched-libfabric-in-aws-stage
branch
from
May 23, 2026 07:04
cdaa68a to
cdca347
Compare
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 23, 2026
…es are merged Removes the "Known Issues" section from docs/kubernetes/cloud-providers/eks/efa.md and prunes the two now-obsolete rows from "Common Failure Modes". Assumes ai-dynamo#9703, ai-dynamo#9704, ai-dynamo#9705, and ai-dynamo#9727 are all merged — after those land, the issues this section documented (GB200 fi_mr_reg(VRAM) failure on the EFA installer's stock libfabric, and GDRCopy v2.5.1 kmod build failure on kernel >= 6.15) no longer affect default --make-efa builds, so the inline workarounds the section provided would mislead readers. Also removes the ofiwg/libfabric#12019 reference from the bottom links list since it points at the same now-resolved upstream issue. Net diff: -34 / +1. Signed-off-by: Yifan Jiang <yifjiang@users.noreply.github.com>
Contributor
Author
|
@dillon-cullinan — addressed in the refactor at # aws.Dockerfile
RUN --mount=type=bind,from=wheel_builder,source=/opt/amazon/efa-patched,target=/tmp/patched-libfabric \
set -e && \
cp -Pf /tmp/patched-libfabric/lib/libfabric.so* /opt/amazon/efa/lib/ && \
cp -f /tmp/patched-libfabric/bin/fi_info /opt/amazon/efa/bin/fi_info && \
...Build deps ( Measured image-size impact (same-base
So total cost is <1 MB / +2 layers per arch — basically just the libfabric .so + fi_info binary copy, plus the LIBFABRIC plugin .so copy added by the v8 NIXL packaging fix. The PR description's earlier "+200-300 MB" estimate (from the pre-refactor inline build) is now corrected. Image refs used for the comparison are tagged |
erezzarum
reviewed
May 24, 2026
erezzarum
reviewed
May 24, 2026
erezzarum
reviewed
May 24, 2026
erezzarum
reviewed
May 24, 2026
erezzarum
reviewed
May 24, 2026
erezzarum
reviewed
May 24, 2026
Contributor
|
I have put some comments, something like this should work erezzarum@aff61e1 |
yifjiang
force-pushed
the
yifjiang/patched-libfabric-in-aws-stage
branch
from
May 26, 2026 17:26
fa07b0d to
6650b44
Compare
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 26, 2026
… + NIXL plugin consolidation Reuses the existing NIXL libfabric build in wheel_builder instead of adding a separate patched-libfabric compilation (per erezzarum's review on PR ai-dynamo#9727). Changes: - context.yaml: add nixl_libfabric_repo (makes the URL configurable); bump nixl_libfabric_ref from v2.3.0 to v2.5.1 (first release with the CUDA dmabuf try/fallback fix for GB200 EFA VRAM registration). - args.Dockerfile: declare NIXL_LIBFABRIC_REPO ARG in the CUDA block. - wheel_builder.Dockerfile: use $NIXL_LIBFABRIC_REPO instead of hardcoded URL in the existing libfabric git clone; add PKG_CONFIG_PATH for downstream consumers; add --setopt=tsflags=nocontexts to dnf installs (SELinux fix for hosts that lack the policy store). - aws.Dockerfile: version-compare bind-mount overlay from wheel_builder — copies libfabric into /usr/local/libfabric and registers with ldconfig ONLY if the wheel_builder-built version is newer than the EFA installer's stock. Auto no-op once EFA ships >= v2.5.x natively. Also: correct ofi-nccl rm path (previously targeted /opt/amazon/aws-ofi-nccl which doesn't exist; actual path is /opt/amazon/ofi-nccl). For trtllm: consolidate NIXL plugins (copy libplugin_LIBFABRIC.so from venv mesonpy into canonical plugins dir, symlink /opt/nvidia/nvda_nixl/plugins, set NIXL_PLUGIN_DIR, clear LD_PRELOAD). Remove redundant ENTRYPOINT/CMD (parent stages set their own). Image-size impact: <1 MB per arch (measured on v8/v9 builds). Signed-off-by: Yifan Jiang <yifjiang@users.noreply.github.com>
… + NIXL plugin consolidation Reuses the existing NIXL libfabric build in wheel_builder instead of adding a separate patched-libfabric compilation (per erezzarum's review on PR ai-dynamo#9727). Changes: - context.yaml: add nixl_libfabric_repo (makes the URL configurable); bump nixl_libfabric_ref from v2.3.0 to v2.5.1 (first release with the CUDA dmabuf try/fallback fix for GB200 EFA VRAM registration). - args.Dockerfile: declare NIXL_LIBFABRIC_REPO ARG in the CUDA block. - wheel_builder.Dockerfile: use $NIXL_LIBFABRIC_REPO instead of hardcoded URL in the existing libfabric git clone; add PKG_CONFIG_PATH for downstream consumers; add --setopt=tsflags=nocontexts to dnf installs (SELinux fix for hosts that lack the policy store). - aws.Dockerfile: version-compare bind-mount overlay from wheel_builder — copies libfabric into /usr/local/libfabric and registers with ldconfig ONLY if the wheel_builder-built version is newer than the EFA installer's stock. Auto no-op once EFA ships >= v2.5.x natively. Also: correct ofi-nccl rm path (previously targeted /opt/amazon/aws-ofi-nccl which doesn't exist; actual path is /opt/amazon/ofi-nccl). For trtllm: consolidate NIXL plugins (copy libplugin_LIBFABRIC.so from venv mesonpy into canonical plugins dir, symlink /opt/nvidia/nvda_nixl/plugins, set NIXL_PLUGIN_DIR, clear LD_PRELOAD). Remove redundant ENTRYPOINT/CMD (parent stages set their own). Image-size impact: <1 MB per arch (measured on v8/v9 builds). Signed-off-by: Yifan Jiang <yifjiang@users.noreply.github.com>
yifjiang
force-pushed
the
yifjiang/patched-libfabric-in-aws-stage
branch
from
May 26, 2026 18:06
6650b44 to
6edcd1d
Compare
Contributor
Author
|
Thanks @erezzarum — restructured in One adjustment vs your reference commit: the overlay copies directly into Also kept: NIXL plugin consolidation for trtllm (separate concern from libfabric — copies Diff is now +71/-9 across 4 files. Lychee failure is a pre-existing |
yifjiang
changed the title
dillon-cullinan
approved these changes
May 26, 2026
MartinRepo
pushed a commit
to MartinRepo/dynamo
that referenced
this pull request
May 27, 2026
…ai-dynamo#9727) Signed-off-by: Yifan Jiang <yifjiang@users.noreply.github.com> Co-authored-by: Yifan Jiang <yifjiang@users.noreply.github.com> Signed-off-by: Chi Xing <cxing@nvidia.com>
2 tasks
jh-nv
pushed a commit
that referenced
this pull request
Jun 8, 2026
…#9727) Signed-off-by: Yifan Jiang <yifjiang@users.noreply.github.com> Co-authored-by: Yifan Jiang <yifjiang@users.noreply.github.com> Signed-off-by: Jie Hao <jihao@nvidia.com>
saturley-hall
pushed a commit
that referenced
this pull request
Jun 11, 2026
…1.2.1 (#10425) Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com> Signed-off-by: Jie Hao <jihao@nvidia.com> Signed-off-by: Yifan Jiang <yifjiang@users.noreply.github.com> Signed-off-by: Erez Zarum <erezz@amazon.com> Co-authored-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com> Co-authored-by: Yifan Jiang <yifjiang@users.noreply.github.com> Co-authored-by: Erez Zarum <erezzarum@users.noreply.github.com>
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds upstream
ofiwg/libfabricv2.5.1 (the first release with the CUDA dmabuf try/fallback fix for GB200 EFA) as part of the existing NIXL libfabric compilation inwheel_builder, and overlays it onto the EFA installer's stock binary in theawsstage — only when the built version is newer than stock. Auto no-op once a future EFA installer ships libfabric >= v2.5.x.Also: consolidates the NIXL LIBFABRIC plugin into the canonical plugins directory for trtllm (post-mesonpy-refactor fix), corrects the ofi-nccl rm path, and adds a SELinux build fix for manylinux hosts.
What changes
container/context.yamlnixl_libfabric_repo(makes URL configurable); bumpnixl_libfabric_refv2.3.0 → v2.5.1.container/templates/args.DockerfileNIXL_LIBFABRIC_REPOARG in the CUDA block.container/templates/wheel_builder.Dockerfile$NIXL_LIBFABRIC_REPOinstead of hardcoded URL in the existing libfabricgit clone; addPKG_CONFIG_PATH; add--setopt=tsflags=nocontextsto dnf installs (SELinux fix).container/templates/aws.Dockerfile/usr/local/libfabric, compare vialibfabric.pc, copy into/opt/amazon/efa/{lib,lib64,bin}only if newer. Correct ofi-nccl rm path. For trtllm: consolidate NIXL plugins (libplugin_LIBFABRIC.so→ canonical dir + symlink), setNIXL_PLUGIN_DIR, clearLD_PRELOAD. Remove redundantENTRYPOINT/CMD.Non-EFA images unchanged. The libfabric version bump affects the NIXL build (which already compiled libfabric); the aws-stage overlay and plugin consolidation only render with
--make-efa.Diff: +71/-9 across 4 files.
Why
The EFA installer (through 1.48.x) ships stock libfabric in the
2.4.0amznline. Its CUDA HMEM path falls through toibv_reg_mr()with a GPU VA and returns EFAULT on GB200.ofiwg/libfabricv2.5.x carries the fix (prov/efa: Implement dmabuf try/fallback logic). No sed-patch needed.Design (per erezzarum's review)
wheel_builder— just makes the repo URL configurable (NIXL_LIBFABRIC_REPO) and bumps the version. No separate{% if make_efa %}build block.libfabric.pcversion, compares withsort -V, and copies only if the wheel_builder version is newer. When a future EFA installer ships >= v2.5.x natively, the overlay becomes a no-op automatically.libfabric.so*+fi_infodirectly into/opt/amazon/efa/{lib,lib64,bin}(not a separate/usr/local/libfabric+ ldconfig — that approach did not reliably win SONAME resolution at runtime due to EFA binary RPATHs).Companion changes in this PR
rm -rf /opt/amazon/aws-ofi-nccl /opt/amazon/ofi-nccl /etc/ld.so.conf.d/aws-ofi-nccl.conf(the oldrmtargeted a path that didn't exist).libplugin_LIBFABRIC.solands under the Dynamo venv while GDS/UCX/POSIX stay at the arch-specific NIXL dir. Copies LIBFABRIC alongside the others, symlinks/opt/nvidia/nvda_nixl/plugins, setsNIXL_PLUGIN_DIR, clearsLD_PRELOAD(the upstream trtllm_runtime ci(trtllm): switch TRT-LLM container to upstream base image #9654 workaround).--setopt=tsflags=nocontextson wheel_builder dnf installs (hosts without the SELinux policy store fail otherwise).Validation
Built and smoke-tested on both arches (v10.1):
fi_info --versionreportslibfabric: 2.5.1[aws] libfabric overlay: 2.5.1 (overwrites EFA stock 2.4.0amzn1.0)[aws] NIXL plugins consolidated under .../pluginsNIXL_PLUGIN_DIRset, is dir, containslibplugin_LIBFABRIC.soLD_PRELOADemptynixlDescList<nixlBlobDesc>+<nixlBasicDesc>/opt/amazon/ofi-ncclremovedImage-size impact: <1 MB per arch (measured vs same-base
--make-efabuild without these changes).Risk
LOW. The libfabric version bump (v2.3.0 → v2.5.1) affects the NIXL build globally but v2.5.1 is ABI-compatible. The aws-stage overlay and plugin consolidation are additive and gated. Non-EFA paths unchanged.
Test plan
fi_info --versionreportslibfabric: 2.5.1(not stock2.4.0amzn1.0)probe_nixl_libfabric_vram.py, no env overrides)Signed-off-by: Yifan Jiang yifjiang@users.noreply.github.com