WIP - #1506
Closed
alec-flowers wants to merge 1 commit into
Closed
WIP#1506alec-flowers wants to merge 1 commit into
alec-flowers wants to merge 1 commit into
Conversation
Contributor
|
As you work on this - just also want to shout out the following PR I like this idea of having |
Contributor
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
Contributor
|
This PR has been closed due to inactivity. If you believe this PR is still relevant, please feel free to reopen it with additional context or information. |
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 19, 2026
Bumps `nixl_ref: 0.10.1 → v1.1.0` in all four occurrences in container/context.yaml (one in `dynamo:` common section, one each in the `vllm:`, `sglang:`, `trtllm:` framework sections). NIXL v1.1.0 brings four fixes that matter on AWS p6e-gb200 EFA fabric: ai-dynamo#1461 NUMA-aware EFA rail selection — without this, the LIBFABRIC backend assigns all initiator GPUs to a single rail and caps aggregate bandwidth at ~1.79 GB/s on p6e-gb200 instead of the expected ~190 GB/s with 4 GPUs + 4 EFA NICs per pod. ai-dynamo#1510 Active rail tracking for multi-rail concurrent transfers. ai-dynamo#1506 Multi-GPU memory-region fix (relevant for TP > 1 workers registering large VRAM blocks across GPUs). ai-dynamo#1433 Transfer handle repost notification fix. These performance gains were validated by yutwu (NVIDIA teammate) during the GLM-5.1 v7→v7.5 image bump on AWS p6e-gb200. For dynamo-trtllm disagg over EFA, NIXL 0.10.1's rail policy is the difference between "looks correct, caps at 1.79 GB/s" and "190 GB/s aggregate". ### Note on the version-tag format `ai-dynamo/nixl` tags are mixed: - Older releases: `0.1.1`, `0.10.0`, `0.10.1` (no `v` prefix) - Newer releases: `v1.1.0` (with `v` prefix) This bump uses `v1.1.0` to match the upstream tag. Dynamo's wheel_builder.Dockerfile clones via `git checkout ${NIXL_REF}` so the ref must be exactly the tag name. Verified the `v1.1.0` tag exists at https://github.com/ai-dynamo/nixl/releases/tag/v1.1.0. ### Risk MEDIUM. This is a major version bump (0.x → 1.x). Specifically: - The C++ `libnixl.so` ABI is the primary compat concern. NIXL 1.1.0 introduced Abseil >= 20240116 as a build dependency (VLOG/absl_log). Dynamo's wheel_builder uses Ubuntu 24.04 system Abseil 20220623, which lacks these symbols. The EFA patch script in our image-build path (yutwu's `install_efa_libfabric_nixl.sh`) builds a newer Abseil from source as a workaround; dynamo's wheel_builder needs the same. Either: (a) Add `libabsl-dev` >= 20240116 to wheel_builder.Dockerfile's apt install list (Ubuntu 24.04.4 LTS or backports may have it). (b) Build Abseil from source in wheel_builder.Dockerfile similar to how we build libfabric. (c) Use the system Abseil and patch NIXL 1.1.0 to not require absl_log — non-starter, upstream change. This PR does NOT include the Abseil bump — it's only the version pin. Maintainers will need to validate the build and add the Abseil dep in a coordinated change (or a follow-up PR I can write once reviewers confirm option a/b). - dynamo's Python NIXL bindings (`nixl-cu13` wheel) may have API changes between 0.10.1 and 1.1.0. Dynamo's serving code that imports `nixl` needs to be verified compatible. A quick smoke (worker init + KV transfer) is sufficient. - dynamo's plugin loader behavior may differ. The fix2 image-build cycle hit a related issue where dynamo's NIXL 0.10.1 plugin's Abseil ABI conflicted with NIXL 1.1.0's plugin in the SAME image (the `install_efa_libfabric_nixl.sh` post-process layered NIXL 1.1.0 nixlbench on top of dynamo's bundled 0.10.1). With this bump, dynamo's serving plugin AND nixlbench's plugin both use NIXL 1.1.0 → that conflict goes away. Marking this PR as **draft** until: 1. Maintainer agrees on the Abseil dep approach (a/b/c above). 2. `trtllm-pipeline` / `vllm-pipeline` / `sglang-pipeline` CI runs pass with the bump. 3. A quick smoke of dynamo serving with the rebuilt NIXL confirms no API break. I can iterate as needed. ### References - NIXL v1.1.0 release: https://github.com/ai-dynamo/nixl/releases/tag/v1.1.0 - Companion PRs: ai-dynamo#9703 (template fixes), ai-dynamo#9704 (render --has-trtllm-context), ai-dynamo#9705 (gdrcopy v2.5.2 bump) Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 19, 2026
Bumps `nixl_ref: 0.10.1 → v1.1.0` in all four occurrences in container/context.yaml (one in `dynamo:` common section, one each in the `vllm:`, `sglang:`, `trtllm:` framework sections), and adds the Abseil source-build prereq that NIXL >= 1.0.0 requires. ### Why bump NIXL NIXL v1.1.0 brings four fixes that matter on AWS p6e-gb200 EFA fabric: ai-dynamo#1461 NUMA-aware EFA rail selection — without this, the LIBFABRIC backend assigns all initiator GPUs to a single rail and caps aggregate bandwidth at ~1.79 GB/s on p6e-gb200 instead of the expected ~190 GB/s with 4 GPUs + 4 EFA NICs per pod. ai-dynamo#1510 Active rail tracking for multi-rail concurrent transfers. ai-dynamo#1506 Multi-GPU memory-region fix (relevant for TP > 1 workers registering large VRAM blocks across GPUs). ai-dynamo#1433 Transfer handle repost notification fix. These gains were validated by yutwu (NVIDIA teammate) during the GLM-5.1 v7→v7.5 image bump on AWS p6e-gb200. For dynamo-trtllm disagg over EFA, NIXL 0.10.1's rail policy is the difference between "looks correct, caps at 1.79 GB/s" and "190 GB/s aggregate". ### The Abseil prereq NIXL >= 1.0.0 uses VLOG(1)/DVLOG(2) in nixl_log.h, which require Abseil >= 20240116. NIXL's meson.build first searches for system Abseil via pkg-config: absl_base_dep = dependency('absl_base', required: false) absl_log_dep = dependency('absl_log', required: false) ... if absl_base_dep.found() and not absl_log_dep.found() error('Your Abseil version is too old: found absl_base but missing support for absl_log. Cannot fallback to subproject because that would result in a mix of Abseil versions at runtime.') If pkg-config finds an old Abseil (absl_base present but no absl_log), NIXL HARD-ERRORS at configure time. Subproject fallback only triggers when NO system Abseil is found. This trips two ways for dynamo: 1. **wheel_builder (AlmaLinux 8 / manylinux_2_28)** typically lacks system Abseil, so subproject fallback would work — BUT the subproject builds shared libs with SONAMEs like libabsl_*.so.20250814. 2. **runtime images (cuda-dl-base Ubuntu 24.04)** ship stock libabsl-dev 20220623 (libabsl_*.so.20220623 SONAMEs). NIXL's libs built against subproject 20250814 can't dlopen against the runtime image's 20220623 — SONAME mismatch. The clean fix is to pre-install Abseil consistently at /usr/local in wheel_builder so meson uses it deterministically, then propagate the .so files to runtime stages. This matches yutwu's validated approach in the GLM-5.1 EFA-patch script (`install_abseil_from_source`). ### What changes 1. `container/context.yaml`: - Bump `nixl_ref: 0.10.1 → v1.1.0` in all 4 sections. - Add `dynamo.abseil_ref: 20240722.0` (the Abseil LTS yutwu validated). 2. `container/templates/args.Dockerfile`: - Declare `ARG ABSEIL_REF` (gated by `{% if device == "cuda" %}`, same as the other NIXL-related ARGs). 3. `container/templates/wheel_builder.Dockerfile`: - New RUN block BEFORE the NIXL clone+build that source-builds Abseil ${ABSEIL_REF} to /usr/local with `BUILD_SHARED_LIBS=ON` + `ABSL_ENABLE_INSTALL=ON`. - Gated by `pkg-config --exists absl_log` as a no-op if a future base image already ships a recent Abseil. - Validates `pkg-config --modversion absl_log` succeeds after install (build fails if not). 4. `container/templates/dynamo_runtime.Dockerfile`: - New COPY line bringing /usr/local/lib/libabsl_*.so* from wheel_builder so libnixl can resolve its Abseil deps at dlopen. 5. `container/templates/trtllm_runtime.Dockerfile`: - Same Abseil .so COPY. trtllm_runtime overrides the dynamo runtime stage, so it needs its own COPY independently. vllm and sglang frameworks are unaffected at the runtime level — they use upstream image NIXL packages (nixl-cu12 from vllm-openai, sglang's bundled NIXL), not dynamo's wheel_builder NIXL. Their wheel_builder stage still builds Abseil (gated by `device == "cuda"`), which is consistent with the existing pattern of building UCX/NIXL/etc. in wheel_builder regardless of whether the framework runtime uses them. ### Why source-build (and not apt-install a newer libabsl-dev) I evaluated `apt-install libabsl-dev` as an option. Ubuntu 24.04's main archive ships `libabsl-dev 20220623.1-1build1`, which is exactly the version NIXL hard-errors on. Backports / -updates / -proposed don't have a newer version. yutwu hit the same investigation and arrived at source-build as the only working path. If a maintainer knows of a repo (NVIDIA apt, PPA, etc.) with a newer libabsl-dev that works on Ubuntu 24.04 / cuda-dl-base, I'm happy to swap source-build for apt-install — the source-build adds ~2-3 min of build time per arch. ### Risk MEDIUM. This is a major version bump (0.x → 1.x) plus a new build dependency. Specifically: - The C++ libnixl.so ABI is the primary compat concern. Subproject fallback would have produced a 20250814 Abseil SONAME mismatch at runtime; this PR avoids that by pinning to 20240722.0 consistently across build + runtime stages. - Python NIXL bindings may have changed across 0.10.1 → v1.1.0. Dynamo's serving code that imports `nixl` needs verification. Worker init + a basic KV transfer smoke is sufficient. - Image size: +~30-40 MB for the Abseil shared libs in runtime stages (libabsl_log, libabsl_base, libabsl_strings, libabsl_status, libabsl_synchronization, libabsl_time, libabsl_flat_hash_map, etc. plus their dependency closure). - Build time: +~2-3 min in wheel_builder for the Abseil compile. ### Tag-format note `ai-dynamo/nixl` mixes tag conventions (older releases lack `v` prefix, newer ones use it): - Older releases: `0.1.1`, `0.10.0`, `0.10.1` (no `v` prefix) - Newer releases: `v1.1.0` (with `v` prefix) This bump uses `v1.1.0` to match the upstream tag. wheel_builder uses `git checkout ${NIXL_REF}` so the value must be exactly the tag name. ### What's needed before this can merge - [ ] `trtllm-pipeline` / `vllm-pipeline` / `sglang-pipeline` / `dynamo-pipeline` CI pass with both Abseil source-build and NIXL v1.1.0. - [ ] Manual smoke: rebuild a trtllm-runtime image, deploy a 1P1D Qwen3-8B disagg, confirm KV transfer works end-to-end. - [ ] (Stretch) `nixlbench` from inside the rebuilt image hits the expected ~190 GB/s on full-node p6e-gb200 allocation. ### References - NIXL v1.1.0 release: https://github.com/ai-dynamo/nixl/releases/tag/v1.1.0 - NIXL meson Abseil logic: https://github.com/ai-dynamo/nixl/blob/v1.1.0/meson.build (the hard-error block we're working around) Companion PRs: - ai-dynamo#9703 — `fix(container)`: ofi-nccl rm path + clearer HAS_TRTLLM_CONTEXT error - ai-dynamo#9704 — `feat(container)`: render.py --has-trtllm-context flag - ai-dynamo#9705 — `build(container)`: nixl_gdrcopy_ref v2.5.1 → v2.5.2 - this PR — `build(container)`: nixl_ref v1.1.0 + Abseil prereq - ai-dynamo#9727 — `feat(container)`: patched libfabric in aws stage Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 19, 2026
Bumps `nixl_ref: 0.10.1 → v1.1.0` in all four occurrences in container/context.yaml (one in `dynamo:` common section, one each in the `vllm:`, `sglang:`, `trtllm:` framework sections), and adds the Abseil source-build prereq that NIXL >= 1.0.0 requires. ### Why bump NIXL NIXL v1.1.0 brings four fixes that matter on AWS p6e-gb200 EFA fabric: ai-dynamo#1461 NUMA-aware EFA rail selection — without this, the LIBFABRIC backend assigns all initiator GPUs to a single rail and caps aggregate bandwidth at ~1.79 GB/s on p6e-gb200 instead of the expected ~190 GB/s with 4 GPUs + 4 EFA NICs per pod. ai-dynamo#1510 Active rail tracking for multi-rail concurrent transfers. ai-dynamo#1506 Multi-GPU memory-region fix (relevant for TP > 1 workers registering large VRAM blocks across GPUs). ai-dynamo#1433 Transfer handle repost notification fix. These gains were validated by yutwu (NVIDIA teammate) during the GLM-5.1 v7→v7.5 image bump on AWS p6e-gb200. For dynamo-trtllm disagg over EFA, NIXL 0.10.1's rail policy is the difference between "looks correct, caps at 1.79 GB/s" and "190 GB/s aggregate". ### The Abseil prereq NIXL >= 1.0.0 uses VLOG(1)/DVLOG(2) in nixl_log.h, which require Abseil >= 20240116. NIXL's meson.build first searches for system Abseil via pkg-config: absl_base_dep = dependency('absl_base', required: false) absl_log_dep = dependency('absl_log', required: false) ... if absl_base_dep.found() and not absl_log_dep.found() error('Your Abseil version is too old: found absl_base but missing support for absl_log. Cannot fallback to subproject because that would result in a mix of Abseil versions at runtime.') If pkg-config finds an old Abseil (absl_base present but no absl_log), NIXL HARD-ERRORS at configure time. Subproject fallback only triggers when NO system Abseil is found. This trips two ways for dynamo: 1. **wheel_builder (AlmaLinux 8 / manylinux_2_28)** typically lacks system Abseil, so subproject fallback would work — BUT the subproject builds shared libs with SONAMEs like libabsl_*.so.20250814. 2. **runtime images (cuda-dl-base Ubuntu 24.04)** ship stock libabsl-dev 20220623 (libabsl_*.so.20220623 SONAMEs). NIXL's libs built against subproject 20250814 can't dlopen against the runtime image's 20220623 — SONAME mismatch. The clean fix is to pre-install Abseil consistently at /usr/local in wheel_builder so meson uses it deterministically, then propagate the .so files to runtime stages. This matches yutwu's validated approach in the GLM-5.1 EFA-patch script (`install_abseil_from_source`). ### What changes 1. `container/context.yaml`: - Bump `nixl_ref: 0.10.1 → v1.1.0` in all 4 sections. - Add `dynamo.abseil_ref: 20240722.0` (the Abseil LTS yutwu validated). 2. `container/templates/args.Dockerfile`: - Declare `ARG ABSEIL_REF` (gated by `{% if device == "cuda" %}`, same as the other NIXL-related ARGs). 3. `container/templates/wheel_builder.Dockerfile`: - New RUN block BEFORE the NIXL clone+build that source-builds Abseil ${ABSEIL_REF} to /usr/local with `BUILD_SHARED_LIBS=ON` + `ABSL_ENABLE_INSTALL=ON`. - Gated by `pkg-config --exists absl_log` as a no-op if a future base image already ships a recent Abseil. - Validates `pkg-config --modversion absl_log` succeeds after install (build fails if not). 4. `container/templates/dynamo_runtime.Dockerfile`: - New COPY line bringing /usr/local/lib/libabsl_*.so* from wheel_builder so libnixl can resolve its Abseil deps at dlopen. 5. `container/templates/trtllm_runtime.Dockerfile`: - Same Abseil .so COPY. trtllm_runtime overrides the dynamo runtime stage, so it needs its own COPY independently. vllm and sglang frameworks are unaffected at the runtime level — they use upstream image NIXL packages (nixl-cu12 from vllm-openai, sglang's bundled NIXL), not dynamo's wheel_builder NIXL. Their wheel_builder stage still builds Abseil (gated by `device == "cuda"`), which is consistent with the existing pattern of building UCX/NIXL/etc. in wheel_builder regardless of whether the framework runtime uses them. ### Why source-build (and not apt-install a newer libabsl-dev) I evaluated `apt-install libabsl-dev` as an option. Ubuntu 24.04's main archive ships `libabsl-dev 20220623.1-1build1`, which is exactly the version NIXL hard-errors on. Backports / -updates / -proposed don't have a newer version. yutwu hit the same investigation and arrived at source-build as the only working path. If a maintainer knows of a repo (NVIDIA apt, PPA, etc.) with a newer libabsl-dev that works on Ubuntu 24.04 / cuda-dl-base, I'm happy to swap source-build for apt-install — the source-build adds ~2-3 min of build time per arch. ### Risk MEDIUM. This is a major version bump (0.x → 1.x) plus a new build dependency. Specifically: - The C++ libnixl.so ABI is the primary compat concern. Subproject fallback would have produced a 20250814 Abseil SONAME mismatch at runtime; this PR avoids that by pinning to 20240722.0 consistently across build + runtime stages. - Python NIXL bindings may have changed across 0.10.1 → v1.1.0. Dynamo's serving code that imports `nixl` needs verification. Worker init + a basic KV transfer smoke is sufficient. - Image size: +~30-40 MB for the Abseil shared libs in runtime stages (libabsl_log, libabsl_base, libabsl_strings, libabsl_status, libabsl_synchronization, libabsl_time, libabsl_flat_hash_map, etc. plus their dependency closure). - Build time: +~2-3 min in wheel_builder for the Abseil compile. ### Tag-format note `ai-dynamo/nixl` mixes tag conventions (older releases lack `v` prefix, newer ones use it): - Older releases: `0.1.1`, `0.10.0`, `0.10.1` (no `v` prefix) - Newer releases: `v1.1.0` (with `v` prefix) This bump uses `v1.1.0` to match the upstream tag. wheel_builder uses `git checkout ${NIXL_REF}` so the value must be exactly the tag name. ### What's needed before this can merge - [ ] `trtllm-pipeline` / `vllm-pipeline` / `sglang-pipeline` / `dynamo-pipeline` CI pass with both Abseil source-build and NIXL v1.1.0. - [ ] Manual smoke: rebuild a trtllm-runtime image, deploy a 1P1D Qwen3-8B disagg, confirm KV transfer works end-to-end. - [ ] (Stretch) `nixlbench` from inside the rebuilt image hits the expected ~190 GB/s on full-node p6e-gb200 allocation. ### References - NIXL v1.1.0 release: https://github.com/ai-dynamo/nixl/releases/tag/v1.1.0 - NIXL meson Abseil logic: https://github.com/ai-dynamo/nixl/blob/v1.1.0/meson.build (the hard-error block we're working around) Companion PRs: - ai-dynamo#9703 — `fix(container)`: ofi-nccl rm path + clearer HAS_TRTLLM_CONTEXT error - ai-dynamo#9704 — `feat(container)`: render.py --has-trtllm-context flag - ai-dynamo#9705 — `build(container)`: nixl_gdrcopy_ref v2.5.1 → v2.5.2 - this PR — `build(container)`: nixl_ref v1.1.0 + Abseil prereq - ai-dynamo#9727 — `feat(container)`: patched libfabric in aws stage Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 19, 2026
Bumps `nixl_ref: 0.10.1 → v1.1.0` in all four occurrences in container/context.yaml (one in `dynamo:` common section, one each in the `vllm:`, `sglang:`, `trtllm:` framework sections), and adds the Abseil source-build prereq that NIXL >= 1.0.0 requires. ### Why bump NIXL NIXL v1.1.0 brings four fixes that matter on AWS p6e-gb200 EFA fabric: ai-dynamo#1461 NUMA-aware EFA rail selection — without this, the LIBFABRIC backend assigns all initiator GPUs to a single rail and caps aggregate bandwidth at ~1.79 GB/s on p6e-gb200 instead of the expected ~190 GB/s with 4 GPUs + 4 EFA NICs per pod. ai-dynamo#1510 Active rail tracking for multi-rail concurrent transfers. ai-dynamo#1506 Multi-GPU memory-region fix (relevant for TP > 1 workers registering large VRAM blocks across GPUs). ai-dynamo#1433 Transfer handle repost notification fix. These gains were validated by yutwu (NVIDIA teammate) during the GLM-5.1 v7→v7.5 image bump on AWS p6e-gb200. For dynamo-trtllm disagg over EFA, NIXL 0.10.1's rail policy is the difference between "looks correct, caps at 1.79 GB/s" and "190 GB/s aggregate". ### The Abseil prereq NIXL >= 1.0.0 uses VLOG(1)/DVLOG(2) in nixl_log.h, which require Abseil >= 20240116. NIXL's meson.build first searches for system Abseil via pkg-config: absl_base_dep = dependency('absl_base', required: false) absl_log_dep = dependency('absl_log', required: false) ... if absl_base_dep.found() and not absl_log_dep.found() error('Your Abseil version is too old: found absl_base but missing support for absl_log. Cannot fallback to subproject because that would result in a mix of Abseil versions at runtime.') If pkg-config finds an old Abseil (absl_base present but no absl_log), NIXL HARD-ERRORS at configure time. Subproject fallback only triggers when NO system Abseil is found. This trips two ways for dynamo: 1. **wheel_builder (AlmaLinux 8 / manylinux_2_28)** typically lacks system Abseil, so subproject fallback would work — BUT the subproject builds shared libs with SONAMEs like libabsl_*.so.20250814. 2. **runtime images (cuda-dl-base Ubuntu 24.04)** ship stock libabsl-dev 20220623 (libabsl_*.so.20220623 SONAMEs). NIXL's libs built against subproject 20250814 can't dlopen against the runtime image's 20220623 — SONAME mismatch. The clean fix is to pre-install Abseil consistently at /usr/local in wheel_builder so meson uses it deterministically, then propagate the .so files to runtime stages. This matches yutwu's validated approach in the GLM-5.1 EFA-patch script (`install_abseil_from_source`). ### What changes 1. `container/context.yaml`: - Bump `nixl_ref: 0.10.1 → v1.1.0` in all 4 sections. - Add `dynamo.abseil_ref: 20240722.0` (the Abseil LTS yutwu validated). 2. `container/templates/args.Dockerfile`: - Declare `ARG ABSEIL_REF` (gated by `{% if device == "cuda" %}`, same as the other NIXL-related ARGs). 3. `container/templates/wheel_builder.Dockerfile`: - New RUN block BEFORE the NIXL clone+build that source-builds Abseil ${ABSEIL_REF} to /usr/local with `BUILD_SHARED_LIBS=ON` + `ABSL_ENABLE_INSTALL=ON`. - Gated by `pkg-config --exists absl_log` as a no-op if a future base image already ships a recent Abseil. - Validates `pkg-config --modversion absl_log` succeeds after install (build fails if not). 4. `container/templates/dynamo_runtime.Dockerfile`: - New COPY line bringing /usr/local/lib/libabsl_*.so* from wheel_builder so libnixl can resolve its Abseil deps at dlopen. 5. `container/templates/trtllm_runtime.Dockerfile`: - Same Abseil .so COPY. trtllm_runtime overrides the dynamo runtime stage, so it needs its own COPY independently. vllm and sglang frameworks are unaffected at the runtime level — they use upstream image NIXL packages (nixl-cu12 from vllm-openai, sglang's bundled NIXL), not dynamo's wheel_builder NIXL. Their wheel_builder stage still builds Abseil (gated by `device == "cuda"`), which is consistent with the existing pattern of building UCX/NIXL/etc. in wheel_builder regardless of whether the framework runtime uses them. ### Why source-build (and not apt-install a newer libabsl-dev) I evaluated `apt-install libabsl-dev` as an option. Ubuntu 24.04's main archive ships `libabsl-dev 20220623.1-1build1`, which is exactly the version NIXL hard-errors on. Backports / -updates / -proposed don't have a newer version. yutwu hit the same investigation and arrived at source-build as the only working path. If a maintainer knows of a repo (NVIDIA apt, PPA, etc.) with a newer libabsl-dev that works on Ubuntu 24.04 / cuda-dl-base, I'm happy to swap source-build for apt-install — the source-build adds ~2-3 min of build time per arch. ### Risk MEDIUM. This is a major version bump (0.x → 1.x) plus a new build dependency. Specifically: - The C++ libnixl.so ABI is the primary compat concern. Subproject fallback would have produced a 20250814 Abseil SONAME mismatch at runtime; this PR avoids that by pinning to 20240722.0 consistently across build + runtime stages. - Python NIXL bindings may have changed across 0.10.1 → v1.1.0. Dynamo's serving code that imports `nixl` needs verification. Worker init + a basic KV transfer smoke is sufficient. - Image size: +~30-40 MB for the Abseil shared libs in runtime stages (libabsl_log, libabsl_base, libabsl_strings, libabsl_status, libabsl_synchronization, libabsl_time, libabsl_flat_hash_map, etc. plus their dependency closure). - Build time: +~2-3 min in wheel_builder for the Abseil compile. ### Tag-format note `ai-dynamo/nixl` mixes tag conventions (older releases lack `v` prefix, newer ones use it): - Older releases: `0.1.1`, `0.10.0`, `0.10.1` (no `v` prefix) - Newer releases: `v1.1.0` (with `v` prefix) This bump uses `v1.1.0` to match the upstream tag. wheel_builder uses `git checkout ${NIXL_REF}` so the value must be exactly the tag name. ### What's needed before this can merge - [ ] `trtllm-pipeline` / `vllm-pipeline` / `sglang-pipeline` / `dynamo-pipeline` CI pass with both Abseil source-build and NIXL v1.1.0. - [ ] Manual smoke: rebuild a trtllm-runtime image, deploy a 1P1D Qwen3-8B disagg, confirm KV transfer works end-to-end. - [ ] (Stretch) `nixlbench` from inside the rebuilt image hits the expected ~190 GB/s on full-node p6e-gb200 allocation. ### References - NIXL v1.1.0 release: https://github.com/ai-dynamo/nixl/releases/tag/v1.1.0 - NIXL meson Abseil logic: https://github.com/ai-dynamo/nixl/blob/v1.1.0/meson.build (the hard-error block we're working around) Companion PRs: - ai-dynamo#9703 — `fix(container)`: ofi-nccl rm path + clearer HAS_TRTLLM_CONTEXT error - ai-dynamo#9704 — `feat(container)`: render.py --has-trtllm-context flag - ai-dynamo#9705 — `build(container)`: nixl_gdrcopy_ref v2.5.1 → v2.5.2 - this PR — `build(container)`: nixl_ref v1.1.0 + Abseil prereq - ai-dynamo#9727 — `feat(container)`: patched libfabric in aws stage Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 19, 2026
NIXL 1.1.0 brings the rail-selection improvements (ai-dynamo#1461, ai-dynamo#1506, ai-dynamo#1510) needed for p6e-gb200 multi-rail performance. The bump is a 1-line context.yaml change. Abseil 20240116+ is required by NIXL 1.1.0 but wheel_builder uses manylinux_2_28 (no system Abseil), so meson falls through to NIXL's bundled subprojects/abseil-cpp.wrap (20250814.1) and statically links it into libnixl.so. No system-wide Abseil install or runtime propagation needed. Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
yifjiang
added a commit
to yifjiang/dynamo
that referenced
this pull request
May 19, 2026
NIXL 1.1.0 brings the rail-selection improvements (ai-dynamo#1461, ai-dynamo#1506, ai-dynamo#1510) needed for p6e-gb200 multi-rail performance. The bump is a 1-line context.yaml change. Abseil 20240116+ is required by NIXL 1.1.0 (NIXL's meson.build calls absl::LogMessage and absl_log). wheel_builder uses manylinux_2_28 as its base (no system Abseil installed), so NIXL's meson falls through to its bundled subprojects/abseil-cpp.wrap (currently Abseil 20250814.1) and statically links Abseil into libnixl.so. No system-wide Abseil install is needed and no Abseil propagation to runtime stages is required — libnixl.so is self-contained on Abseil. Tested: - render trtllm runtime with --make-efa, build wheel_builder stage, confirm libnixl.so loads without external libabsl_*.so dependency - rendered Dockerfile content unchanged for non-NIXL stages Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview:
Details:
Where should the reviewer start?
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)