fix(container): cherry-pick SGLang UCX compatibility (#12558) - #12647
Conversation
…2558) Signed-off-by: Zhuangcheng(Jesse) Gu <zcgu@connect.hku.hk>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if [[ "${#PLUGINS[@]}" -eq 0 ]] && \ | ||
| [[ "${NIXL_UCX_COMPAT_ALLOW_NO_CONSUMER:-0}" != "1" ]]; then | ||
| die "no NVSHMEM UCX transport found; aliases were not validated against a consumer;" \ | ||
| "set NIXL_UCX_COMPAT_ALLOW_NO_CONSUMER=1 to accept this" | ||
| fi |
There was a problem hiding this comment.
🔍 Build fails hard when no NVSHMEM UCX transport is present
If the upstream SGLang runtime image does not ship an nvidia-nvshmem-cu* wheel with nvshmem_transport_ucx.so, PLUGINS is empty and the script calls die unless NIXL_UCX_COMPAT_ALLOW_NO_CONSUMER=1 is set. The Dockerfile invocation (container/templates/sglang_runtime.Dockerfile:176-178) never sets that variable, so any base-image variant (or a future upstream change) that drops NVSHMEM will break the entire non-xpu sglang runtime build rather than just skipping the validation. Validation in the description was only performed on GB200/aarch64; worth confirming the x86 base image also installs the NVSHMEM UCX transport, or setting the escape hatch in the Dockerfile.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # Keep the generic names beside NIXL's private libraries. Loading libucp or | ||
| # libucs through a separate symlink directory changes $ORIGIN and prevents UCX | ||
| # from finding both its hashed core dependencies and its ucx/ module directory. | ||
| # These aliases are not tracked by the wheel's RECORD, so a later NIXL uninstall | ||
| # or upgrade can leave stale or dangling links. Run this after the final install | ||
| # step that can modify the nixl-cu* wheel. | ||
| for alias in "${!ALIASES[@]}"; do | ||
| target="${ALIASES[${alias}]}" | ||
| [[ -f "${target}" ]] || die "alias target is missing: ${target}" | ||
| [[ "${target%/*}" == "${NIXL_LIB_DIR}" ]] || \ | ||
| die "alias target is outside the NIXL private library directory: ${target}" | ||
| alias_path="${NIXL_LIB_DIR}/${alias}" | ||
| if [[ -e "${alias_path}" || -L "${alias_path}" ]]; then | ||
| [[ -L "${alias_path}" && "$(readlink -f "${alias_path}")" == "${target}" ]] || \ | ||
| die "refusing to replace existing path: ${alias_path}" | ||
| else | ||
| ln -s "${target##*/}" "${alias_path}" | ||
| fi | ||
| [[ "$(readlink -f "${alias_path}")" == "${target}" ]] || \ | ||
| die "failed to install ${alias}" | ||
| done |
There was a problem hiding this comment.
🔍 Aliases and the stable symlink are untracked by the wheel RECORD
The generic aliases are written into the NIXL wheel's auditwheel .libs directory and /opt/dynamo/nixl-ucx-compat symlinks into it. Neither is recorded in the wheel's RECORD, so any later pip install/uninstall of nixl-cu* (e.g. in a derived image or at runtime by a user) leaves dangling links while LD_LIBRARY_PATH still points at the path — resulting in silent loader failures. The script comment acknowledges this and requires it be run last; worth confirming no downstream Dockerfile stage or recipe reinstalls nixl after this step.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
🎯 Code Coverage (details) 🔗 Commit SHA: 62e0bab | Docs | Datadog PR Page | Give us feedback! |
2 participants
Summary
Cherry-pick of #12558 to
release/1.4.0.Backports the SGLang arm64 runtime UCX compatibility fix for DYN-3695. The upstream SGLang runtime can expose generic UCX SONAMEs from the system UCX installation while NIXL loads its wheel-bundled UCX, producing a mixed UCX stack. On GB200 this broke multimodal E/P/D embedding transfer when the embedding cache was enabled.
This change installs stable generic UCX aliases that resolve to the NIXL wheel libraries and makes that directory take precedence in
LD_LIBRARY_PATH.Original PR
f865db0808c1a380da41ca48ef95c4b3198a3ef862e0babdf1f92eab5cdf3a8529e895e26da01d10Validation
main(SHA-256:5b586fdb97577bf28b9c654382521ae3388fb06d007b96f524dee87475f16e26).Qwen/Qwen3-VL-2B-Instruct, multimodal E/P/D,--single-gpu, and defaultnixl-read:Related Issues
Fixes DYN-3695