Repository navigation
[ROCm] Bump UCX to v1.23.0-rc1 so NIXL uses rocm_ipc - #62827
Merged
AndreasKaratzas merged 4 commits intoOct 23, 2026
Merged
AndreasKaratzas merged 4 commits into
AndreasKaratzas merged 4 commits into
Conversation
…NABLE NIXL creates UCX endpoints with peer error handling. At the old pin, rocm_ipc does not advertise UCT_IFACE_FLAG_ERRHANDLE_PEER_FAILURE, so NIXL GPU-to-GPU reads never use it: they fall back to TCP emulation, or, with UCX_RMA_PPLN_ENABLE=y, to a host-staged pipeline whose fragment pool exhausts shared memory under load. openucx/ucx#11977, in v1.23.0-rc1, adds the capability; with it rocm_ipc is selected again and the pipeline override is no longer needed. Lift the rocm_ipc signal pool cap the same way as rocm_copy's, since every in-flight copy holds one signal. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
daniel37907-spec
approved these changes
Oct 21, 2026
Member
Author
|
/amd-ci run nightly |
|
✅ Triggered Buildkite AMD CI #15198 for commit |
AndreasKaratzas
marked this pull request as ready for review
October 23, 2026 02:39
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #96782 for commit ℹ️ This PR is 1 commit behind upstream |
AndreasKaratzas
added a commit
to AndreasKaratzas/vllm
that referenced
this pull request
Oct 23, 2026
Port of vllm-project#62827 to this branch's Dockerfile.rocm: - UCX_BRANCH -> 2ff0b47c (v1.23.0-rc1), which adds peer error handling to rocm_ipc (openucx/ucx#11977), so NIXL selects it for GPU-to-GPU reads instead of the TCP fallback or the host-staged pipeline. - Drop UCX_RMA_PPLN_ENABLE=y, which is no longer needed. - Set UCX_ROCM_IPC_SIGPOOL_MAX_ELEMS=inf, matching rocm_copy: each in-flight rocm_ipc copy holds one HSA signal. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
AndreasKaratzas
added a commit
to AndreasKaratzas/vllm
that referenced
this pull request
Oct 23, 2026
Conflicts, both from main's vllm-project#62827 (UCX v1.23.0-rc1 for NIXL rocm_ipc), which this branch already carries in docker/Dockerfile.rocm (d7cb939): - docker/Dockerfile.rocm: main's only change there is that UCX bump, so this branch's version is kept unchanged. - docker/Dockerfile.rocm_72 (modify/delete): stays deleted with the ROCm 7.2 stack. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
AndreasKaratzas
added a commit
that referenced
this pull request
Oct 23, 2026
Conflict: .buildkite/test_areas/models_basic.yaml. Main now gives the MiniMax M3 MonoKernel step 45 minutes; this branch had raised it from 20 to 40. Take main's 45, which reverts this branch's change to the file. docker/Dockerfile.rocm merged cleanly: main's UCX v1.23.0-rc1 bump (#62827) plus this branch's symbol-map stages and tools/ci export. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
UCX_BRANCHtov1.23.0-rc1inDockerfile.rocmandDockerfile.rocm_72UCX_RMA_PPLN_ENABLE=yUCX_ROCM_IPC_SIGPOOL_MAX_ELEMS=inf, matching the existingrocm_copysettingNIXL opens its UCX endpoints with peer error handling, and at the current UCX pin
rocm_ipcdoes not advertise that capability, so NIXL never uses it for GPU-to-GPU reads on ROCm. Reads fall back to TCP emulation, or, with theUCX_RMA_PPLN_ENABLE=yoverride from #49251, to a host-staged pipeline whose fragment pool grows until/dev/shmallocations fail; under long-context P/D load that stalled transfers and killed the decode engine. openucx/ucx#11977, included inv1.23.0-rc1, adds the capability. With it,rocm_ipcis selected again and the override is no longer needed. Each in-flightrocm_ipccopy holds one HSA signal, and the 1024 default made multi-request NIXL reads fail, hence the signal pool setting.On MI355, a 256 MiB NIXL READ between two GPUs goes from 0.48 GB/s (8.8 GB/s with the override) to 60.9 GB/s. With NIXL at its current pin built against the new UCX the same way as the Dockerfile, the MI355 NixlConnector groups pass: spec decode acceptance, PD accuracy sweep, AITER PD accuracy and DP EP PD accuracy. The PD accuracy sweep's lm_eval time drops from 370s to 183s. Step-5 1P1D (TP4 + TP4) passed 10/10 rounds of GSM8K and long-context NIAH; on the current image it hung in round 5.
AI assistance (Claude Code) was used to draft this change. Every line was reviewed by the submitter.