Repository navigation
[AMD][DI] Keep loopback in UCX_NET_DEVICES - #40641
Merged
Merged
Conversation
Lzy17
changed the title
…eeds sgl-project#37874 pins UCX_NET_DEVICES to the RDMA devices used by the test. UCX applies that list to every network resource, not just the RDMA ones, so it also drops tcp. NIXL opens an endpoint to itself in createBackend and needs an active-message transport for it. Without tcp none is left: self, sysv and posix have no peer failure handler, rocm_copy has no am bcopy, and ud_verbs is unreachable on this fabric. create_backend then fails with NIXL_ERR_BACKEND and the prefill server exits. Keep the RDMA pin and add loopback so that endpoint still works.
Lzy17
force-pushed
the
amd/fix-ucx-net-devices-nixl
branch
from
September 21, 2026 22:51
8a673dd to
150a927
Compare
Lzy17
changed the title
This was referenced Sep 22, 2026
HaiShaw
approved these changes
Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
test_nixl_transfer_engine_e2ehas failed on the MI35X disaggregation runner since #37874. The prefill server dies at startup withNIXL_ERR_BACKEND.#37874 added
_maybe_set_ucx_net_devices, which pinsUCX_NET_DEVICESto the RDMA devices used by the test. UCX applies that list to every network device, not just the RDMA ones, so it also drops tcp.NIXL opens an endpoint to itself during
createBackendand needs an active-message transport for it. Without tcp none is left:self,sysvandposixhave no peer failure handler,rocm_copyhas no am bcopy, andud_verbsis unreachable on this fabric.The same CI job shows both sides: the
Verify NIXL in Containerstep creates a UCX backend fine before any test runs, and the same call fails once the fixture has set the pin.Modifications
Keep the RDMA pin and add loopback, so the endpoint still has a transport.
ud_verbsbeing unreachable is a separate fabric issue and is not addressed here.Checklist
pre-commit run --all-filesCI States
Latest PR Test (Base): ❌ Run #35664824495
Latest PR Test (Extra): ❌ Run #35664824307
Latest PR Test (AMD ROCm 10): ❌ Run #35664824445