Repository navigation
[ROCm][Bugfix] Warm elastic EP target groups at commit on ROCm - #59684
Closed
aarushjain29 wants to merge 18 commits into
Closed
aarushjain29 wants to merge 18 commits into
aarushjain29 wants to merge 18 commits into
Conversation
vllm-project#56610 stopped warming the target DP/EP torch device groups during asynchronous preparation on ROCm and relied on warm_and_capture to warm them at commit. warm_and_capture only drives the PyNccl communicators, so the first collective on the new EP device group is the EPLB all-reduce: in commit_scale_up's reshuffle in eager mode, and in the first periodic EPLB step after a scale-down with CUDA graphs. On MI355 that first all-reduce fails in RCCL with "hipIpcGetMemHandle failed : invalid argument" (vllm/ci builds 92231 and 92464, Elastic EP Scaling). Warm the target groups on ROCm at commit for scale-up and for the non-removing scale-down path, while the engine is paused, regardless of whether the fused MoE kernel is reused. Also add the MI355 AMD mirror of Elastic EP Scaling from vllm-project#56679 so the fix runs on MI355. Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
❌ @aarushjain29, A reviewer with write access must run |
Contributor
|
/ci run |
|
✅ Triggered Buildkite CI #92486 for commit |
…oups Warming the target groups at commit moved the MI355 failure into the warm-up all-reduce (vllm/ci build 92486): the first collective on a torch device group created during scaling fails in RCCL with "hipIpcGetMemHandle failed : invalid argument", while groups created at startup, before the model and KV cache are allocated, work. Release cached allocator blocks before the warm-up so RCCL sets up the new communicators with the cached memory returned to the device. Run the MI355 Elastic EP mirror with NCCL_DEBUG=INFO for the INIT, P2P and ALLOC subsystems to show which RCCL buffer fails to export. Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Temporary diagnostics for the MI355 Elastic EP Scaling failure. On ROCm, log the worker environment, RCCL and HIP versions, GPU memory and allocator state, and whether hipIpcGetMemHandle succeeds on default, fine-grained, uncached and caching-allocator buffers, when the standby groups are created and before and after emptying the cache at commit. Warm the DP and EP groups one at a time and log which one fails. Also enable HIP error logging and forward AMD_LOG_LEVEL to the Ray workers. Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
aarushjain29
requested review from
dllehr-amd,
shen-shanshan and
tjtanaa
as code owners
October 1, 2026 22:38
… EP job Run enforce_eager_none with HSA_ENABLE_IPC_MODE_LEGACY=1, which the ROCm Docker image sets but the native MI355 pod may not, and the remaining variants with the pod environment, so one run shows whether the IPC mode affects the hipIpcGetMemHandle failure. Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
AndreasKaratzas
added
the
ready
|
✅ @aarushjain29, CI is now available for this PR.
|
…diagnostics Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Member
|
/ci run |
|
✅ Triggered Buildkite CI #92510 for commit |
The existing IPC probe only covers hipExtMallocWithFlags, but RCCL exports handles for plain hipMalloc buffers, so that path was never exercised. Add it, plus a burst probe for export-handle exhaustion and a probe that exports while a different device is current. Both of those perturb state, so they only run at prepare and after a failure, never right before the collective under observation. Also log HIP's current device next to torch's before each collective, since the stream context does not set it, and add one run with NCCL_P2P_DISABLE=1 to tell whether the failure is confined to the IPC transport. Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Collaborator
|
/ci run |
|
❌ @stefankoncarevic, Only reviewers with write access can use CI commands before CI is delegated to the PR author. |
Every probe so far only covered the export side. In CI the run with HSA_ENABLE_IPC_MODE_LEGACY=1 did not fail on export at all, it aborted on attach with a 2 MiB handle the exporter had allocated with flags 3, so the side that actually failed there was never instrumented. Each rank now exports one buffer and attaches every peer's, with the two size and flag combinations taken from the two observed failures. Handles travel over the CPU group so the device group under test stays untouched. Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Collaborator
CI selector (shadow): 40 test steps (47 jobs) instead of 64 (80 jobs)Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works. Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.
Selector would run (40)
Would skip (today's rules run them) (55)
Would add (today's rules do not run them) (31)
AMD mirrors: would skip (0)none AMD mirrors: would add (95)
6 changed files · base |
UCX's ROCm memory hooks (libucm_rocm) intercept GPU frees once NIXL starts its UCX backend. On MI355 this breaks HIP IPC for buffers that RCCL allocates afterwards, so the RCCL communicators created for elastic EP scale-up fail with hipIpcGetMemHandle/hipIpcOpenMemHandle invalid argument. Set UCX_MEM_EVENTS=no on ROCm before NIXL is loaded unless the user already set it. Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Collaborator
Author
|
/ci run |
|
❌ This PR is 69 commits behind upstream |
Collaborator
Author
|
/ci run |
|
✅ Triggered Buildkite CI #92694 for commit |
…ic EP job Run a vLLM-free script that creates, all-reduces and destroys RCCL groups of 4 and 2 ranks, alone, with most GPU memory held, and with that memory registered in NIXL, probing hipIpcGetMemHandle after every round. Trim the job to the repro plus one elastic EP case to shorten CI. Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Collaborator
Author
|
/ci run |
|
❌ This PR is 41 commits behind upstream |
Collaborator
Author
|
/ci run |
|
✅ Triggered Buildkite CI #92829 for commit |
Phases A-C pass on MI355 CI. Add phase D, which also exchanges NIXL agent metadata and READs from every peer's registered GPU memory each round, as the EPLB NIXL communicator does, before churning RCCL groups. Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Collaborator
Author
|
/ci run |
|
✅ Triggered Buildkite CI #92832 for commit |
Opt the AMD image steps into the TheRock Dockerfiles from vllm-project#56351 so the MI355 Elastic EP job runs on ROCm SDK 10.0.0 (HIP, RCCL, PyTorch) and we can see whether the HIP IPC failure is fixed in the newer ROCm userspace. Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Signed-off-by: aarushjain29 <aarushi.jain2@amd.com>
Collaborator
Author
|
/ci run |
|
❌ This PR is 1 commit behind upstream |
Collaborator
Author
|
/ci run |
|
✅ Triggered Buildkite CI #94251 for commit |
stefankoncarevic
added a commit
to stefankoncarevic/vllm
that referenced
this pull request
Oct 14, 2026
Carries the diagnostic work from vllm-project#59684 onto current main unchanged. None of these six files were touched on main since that branch was cut, so this is the same content, only rebased so CI runs against a current tree. The follow-up commit adds the instrumentation that narrows the failure down. Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
stefankoncarevic
added a commit
to stefankoncarevic/vllm
that referenced
this pull request
Oct 14, 2026
Follow-up diagnostics for vllm-project#59684, where the MI355 Elastic EP Scaling step fails on every run and nothing reproduces on a bare metal host. The CI logs report ValueError('Process group <torch...'). That is not the real error: while handling the collective failure, torch's c10d logger calls dist.get_backend() on the already invalidated group, which raises and replaces "hipIpcGetMemHandle failed : invalid argument" in everything downstream. Walk the exception chain and print the traceback so the original error survives. The mesh probe already runs immediately before the all_reduce, with nothing between them, and it passes on the very rank that then fails. So the export capability dies inside RCCL's lazy communicator init, not when the group is created. Ask RCCL directly which transport it picks per pair and which buffer it exports, and add two runs that avoid the IPC transport to establish whether it is the trigger rather than only a symptom. After the failure every export fails, including a fresh plain hipMalloc and a buffer owned by another device, so this is not one bad pointer. The fixed 64 handle burst cannot see a ceiling, since the same probe holds 8192 handles on a bare metal host with amdgpu 6.16.6, so report the count actually reached. amd-smi on the CI node reports XGMI as N/A for four of the six pairs, which could mean no link or merely no visibility from inside the VM. Dump the KFD node properties and io_links, which do not carry that ambiguity. Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#56610 stopped warming the target DP/EP torch device groups during asynchronous preparation on ROCm and relied on warm_and_capture to warm them at commit. warm_and_capture only drives the PyNccl communicators, so the first collective on the new EP device group is the EPLB all-reduce: in commit_scale_up's reshuffle in eager mode, and in the first periodic EPLB step after a scale-down with CUDA graphs. On MI355 that first all-reduce fails in RCCL with "hipIpcGetMemHandle failed : invalid argument" (vllm/ci builds 92231 and 92464, Elastic EP Scaling).
Warm the target groups on ROCm at commit for scale-up and for the non-removing scale-down path, while the engine is paused, regardless of whether the fused MoE kernel is reused.
Also add the MI355 AMD mirror of Elastic EP Scaling from #56679 so the fix runs on MI355.