Conversation
djramic
requested review from
mgoin,
pavanimajety and
zyongye
as code owners
September 12, 2026 18:41
djramic
force-pushed
the
rocm-gate-aux-stream
branch
from
September 12, 2026 18:43
1b46feb to
ad96a46
Compare
Signed-off-by: Djordje Ramic <djoramic@amd.com>
djramic
force-pushed
the
rocm-gate-aux-stream
branch
from
September 12, 2026 18:44
ad96a46 to
01b747c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
#56098 enabled the MoE shared-experts auxiliary stream across ROCm. On gfx942, that causes Elastic EP scaling to hang: AMD CI build 12870 failed three eager parameterizations of
test_elastic_ep_scaling, andenforce_eager_lightfails 5 of 5 runs locally onmain.Tracing the rescale through
StatelessGroupCoordinatorandPyNcclCommunicatorshows that all four ranks of the new DP group return fromncclCommInitRank, but the firstall_reducenever completes and c10d times out after 300 seconds. Withthe aux stream disabled, the same collective completes within 35 ms;
ncclCommInitRanktakes about 10 seconds in both cases.This change restricts shared-expert auxiliary-stream overlap on ROCm to gfx950, so #56098's behavior is preserved there. The temporary gate prevents the gfx942 Elastic EP scaling failure until the interaction between the aux stream and RCCL initialization is resolved.
Test Plan
Four MI300X GPUs:
cd /vllm-workspace/tests pytest -v -s distributed/test_elastic_ep.pyTest Result
Before: 0 of 5 runs of
test_elastic_ep_scaling[enforce_eager_light]passed,each failing after a 300 s
[c10d] waitForInputtimeout followed byScale failed: The actor died unexpectedly before finishing this task.After: the full file passes, 5 of 5 tests in 26m30s.
AI assistance was used.