Skip to content

[ROCm][Bugfix] Restrict multi-stream shared experts to gfx950 - #56630

Closed
djramic wants to merge 1 commit into
vllm-project:mainfrom
djramic:rocm-gate-aux-stream
Closed

djramic wants to merge 1 commit into
vllm-project:mainfrom
djramic:rocm-gate-aux-stream

Conversation

@djramic

@djramic djramic commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

#56098 enabled the MoE shared-experts auxiliary stream across ROCm. On gfx942, that causes Elastic EP scaling to hang: AMD CI build 12870 failed three eager parameterizations of test_elastic_ep_scaling, and enforce_eager_light fails 5 of 5 runs locally on main.

Tracing the rescale through StatelessGroupCoordinator and PyNcclCommunicator shows that all four ranks of the new DP group return from ncclCommInitRank, but the first all_reduce never completes and c10d times out after 300 seconds. With
the aux stream disabled, the same collective completes within 35 ms; ncclCommInitRank takes about 10 seconds in both cases.

This change restricts shared-expert auxiliary-stream overlap on ROCm to gfx950, so #56098's behavior is preserved there. The temporary gate prevents the gfx942 Elastic EP scaling failure until the interaction between the aux stream and RCCL initialization is resolved.

Test Plan

Four MI300X GPUs:

cd /vllm-workspace/tests
pytest -v -s distributed/test_elastic_ep.py

Test Result

Before: 0 of 5 runs of test_elastic_ep_scaling[enforce_eager_light] passed,
each failing after a 300 s [c10d] waitForInput timeout followed by
Scale failed: The actor died unexpectedly before finishing this task.

After: the full file passes, 5 of 5 tests in 26m30s.

AI assistance was used.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added rocm Related to AMD ROCm bug Something isn't working labels Sep 12, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Sep 12, 2026
@djramic
djramic force-pushed the rocm-gate-aux-stream branch from 1b46feb to ad96a46 Compare September 12, 2026 18:43
Signed-off-by: Djordje Ramic <djoramic@amd.com>
@djramic
djramic force-pushed the rocm-gate-aux-stream branch from ad96a46 to 01b747c Compare September 12, 2026 18:44
@djramic djramic closed this Sep 12, 2026
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant