[ROCm][Perf][Bugfix] Multi-stream perf improvements; rocprofiler fixes by mawong-amd · Pull Request #55099 · vllm-project/vllm

mawong-amd · 2026-09-03T06:59:51Z

Purpose

This PR does the following:

Relocates the ROCR+CLR build to its separate stage, instead of being a prerequisite of the base stage. This allows ROCR+CLR updates to not invalidate + force rebuilds of all build stages. Downstream libraries currently only ever link against them dynamically, so this relocation is safe.
Brings a backport of changes in rocr: Add device-resident ordering edges ROCm/rocm-systems#11212 to ROCm 7.2.4's ROCR+CLR, which resolves slow cross-stream dependencies in various workloads (primarily observed in multi-stream use cases, e.g. [Perf][ROCm] Dual-stream decode with hipgraphs #52033, [ROCm][Perf] Enable CSA multi-stream overlap for DeepSeek-V4 #51794, [ROCm] Enable shared-expert multi-stream overlap on ROCm #51117) and completes the fix series started in [Hardware][AMD][Perf][Bugfix] Update ROCr and clr in base image #53712 which only worked on certain stream events.
Brings in the aforementioned multi-stream improvements into the ROCm 10.0 image as well.
Update ROCR to contain a 7.2.4-specific backport of fix(rocr): release GWS queue in ReleaseResources to fix shutdown UAF ROCm/rocm-systems#6942. This resolves the issues previously reported in [ROCm] Work around DeepEP teardown SIGSEGV in MoE test harness #51174 and [ROCm][CI] Keep rocprofiler-sdk out of DeepEP HT MoE test workers #51173 (also occasionally seen during test teardown in some AMD CI jobs e.g. here), so those PRs are reverted as part of this change. Fixes [CI Failure]: tests/kernels/moe/test_deepep_moe.py SIGSEGV on ROCm, drop the two teardown workarounds once the image ships rocm-systems#6942 #51644.
Sets the env var ROCPROFILER_QUEUE_INTERPOSITION=0 to avoid the hangs mentioned in [Bugfix][ROCm][Build] fix profiler hang due to queue interposition bug #54171. While most of them were resolved by said PR, a few remain in some profiling workloads.

Test Plan

Full AMD CI will run on the image built from this PR.

Test Result

There are no new regressions.

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

github-actions · 2026-09-03T07:01:37Z

✅ @mawong-amd, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

mawong-amd · 2026-09-03T07:03:22Z

/ci run

github-actions · 2026-09-03T07:03:41Z

✅ Triggered Buildkite CI #87032 for commit 8df4f1675c78.

mawong-amd · 2026-09-03T15:12:58Z

/ci run

github-actions · 2026-09-03T15:13:19Z

✅ Triggered Buildkite CI #87091 for commit 3c4d7d6479fd.

Rohan138 · 2026-09-03T16:51:15Z

        raise
    finally:
        torch.distributed.destroy_process_group()
-        if current_platform.is_rocm():


oh is this fixed by your changes/the ROCr+CLR bump? awesome!

Rohan138 · 2026-09-03T17:04:02Z

    low_latency_mode = False
    use_fp8_dispatch = False

-    if current_platform.is_rocm():


Rohan138 · 2026-09-03T17:04:54Z

    && git config --global user.email "you@example.com" && git config --global user.name "Your Name" \
    && if [ ! -f setup.py ]; then cd python; fi \
-    && python3 setup.py bdist_wheel --dist-dir=dist \
+    && MAX_JOBS=16 python3 setup.py bdist_wheel --dist-dir=dist \


drop the MAX_JOBS?

There's a race condition in Triton build which we hit somewhat often when we use the default MAX_JOBS (when unset it's defaulted to $(nproc) in Triton setup.py). It's related to the issue mentioned in triton-lang/triton#6718 but is not necessarily the same manifestation. They haven't bothered to fix it and honestly it's not a blocker for us -- Triton builds quickly by itself and if it's a cold build, the build time of the Triton stage is hidden by the long build time of the other stages it builds in parallel with.

good point; I'd forgotten about this occasional error, LGTM

Rohan138 · 2026-09-03T17:06:02Z

 RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
    cp /install/*.whl /app/debs

+###


sorry just to clarify, why'd we move this down here instead of up top before the other build stages? It's a minor nit I'd just prefer ROCm stuff to be before component libs

Some Docker build backends (not BuildKit though) invalidate all stages below when a change is made above it, even in a stage it doesn't depend on. Since the ROCR/CLR runtime builds are quick, it's pretty cheap if it's invalidated. And conversely, we don't want to invalidate expensive stages if we bump the runtime, because with the proper caching it should just be a <5 min build of the new ROCm base.

I'm fine with this for now, I think we use buildkit by default across both CI and release pipelines and it'd be a little more organized to put it up top to track+remove it later once the fixes land in ROCm

coderabbitai · 2026-09-03T19:36:20Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: 8366fc6f-0934-4fc1-b3b6-f46b181b322d

📥 Commits

Reviewing files that changed from the base of the PR and between 0475b7b and 27fc2d3.

📒 Files selected for processing (1)

docker/Dockerfile.rocm_base

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

The ROCm image now uses a shared ROCm systems repository for runtime and profiler builds. Runtime installation moves to the final image stage. DeepEP workers use common exception teardown, and the ROCm profiler environment workaround is removed.

Changes

ROCm runtime and DeepEP integration

Layer / File(s)	Summary
Shared ROCm systems build `docker/Dockerfile.rocm_base`	Runtime and rocprofiler SDK builds use `ROCM_SYSTEMS_REPO`. The runtime commit and package set are updated. Triton builds use `MAX_JOBS=16`.
Runtime installation and image metadata `docker/Dockerfile.rocm_base`	Runtime copying, library linking, and `ldconfig` move to `final`. The image sets `ROCPROFILER_QUEUE_INTERPOSITION=0` and records the shared repository.
DeepEP worker teardown `tests/kernels/moe/parallel_utils.py`, `tests/kernels/moe/test_deepep_moe.py`	Workers re-raise exceptions after process-group destruction. The ROCm profiler environment workaround is removed.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 27fc2

The ROCm image now uses the updated shared runtime build, installs it in the final image, and removes the DeepEP teardown workarounds. No concrete merge-blocking regression is identified.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The pull request implements issue `#51644` by updating the ROCr source to include the upstream fix and removing both ROCm-specific DeepEP teardown workarounds. The remaining build and profiler changes s…
Out of Scope Changes check	✅ Passed	The changes remain within the ROCm runtime, rocprofiler, container build, and DeepEP teardown scope described by the pull request and issue `#51644`. No unrelated code changes are evident.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Title check	✅ Passed	The title clearly identifies the ROCm performance bugfixes and rocprofiler fixes that are central to the changeset.
Description check	✅ Passed	The description directly explains the build-stage relocation, ROCr fixes, DeepEP workaround removal, profiler environment change, and planned AMD CI validation.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

AndreasKaratzas · 2026-09-03T20:28:52Z

/ci run

github-actions · 2026-09-03T20:29:13Z

✅ Triggered Buildkite CI #87141 for commit 0475b7b8b43a.

torch 2.12 attaches rocprofiler-sdk implicitly and its queue interposition hangs at HSA teardown. Set ROCPROFILER_QUEUE_INTERPOSITION=0 in the final image, matching the mitigation in vllm-project#55099 for Dockerfile.rocm_base. The ROCr/CLR and rocprofiler-sdk source rebuilds from rocm_base are not ported: they target ROCm 7.2.x and ROCm 10.0 already carries those fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>

mawong-amd · 2026-09-04T23:39:37Z

/ci run

github-actions · 2026-09-04T23:39:55Z

✅ Triggered Buildkite CI #87339 for commit 27fc2d399fcb.

Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

vllm-project#51174)" This reverts commit c2d8009. Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

…kers (vllm-project#51173)" This reverts commit 65addac. Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

mawong-amd · 2026-09-08T02:55:19Z

/ci run

github-actions · 2026-09-08T02:55:54Z

✅ Triggered Buildkite CI #87617 for commit f00cd34c047c.

Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

mawong-amd · 2026-09-08T04:34:27Z

/ci run

github-actions · 2026-09-08T04:34:45Z

✅ Triggered Buildkite CI #87629 for commit 961c3d457d2a.

AndreasKaratzas

Stamping for clarity -- all AMD failures here are pre-existing.

vllm-project#55099) Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com> Signed-off-by: Jyotirmoy Roy <jyotirmoyroy649@gmail.com>

…e baseline Remove the ROCr + CLR source rebuild (build_rocm_runtime stage, final-stage COPY/relink, and the ROCM_SYSTEMS_REPO/ROCM_RUNTIME_COMMIT ARGs) from the rock base, runs the stock TheRock ROCr/CLR runtimes. Baseline for comparing ROCm 10 with and without the multi-stream runtime patches. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

…e baseline Remove the ROCr + CLR source rebuild (build_rocm_runtime stage, final-stage COPY/relink, and the ROCM_SYSTEMS_REPO/ROCM_RUNTIME_COMMIT ARGs) from the rock base so the image runs the stock TheRock ROCr/CLR runtimes. Baseline for comparing ROCm 10 with and without the multi-stream runtime patches. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

mawong-amd requested review from AndreasKaratzas, dllehr-amd and tjtanaa as code owners September 3, 2026 06:59

claude Bot reviewed Sep 3, 2026

View reviewed changes

mawong-amd requested review from WoosukKwon, mgoin, tlrmchlsmth, yewentao256 and zyongye as code owners September 3, 2026 07:00

mergify Bot added ci/build rocm Related to AMD ROCm labels Sep 3, 2026

github-project-automation Bot added this to AMD Sep 3, 2026

mergify Bot added the bug Something isn't working label Sep 3, 2026

github-project-automation Bot moved this to Todo in AMD Sep 3, 2026

AndreasKaratzas added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 3, 2026

Rohan138 reviewed Sep 3, 2026

View reviewed changes

Rohan138 approved these changes Sep 3, 2026

View reviewed changes

AndreasKaratzas approved these changes Sep 3, 2026

View reviewed changes

Rohan138 mentioned this pull request Sep 4, 2026

[ROCm][CI] Bump ROCk base to ROCm 10.0 #55246

Merged

AndreasKaratzas mentioned this pull request Sep 4, 2026

[CI][ROCm] Make image dependencies cache-safe #55093

Draft

mawong-amd added 6 commits September 8, 2026 02:40

Move ROCR/CLR stage; update with rocprofiler changes

3f2b9d8

Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

Revert "[ROCm] Work around DeepEP teardown SIGSEGV in MoE test harness (

e75c532

vllm-project#51174)" This reverts commit c2d8009. Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

Revert "[ROCm][CI] Keep rocprofiler-sdk out of DeepEP HT MoE test wor…

71b9d38

…kers (vllm-project#51173)" This reverts commit 65addac. Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

Relocate runtimes build to the components build section

b726ed2

Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

Limit Triton MAX_JOBS to 16 to limit race condition

41407da

Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

Bring in multi-stream perf improvements

f00cd34

Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

mawong-amd force-pushed the mawong/rocm_libs_update branch from 27fc2d3 to f00cd34 Compare September 8, 2026 02:44

mawong-amd changed the title ~~[Hardware][AMD][Bugfix] Relocate ROCR/CLR build + rocprofiler improvements~~ [Hardware][AMD][Bugfix] Multi-stream perf improvements; rocprofiler fixes Sep 8, 2026

mawong-amd changed the title ~~[Hardware][AMD][Bugfix] Multi-stream perf improvements; rocprofiler fixes~~ [ROCm][Perf][Bugfix] Multi-stream perf improvements; rocprofiler fixes Sep 8, 2026

Bring in multi-stream improvements for ROCm 10.0 image too

961c3d4

Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>

AndreasKaratzas approved these changes Sep 8, 2026

View reviewed changes

ywang96 merged commit bc8587f into vllm-project:main Sep 9, 2026
143 of 146 checks passed

github-project-automation Bot moved this from Todo to Done in AMD Sep 9, 2026

mawong-amd deleted the mawong/rocm_libs_update branch September 9, 2026 10:41

mawong-amd mentioned this pull request Sep 9, 2026

[ROCm][Bugfix][Perf] Tune multi-stream shared experts use; wvSplitKrc fixes #56098

Merged

4 tasks

JohnQinAMD mentioned this pull request Sep 11, 2026

[Bug][ROCm]: Intermittent worker segfault in libhsa-runtime64 — ROCPROFILER_QUEUE_INTERPOSITION=0 forces an unfixed ROCr heap overflow #56521

Open

karverma-amd mentioned this pull request Sep 13, 2026

[AMD][ROCm] Add optional device-resident ordering-edge runtime build (rocm-systems#11212) sgl-project/sglang#39282

Open

Uh oh!

Uh oh!

Conversation

mawong-amd commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

mawong-amd commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

mawong-amd commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

AndreasKaratzas commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

mawong-amd commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

mawong-amd commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

mawong-amd commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

AndreasKaratzas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mawong-amd commented Sep 3, 2026 •

edited

Loading

coderabbitai Bot commented Sep 3, 2026 •

edited

Loading