[ROCm][CI] Keep rocprofiler-sdk out of DeepEP HT MoE test workers - #51173
Merged
AndreasKaratzas merged 2 commits intoAug 5, 2026
Merged
Conversation
All high-throughput test_deep_ep_moe variants fail with SIGSEGV on ROCm. The crash happens after the test body, during process teardown inside the HSA runtime, where AqlQueue's destructor writes to signals whose backing memory is already unmapped. That path is only reached when rocprofiler-sdk is attached, which torch attaches implicitly, and it reproduces with a plain HIP program under rocprofv3, so the fix belongs to ROCm. Until that fix lands, disable rocprofiler registration for the workers spawned by that test, on ROCm only. Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
stefankoncarevic
requested review from
AndreasKaratzas,
WoosukKwon,
mgoin,
tlrmchlsmth,
yewentao256 and
zyongye
as code owners
August 5, 2026 15:23
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Member
|
/ci run |
|
✅ Triggered Buildkite CI #82525 for commit |
Member
|
/ci run |
AndreasKaratzas
added
the
ready
|
✅ @stefankoncarevic, CI is now available for this PR.
|
Member
|
/ci run |
|
✅ Triggered Buildkite CI #82543 for commit |
Contributor
Author
|
The root cause here is already fixed upstream in ROCm/rocm-systems#6942. Reminder, drop this workaround, together with the os._exit one from #51174 once the base image ships a ROCm build that contains #6942. |
Member
|
@stefankoncarevic maybe create an issue then and share it with me to assign it to you so that we do not forget |
Contributor
Author
Opened it #51644. It covers both this workaround and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
On ROCm, every high-throughput variant of
test_deep_ep_moefails withSIGSEGV(28 of 28), while every low-latency variant passes. The crash is not in the test
and not in DeepEP: the test body completes, and the process then dies during
teardown inside the ROCm HSA runtime, where
AqlQueue's destructor writes tosignals whose backing memory has already been unmapped.
That destructor is only reached when rocprofiler-sdk is attached to the process,
which recent torch builds do implicitly, so the failure appeared without any
change on the vLLM side. It reproduces with HIP program run under
rocprofv3, with no torch, DeepEP or vLLM involved, so the defect belongs toROCm and it will be report to the AMD team. The issue link will be added here
once it is filed.
Until the ROCm fix lands, this keeps the profiler out of the spawned workers for
the affected test only: guarded by
current_platform.is_rocm(), applied only tothe high-throughput test, and set through
monkeypatchso it is restoredafterwards.
Test Plan
on 2x MI355X (gfx950), ROCm 7.2.3.
Test Result
Before:
Every
test_deep_ep_moevariant fails withSIGSEGVat process exit, everytest_low_latency_deep_ep_moevariant passes.After:
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.