[ROCm] Work around DeepEP teardown SIGSEGV in MoE test harness - #51174
Merged
AndreasKaratzas merged 3 commits intoAug 5, 2026
Conversation
Rohan138
force-pushed
the
rocm-deepep-teardown-osexit
branch
from
August 5, 2026 15:47
c0b4090 to
6ba8d48
Compare
On ROCm 7.2.x, `test_deep_ep_moe` fails 28/28 with each rank dying via `SIGSEGV` at process exit (no Python traceback). The MoE compute is correct; the crash is teardown-only, inside HIP's atexit handler. Root cause is a ROCr use-after-free: `Runtime::Unload()` clears the shared signal pool before `DestroyAgents()`, and `GpuAgent::ReleaseResources()` never releases the GWS (cooperative-dispatch) queue that DeepEP's `hipLaunchCooperativeKernel` leaves live, so `~AqlQueue` stores into freed signal memory. The real fix is in ROCr (submitted separately); this only unblocks CI. Guard the spawn worker's teardown so that on ROCm it `os._exit(exit_code)` after `destroy_process_group()`, bypassing the broken atexit path while preserving failure reporting (non-zero exit when the worker raised). No effect off ROCm. Verified locally: `test_deep_ep_moe` 28 passed with the ROCr fix; this guard lets the same run pass on a stock base image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Rohan138
force-pushed
the
rocm-deepep-teardown-osexit
branch
from
August 5, 2026 16:32
6ba8d48 to
e07de74
Compare
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Rohan138
marked this pull request as ready for review
August 5, 2026 17:17
Rohan138
requested review from
AndreasKaratzas,
WoosukKwon,
mgoin,
tlrmchlsmth,
yewentao256 and
zyongye
as code owners
August 5, 2026 17:17
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Collaborator
Author
|
/ci run |
|
✅ @Rohan138, CI is now available for this PR.
|
|
✅ Triggered Buildkite CI #82528 for commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
On ROCm,
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moefails with eachspawned rank dying via
SIGSEGVat process exit (no traceback). The MoE computeis correct; the crash is teardown-only — a ROCr use-after-free in HIP's atexit
handler, fixed upstream in ROCm/rocm-systems#6942 but not
yet in our base image's ROCm build.
Guard the DeepEP spawn worker so on ROCm it
os._exit(exit_code)afterdestroy_process_group(), bypassing the broken atexit path while preserving thepass/fail signal (non-zero exit if the worker raised). No effect off ROCm;
test-harness only (no kernel/model change). Revert once the base image ships a
ROCm build with the ROCr fix.
Test plan
MI300X (gfx942), ROCm 7.2.3, torch 2.12, stock runtime (isolates the workaround):
Notes
support, [ROCm] Enable DeepEP ROCm as all2allbackend for AMD GPUs. #34692 DeepEP enablement, [WideEP] Remove DeepEP high-throughput backend, redirect to DeepEP v2 #46887 HT backend rework — none touch exit).
before submission.