Skip to content

[ROCm] Work around DeepEP teardown SIGSEGV in MoE test harness #51174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AndreasKaratzas merged 3 commits into from
Aug 5, 2026
Merged

[ROCm] Work around DeepEP teardown SIGSEGV in MoE test harness #51174

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e07de74
[ROCm] Work around DeepEP teardown SIGSEGV in MoE test harness
Rohan138 Aug 4, 2026
1f2cef5
Merge branch 'main' into rocm-deepep-teardown-osexit
Rohan138 Aug 5, 2026
2ded847
Add TODO for os.exit fix
Rohan138 Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions tests/kernels/moe/parallel_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

import dataclasses
import os
import sys
import traceback
from collections.abc import Callable
from typing import Concatenate
Expand All @@ -15,6 +16,7 @@
from torch.multiprocessing import spawn # pyright: ignore[reportPrivateImportUsage]
from typing_extensions import ParamSpec

from vllm.platforms import current_platform
from vllm.utils.import_utils import has_deep_ep, has_deep_ep_v2
from vllm.utils.network_utils import get_open_port

Expand Down Expand Up @@ -72,6 +74,7 @@ def _worker_parallel_launch(
barrier = torch.tensor([rank], device=device)
torch.distributed.all_reduce(barrier)

exit_code = 0
try:
worker(
ProcessGroupInfo(
Expand All @@ -88,9 +91,18 @@ def _worker_parallel_launch(
except Exception as ex:
print(ex)
traceback.print_exc()
exit_code = 1
raise
finally:
torch.distributed.destroy_process_group()
if current_platform.is_rocm():
# Bypass a ROCm teardown use-after-free (SIGSEGV in HIP's atexit
# handler; fixed by https://github.com/ROCm/rocm-systems/pull/6942).
# Flush first since os._exit skips it, to keep a failing traceback.
# TODO (Rohan138): Remove after ROCm 10.0 lands
sys.stdout.flush()
sys.stderr.flush()
os._exit(exit_code)


def parallel_launch(
Expand Down
Loading