Skip to content

[AMD] Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation #23581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andyluo7 wants to merge 1 commit into
base: main
Choose a base branch
from
+13 −2
Open

[AMD] Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation #23581

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions python/sglang/srt/distributed/device_communicators/custom_all_reduce.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -376,13 +376,24 @@ def dispatch_custom_allreduce():
if _use_amd_deterministic_impl():
return CustomAllreduce

if get_bool_env_var("SGLANG_USE_AITER_AR", default="true"):
# NOTE(rocm): AiterCustomAllreduce launches helper kernels on an internal
# stream during HIP graph capture, which invalidates the captured graph
# (hipErrorStreamCaptureInvalidated) and triggers HSA_STATUS_ERROR_EXCEPTION
# 0x1016 on the first decode replay. This was first observed with the
# tencent/Hy3-preview model on MI300X/MI355X (see
# https://github.com/sgl-project/sglang/issues/23580). Until the AITER
# kernel is fixed, default to sglang's own CustomAllreduce on HIP and let
# the user explicitly opt back into AITER's implementation by setting
# SGLANG_USE_AITER_AR=1.
if get_bool_env_var("SGLANG_USE_AITER_AR", default="false"):
try:
from aiter.dist.device_communicators.custom_all_reduce import (
CustomAllreduce as AiterCustomAllreduce,
)

logger.info("[AR] Using AiterCustomAllreduce (AMD default)")
logger.info(
"[AR] Using AiterCustomAllreduce (opted in via SGLANG_USE_AITER_AR=1)"
)
tms_cudagraph = envs.SGLANG_MEMORY_SAVER_CUDA_GRAPH.get()
return partial(
AiterCustomAllreduce,
Expand Down
Loading