[yctseng0211]

Motivation

On gfx1250-rocm1010, torch.profiler hangs on TP workloads. Right after profiling starts, the server stops making progress and the log keeps printing:

queue_interposition.cpp:470] Async signal handler still waiting on signal {.handle=...} after 5242880 iterations (value=1, starting_value=1)

The bug is in rocprofiler-sdk, and ROCm 10.1 does not have the fixes.

Modifications

Rebuild ROCr and rocprofiler-sdk from the rocm-systems commit the SDK was built from, cherry-pick the commits below, and replace the SDK libraries, like the existing RCCL step.

• fix(rocprofiler-sdk): Fix ROCPROFILER_ASYNC_SIGNAL_HANDLER_THREADS deadlock ROCm/rocm-systems#9984 (rocprofiler-sdk): fixes the hang. Before it, rocprofiler-sdk waits for kernels with only 4 threads, one kernel per thread, so all 4 get stuck when kernels wait on each other (for example, RCCL all-reduce on TP).
• fix(rocr): fix out-of-bounds access in hsa_amd_signal_wait_any ROCm/rocm-systems#8564 (ROCr): fix(rocprofiler-sdk): Fix ROCPROFILER_ASYNC_SIGNAL_HANDLER_THREADS deadlock ROCm/rocm-systems#9984 needs it, as noted in its review. Without it, hsa_amd_signal_wait_any writes out of bounds, and the server crashes as soon as profiling starts:

  free(): invalid pointer
  Fatal Python error: Aborted
  Fatal glibc error: malloc.c:2599 (sysmalloc): assertion failed: ...
  

• fix(rocprofiler-sdk): Add null signal for backwards compatibility rocr-runtime OOB error ROCm/rocm-systems#12957 (rocprofiler-sdk): a small upstream follow-up to fix(rocprofiler-sdk): Fix ROCPROFILER_ASYNC_SIGNAL_HANDLER_THREADS deadlock ROCm/rocm-systems#9984, tested together with it.

On MI450, Kimi-K2 TP8 profiling now works without any env var workaround, and the traces are complete. GSM8K accuracy (Kimi-K2, gpt-oss) and gpt-oss serving speed are the same as before.

Speed/Accuracy Tests

Kimi-K2 TP8

unset NCCL_MIN_NCHANNELS NCCL_MAX_NCHANNELS
export HIP_FORCE_DEV_KERNARG=1 ENABLE_CK=0 NCCL_IGNORE_CPU_AFFINITY=1 NCCL_MNNVL_ENABLE=1 \
  NCCL_CUMEM_ENABLE=1 HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
python3 -m sglang.launch_server --model-path /data/amd/Kimi-K2-Thinking-MXFP4-v1 --tp-size 8 \
  --trust-remote-code --reasoning-parser kimi_k2 --tool-call-parser kimi_k2 --mem-fraction-static 0.8 \
  --max-running-requests 128 --cuda-graph-max-bs-decode 128 --chunked-prefill-size 8192 \
  --disable-radix-cache --attention-backend aiter --prefill-attention-backend aiter \
  --decode-attention-backend triton --disable-custom-all-reduce --kv-cache-dtype fp8_e4m3

gpt-oss-120b TP2 (GPU 0 and 1)

export TRITON_HIP_USE_EXPERT_SCHEDULING=0 HIP_FORCE_DEV_KERNARG=1 ENABLE_CK=0 \
  SGLANG_GFX1250_W4A8_BACKEND=triton SGLANG_GFX1250_W4A8_KERNEL_GATHER=1 SGLANG_LM_HEAD_AITER_TGEMM=1
python3 -m sglang.launch_server --model-path /mnt/raid/models/gpt-oss-120b-w-mxfp4-a-fp8 --tp 2 \
  --trust-remote-code --max-running-requests 512 --mem-fraction-static 0.9 --attention-backend aiter \
  --disable-radix-cache --page-size 64 --chunked-prefill-size 3072 --max-prefill-tokens 3072 \
  --disable-custom-all-reduce

# warm-up, for C in 1 16 64 128
python3 -m sglang.benchmark.serving --backend sglang --port 30000 --dataset-name random \
  --random-range-ratio 1.0 --warmup-requests 1 --random-input-len 1024 \
  --random-output-len 32 --num-prompts $C --max-concurrency $C

# measured, for C in 1 16 64 128 (N = 8, 64, 256, 512)
python3 -m sglang.benchmark.serving --backend sglang --port 30000 --dataset-name random \
  --random-range-ratio 1.0 --warmup-requests 1 --random-input-len 1024 \
  --flush-cache --random-output-len 512 --num-prompts $N --max-concurrency $C

MI450 (gfx1250). Stock: rocm/sgl-dev:v0.5.23-rocm101-mi45x-20261107 (stock TheRock 10.1.0 ROCr and rocprofiler-sdk). This PR: rocm/sgl-dev:v0.5.23-rocm101-mi45x-20261107_8066cd93b__test. Both images use the same TheRock 10.1.0, aiter, RCCL, Triton and torch; the stock image's sglang is 66 commits newer. gpt-oss-120b runs with --disable-custom-all-reduce (RCCL all-reduce).

Serving without profiling

Random 1024 in / 512 out, 4 x concurrency prompts, mean of 2 sessions per setup.

Model	Concurrency	Median TPOT (ms): Stock	Stock + QI=0	This PR	Output tok/s: Stock	Stock + QI=0	This PR
gpt-oss-120b TP2	1	3.77	3.86	3.77	251.2	251.0	256.5
gpt-oss-120b TP2	16	8.55	8.62	8.55	1,732.7	1,717.3	1,731.3
gpt-oss-120b TP2	64	16.57	16.66	16.60	3,449.7	3,432.8	3,447.3
gpt-oss-120b TP2	128	22.10	22.12	22.11	4,978.6	4,978.1	4,981.2
Kimi-K2 TP8	1	12.14	12.02	11.95	80.3	81.0	81.3
Kimi-K2 TP8	16	23.42	24.72	23.87	649.5	619.4	638.3
Kimi-K2 TP8	64	43.91	46.17	44.07	1,346.1	1,289.6	1,346.6
Kimi-K2 TP8	128	59.35	60.37	58.48	1,940.5	1,909.1	1,966.0

QI=0 = ROCPROFILER_QUEUE_INTERPOSITION=0.

Profiling

sglang.benchmark.serving --profile, random 1024 in / 128 out, 16 prompts, concurrency 8, two profiled runs per setup.

Model	Setup	Result	Median TPOT while profiling (ms)	Median TPOT without profiling (ms)	Kernel events per rank	Median decode step in trace (ms)	GPU idle in profiled span (ms)
Kimi-K2 TP8	Stock	hangs*	-	-	-	-	-
Kimi-K2 TP8	Stock + QI=0	completes	28.66 / 28.37	20.76	528,346	27.16 / 27.10	1,882 / 1,762
Kimi-K2 TP8	This PR	completes	24.45 / 25.46	19.42	528,346	23.22 / 23.95	1,029 / 1,022
gpt-oss-120b TP2	Stock	hangs*	-	-	-	-	-
gpt-oss-120b TP2	Stock + QI=0	completes	9.70 / 9.10	7.39	170,095	9.16 / 9.12	989 / 1,126
gpt-oss-120b TP2	This PR	completes	7.74 / 7.78	7.20	170,095	7.58 / 7.52	819 / 843

* Measured on the 20261105 image, which has the same stock ROCr and rocprofiler-sdk.

GSM8K

5-shot, 1319 questions. Measured on the 20261105 image with the same library swap (Stock vs ROCr + 8564 and rocprofiler-sdk + 9984, 12957).

# gpt-oss-120b TP2
python3 -m sglang.test.few_shot_gsm8k --num-shots 5 --num-questions 1319 --parallel 512 --temperature 0

# Kimi-K2 TP8
python3 -m sglang.test.few_shot_gsm8k --num-shots 5 --num-questions 1319 --parallel 128

Model	Stock	This PR
gpt-oss-120b TP2	0.839 / 0.836	0.835 / 0.833
Kimi-K2 TP8	0.936	0.938 / 0.934

On the 20261107 image with this PR: gpt-oss-120b 0.837, Kimi-K2 0.929.

Speed Tests and Profiling

as mentioned, above

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #42330198586
Latest PR Test (Extra): ❌ Run #42330198527
Latest PR Test (AMD ROCm 10): ❌ Run #42330198639

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

[yctseng0211]

https://github.com/sgl-project/sglang/actions/runs/42330341416