[AMD] Support alt stream for Qwen3.5 on AMD platform

[At1a8]

Motivation

• Enable cuda alt stream for AMD platform with SGLANG_ALT_STREAM flag
• Use SGLANG_QK_NORM_ALT_STREAM flag to disable qknorm alt stream, because fused_qk_gemma_rmsnorm could be more effcient than alt stream in AMD platform. On the other device like Nvidia platform, q&k will continue use alt stream path based on default value of SGLANG_QK_NORM_ALT_STREAM

• When both alt stream and shared expert fusion are enabled simultaneously, the shared expert and routed expert would run together on the alt stream instead of the main stream, the alt stream should be disabled when moe layer enable shared expert fusion, we use server args disable_shared_experts_fusion flag to control it.
• Use alt stream to run qkvz and ba in GDN module also cause the perf regression. we use SGLANG_GDN_QKVZ_BA_ALT_STREAM to disable it when enable alt stream

Modifications

• Alt stream(Enable alt stream with _hip_use_alt_stream & _is_hip flag)
  • MOE: Using server args disable_shared_experts_fusion, it set as false by default. When enable _shared_expert_fusion, sglang will calculate num_fused_shared_experts and disable moe alt_stream. Otherwise, will use alt stream to execute shared experts and routed experts at different stream.
  • QKVZ & BA in GDN: Use SGLANG_GDN_QKVZ_BA_ALT_STREAM to control if QKVZBA need use alt stream, the flag set as false by default, cause it make perf regression.
  • QK NORM: Use SGLANG_QK_NORM_ALT_STREAM to control if qknorm need use alt stream, the flag set as false by default, cause it make perf regression.

Accuracy Tests

root@smci355-ccs-aus-n12-13:/mnt/raid0/fangyuan/qwen3.5# bash client.sh
100%|████████████████████████████████████████████████████████████████████████████| 200/200 [00:37<00:00, 5.39it/s]
Accuracy: 0.955
Invalid: 0.000
Latency: 37.119 s
Output throughput: 888.580 token/s

Speed Tests and Profiling

The results based on Qwen3.5-397B-A17B-FP8 @ MI355X , I/O=8k/1k

disable_shared_experts_fusion	SGLANG_ALT_STREAM	concurrency	TTFT	TPOT	GSM8K
True	False	1	184.18ms	10.79ms	0.975
False	False	1	180.47ms	9.00ms	0.955
True	True	1	178.81ms	10.60ms	0.965

• disable_shared_experts_fusion =True, SGLANG_ALT_STREAM=False
  • baseline, no any optimi method
• disable_shared_experts_fusion =False, SGLANG_ALT_STREAM=False
  • current method in sglang, TPOT uplift 16.6% compared to baseline
• disable_shared_experts_fusion =True, SGLANG_ALT_STREAM=True
  • this PR enabled, TPOT uplift 1.76% compared to baseline
• disable_shared_experts_fusion =False, SGLANG_ALT_STREAM=True
  • disable_shared_experts_fusion will disable SGLANG_ALT_STREAM, same as current method in sglang

More results based on Qwen3.5-397B-A17B-FP8 @ MI355X , I/O=8k/1k with higher concurrency(TP8 and TP8EP8)
It's very similar to what we see in small concurrency.

disable_shared_experts_fusion	SGLANG_ALT_STREAM	max_concurrency	num_prompts	ttft_ms(TP8)	tpot_ms(TP8)	ttft_ms(TP8 EP8)	tpot_ms(TP8 EP8)
TRUE	FALSE	1	5	177.93	10.75	173.66	11.54
FALSE	FALSE	1	5	204.56	8.96	205.35	9.69
TRUE	TRUE	1	5	174.51	10.6	172.05	11.55
TRUE	FALSE	2	10	259.22	11.37	242.27	12.32
FALSE	FALSE	2	10	250.65	9.48	260.6	10.42
TRUE	TRUE	2	10	260.15	11.32	246.41	12.43
TRUE	FALSE	4	20	424.93	12.05	397.56	13.1
FALSE	FALSE	4	20	418.91	10.1	425.66	11.17
TRUE	TRUE	4	20	423.56	11.96	392.24	12.99
TRUE	FALSE	8	40	719.95	13.19	663.54	14.12
FALSE	FALSE	8	40	707.11	11.13	714.35	11.92
TRUE	TRUE	8	40	719.71	13.02	666.5	13.83
TRUE	FALSE	16	80	1284.22	14.78	1181.45	15.64
FALSE	FALSE	16	80	1256.88	12.52	1275.31	13.02
TRUE	TRUE	16	80	1284.11	14.23	1183.86	15.16
TRUE	FALSE	32	160	2423.16	18.31	2220.07	19.59
FALSE	FALSE	32	160	2384.87	15.89	2389.1	16.05
TRUE	TRUE	32	160	2449.43	17.76	2230.46	19.18
TRUE	FALSE	64	320	4687.13	25.44	4318.94	25.82
FALSE	FALSE	64	320	4587.92	23.38	4629.1	22.52
TRUE	TRUE	64	320	4721.98	24.9	4302.45	25.15
TRUE	FALSE	128	640	9196.03	36.61	8466.09	36.94
FALSE	FALSE	128	640	9046.8	34.28	9012.84	34.1
TRUE	TRUE	128	640	9241.55	36.13	8433.52	36.52

Therefore, alt streams can currently provide performance improvements.
However, the gains in the MoE component are not as significant as those from shared expert fusion, and they also introduce performance regressions in QKNorm and GDN.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #26988695592
Latest PR Test (Extra): ❌ Run #26988695503

[gemini-code-assist]

Code Review

This pull request introduces the SGLANG_HIP_ALT_STREAM environment variable to enable alternative stream support on HIP platforms and adds a configuration check for SGLANG_QK_NORM_ALT_STREAM during Q/K normalization. The review feedback suggests caching the SGLANG_QK_NORM_ALT_STREAM environment variable at the module level to avoid redundant lookups in the performance-sensitive _apply_qk_norm method.

[hubertlu-tw]

@At1a8
For multistream usage on our hardware, you will need to use at least export GPU_MAX_HW_QUEUES=5 if you are not aware of it.

You can refer to this merged PR and this comment.
CC: @HaiShaw

[HaiShaw]

/tag-and-rerun-ci

[At1a8]

@At1a8 For multistream usage on our hardware, you will need to use at least export GPU_MAX_HW_QUEUES=5 if you are not aware of it.

You can refer to this merged PR and this comment. CC: @HaiShaw

Hi @hubertlu-tw, just add some experiments and analysis in this page.

I found alt streams can currently provide performance improvements.
However, the gains in the MoE component are not as significant as those from shared expert fusion(which has been enabled in sglang), and they also introduce performance regressions in QKNorm and GDN.

CC: @HaiShaw

[hubertlu-tw]

@At1a8 could you please clarify what --max-concurrency you used to profile with bench_serving.py? As Mingzhi mentioned in the discussion chat,

Trade-off: multi-stream vs. single-stream for EP

Benefit: Multi-stream execution can overlap the cost of sharded expert GEMMs.
Cost: Multi-stream also introduces async EP kernels, which add more latency than non-async EP kernels.
So there is a workload-dependent trade-off:

Small batch size (low-latency workloads): The expert GEMMs are small, so the async EP overhead (2) often outweighs the GEMM overlap benefit (1) → 2 > 1.
Large batch size: The GEMM overlap benefit (1) grows with batch size, while the async EP overhead (2) does not grow as quickly → 1 > 2, so multi-stream can be worthwhile.

If we have perf numbers for low concurrency and high concurrency, it would be great.
Besides that, I think the code changes in this PR look good to me.

cc: @HaiShaw @kkHuang-amd @1am9trash

[At1a8]

@At1a8 could you please clarify what --max-concurrency you used to profile with bench_serving.py? As Mingzhi mentioned in the discussion chat,

Trade-off: multi-stream vs. single-stream for EP
Benefit: Multi-stream execution can overlap the cost of sharded expert GEMMs.
Cost: Multi-stream also introduces async EP kernels, which add more latency than non-async EP kernels.
So there is a workload-dependent trade-off:
Small batch size (low-latency workloads): The expert GEMMs are small, so the async EP overhead (2) often outweighs the GEMM overlap benefit (1) → 2 > 1.
Large batch size: The GEMM overlap benefit (1) grows with batch size, while the async EP overhead (2) does not grow as quickly → 1 > 2, so multi-stream can be worthwhile.

If we have perf numbers for low concurrency and high concurrency, it would be great. Besides that, I think the code changes in this PR look good to me.

cc: @HaiShaw @kkHuang-amd @1am9trash

Hi @hubertlu-tw, this results based on max_concurrency/num_prompts=1, will add more experiment data for larger concurrency later.

cc: @HaiShaw @kkHuang-amd @1am9trash

[At1a8]

Trade-off: multi-stream vs. single-stream for EP

Benefit: Multi-stream execution can overlap the cost of sharded expert GEMMs. Cost: Multi-stream also introduces async EP kernels, which add more latency than non-async EP kernels. So there is a workload-dependent trade-off:

Small batch size (low-latency workloads): The expert GEMMs are small, so the async EP overhead (2) often outweighs the GEMM overlap benefit (1) → 2 > 1. Large batch size: The GEMM overlap benefit (1) grows with batch size, while the async EP overhead (2) does not grow as quickly → 1 > 2, so multi-stream can be worthwhile.

Hi @hubertlu-tw, I have added more cases with larger concurrency (concurrency from 1 to 128, TP8 && TP8EP8), but overall the conclusion is basically consistent with that under low concurrency, i.e., shared expert fusion>alt stream>baseline.

[hubertlu-tw]

LGTM

[hubertlu-tw]

cc: @mqhc2020

[mqhc2020]

Running on Qwen3.5 397B MXFP4, which disables shared expert fusion:

(1k,1k) — full sweep cc=4..256

cc	tok/s	Δ%	median TPOT (ms)	Δ%	median ITL (ms)
4	342.9	−0.5%	11.20	+0.5%	10.90
8	604.4	+1.4%	12.85	−1.6%	12.24
16	962.5	+2.5%	16.28	−2.4%	14.92
32	1532.7	+2.0%	20.34	−2.3%	17.77
64	2211.7	+2.2%	28.45	−2.4%	23.10
128	3057.3	+3.1%	41.56	−2.9%	31.72
256	4076.4	+1.5%	62.56	−1.6%	44.10

Baseline reference (1k,1k): cc4 344.7 / cc256 4014.9 tok/s.

(8k,1k) — matched cc=4..64

cc	tok/s	Δ%	median TPOT (ms)	Δ%	median ITL (ms)
4	315.8	−0.1%	11.85	−0.2%	11.00
8	520.5	+1.3%	14.40	−1.2%	12.25
16	772.7	+1.7%	19.45	−1.7%	15.06
32	1101.8	+1.3%	27.38	−1.5%	18.03
64	1396.9	+0.8%	43.90	−0.5%	24.51

[HaiShaw]

@amd-bot ci-status

[amd-bot]

@HaiShaw

CI Status for PR #25885

PR: [AMD] Support alt stream for Qwen3.5 on AMD platform
Changed files: python/sglang/srt/models/qwen3_5.py (+20/-7)
Head SHA: c4b0bd70

AMD: 2 failures (0 likely related) | Others: 7 failures (1 likely related)

AMD CI Failures

Job	Test File	Test Function	Error	Related?	Explanation	Log
stage-b-test-large-8-gpu-mi35x-disaggregation-amd	test/registered/amd/disaggregation/test_mori_transfer_engine_e2e.py	test_generate_smoke_hybrid_mamba	AssertionError: 500 != 200 : Internal Server Error (prefill server returns 500 during /generate)	🟢 Unlikely	Test uses Qwen/Qwen3-Next-80B-A3B-Instruct (hybrid-mamba), not Qwen3.5; qwen3_5.py is not imported by this code path. Disagg server crash appears infrastructure-related.	Log
pr-test-amd-finish	N/A	N/A	stage-b-test-large-8-gpu-mi35x-disaggregation-amd: failure	🟢 Unlikely	Aggregator job — cascades from the disagg failure above.	Log

Other CI Failures

Job	Test File	Test Function	Error	Related?	Explanation	Log
base-a-test-cpu (3)	test/registered/unit/models/test_qwen3_5_packed_weight_loader.py	module import	ValueError: Global server args is not set yet! at qwen3_5.py:108 during from sglang.srt.models.qwen3_5 import Qwen3_5GatedDeltaNet	🔴 Likely	PR-caused. The diff adds _disable_shared_experts_fusion = get_global_server_args().disable_shared_experts_fusion at module top-level (line 108). get_global_server_args() raises when called before server args are initialized, which is exactly the case for this unit test that imports the model class directly. Same applies to the other 3 new module-level get_bool_env_var(...) flags — those won't raise, but the get_global_server_args() call does.	Log
wait-for-base-a	N/A	N/A	base-a jobs failed: base-a-test-cpu (3)	🟢 Unlikely	Wait-for job — cascades from base-a-test-cpu (3) above.	Log
pr-test-finish	N/A	N/A	wait-for-base-a: failure	🟢 Unlikely	Aggregator — cascades from the same root cause.	Log
call-gate / pr-gate	N/A	N/A	Missing required label 'run-ci-extra' → exit 1	🟢 Unlikely	Label gate — run-ci-extra is not on the PR (only amd and run-ci). Add the label (or /tag-and-rerun-ci extra) to opt into the extra workflow.	Log
pr-test-extra-finish	N/A	N/A	call-gate: failure	🟢 Unlikely	Aggregator — cascades from missing run-ci-extra label.	Log
stage-b-test-1-npu-a2 (0)	test/registered/ascend/basic_function/quant/test_npu_w8a8_quantization.py	test_gsm8k	AssertionError: 639.4461840551697 not greater than or equal to 700 (output throughput threshold)	🟢 Unlikely	NPU-only Ascend w8a8 throughput threshold; Qwen3.5 model code is not in this path. Likely NPU perf flake / threshold tuning.	Log
pr-test-npu-finish	N/A	N/A	stage-b-test-1-npu-a2: failure	🟢 Unlikely	Aggregator — cascades from the NPU perf failure above.	Log

Details

🔴 base-a-test-cpu (3) — test_qwen3_5_packed_weight_loader.py

The PR adds these module-level statements to python/sglang/srt/models/qwen3_5.py (lines 108–115 in the diff):

_disable_shared_experts_fusion = get_global_server_args().disable_shared_experts_fusion
_hip_use_alt_stream = get_bool_env_var("SGLANG_ALT_STREAM") and _is_hip
_gdn_use_alt_stream = (
    get_bool_env_var("SGLANG_GDN_QKVZ_BA_ALT_STREAM", "False") and _hip_use_alt_stream
)
_qknorm_use_alt_stream = (
    get_bool_env_var("SGLANG_QK_NORM_ALT_STREAM", "False") and _hip_use_alt_stream
)

The unit test test_qwen3_5_packed_weight_loader.py does from sglang.srt.models.qwen3_5 import Qwen3_5GatedDeltaNet without first initializing global server args, so the very first statement raises in python/sglang/srt/server_args.py:7890:

ValueError: Global server args is not set yet!

Stack trace from the log:

File ".../test/registered/unit/models/test_qwen3_5_packed_weight_loader.py", line 22, in <module>
    from sglang.srt.models.qwen3_5 import Qwen3_5GatedDeltaNet
File ".../python/sglang/srt/models/qwen3_5.py", line 108, in <module>
    _disable_shared_experts_fusion = get_global_server_args().disable_shared_experts_fusion
File ".../python/sglang/srt/server_args.py", line 7890, in get_global_server_args
    raise ValueError("Global server args is not set yet!")

Suggested fix direction (you decide): read disable_shared_experts_fusion lazily inside __init__ (or via a small cached helper that calls get_global_server_args() on first use) rather than at module import time, mirroring how other model files avoid this exact import-time dependency on global server args. The three get_bool_env_var(...) flags are import-time-safe and can stay where they are.

Other failures: all 8 remaining failures are either downstream aggregator/wait jobs cascading from the one real failure (or from the missing run-ci-extra label), or unrelated AMD-disagg / NPU-perf signals that don't touch qwen3_5.py.

Generated by amd-bot using Claude Code CLI

[HaiShaw]

@At1a8 would you please take a look at the possible CI failure?

[At1a8]

@At1a8 would you please take a look at the possible CI failure?

Hi @HaiShaw @hubertlu-tw

I have fixed this issue by lazy import/init this variable, CI passed at my side

[At1a8]

@amd-bot ci-status