Skip to content

[AMD] Update qwen3.5 cookbook #31258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
1am9trash merged 2 commits into from
Jul 15, 2026
Merged

[AMD] Update qwen3.5 cookbook #31258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
afdf906
Update qwen3.5 cookbook
1am9trash Jul 15, 2026
95c8e7a
Update qwen3.5 cookbook
1am9trash Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ This section provides deployment configurations optimized for different hardware

- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
- **H100 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings.
- **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`.
- **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`. Additionally set `AITER_FLYDSL_FORCE=1` to force the AITER FlyDSL MoE kernels and `SGLANG_MAMBA_SSM_DTYPE=bfloat16` to store the Mamba SSM state in bfloat16 (instead of the default float32).
- **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time.
- **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs.
Expand Down Expand Up @@ -258,6 +258,8 @@ Deploy Qwen3.5-397B-A17B with the following command (MI300X/MI325X/MI355X):
```shell Command
SGLANG_USE_AITER=1 \
SGLANG_USE_AITER_UNIFIED_ATTN=1 \
AITER_FLYDSL_FORCE=1 \
SGLANG_MAMBA_SSM_DTYPE=bfloat16 \
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3.5-397B-A17B \
--tp 8 \
Expand Down
6 changes: 5 additions & 1 deletion docs_new/src/snippets/autoregressive/qwen35-deployment.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -435,7 +435,11 @@ export const Qwen35Deployment = () => {
// SGLANG_USE_AITER=1 and SGLANG_USE_AITER_UNIFIED_ATTN=1; see cookbook prose),
// which requires --page-size 16. Enable AITER allreduce fusion for multi-GPU.
if (amdGpu) {
cmd = "SGLANG_USE_AITER=1 \\\nSGLANG_USE_AITER_UNIFIED_ATTN=1 \\\n" + cmd;
let amdEnv = "SGLANG_USE_AITER=1 \\\nSGLANG_USE_AITER_UNIFIED_ATTN=1 \\\nAITER_FLYDSL_FORCE=1 \\\n";
if (MOE_MODELS.has(model)) {
amdEnv += "SGLANG_MAMBA_SSM_DTYPE=bfloat16 \\\n";
}
cmd = amdEnv + cmd;
cmd += " \\\n --attention-backend aiter";
cmd += " \\\n --page-size 16";
if (hwConfig.tp > 1) {
Expand Down
Loading