[XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 by Xia-Weiwen · Pull Request #33354 · sgl-project/sglang

Xia-Weiwen · 2026-08-03T06:33:14Z

Motivation

To use a fused GDN kernel provided by sgl-kernel-xpu for better performance than what the existing triton kernels in SGLang produce.

Modifications

Added a subclass of GDNAttnBackend for XPU, which does some checks and calls the fused GDN kernel.
Added a simple dispatch mechanism for the fused GDN kernel on XPU with existing paths unchanged.

Accuracy Tests

Qwen3.5-9B, GSM8K, 200 samples

Fused GDN kernel from sgl-kernel-xpu

Accuracy: 0.910
Invalid: 0.000

Triton GDN in SGLang

Accuracy: 0.900
Invalid: 0.000

Speed Tests and Profiling

Qwen3.5-9B on a Intel Arc BMG60 machine

python -I -m sglang.bench_one_batch --batch-size 1 --input 4096 --output 1024 --model Qwen/Qwen3.5-9B --tp 1 --device xpu --mem-fraction-static 0.9 --attention-backend intel_xpu --linear-attn-backend intel_xpu --disable-radix-cache --profile --profile-stage all --profile-activities CPU XPU --chunked-prefill-size -1 --max-prefill-tokens 20480 --max-total-tokens 20480

Latency	Old	New
Prefill E2E	0.937 s	0.808 s
Prefill GDN	165 ms	59.17 ms
Decode E2E	0.0404 s	0.0402 ms
Decode GDN	0.522 ms	0.452 ms

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #32383609683
Latest PR Test (Extra): ⏳ Run #32688872423
Latest PR Test (AMD ROCm 7.2): ❌ Run #32383610036

gemini-code-assist · 2026-08-03T06:33:17Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

gemini-code-assist · 2026-08-04T05:43:43Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Xia-Weiwen · 2026-08-04T05:43:54Z

/tag-and-rerun-ci

Xia-Weiwen · 2026-08-04T08:50:37Z

Hi @mingfeima @airMeng Can you please review? Thanks.

mingfeima · 2026-08-13T01:29:44Z

+        core_attn_out = z = None
+        from sglang.srt.model_executor.forward_context import get_attn_backend
+
+        backend = get_attn_backend()
+        backend = getattr(backend, "linear_attn_backend", backend)
+        if hasattr(backend, "forward_fused_gdn") and backend.supports_fused_gdn(
+            self.attn, forward_batch
+        ):


doing such a backend chosen in a forward seems not appropriate.

how does the original code choose the backend? when it is settled?

The original path does not use a fused kernel (CUDA, CPU, etc.). So, they just go to the else branch. We need to intercept here because we call a fused kernel.

Updated. Now we use a separate _forward_xpu.

… of a runtime flag

mingfeima

some minor issues to address.

mingfeima · 2026-08-20T02:00:45Z

@Xia-Weiwen could you please check with validation team why xpu ci failed again?

Xia-Weiwen · 2026-08-24T03:10:40Z

@Xia-Weiwen could you please check with validation team why xpu ci failed again?

CI should be good now.

…n-transport1 * 'main' of https://github.com/sgl-project/sglang: (326 commits) [diffusion] feat: cache LoRA-merged weights in files the page cache can hold (sgl-project#36062) [diffusion] Speed up LingBot high-quality VAE decode (sgl-project#36024) [diffusion] Honor XDG cache for model overlays (sgl-project#36019) Support streaming session on NPU (sgl-project#32597) fix(xpu): read enable_deterministic_inference from the config bag (sgl-project#36149) xeon ci fail fast strategy change (sgl-project#36146) [diffusion] Fix Hunyuan QKV pack indexing at production video shapes (sgl-project#36009) [diffusion] Refresh quality and BCG benchmark skills (sgl-project#36016) [MoE] Gather the cutlass MoE activation and its scales in one launch (sgl-project#34915) [diffusion] feat: add plain component weight overrides (sgl-project#36086) [diffusion] feat: support loading mixed w4a8 text encoders (sgl-project#36037) [diffusion] Default Hunyuan VAE to tiled decode (sgl-project#36012) fix(xpu): enable compressed-tensors FP8 W8A8 on XPU (RedHatAI FP8-dynamic models) (sgl-project#33057) chore: move cuda_vmm_utils.py under srt/utils/ (sgl-project#36053) [Intel XPU] Add xpu pass for biased_topk and hash_topk (sgl-project#33323) [CPU] Fix NUMA/core binding for DP ranks (sgl-project#32856) [Fix] Harden FlashAttention CUDA graph metadata bounds (sgl-project#35454) [XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 (sgl-project#33354) [diffusion] Fuse LongCat-Image QKNorm and interleaved RoPE (sgl-project#35995) [diffusion] Keep LongLive2 components resident on large GPUs (sgl-project#35993) ... # Conflicts: # python/sglang/srt/multimodal/processors/base_processor.py # python/sglang/srt/server_args.py

…#33354)

Xia-Weiwen added 10 commits July 17, 2026 18:25

[XPU] Using GDN kernels from sgl-kernel instead of triton

9feeb64

Fix mrope dtype mismatch

018c98e

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

5e9641f

Support TP>1

297fda5

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

55bd29c

Use workspace buffer for MOE and GDN

4506e9d

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

020d83a

Update call site for the new op signature

612fbc5

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

a519bcb

Update python/sglang/srt/layers/quantization/unquant.py

6e6d54e

Move changes to a dedicated place for XPU

45db74b

Xia-Weiwen commented Aug 4, 2026

View reviewed changes

Comment thread python/sglang/srt/layers/rotary_embedding/mrope.py Outdated

Xia-Weiwen added 2 commits August 4, 2026 13:39

Refine code

6662e5e

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

50d8472

Xia-Weiwen marked this pull request as ready for review August 4, 2026 05:43

Xia-Weiwen requested review from Fridge003, HaiShaw, Qiaolin-Yu, Ying1123, hebiao064, ispobock and merrymercy as code owners August 4, 2026 05:43

Xia-Weiwen requested review from BBuf, Edwardf0t1 and ch-wan as code owners August 4, 2026 05:43

github-actions Bot added the run-ci label Aug 4, 2026

mingfeima approved these changes Aug 13, 2026

View reviewed changes

mingfeima reviewed Aug 13, 2026

View reviewed changes

Xia-Weiwen added 8 commits August 13, 2026 10:21

Add _forward_xpu for Qwen3_5GatedDeltaNet

093fb5f

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

0203061

Refine code

77ff7d9

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

389ff9e

Refine code

0d429ec

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

23490fe

Use the server arg --linear-attn-backend to specify intel_xpu instead…

d44bee6

… of a runtime flag

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

3e9deab

airMeng mentioned this pull request Aug 18, 2026

Improve xpu gdn #27020

Draft

5 tasks

mingfeima requested changes Aug 18, 2026

View reviewed changes

Comment thread python/sglang/srt/models/qwen3_5.py

Xia-Weiwen added 2 commits August 19, 2026 09:33

Refine GDN forward_xpu

a792a0f

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

2ab0958

Xia-Weiwen requested a review from mingfeima August 19, 2026 01:36

mingfeima requested changes Aug 20, 2026

View reviewed changes

Xia-Weiwen mentioned this pull request Aug 20, 2026

[Bug] [XPU] Qwen3.5 GDN + speculative decode: causal_conv1d_update_xpu() got an unexpected keyword argument 'intermediate_conv_window' #34720

Closed

5 tasks

Xia-Weiwen added 3 commits August 20, 2026 13:24

Refine per comments

7036be6

Merge branch 'main' into qwen3.5_gdn_xpu_kernel

1e3b207

Update UT

c8ed12a

jiayisunx mentioned this pull request Aug 21, 2026

[GDN] remove XPU path of causal_conv1d_fn and causal_conv1d_update #35775

Merged

5 tasks

mingfeima approved these changes Aug 24, 2026

View reviewed changes

mingfeima merged commit 5b5b29d into sgl-project:main Aug 24, 2026
137 of 150 checks passed

mingfeima added intel xpu intel gpu with device `torch.xpu` labels Aug 24, 2026

saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 31, 2026

[XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 (sgl-project…

5cf4566

…#33354)

jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026

[XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 (sgl-project…

dcf3022

…#33354)

Uh oh!

Conversation

Xia-Weiwen commented Aug 3, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

gemini-code-assist Bot commented Aug 3, 2026

Uh oh!

Uh oh!

gemini-code-assist Bot commented Aug 4, 2026

Uh oh!

Xia-Weiwen commented Aug 4, 2026

Uh oh!

Xia-Weiwen commented Aug 4, 2026

Uh oh!

mingfeima Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Xia-Weiwen Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Xia-Weiwen Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mingfeima left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mingfeima commented Aug 20, 2026

Uh oh!

Xia-Weiwen commented Aug 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Xia-Weiwen commented Aug 3, 2026 •

edited by github-actions Bot

Loading