[amd-nprotaso]

Requires AITER with ROCm/aiter#5992 (merged), which adds causal_conv1d_prefill_flydsl_fn.

Motivation

On ROCm, the GDN layers of Qwen3.5/Qwen3.8 run the prefill causal conv1d through Triton. AITER has a FlyDSL kernel for it on gfx950 that is 1.1x to 1.8x faster. This PR adds it behind an opt-in flag. Triton stays the default and the fallback.

Modifications

layers/attention/linear/gdn_backend.py only:

• The prefill convolution dispatches through _prefill_causal_conv1d. With SGLANG_USE_AITER=1 and SGLANG_AITER_GDN_CONV=1 it calls AITER's causal_conv1d_prefill_flydsl_fn, which has the same contract as causal_conv1d_fn: one output, conv_states updated in place.
• Fallback: an AITER ValueError (unsupported shape or dtype) falls back to Triton causal_conv1d_fn. When the flags are off or the import fails, that choice is cached and Triton is used.
• Block-table launches (conv_block_table set) stay on Triton.

Nothing changes on CUDA or when the flags are unset.

Accuracy Tests

Kernel level, on Qwen3.8 shapes, against Triton causal_conv1d_fn:

• Output: max abs diff 3.1e-2, which is 1 bf16 ULP.
• conv_states: bit-identical.

Speed Tests and Profiling

MI355X (gfx950), BF16, Qwen3.8 TP1 shapes (dim 10240, width 4), eager timing:

tokens in batch	Triton (µs)	FlyDSL (µs)	speedup
1 × 1024	22.5	19.6	1.14x
1 × 4096	62.6	47.6	1.32x
1 × 16384	313.8	181.2	1.73x
16 × 1024	324.9	179.8	1.81x

End to end

Setup:

• Hardware and model: MI355X, TP1, amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8.
• Server: NEXTN MTP (3 steps, 4 draft tokens), chunked prefill 16384, page size 64, the model's default fp32 SSM state.
• A/B: the arms differ only in SGLANG_AITER_GDN_CONV.
• Source tree: main cannot serve this model on ROCm yet, so these runs used the Qwen3.8 bring-up branch with this PR's hook applied. The hook code is byte-identical to this PR.

Synthetic (bench_serving random, radix cache off, simulated MTP acceptance 2.32, mean of 3 repeats per arm). Rows with OSL 1 are prefill-only.

ISL / OSL	conc	throughput	mean TTFT
16k / 1	12	+2.6%	−2.5%
16k / 1	16	+2.5%	−2.4%
64k / 1	16	+1.3%	−1.3%
128k / 1	16	+1.0%	−1.1%
128k / 1	1	+1.6%	−1.5%
4k / 1	16	+1.0%	−0.6%
4k / 1	1	−1.7%	+3.8%
8k / 1k	16	+0.6%	−1.4%
32k / 1k	8	+1.1%	−2.1%
32k / 1k	16	+1.3%	−1.9%

---

CI States

Latest PR Test (Base): ✅ Run #41421708727
Latest PR Test (Extra): ❌ Run #41421708779
Latest PR Test (AMD ROCm 10): ❌ Run #41421708692

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

[1am9trash]

Hi, @amd-nprotaso

I am a bit confused by the decode part change. I am not sure the decode kernel will be called or not?
Because supports_packed_decode may be True, so the decode part runs into packed_decode() and returns there. It may not reach decode().

# gdn_triton.py
supports_packed_decode = not is_cpu() and not is_npu()
# gdn_backend.py
# early return
if self.kernel_dispatcher.supports_packed_decode:
    core_attn_out = self.kernel_dispatcher.packed_decode(
        ......
    )
    return
# your change is inside this decode()
core_attn_out = self.kernel_dispatcher.decode(
    ......
)

Also, the current results look kernel-level only, so we cannot tell whether sglang actually runs these kernels. Could we add an e2e test? We may see improvement on TTFT for prefill(), ITL for decode(), and ITL with speculative decoding for verify().

[1am9trash]

Hi, @amd-nprotaso

Some other notes:

• This branch conflicts with main. I added the run-ci tag, but CI needs the conflict resolved before it can run.
• Could SGLANG_AITER_GDN_CONV and SGLANG_AITER_GDN_DECODE go in environ.py, default on for HIP? I know you mentioned that decode regresses at high conc, but users may not know where that threshold is. A batch-size guard in the dispatch may be easier than leaving it opt-in.

Thanks.

[amd-nprotaso]

Hi, @amd-nprotaso

I am a bit confused by the decode part change. I am not sure the decode kernel will be called or not? Because supports_packed_decode may be True, so the decode part runs into packed_decode() and returns there. It may not reach decode().

# gdn_triton.py
supports_packed_decode = not is_cpu() and not is_npu()
# gdn_backend.py
# early return
if self.kernel_dispatcher.supports_packed_decode:
    core_attn_out = self.kernel_dispatcher.packed_decode(
        ......
    )
    return
# your change is inside this decode()
core_attn_out = self.kernel_dispatcher.decode(
    ......
)

Also, the current results look kernel-level only, so we cannot tell whether sglang actually runs these kernels. Could we add an e2e test? We may see improvement on TTFT for prefill(), ITL for decode(), and ITL with speculative decoding for verify().

Correct. I move it to TritonGDNKernel.packed_decode, so new kernels, will be use even for packed scenario. End-to-end gains are small (1–2%), for conc 16, ISL=32k OSL=32k

[1am9trash]

Hi, @amd-nprotaso

I mentioned this in previous comments. Maybe overlook. XD!

Could we add a table for e2e perf in PR head?
It may help to clear ttft/itl change before/after the PR. Currently we only have kernel-side comparison, hard to know e2e impact of this PR.

Could SGLANG_AITER_GDN_CONV and SGLANG_AITER_GDN_DECODE go in environ.py, default on for HIP? I know you mentioned that decode regresses at high conc, but users may not know where that threshold is. A batch-size guard in the dispatch may be easier than leaving it opt-in.

Thanks.

[amd-nprotaso]

. Maybe overlook. XD!

I measure again with updated SGLang:

ISL	OSL	Conc	Interactivity (tok/s/user)	Throughput (tok/s/GPU)	Mean TTFT (ms)	Runs
32k	100	8	–	35,921 → 36,540 (+1.7%)	4,104 → 4,030 (−1.8%)	3
32k	100	12	–	36,655 → 37,297 (+1.8%)	5,786 → 5,678 (−1.9%)	3
32k	100	16	–	36,879 → 37,508 (+1.7%)	7,576 → 7,465 (−1.5%)	3
64k	100	8	–	35,288 → 35,890 (+1.7%)	8,378 → 8,211 (−2.0%)	3
64k	100	12	–	35,790 → 36,274 (+1.4%)	11,889 → 11,733 (−1.3%)	3
64k	100	16	–	35,928 → 36,429 (+1.4%)	15,591 → 15,398 (−1.2%)	3
32k	32k	8	146.8 → 148.6 (+1.2%)	2,356 → 2,386 (+1.3%)	4,107 → 4,010 (−2.4%)	1
32k	32k	12	124.4 → 123.8 (−0.5%)	2,969 → 2,973 (+0.1%)	5,843 → 5,674 (−2.9%)	1
32k	32k	16 †	113.1 → 113.5 (+0.3%)	2,843 → 2,874 (+1.1%)	24,953 → 24,720 (−0.9%)	1
64k	32k	8	140.6 → 142.7 (+1.4%)	3,290 → 3,326 (+1.1%)	8,358 → 8,258 (−1.2%)	1
64k	32k	12	120.0 → 118.9 (−1.0%)	4,130 → 4,131 (0.0%)	11,935 → 11,681 (−2.1%)	1
64k	32k	16 †	108.0 → 108.5 (+0.4%)	3,936 → 3,982 (+1.2%)	33,786 → 33,430 (−1.1%)	1

And see Interactivity degradation for conc=12, so maybe will close this PR.

[1am9trash]

All pr-test-base passed.
Link: https://github.com/sgl-project/sglang/actions/runs/40177142363/job/122618304704?pr=41879

[1am9trash]

Wrong approval.
Didn't see the regression issue.
Please feel free to close or modify the PR.

[amd-nprotaso]

Wrong approval. Didn't see the regression issue. Please feel free to close or modify the PR.

Updated PR, now it's only about convolution kernel. Add e2e results for different scenarios.