Skip to content

qwen 3.8 rebase - #35758

Merged
Fridge003 merged 42 commits into
mainfrom
qwen-qiaolin
Aug 29, 2026
Merged

qwen 3.8 rebase#35758
Fridge003 merged 42 commits into
mainfrom
qwen-qiaolin

Conversation

@Qiaolin-Yu

@Qiaolin-Yu Qiaolin-Yu commented Aug 20, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Collaborator

initial pr: #34585

Following work

  • upgrade flashinfer to 0.6.18
  • remove all the flashinfer patches in this pr

Note that comparing with the day 0 image, this pr is not using flashinfer gdn prefill cp kernel.

CI States

Latest PR Test (Base): 🚫 Run #33213936462
Latest PR Test (Extra): ✅ Run #33230635480
Latest PR Test (AMD ROCm 7.2): 🚫 Run #33213936196

Qiaolin-Yu and others added 11 commits August 20, 2026 20:21
@Qiaolin-Yu @cherichy
@guangyunh-nv @jiahanc @jinyangyuan-nvidia @Njuapp @qiangyicheng @samuellees @Tom-Zheng @YAMY1234 @xiaoweiw-nv @zju-stu-lizheng @yizhang2077 @ispobock @JustinTong0323 @yhyang201 @zijiexia
introduce qwen 3.8
251a0bf 
Co-authored-by: cherichy <cherichy@outlook.com>
Co-authored-by: guangyunh-nv <guangyunh@nvidia.com>
Co-authored-by: jiahanc <jiahanc@nvidia.com>
Co-authored-by: jinyangyuan-nvidia <joyuan@nvidia.com>
Co-authored-by: Cheng Hang <chang@nvidia.com>
Co-authored-by: Yicheng Qiang <yqiang@nvidia.com>
Co-authored-by: Sam Li <lsam@nvidia.com>
Co-authored-by: Tom-Zheng <tizheng@nvidia.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: xiaoweiw-nv <xiaoweiw@nvidia.com>
Co-authored-by: Zheng Li <lizheng.cs@zju.edu.cn>
Co-authored-by: yizhang2077 <1109276519@qq.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
@Qiaolin-Yu
upd
cc2ae7a
@Qiaolin-Yu
upd
e75eb52
@Qiaolin-Yu
upd
26fabc5
@Qiaolin-Yu
upd
467c0d5
@Qiaolin-Yu
upd
63ce746
@Qiaolin-Yu
upd
a0a79c5
@Qiaolin-Yu
upd
7643a3f
@Qiaolin-Yu
upd
2316a26
@Qiaolin-Yu
upd
3200d0b
@Qiaolin-Yu
upd
f4d3335
@YAMY1234

YAMY1234 commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

All CI passed! https://github.com/sgl-project/sglang/actions/runs/33163337156/job/98867705789?pr=35758

Qiaolin-Yu
Qiaolin-Yu commented Aug 28, 2026
View reviewed changes
Comment thread python/sglang/srt/managers/overlap_utils.py Outdated
@Fridge003 Fridge003 added the release-highlight Candidate PR for release note highlight label Aug 29, 2026
@Fridge003
Fridge003 merged commit 5f216fc into main Aug 29, 2026
201 of 225 checks passed
@Fridge003
Fridge003 deleted the qwen-qiaolin branch August 29, 2026 03:41
@whn09 whn09 mentioned this pull request Aug 31, 2026
Open
5 tasks
saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 31, 2026
@Qiaolin-Yu @cherichy
@guangyunh-nv @jiahanc @jinyangyuan-nvidia @Njuapp @qiangyicheng @samuellees @Tom-Zheng @YAMY1234 @xiaoweiw-nv @zju-stu-lizheng @yizhang2077 @ispobock @JustinTong0323 @yhyang201 @zijiexia @github-actions @saturn-acc
qwen 3.8 rebase (sgl-project#35758)
c25a47e 
Co-authored-by: cherichy <cherichy@outlook.com>
Co-authored-by: guangyunh-nv <guangyunh@nvidia.com>
Co-authored-by: jiahanc <jiahanc@nvidia.com>
Co-authored-by: jinyangyuan-nvidia <joyuan@nvidia.com>
Co-authored-by: Cheng Hang <chang@nvidia.com>
Co-authored-by: Yicheng Qiang <yqiang@nvidia.com>
Co-authored-by: Sam Li <lsam@nvidia.com>
Co-authored-by: Tom-Zheng <tizheng@nvidia.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: xiaoweiw-nv <xiaoweiw@nvidia.com>
Co-authored-by: Zheng Li <lizheng.cs@zju.edu.cn>
Co-authored-by: yizhang2077 <1109276519@qq.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
nzr-niu pushed a commit to nzr-niu/sglang that referenced this pull request Sep 1, 2026
@Qiaolin-Yu @cherichy
@guangyunh-nv @jiahanc @jinyangyuan-nvidia @Njuapp @qiangyicheng @samuellees @Tom-Zheng @YAMY1234 @xiaoweiw-nv @zju-stu-lizheng @yizhang2077 @ispobock @JustinTong0323 @yhyang201 @zijiexia @github-actions Niu Ziru
qwen 3.8 rebase (sgl-project#35758)
3de280a 
Co-authored-by: cherichy <cherichy@outlook.com>
Co-authored-by: guangyunh-nv <guangyunh@nvidia.com>
Co-authored-by: jiahanc <jiahanc@nvidia.com>
Co-authored-by: jinyangyuan-nvidia <joyuan@nvidia.com>
Co-authored-by: Cheng Hang <chang@nvidia.com>
Co-authored-by: Yicheng Qiang <yqiang@nvidia.com>
Co-authored-by: Sam Li <lsam@nvidia.com>
Co-authored-by: Tom-Zheng <tizheng@nvidia.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: xiaoweiw-nv <xiaoweiw@nvidia.com>
Co-authored-by: Zheng Li <lizheng.cs@zju.edu.cn>
Co-authored-by: yizhang2077 <1109276519@qq.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
TobyMint added a commit to TobyMint/sglang that referenced this pull request Sep 1, 2026
@TobyMint
[qwen3_5] Fused GDN decode QKVZ/BA unpack + indexed Conv1D (sgl-project
85c351e 
sgl-project#35758)

Decode fusion: the projected QKVZ/BA pair stays packed and one Triton
kernel unpacks it, applies the indexed causal Conv1D state update, and
materializes mixed_qkv/z/b/a, replacing the split kernel plus a second
conv launch. Eligibility-checked with an explicit fallback, env-gated
(SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV, default on; names kept
upstream-verbatim for future syncs), with an opt-in live parity check
(SGLANG_GDN_DECODE_FUSION_VERIFY_REAL_TENSORS) and per-layer hit logs.
Env vars registered per the Envs convention.

Ported test covers parity vs split+causal_conv1d_update, widths 2-4,
dtypes, CUDA-graph replay, OOB-slot masking, and the fp8 fallback
(5/5 on H20).

Measured on H20 TP1, dense 27B: neutral-to-slightly-positive decode
(bs8 TPOT 18.91 vs 19.04 ms; bs64 21.91 vs 22.02 ms); gsm8k 200q
0.965 vs 0.975 (within run noise). Caveat: the live parity check
fires during CUDA graph capture because capture dummy batches alias
mamba slots across rows (reference uses a private state copy); it is
a debug-only mode and real batches have unique slots.
@nvpohanh nvpohanh mentioned this pull request Sep 1, 2026
Merged
@TobyMint TobyMint mentioned this pull request Sep 2, 2026
Open
@HanHan009527 HanHan009527 mentioned this pull request Sep 2, 2026
Open
@Qiaolin-Yu Qiaolin-Yu mentioned this pull request Sep 2, 2026
Closed
b8zhong added a commit that referenced this pull request Sep 4, 2026
@b8zhong
[Comm] Drop the CuTe DSL workspace instance-count env var
14296e3 
SGLANG_FLASHINFER_MNNVL_CUTEDSL_AR_FUSION_MAX_INSTANCES had no valid setting.
The registry keys workspaces by full signature and one model configuration
exists per process, so the count is an invariant rather than a knob; it is now
a hardcoded guard with the same fail-closed behavior at the same call site.

The optional-argument form of get_flashinfer_mnnvl_cutedsl_ar_fusion() went
with it: the no-argument lookup only had to disambiguate between several
workspaces, and its sole caller always supplied all five. The arguments are
now required.

Added in #35758 alongside the gate env var and never released, so no
deprecation alias.
b8zhong added a commit that referenced this pull request Sep 4, 2026
@b8zhong
[Comm] Delete dead surface in the CuTe DSL AR fusion
7a90133 
Removes machinery that no caller reaches, most of it inherited from #35758.

Workspace module:
- The registry (signature dataclass, dict, RLock, best-fit search) held at
  most one entry once the instance cap became an invariant; it collapses to a
  single module global.
- destroy() was never called and _destroyed was therefore always False, so
  supports() carried a check that could not fire.
- moe_finalize_all_reduce_rms_norm / all_reduce_residual_rms_norm took
  optional norm_output / residual_output that only the unit test supplied;
  production always let them allocate.

Fusion core:
- The service re-validated shapes, dtypes and contiguity on every eligible
  layer of every forward. The handoff builds its own views, and the workspace
  already refuses an M it cannot serve, so this was duplicated work on the
  hot path.
- supports() also duplicated the workspace's range check, and is_prepared was
  subsumed by it (supports() is False before prepare()).

Qwen3.5 final norm:
- SGLANG_TRACE_QWEN35_FINAL_NORM and SGLANG_QWEN35_NATIVE_FINAL_NORM were
  bring-up scaffolding: print() plus torch.cuda.synchronize() around the norm,
  and a forward_native A/B switch. Both env vars are deleted; like the other
  two they shipped in #35758 and were never released.
@ch-wan ch-wan mentioned this pull request Sep 5, 2026
Open
6 tasks
@bingxche bingxche mentioned this pull request Sep 8, 2026
Open
@ch-wan ch-wan mentioned this pull request Sep 8, 2026
Open
4 tasks
@cyhdmjzzy cyhdmjzzy mentioned this pull request Sep 8, 2026
Open
1 task
@Klaud-Cold Klaud-Cold mentioned this pull request Sep 9, 2026
Merged
9 tasks
joev-harmonic added a commit to joev-harmonic/sglang that referenced this pull request Sep 13, 2026
@joev-harmonic
perf(gdn): port fused decode projection and conv from sgl-project#35758
17513d2
joev-harmonic added a commit to joev-harmonic/sglang that referenced this pull request Sep 13, 2026
@joev-harmonic
fix(gdn): port FlashInfer alignment handling from sgl-project#35758
bd5c586
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@YAMY1234 YAMY1234 YAMY1234 left review comments
@kpham-sgl kpham-sgl Awaiting requested review from kpham-sgl
@pyc96 pyc96 Awaiting requested review from pyc96 pyc96 is a code owner
@Ying1123 Ying1123 Awaiting requested review from Ying1123
@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner
@fzyzcjy fzyzcjy Awaiting requested review from fzyzcjy
@ispobock ispobock Awaiting requested review from ispobock
@ch-wan ch-wan Awaiting requested review from ch-wan
@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner
@hanming-lu hanming-lu Awaiting requested review from hanming-lu hanming-lu is a code owner
@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner
@hzh0425 hzh0425 Awaiting requested review from hzh0425 hzh0425 is a code owner
@alphabetc1 alphabetc1 Awaiting requested review from alphabetc1 alphabetc1 is a code owner
@huangtingwei9988 huangtingwei9988 Awaiting requested review from huangtingwei9988 huangtingwei9988 is a code owner
@ShangmingCai ShangmingCai Awaiting requested review from ShangmingCai
@BBuf BBuf Awaiting requested review from BBuf
@Edwardf0t1 Edwardf0t1 Awaiting requested review from Edwardf0t1 Edwardf0t1 is a code owner
@FlamingoPg FlamingoPg Awaiting requested review from FlamingoPg FlamingoPg is a code owner
@AniZpZ AniZpZ Awaiting requested review from AniZpZ AniZpZ is a code owner
@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner
@b8zhong b8zhong Awaiting requested review from b8zhong b8zhong is a code owner
@OrangeRedeng OrangeRedeng Awaiting requested review from OrangeRedeng OrangeRedeng is a code owner
@Alisehen Alisehen Awaiting requested review from Alisehen Alisehen is a code owner
@mmangkad mmangkad Awaiting requested review from mmangkad mmangkad is a code owner
@hebiao064 hebiao064 Awaiting requested review from hebiao064 hebiao064 is a code owner
@yuan-luo yuan-luo Awaiting requested review from yuan-luo yuan-luo is a code owner
@CatherineSue CatherineSue Awaiting requested review from CatherineSue CatherineSue is a code owner
@slin1237 slin1237 Awaiting requested review from slin1237 slin1237 is a code owner
@JustinTong0323 JustinTong0323 Awaiting requested review from JustinTong0323 JustinTong0323 is a code owner
@ByronHsu ByronHsu Awaiting requested review from ByronHsu ByronHsu is a code owner
@Duyi-Wang Duyi-Wang Awaiting requested review from Duyi-Wang Duyi-Wang is a code owner
@sogalin sogalin Awaiting requested review from sogalin sogalin is a code owner
@DarkSharpness DarkSharpness Awaiting requested review from DarkSharpness DarkSharpness is a code owner
@celve celve Awaiting requested review from celve celve is a code owner
@HydraQYH HydraQYH Awaiting requested review from HydraQYH HydraQYH is a code owner
+2 more reviewers
@Fridge003 Fridge003 Fridge003 approved these changes
@hnyls2002 hnyls2002 hnyls2002 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@YAMY1234 YAMY1234

Labels

bypass-fastfail deepseek dependencies Pull requests that update a dependency file high priority jit-kernel quant LLM Quantization release-highlight Candidate PR for release note highlight run-ci run-ci-extra

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Qiaolin-Yu @YAMY1234 @Fridge003 @hnyls2002