Introduce Qwen 3.8 Flash Next by JustinTong0323 · Pull Request #36497 · sgl-project/sglang

JustinTong0323 · 2026-08-26T12:32:08Z

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.

Co-authored-by: Qiaolin-Yu <90088090+Qiaolin-Yu@users.noreply.github.com> Co-authored-by: ch-wan <54331508+ch-wan@users.noreply.github.com> Co-authored-by: ispobock <26454835+ispobock@users.noreply.github.com> Co-authored-by: JustinTong0323 <115166877+JustinTong0323@users.noreply.github.com> Co-authored-by: samuellees <26428561+samuellees@users.noreply.github.com> Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com> Co-authored-by: yhyang201 <47235274+yhyang201@users.noreply.github.com> Co-authored-by: yizhang2077 <25844240+yizhang2077@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>

…(GB10 / DGX Spark) (#37068)

mintlify · 2026-09-05T11:19:10Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
lmsysorg	🟢 Ready	View Preview	Sep 5, 2026, 11:21 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

…ISION) on qwen4-main-squashed (#38121) Co-authored-by: Yangmin Li <yangminl@nvidia.com> Co-authored-by: rdxa <rdxa@rdxa-int-spark-01.yvb.moe> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Alison Shao <54658187+alisonshao@users.noreply.github.com>

…in fused softmax routing (fixes GB10 NEXTN collapse) (#38308) Co-authored-by: YAMY <74099316+YAMY1234@users.noreply.github.com>

Leslie360 · 2026-09-07T09:36:05Z

@mochgolf Same finding on our side — we opened #38144 with an exact fallback for the threshold-bin overflow in fast_topk (when the coarse fp16 bin saturates, fall back to exact top-k over the candidate set instead of returning the clipped bin boundary), plus a concentrated-distribution regression test.

Validated on 8×A800 (SM80), qwen3.8-flash-next TP8, against current qwen4-main-squashed:

pre-patch: overflow-triggering inputs → 80/4 runs wrong, 20 distinct wrong signatures (wrong and unstable across reruns)
post-patch: 0/4 wrong rows, single deterministic signature; randn control unchanged; 10× CUDA-graph replay clean

Happy to fold this into your candidate if you already have one in flight — the repro and regression test are in the PR.

Leslie360 · 2026-09-07T09:36:17Z

@kun10 Reproduced the same assert (prefix_lens % compress_ratio == 0 in _qsa_build_write_plan) on 8×A800 (SM80) with qwen3.8-flash-next TP8 under mixed load. Root cause: mixed-chunk injects running decode rows into an EXTEND batch with extend_len == 1, so prefix_len is almost never compress-ratio aligned.

#38180 adds a startup validation that rejects --enable-mixed-chunk with compressed QSA (fail-fast instead of a mid-serving device assert), plus a CPU unit test. Verified: post-fix mixed-load stress passes and the assert no longer fires.

Making the write plan tolerate unaligned prefixes would be the deeper fix, but the guard unblocks mixed-chunk users in the meantime.

…Triton + radix) (#38290) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Backport of sgl-project#36811 (as carried in sgl-project#36497 c8457de). Plain softmax routing has no correction bias, but the JIT router took a required bias tensor, so the caller built one: every routing call ran torch.zeros(num_experts, fp32) plus its fill kernel purely to satisfy the signature. Qwen3.8-Flash-Next routes 512 experts on all 48 layers with SGLANG_OPT_USE_JIT_KERNEL_FUSED_TOPK on by default, so that is 48 allocations and 48 fill launches per forward. Measured on A100-80GB at 370 us per forward -- 1.2%-3.7% of a decode step depending on its length, and the shorter the step the larger the share. The kernel now takes HAS_BIAS as a constexpr and keeps the zero in registers when there is none, and the bias may be fp16/bf16 as well as fp32 (a bf16 correction bias no longer needs a host-side cast). Verified on A100 (sm80): topk_ids identical and weight max|diff| = 0 against the previous zero-bias path at M = 1, 8, 64, 512, 8192 with 512 experts / top-10; the upstream unit suite passes (42 tests). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGAy9m3Q2VxMk5LLamv7HG

Qiaolin-Yu · 2026-09-08T20:56:42Z

merged in #37500

Mixed chunked prefill injects running decode rows into an EXTEND batch (ForwardMode.MIXED) with prefix_len == seq_len - 1, which is almost never a multiple of indexer_compress_ratio. That violates the ratio-aligned prefix invariant the compressed QSA write plan relies on, and torch._assert_async((prefix_lens % ratio == 0).all()) in _qsa_build_write_plan fires device-side, killing every TP rank (reported in the PR sgl-project#36497 thread; reproduced on 8x A800 SM80 TP8: 32x device-side assert, all ranks). - Startup: _qwen4_exp_overrides declares enable_mixed_chunk=False via declare_resolution for compressed QSA (the spec/dllm hook pattern); the publish gate's resolvable_fields whitelist does not admit enable_mixed_chunk as a model-overridable declaration, so it cannot ride the returned overrides dict. - Runtime: QwenSparseAttnBackend raises NotImplementedError on ForwardMode.MIXED so the assert can never be reached from any path. - Pure-CPU regression test: MIXED batch raises with the disable directive; DECODE batch unaffected. Verified on 8x A800 TP8: startup warning, health 200, 720s mixed load ok=1329 fail=0, negative control (no flag) ok=911 fail=0.

Mixed chunked prefill injects running decode rows into an EXTEND batch (ForwardMode.MIXED) with prefix_len == seq_len - 1, which is almost never a multiple of indexer_compress_ratio. That violates the ratio-aligned prefix invariant the compressed QSA write plan relies on, and torch._assert_async((prefix_lens % ratio == 0).all()) in _qsa_build_write_plan fires device-side, killing every TP rank (reported in the PR sgl-project#36497 thread; reproduced on 8x A800 SM80 TP8: 32x device-side assert, all ranks). - Startup: _qwen4_exp_overrides declares enable_mixed_chunk=False via declare_resolution for compressed QSA (the spec/dllm hook pattern); the publish gate's resolvable_fields whitelist does not admit enable_mixed_chunk as a model-overridable declaration, so it cannot ride the returned overrides dict. - Runtime: QwenSparseAttnBackend raises NotImplementedError on ForwardMode.MIXED under the same condition as the write plan (qsa_profile is None or variant == compressed), so the ratio assert can never be reached. Tokenwise QSA is unaffected: it never enters the write plan and compress_ratio == 1 makes the alignment trivially hold, so its MIXED batches are not rejected. - Pure-CPU regression test: MIXED batch raises with the disable directive for compressed QSA; tokenwise MIXED and DECODE batches are not guarded. Verified on 8x A800 TP8: startup warning, health 200, 720s mixed load ok=1329 fail=0, negative control (no flag) ok=911 fail=0.

nbald mentioned this pull request Sep 3, 2026

Add --offload-embedding-to-host: keep the input embedding table in pinned host memory #37826

Open

5 tasks

NVShreyas mentioned this pull request Sep 3, 2026

feat(cache): expose hybrid cache observability #37868

Closed

5 tasks

Jiminator mentioned this pull request Sep 5, 2026

[Qwen4-Exp] Load nvidia/Qwen3.8-Flash-Next-NVFP4 (ModelOpt MIXED_PRECISION) on qwen4-main-squashed #38121

Merged

5 tasks

[Qwen4-Exp] File-backed PLE table backend for unified-memory devices …

3a09f08

…(GB10 / DGX Spark) (#37068)

yhyang201 requested review from wisclmy0611 and zijiexia as code owners September 5, 2026 11:19

mintlify Bot deployed to staging - docs September 5, 2026 11:21 View deployment

Fix PLE file prefetch offsets for TP shards (#38123)

c093db9

Leslie360 mentioned this pull request Sep 5, 2026

fix(jit-kernel): exact fallback for fast_topk threshold-bin overflow (#36807) #38144

Open

sghhhh mentioned this pull request Sep 6, 2026

[Qwen3.8-Flash-Next] Streamline QSA prefill selection #38209

Open

4 tasks

[qwen4-main-squashed] Cherry-pick #36811: avoid zero-bias allocation …

c8457de

…in fused softmax routing (fixes GB10 NEXTN collapse) (#38308) Co-authored-by: YAMY <74099316+YAMY1234@users.noreply.github.com>

[Kernel] Wait for the PDL dependency before loading the router bias (…

4ccff14

…Triton + radix) (#38290) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Leslie360 mentioned this pull request Sep 7, 2026

[decode] Spec-decode with large draft-token counts OOMs mid-traffic on the decode instance - no upfront capacity check (verify workspace vs KV pool is zero-sum) #38341

Open

Qiaolin-Yu closed this Sep 8, 2026

This was referenced Sep 9, 2026

[AMD][MI35X] Add qwen3.8-flash-next-fp8-mi355x-sglang-agentic-mtp on v0.5.19-rocm700-mi35x-20260909 SemiAnalysisAI/InferenceX#2924

Closed

fix(qsa): restore ROCm MFMA head-alignment padding in QSA MQA decode #38910

Closed

Leslie360 mentioned this pull request Sep 11, 2026

fix(qwen4): disable mixed-chunk with compressed QSA + fail-fast guard #38180

Open

Jiminator deleted the qwen4-main-squashed branch September 14, 2026 04:45

alexnails restored the qwen4-main-squashed branch September 14, 2026 05:48

mintlify Bot deployed to staging - docs September 14, 2026 05:50 View deployment

Navigation Menu

Uh oh!

Introduce Qwen 3.8 Flash Next - #36497

Introduce Qwen 3.8 Flash Next#36497
JustinTong0323 wants to merge 11 commits into
mainfrom
qwen4-main-squashed

JustinTong0323 commented Aug 26, 2026 •

edited by github-actions Bot

Loading

Uh oh!

mintlify Bot commented Sep 5, 2026 •

edited

Loading

Uh oh!

Leslie360 commented Sep 7, 2026

Uh oh!

Leslie360 commented Sep 7, 2026

Uh oh!

Qiaolin-Yu commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

Uh oh!

Conversation

JustinTong0323 commented Aug 26, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

mintlify Bot commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Leslie360 commented Sep 7, 2026

Uh oh!

Leslie360 commented Sep 7, 2026

Uh oh!

Qiaolin-Yu commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

JustinTong0323 commented Aug 26, 2026 •

edited by github-actions Bot

Loading

mintlify Bot commented Sep 5, 2026 •

edited

Loading