[Fix] [SM120] quantization: Requantize block-FP8 MoE weights to UE8M0 for DeepGEMM by lucifer1004 · Pull Request #43556 · sgl-project/sglang

lucifer1004 · 2026-10-10T12:53:18Z

Motivation

On SM120, should_deepgemm_weight_requant_ue8m0 returns False for every caller, so block-FP8 MoE experts served by the DeepGEMM runner keep their FP32 checkpoint scales. SM120's grouped GEMM converts FP32 scales to UE8M0 itself, and it faults (CUDA error 719) on scales that are not powers of two.

With nvidia/Qwen3.8-Flash-Next-NVFP4 and --speculative-moe-runner-backend deep_gemm, the block-FP8 MTP draft experts crash at the first prefill graph capture, at every token count. The same weights with power-of-two scales run.

#39065 (open, fixing #39063) makes the same guard change by inferring a grouped layer from weight.ndim == 3, without tests. This PR differs in two ways:

Fp8MoEMethod passes grouped_moe=True explicitly.
It adds a regression test.

Split out of #42566 at a maintainer's request. #42566 now carries only the TP block-scale refinement, which depends on this change for its DeepGEMM path on SM120.

Fixes #39063

Modifications

Keep the SM120 exemption for dense block-FP8 GEMMs, which run on CUTLASS or Triton with FP32 scales there.
Requantize grouped MoE weights to UE8M0 as on SM90 and SM100.
should_deepgemm_weight_requant_ue8m0 and requant_block_scale_ue8m0_for_deepgemm take a keyword-only grouped_moe flag, which Fp8MoEMethod sets for its expert weights.

Accuracy Tests

test/registered/unit/layers/quantization/test_deepgemm_ue8m0_requant.py adds a case for the SM120 grouped-MoE requant.

Two existing tests in that file fail on SM120 hosts both with and without this PR:

test_helper_requants_supported_deepgemm_bf16_once
test_compressed_tensors_block_processing_preserves_ue8m0_marker

They patch DeepGEMM's UE8M0 flags on but not the device, and SM120 deliberately skips the dense requant they exercise. The file passes on the CPU runner it is registered for.

End to end, together with the TP refinement in #42566:

nvidia/Qwen3.8-Flash-Next-NVFP4 at TP2 with its own MTP head (NEXTN 3/1/4, draft experts on DeepGEMM) scores 0.976 on GSM8K through the chat template (lm-eval gsm8k, 1,319 questions, flexible-extract) on 2x RTX PRO 6000.
Without this change, the same draft experts on DeepGEMM fault on SM120 at TP1 already.

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Follow the SGLang code style guidance.

This PR was written with AI assistance (Claude); I reviewed every line.

🤖 Generated with Claude Code

CI States

Latest PR Test (Base): ❌ Run #39355945695
Latest PR Test (Extra): ❌ Run #39355945781
Latest PR Test (AMD ROCm 10): ❌ Run #39355945903

…MM grouped GEMM On SM120, should_deepgemm_weight_requant_ue8m0 returned False for every caller, so block-FP8 MoE experts served by the DeepGEMM runner kept their checkpoint FP32 block scales. SM120's grouped GEMM converts FP32 scales to UE8M0 itself, and on scales that are not powers of two the kernel faults (CUDA error 719): Qwen3.8-Flash-Next's block-FP8 MTP experts with --speculative-moe-runner-backend deep_gemm crashed at the first prefill graph capture, at any token count; the same weights with power-of-two scales run. Keep the SM120 exemption for dense block-FP8 GEMMs, which run on CUTLASS or Triton with FP32 scales there, and requant grouped MoE weights as on SM90 and SM100. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

… grouped MoE weights (upstream PR)

…TP shards that split checkpoint blocks (upstream PR, on sgl-project#43556)

…UE8M0 for the SM120 DeepGEMM grouped GEMM Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>

Fridge003 · 2026-10-20T01:51:58Z

/rerun-test test/registered/unit/layers/quantization/test_deepgemm_ue8m0_requant.py

github-actions · 2026-10-20T01:52:22Z

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit 6a39ddf1f2e0.

Re-run was not dispatched. What to do:

Rebase your branch onto the latest main and push again
Follow issue #21065 for context
CI-fix PRs may request the bypass-maintenance label to skip this check

Fridge003 · 2026-10-20T01:53:48Z

/rerun-test test/registered/unit/layers/quantization/test_deepgemm_ue8m0_requant.py

github-actions · 2026-10-20T01:54:11Z

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/layers/quantization/test_deepgemm_ue8m0_requant.py

…rouped MoE SGLang v0.5.23 (sgl-project/sglang#43556) added a grouped_moe flag to should_deepgemm_weight_requant_ue8m0: on SM120 dense block-FP8 GEMMs keep fp32 scales, but the grouped MoE GEMM runs on DeepGEMM and SGLang now requantizes expert scales to UE8M0 when loading. miles asked without the flag, so on SM120 it sent fp32 scales for expert weights that SGLang holds in UE8M0, and a weight update would write them in the wrong format. Pass grouped_moe for expert weights, as SGLang's own MoE call does. Other GPUs are unaffected: there the flag does not change the answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The grouped_moe fix for SGLang v0.5.23 (sgl-project/sglang#43556) passes `grouped_moe` to should_deepgemm_weight_requant_ue8m0 for every FP8 weight. The ROCm image stays on sglang-miles-v0.5.22-final, whose function has no such parameter, so after this bump merges every ROCm FP8 weight update (e.g. tests/e2e/megatron/test_qwen3_30B_A3B/test_amd_moriep_fp8_bridge.py) would raise TypeError. PR CI does not show it: with the ci-sglang-pr pin, the ROCm job installs the v0.5.23 stack over its image. The SGLang compat module now drops `grouped_moe` when the installed function does not take it, which is exactly the call main makes today. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lucifer1004 requested review from Alisehen, AniZpZ, BBuf, Edwardf0t1, FlamingoPg, HaiShaw, OrangeRedeng, b8zhong, ch-wan and mmangkad as code owners October 10, 2026 12:53

lucifer1004 mentioned this pull request Oct 10, 2026

[Bugfix] Load block-FP8 MoE TP shards that split checkpoint blocks #42566

Open

5 tasks done

github-actions Bot added the quant LLM Quantization label Oct 10, 2026

lucifer1004 added a commit to lucifer1004/sglang that referenced this pull request Oct 13, 2026

Merge sgl-project#43556 at 547bbf4: SM120 UE8M0 requant for block-FP8…

0e5e2ed

… grouped MoE weights (upstream PR)

lucifer1004 added a commit to lucifer1004/sglang that referenced this pull request Oct 13, 2026

Merge sgl-project#42566 at a900a02: refined block-FP8 MoE scales for …

98d4c56

…TP shards that split checkpoint blocks (upstream PR, on sgl-project#43556)

Fridge003 mentioned this pull request Oct 20, 2026

[Bugfix] Fix SM120 DeepGEMM grouped FP8 weight requantization #39065

Closed

3 tasks done

Fridge003 approved these changes Oct 20, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

Merge branch 'main' into fix/sm120-deepgemm-moe-ue8m0-requant

cc9eac9

This comment has been minimized.

Sign in to view

Fridge003 changed the title ~~[Bugfix][SM120] Requant block-FP8 MoE weights to UE8M0 for the DeepGEMM grouped GEMM~~ [Fix] [SM120] quantization: Requantize block-FP8 MoE weights to UE8M0 for DeepGEMM Oct 20, 2026

Fridge003 merged commit 67d9ee1 into sgl-project:main Oct 20, 2026
106 of 123 checks passed

bingxche mentioned this pull request Oct 21, 2026

[CI Monitor] Daily Report - 2026-10-21 bingxche/sglang-ci-bot#237

Open

LingYi-Liang mentioned this pull request Oct 22, 2026

[Fix] moe: Quantize contiguous SiTU with the packed UE8M0 scale #45856

Open

2 of 5 tasks

Rockdu mentioned this pull request Oct 31, 2026

build(docker): bump SGLang to v0.5.23 radixark/miles#4548

Merged

6 tasks done

Navigation Menu

Repository navigation

[Fix] [SM120] quantization: Requantize block-FP8 MoE weights to UE8M0 for DeepGEMM - #43556

[Fix] [SM120] quantization: Requantize block-FP8 MoE weights to UE8M0 for DeepGEMM#43556
Fridge003 merged 2 commits into
sgl-project:mainfrom
lucifer1004:fix/sm120-deepgemm-moe-ue8m0-requant

lucifer1004 commented Oct 10, 2026 •

edited by github-actions Bot

Loading

Uh oh!

This comment has been minimized.

Fridge003 commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

Fridge003 commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lucifer1004 commented Oct 10, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Checklist

CI States

Uh oh!

This comment has been minimized.

Fridge003 commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

⚠️ Rebase Required Before Re-run

Uh oh!

Fridge003 commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lucifer1004 commented Oct 10, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading