Implement online nvfp4 quantization by zianglih · Pull Request #26083 · sgl-project/sglang

zianglih · 2026-05-22T08:55:46Z

Motivation

@HumansAnd
Add online NVFP4 MoE weight quantization under the --quantization nvfp4_online interface.

After #22918, FlashInfer TRTLLM MoE can use runtime per-token activation scaling, so SGLang no longer needs a calibrated static activation FP32 scale for this path. The nvfp4_online interface is explicitly a load-time conversion mode: weights still use static NVFP4 block scales plus static per-tensor FP32 scales, while activations use runtime per-token FP32 scales. This makes it possible to load BF16/FP16 or FP8 checkpoints and quantize eligible MoE expert weights to NVFP4 during weight loading, instead of requiring a pre-quantized NVFP4 checkpoint.

The intended initial scope is narrow: Blackwell GPUs only, MoE only, no dense linear quantization, and only the FlashInfer TRTLLM MoE backends (flashinfer_trtllm and flashinfer_trtllm_routed).

Modifications

Add the nvfp4_online quantization method for online NVFP4 MoE expert weight quantization.
Quantize eligible expert weights as they are loaded, avoiding a separate whole-checkpoint conversion step and keeping peak memory lower during startup.
Use static per-tensor FP32 weight scales and runtime per-token activation scales; no activation-scale calibration is required.
Support FP8 checkpoints by dequantizing eligible FP8 expert weights during load and requantizing them to NVFP4.
Keep dense linear layers in their source precision or checkpoint quantization path; only MoE experts are converted to NVFP4.
Add SGLANG_FP4_IGNORED_LAYERS so users can keep selected MoE layers or shared experts high precision.
Default the MoE runner backend to flashinfer_trtllm when --quantization nvfp4_online is used without an explicit backend, and error out for unsupported backend, hardware, or MoE TP configurations.
Thread per-token activation metadata through FlashInfer TRTLLM MoE quant info and add registered coverage with Qwen3-Next FP8 online NVFP4 per-token activation serving plus GSM8K accuracy.

Accuracy Tests

DeepSeek V3.2 fp8 requantization:

# baseline without NVFP4 per-token activation:
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum
Accuracy: 0.977
Invalid: 0.000
Latency: 13.146 s
Output throughput: 8566.746 token/s
Accuracy: 0.978
Invalid: 0.000
Latency: 12.749 s
Output throughput: 8878.813 token/s
Accuracy: 0.981
Invalid: 0.000
Latency: 17.272 s
Output throughput: 6584.294 token/s

# with online NVFP4 --quantization nvfp4_online
SGLANG_NSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD=0 python3 -m sglang.launch_server --kv-cache-dtype bf16 --model /data/models/ziangli_v32/DeepSeek-V3.2 --tp 8 --ep 8 --dp 8 --enable-dp-attention --moe-runner-backend flashinfer_trtllm_routed --attention-backend nsa --nsa-decode-backend flashmla_sparse --nsa-prefill-backend flashmla_sparse --page-size 64 --trust-remote-code --quantization nvfp4_online
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum

Accuracy: 0.979
Invalid: 0.000
Latency: 14.583 s
Output throughput: 7633.728 token/s
Accuracy: 0.981
Invalid: 0.000
Latency: 9.811 s
Output throughput: 11312.345 token/s
Accuracy: 0.975
Invalid: 0.000
Latency: 10.017 s
Output throughput: 11130.541 token/s

# with MTP drafter:
SGLANG_NSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD=0 python3 -m sglang.launch_server --kv-cache-dtype bf16 --model /data/models/ziangli_v32/DeepSeek-V3.2 --tp 8 --ep 8 --dp 8 --enable-dp-attention --moe-runner-backend flashinfer_trtllm_routed --attention-backend nsa --nsa-decode-backend flashmla_sparse --nsa-prefill-backend flashmla_sparse --page-size 64 --trust-remote-code --quantization nvfp4_online --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --max-running-requests 1209
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum

Accuracy: 0.980
Invalid: 0.000
Latency: 9.668 s
Output throughput: 11736.157 token/s
Accuracy: 0.980
Invalid: 0.000
Latency: 7.151 s
Output throughput: 15864.715 token/s
Accuracy: 0.978
Invalid: 0.000
Latency: 7.177 s
Output throughput: 15847.268 token/s

Also there is an added test TestFlashinferTrtllmGenMoeBackendNvFp4Online that partially requantizes Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 with:

    extra_env = {
        "SGLANG_FP4_IGNORED_LAYERS": ",".join(
            ["shared_expert"]
            + [f"model.layers.{layer_id}" for layer_id in range(40, 48)]
        )
    }

Speed Tests and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #27166586377
Latest PR Test (Extra): ❌ Run #27166586161

gemini-code-assist

Code Review

This pull request introduces the per_token_nvfp4 online quantization method for MoE checkpoints on NVIDIA Blackwell GPUs, supporting BF16, FP16, and FP8 weights. Key changes include the implementation of ModelOptPerTokenNvFp4FusedMoEMethod for dynamic weight quantization during loading, the addition of the SGLANG_FP4_IGNORED_LAYERS environment variable for layer exclusion, and updates to the FlashInfer TRTLLM MoE runner. Feedback highlights a critical unpacking error in nvfp4_quantize that would cause a runtime crash, a potential overflow in the nvfp4_max scale calculation, memory efficiency concerns regarding unnecessary FP32 conversions during weight processing, and thread-safety issues when modifying environment variables.

zianglih · 2026-05-22T09:22:26Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces the per_token_nvfp4 online quantization method for NVIDIA Blackwell GPUs, enabling on-the-fly quantization of MoE expert weights from BF16, FP16, or FP8 checkpoints. Key additions include the ModelOptPerTokenNvFp4FusedMoEMethod class, the SGLANG_FP4_IGNORED_LAYERS environment variable, and integration with the FlashInfer TRTLLM backend. Reviewers identified a missing import and the use of an undefined function per_tensor_dequantize (recommending dequantize_fp8 instead), and suggested optimizing the _weight_scale_2_from_amax method by performing calculations in Python to avoid unnecessary GPU operations.

zianglih · 2026-05-22T20:47:07Z

Testing fp8 requantization on DeepSeek V3.2:

SGLANG_NSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD=0 python3 -m sglang.launch_server --kv-cache-dtype bf16 --model /data/models/ziangli_v32/DeepSeek-V3.2 --tp 8 --ep 8 --dp 8 --enable-dp-attention --moe-runner-backend flashinfer_trtllm_routed --attention-backend nsa --nsa-decode-backend flashmla_sparse --nsa-prefill-backend flashmla_sparse --page-size 64 --trust-remote-code --quantization per_token_nvfp4

Running online NVFP4 quantization for MoE expert weights. log is printed out as expected during server launch:

[2026-05-22 20:39:55 DP1 TP1 EP1] Init torch distributed begin.
[2026-05-22 20:39:56 DP0 TP0 EP0] sglang is using nccl==2.28.9
[2026-05-22 20:40:04 DP0 TP0 EP0] Custom allreduce v2 initialized successfully (rank-check failed: tensor model parallel group is not initialized)
[2026-05-22 20:40:04 DP7 TP7 EP7] Init torch distributed ends. elapsed=9.10 s, mem usage=1.22 GB
[2026-05-22 20:40:04 DP0 TP0 EP0] Init torch distributed ends. elapsed=9.63 s, mem usage=1.47 GB
[2026-05-22 20:40:04 DP6 TP6 EP6] Init torch distributed ends. elapsed=9.11 s, mem usage=1.53 GB
[2026-05-22 20:40:04 DP5 TP5 EP5] Init torch distributed ends. elapsed=9.12 s, mem usage=1.53 GB
[2026-05-22 20:40:04 DP4 TP4 EP4] Init torch distributed ends. elapsed=9.22 s, mem usage=1.53 GB
[2026-05-22 20:40:04 DP3 TP3 EP3] Init torch distributed ends. elapsed=9.32 s, mem usage=1.53 GB
[2026-05-22 20:40:04 DP2 TP2 EP2] Init torch distributed ends. elapsed=9.17 s, mem usage=1.53 GB
[2026-05-22 20:40:04 DP1 TP1 EP1] Init torch distributed ends. elapsed=9.04 s, mem usage=1.53 GB
[2026-05-22 20:40:05 DP5 TP5 EP5] Load weight begin. avail mem=176.17 GB
[2026-05-22 20:40:05 DP7 TP7 EP7] Load weight begin. avail mem=176.48 GB
[2026-05-22 20:40:05 DP0 TP0 EP0] Load weight begin. avail mem=176.23 GB
[2026-05-22 20:40:05 DP2 TP2 EP2] Load weight begin. avail mem=176.17 GB
[2026-05-22 20:40:05 DP1 TP1 EP1] Load weight begin. avail mem=176.17 GB
[2026-05-22 20:40:05 DP3 TP3 EP3] Load weight begin. avail mem=176.17 GB
[2026-05-22 20:40:05 DP6 TP6 EP6] Load weight begin. avail mem=176.17 GB
[2026-05-22 20:40:05 DP4 TP4 EP4] Load weight begin. avail mem=176.17 GB
[2026-05-22 20:40:05 DP5 TP5 EP5] Setting inplace to False for FlashInfer TRTLLM MoE backend.
[2026-05-22 20:40:06 DP7 TP7 EP7] Setting inplace to False for FlashInfer TRTLLM MoE backend.
[2026-05-22 20:40:06 DP0 TP0 EP0] Setting inplace to False for FlashInfer TRTLLM MoE backend.
[2026-05-22 20:40:06 DP2 TP2 EP2] Setting inplace to False for FlashInfer TRTLLM MoE backend.
[2026-05-22 20:40:06 DP3 TP3 EP3] Setting inplace to False for FlashInfer TRTLLM MoE backend.
[2026-05-22 20:40:06 DP6 TP6 EP6] Setting inplace to False for FlashInfer TRTLLM MoE backend.
[2026-05-22 20:40:06 DP1 TP1 EP1] Setting inplace to False for FlashInfer TRTLLM MoE backend.
[2026-05-22 20:40:06 DP4 TP4 EP4] Setting inplace to False for FlashInfer TRTLLM MoE backend.
Multi-thread loading shards:   4% Completed | 6/163 [00:02<00:41,  3.76it/s][2026-05-22 20:40:10 DP6 TP6 EP6] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:10 DP7 TP7 EP7] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:10 DP7 TP7 EP7] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:10 DP7 TP7 EP7] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:10 DP0 TP0 EP0] Running online NVFP4 quantization for MoE expert weights.
Multi-thread loading shards:   4% Completed | 7/163 [00:02<00:47,  3.29it/s][2026-05-22 20:40:10 DP1 TP1 EP1] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:10 DP1 TP1 EP1] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP4 TP4 EP4] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP5 TP5 EP5] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP3 TP3 EP3] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP2 TP2 EP2] Running online NVFP4 quantization for MoE expert weights.
[2026-05-22 20:40:11 DP2 TP2 EP2] Running online NVFP4 quantization for MoE expert weights.
Multi-thread loading shards: 100% Completed | 163/163 [00:44<00:00,  3.64it/s]

Ran gsm8k 3 times:

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum

Accuracy: 0.979
Invalid: 0.000
Latency: 14.583 s
Output throughput: 7633.728 token/s
Accuracy: 0.981
Invalid: 0.000
Latency: 9.811 s
Output throughput: 11312.345 token/s
Accuracy: 0.975
Invalid: 0.000
Latency: 10.017 s
Output throughput: 11130.541 token/s

zianglih · 2026-05-22T21:43:05Z

MTP drafter should follow the online NVFP4 path without extra handling in the current DeepSeek EAGLE setup.

The relevant flow is:

When --speculative-draft-model-quantization is unset, ServerArgs copies the target --quantization value to the draft model, so --quantization per_token_nvfp4 also applies to the draft.
The DeepSeek speculative hook uses the same checkpoint as the draft model when no separate draft path is provided, and the speculative MoE backend inherits the target backend when unset.
The draft worker is created with is_draft_worker=True, so its ModelConfig reads speculative_draft_model_quantization.
DeepseekV3ForCausalLMNextN passes that quant config into the NextN decoder/MoE layer.
For per_token_nvfp4, FusedMoE uses ModelOptPerTokenNvFp4FusedMoEMethod, whose online loader dequantizes FP8 checkpoint expert weights when needed and requantizes them to NVFP4 during load.

So with --quantization per_token_nvfp4 --moe-runner-backend flashinfer_trtllm_routed and no draft quant/backend override, MTP expert weights are online-quantized to NVFP4 as part of normal draft model loading. The main caveats are explicit overrides such as --speculative-draft-model-quantization unquant or ignored-layer patterns matching the MTP MoE layers.

ziang-and · 2026-05-23T02:11:35Z

/gemini review

gemini-code-assist · 2026-05-23T02:11:38Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

zianglih · 2026-05-26T00:51:35Z

/gemini review

gemini-code-assist · 2026-05-26T00:51:39Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

zianglih · 2026-05-26T10:01:55Z

/tag-and-rerun-ci

Use --quantization nvfp4_per_token_activation as the only public spelling for the online NVFP4 MoE path. Rename the quant config, internal marker, server validation, registered tests, and docs_new references to make clear that per-token applies to activations rather than weights. Validation: pre-commit run --all-files; remote TestNvFp4PerTokenActivationConfig registry unit test; remote TestFlashinferTrtllmGenMoeBackendOnlineNvFp4PerTokenActivation::test_gsm8k scored 0.960.

gemini-code-assist · 2026-06-08T20:57:15Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

zianglih · 2026-06-08T20:59:44Z

/rerun-failed-ci

zianglih · 2026-06-09T05:01:42Z

/rerun-failed-ci

zianglih · 2026-06-09T18:37:09Z

/rerun-failed-ci

zianglih · 2026-06-09T21:31:08Z

/rerun-failed-ci

zianglih · 2026-06-10T00:02:09Z

all nv base ci passed

Edwardf0t1

LGTM, left a minor comment.

Edwardf0t1 · 2026-06-10T06:14:34Z

            "modelopt_fp8": ["modelopt"],
            "modelopt_fp4": ["modelopt"],
            "modelopt_mixed": ["modelopt"],
+            "nvfp4_online": ["fp8"],


why nvfp4_online maps to fp8?

This allows an FP8 checkpoint to be served with --quantization nvfp4_online. For eligible MoE expert weights, SGLang dequantizes the serialized FP8 weight to BF16 during loading, then requantizes it to NVFP4.

zianglih requested review from AniZpZ, BBuf, Edwardf0t1, FlamingoPg, Fridge003, HaiShaw, JustinTong0323, Ying1123, b8zhong, ch-wan, ispobock, merrymercy, wisclmy0611 and zijiexia as code owners May 22, 2026 08:55

github-actions Bot added documentation Improvements or additions to documentation quant LLM Quantization labels May 22, 2026

gemini-code-assist Bot reviewed May 22, 2026

View reviewed changes

Comment thread python/sglang/srt/layers/quantization/modelopt_quant.py

Comment thread python/sglang/srt/layers/quantization/modelopt_quant.py Outdated

Comment thread python/sglang/srt/layers/quantization/modelopt_quant.py Outdated

ziang-and force-pushed the online-per-token-fp4 branch 3 times, most recently from b08dbc8 to 5e1c052 Compare May 23, 2026 02:03

amd-bot mentioned this pull request May 25, 2026

[CI Monitor] Daily Report - 2026-05-25 bingxche/sglang-ci-bot#83

Open

zianglih added 4 commits June 8, 2026 13:34

Drop NVFP4 activation registry unit test

82455c7

Rename online NVFP4 quantization interface

385c822

Document nvfp4_online design

b21f646

zianglih marked this pull request as draft June 8, 2026 20:53

Clarify FP4 ignored layers example

f3dc2bb

zianglih marked this pull request as ready for review June 8, 2026 20:57

Merge branch 'main' into online-per-token-fp4

73ee07c

Edwardf0t1 approved these changes Jun 10, 2026

View reviewed changes

Fridge003 approved these changes Jun 10, 2026

View reviewed changes

Fridge003 merged commit 01f10ac into sgl-project:main Jun 10, 2026
333 of 389 checks passed

This was referenced Jun 10, 2026

[CI Monitor] Daily Report - 2026-06-10 bingxche/sglang-ci-bot#99

Open

[CI Monitor] Daily Report - 2026-06-11 bingxche/sglang-ci-bot#100

Open

zianglih mentioned this pull request Jun 15, 2026

[FlashInfer v0.6.13] Use CuTe DSL backend for FlashInfer per-token NVFP4 quantization #28220

Merged

5 tasks

coderabbitai Bot mentioned this pull request Jul 4, 2026

feat(collector): upgrade the SGLang collector to 0.5.14 ai-dynamo/aiconfigurator#1304

Closed

zianglih mentioned this pull request Jul 12, 2026

[Roadmap] Blackwell MXFP8 and NVFP4 RL training radixark/miles#615

Open

70 tasks

mgoin mentioned this pull request Jul 14, 2026

[Quant] Add nvfp4_per_token online MoE quantization vllm-project/vllm#48538

Merged

4 tasks

zianglih mentioned this pull request Jul 31, 2026

[ModelOpt FP4] Support online MoE weight quantization #33115

Merged

5 tasks

zianglih deleted the online-per-token-fp4 branch August 5, 2026 00:13

zianglih mentioned this pull request Aug 6, 2026

[FlashInfer v0.6.16] Support FlashInfer CuTe DSL NVFP4 MoE quantization #28354

Merged

5 tasks

zianglih mentioned this pull request Aug 17, 2026

[SM121] NVFP4 online per-token CuTe DSL execution plan zianglih/sglang#2

Open

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

Implement online nvfp4 quantization (sgl-project#26083)

d2ef919

zianglih mentioned this pull request Sep 6, 2026

[MoE] Support NVFP4 dispatch with standard FlashInfer TRTLLM and CuTe DSL backends #38216

Draft

5 tasks

Conversation

zianglih commented May 22, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zianglih commented May 22, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zianglih commented May 22, 2026

Uh oh!

zianglih commented May 22, 2026

Uh oh!

ziang-and commented May 23, 2026

Uh oh!

gemini-code-assist Bot commented May 23, 2026

Uh oh!

zianglih commented May 26, 2026

Uh oh!

gemini-code-assist Bot commented May 26, 2026

Uh oh!

zianglih commented May 26, 2026

Uh oh!

gemini-code-assist Bot commented Jun 8, 2026

Uh oh!

zianglih commented Jun 8, 2026

Uh oh!

zianglih commented Jun 9, 2026

Uh oh!

zianglih commented Jun 9, 2026

Uh oh!

zianglih commented Jun 9, 2026

Uh oh!

zianglih commented Jun 10, 2026

Uh oh!

Edwardf0t1 left a comment

Choose a reason for hiding this comment

Uh oh!

Edwardf0t1 Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

zianglih Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zianglih commented May 22, 2026 •

edited by github-actions Bot

Loading