Skip to content
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
8409b4b
Initial impl
zianglih May 22, 2026
75cc832
Allow fp8 ckpt by qdq
zianglih May 22, 2026
9c29590
Fix ue8m0
zianglih May 22, 2026
c627dd5
Address online NVFP4 review feedback
zianglih May 22, 2026
232b03a
Simplify online NVFP4 scale helper
zianglih May 22, 2026
17c9214
Optimize online NVFP4 weight loading
zianglih May 22, 2026
c273310
Enable TP for online per-token NVFP4
zianglih May 26, 2026
f56e995
Restrict online NVFP4 FP8 detection to CUDA dtypes
zianglih Jun 4, 2026
0ca5f9c
Move online NVFP4 MoE method to its own file
zianglih Jun 4, 2026
db371d9
Rename online NVFP4 quantization module
zianglih Jun 4, 2026
031bd0c
Move online NVFP4 config to owner module
zianglih Jun 4, 2026
3e56409
Set FlashInfer FP4 fast math env for online NVFP4
zianglih Jun 4, 2026
610086c
Polish FP4 ignored layers docs
zianglih Jun 4, 2026
834bda6
Simplify per-token activation config
zianglih Jun 4, 2026
d9262ae
Clean up online NVFP4 config access
zianglih Jun 4, 2026
2e08590
Use 4over6 in test since already supported
zianglih Jun 5, 2026
14be601
Align online NVFP4 scales with 4over6 config
zianglih Jun 5, 2026
9f4eda9
Keep 4over6 E4M3 override activation-scoped
zianglih Jun 5, 2026
eef16b4
Revert "Keep 4over6 E4M3 override activation-scoped"
zianglih Jun 5, 2026
bf92fe0
Refactor online modelopt FP4 quantization path
zianglih Jun 8, 2026
422e697
Revert "Refactor online modelopt FP4 quantization path"
zianglih Jun 8, 2026
78e203b
Rename NVFP4 per-token activation interface
zianglih Jun 8, 2026
82455c7
Drop NVFP4 activation registry unit test
zianglih Jun 8, 2026
385c822
Rename online NVFP4 quantization interface
zianglih Jun 8, 2026
b21f646
Document nvfp4_online design
zianglih Jun 8, 2026
f3dc2bb
Clarify FP4 ignored layers example
zianglih Jun 8, 2026
73ee07c
Merge branch 'main' into online-per-token-fp4
zianglih Jun 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions docs_new/docs/advanced_features/quantization.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,13 @@ The following table summarizes quantization method support across NVIDIA and AMD
<td>No</td>
<td><a href="https://github.com/NVIDIA/Model-Optimizer">NVIDIA ModelOpt</a>; use Marlin W4A16 fallback on Ampere/Hopper and native FP4 backends on Blackwell</td>
</tr>
<tr>
<td><code>per_token_nvfp4</code></td>
<td>Yes (Blackwell/SM100 or SM103)</td>
<td>No</td>
<td>No</td>
<td>Online MoE-only NVFP4 quantization for BF16/FP16/FP8 checkpoints; requires <code>flashinfer_trtllm</code> or <code>flashinfer_trtllm_routed</code></td>
</tr>
<tr>
<td><code>petit_nvfp4</code></td>
<td>No</td>
Expand Down Expand Up @@ -798,6 +805,23 @@ python3 -m sglang.launch_server \
--port 30000 --host 0.0.0.0
```

### `per_token_nvfp4` online quantization method

SGLang running on NVIDIA Blackwell SM100 or SM103 GPUs supports `--quantization per_token_nvfp4` for MoE checkpoints stored in BF16, FP16, or FP8. It quantizes MoE expert weights online as each expert weight tensor is loaded, using standard 2D NVFP4 weight quantization with per-tensor amax-derived FP32 scales. If an expert checkpoint weight is FP8, SGLang first dequantizes that tensor to BF16 with the checkpoint scale and then requantizes it to NVFP4. Dense linear layers are kept in the original checkpoint precision. The `per_token` name refers to activation scaling: weights use static per-tensor FP32 scales, while activations get runtime per-token FP32 scales from FlashInfer, so activation FP32 calibration scales are not required.

Only `--moe-runner-backend flashinfer_trtllm` and `--moe-runner-backend flashinfer_trtllm_routed` are supported. If `--moe-runner-backend` is omitted, SGLang selects `flashinfer_trtllm`. Tensor parallelism is supported; activation per-token scales are computed locally on each TP rank, while online weight quantization still uses the loaded expert tensor's per-tensor amax-derived FP32 scale.

FlashInfer TRTLLM MoE backends disable shared-expert fusion, so online quantization applies to routed MoE experts while shared experts stay in the checkpoint precision. To keep specific routed MoE layers out of online FP4 conversion, include their module path in `SGLANG_FP4_IGNORED_LAYERS`; FP8 checkpoints keep those ignored experts in FP8 checkpoint precision.

```bash Command
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \
--tp-size 2 \
--ep-size 2 \
--quantization per_token_nvfp4 \
--port 30000 --host 0.0.0.0
```

### `quark_int4fp8_moe` online quantization method

SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/v0.4.8/python/sglang/srt/layers/moe/fused_moe_triton/layer.py#L271) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8.
Expand Down
2 changes: 1 addition & 1 deletion docs_new/docs/advanced_features/server_arguments.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--quantization`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The quantization method.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>awq</code>, <code>fp8</code>, <code>gptq</code>, <code>marlin</code>, <code>gptq_marlin</code>, <code>awq_marlin</code>, <code>bitsandbytes</code>, <code>gguf</code>, <code>modelopt</code>, <code>modelopt_fp8</code>, <code>modelopt_fp4</code>, <code>petit_nvfp4</code>, <code>w8a8_int8</code>, <code>w8a8_fp8</code>, <code>moe_wna16</code>, <code>qoq</code>, <code>w4afp8</code>, <code>mxfp4</code>, <code>mxfp8</code>, <code>auto-round</code>, <code>compressed-tensors</code>, <code>modelslim</code>, <code>quark_int4fp8_moe</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>awq</code>, <code>fp8</code>, <code>gptq</code>, <code>marlin</code>, <code>gptq_marlin</code>, <code>awq_marlin</code>, <code>bitsandbytes</code>, <code>gguf</code>, <code>modelopt</code>, <code>modelopt_fp8</code>, <code>modelopt_fp4</code>, <code>per_token_nvfp4</code>, <code>petit_nvfp4</code>, <code>w8a8_int8</code>, <code>w8a8_fp8</code>, <code>moe_wna16</code>, <code>qoq</code>, <code>w4afp8</code>, <code>mxfp4</code>, <code>mxfp8</code>, <code>auto-round</code>, <code>compressed-tensors</code>, <code>modelslim</code>, <code>quark_int4fp8_moe</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--quantization-param-path`</td>
Expand Down
9 changes: 7 additions & 2 deletions docs_new/docs/references/environment_variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -576,12 +576,12 @@ SGLang supports various environment variables that can be used to configure its
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>FLASHINFER_NVFP4_4OVER6</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable FlashInfer NVFP4 4over6 scaling for the per-token activation path; effective only with <code>SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable FlashInfer NVFP4 4over6 scaling for per-token NVFP4 activation and online NVFP4 MoE weight quantization paths</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>false</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>FLASHINFER_NVFP4_4OVER6_E4M3_USE_256</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use <code>256</code> as the E4M3 scale maximum for FlashInfer NVFP4 4over6 per-token activation scaling; otherwise uses <code>448</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use <code>256</code> as the E4M3 scale maximum for FlashInfer NVFP4 4over6 scaling; otherwise uses <code>448</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>false</code></td>
</tr>
<tr>
Expand All @@ -599,6 +599,11 @@ SGLang supports various environment variables that can be used to configure its
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A comma-separated list of layer names to ignore during FP8 quantization. For example: <code>model.layers.0,model.layers.1.,qkv_proj</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>""</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_FP4_IGNORED_LAYERS</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A comma-separated list of layer names to keep out of FP4 online quantization, including <code>per_token_nvfp4</code>. For example: <code>model.layers.0,model.layers.1,qkv_proj</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>""</code></td>
</tr>
</tbody>
</table>

Expand Down
3 changes: 3 additions & 0 deletions python/sglang/srt/configs/model_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -1109,6 +1109,7 @@ def _validate_quantize_and_serve_config(self):
"modelopt",
"modelopt_fp8",
"modelopt_fp4",
"per_token_nvfp4",
"modelopt_mixed",
]
modelopt_quantization_specified = (
Expand Down Expand Up @@ -1153,6 +1154,7 @@ def _verify_quantization(self) -> None:
"modelopt_fp8",
"modelopt_fp4",
"modelopt_mixed",
"per_token_nvfp4",
"gptq_marlin_24",
"gptq_marlin",
"awq_marlin",
Expand All @@ -1174,6 +1176,7 @@ def _verify_quantization(self) -> None:
"modelopt_fp8": ["modelopt"],
"modelopt_fp4": ["modelopt"],
"modelopt_mixed": ["modelopt"],
"per_token_nvfp4": ["fp8"],
"petit_nvfp4": ["modelopt"],
"w8a8_int8": ["compressed-tensors", "compressed_tensors"],
"w8a8_fp8": ["compressed-tensors", "compressed_tensors"],
Expand Down
1 change: 1 addition & 0 deletions python/sglang/srt/environ.py
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,7 @@ class Envs:
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE = EnvBool(False)
SGLANG_QUANT_ALLOW_DOWNCASTING = EnvBool(False)
SGLANG_FP8_IGNORED_LAYERS = EnvStr("")
SGLANG_FP4_IGNORED_LAYERS = EnvStr("")

# Flashinfer
SGLANG_IS_FLASHINFER_AVAILABLE = EnvBool(True)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -810,6 +810,7 @@ class FlashInferTrtllmFp4MoeQuantInfo(MoeQuantInfo):
intermediate_size_per_partition: int

routing_method_type: int
use_per_token_activation: bool = False


def quantize_hidden_states_fp4(
Expand Down Expand Up @@ -873,7 +874,7 @@ def fused_experts_none_to_flashinfer_trtllm_fp4(
topk_output = dispatch_output.topk_output

# Quantize hidden states to FP4
if envs.SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION.get():
if quant_info.use_per_token_activation:
from flashinfer import SfLayout, nvfp4_quantize

e4m3_max = 448.0
Expand Down
2 changes: 2 additions & 0 deletions python/sglang/srt/layers/quantization/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ def override_quantization_method(self, *args, **kwargs):
from sglang.srt.layers.quantization.modelslim.modelslim import ModelSlimConfig
from sglang.srt.layers.quantization.moe_wna16 import MoeWNA16Config
from sglang.srt.layers.quantization.mxfp4 import Mxfp4Config
from sglang.srt.layers.quantization.nvfp4_online import PerTokenNvFp4Config
from sglang.srt.layers.quantization.petit import PetitNvFp4Config
from sglang.srt.layers.quantization.qoq import QoQConfig
from sglang.srt.layers.quantization.quark.quark import QuarkConfig
Expand Down Expand Up @@ -74,6 +75,7 @@ def override_quantization_method(self, *args, **kwargs):
"modelopt": ModelOptFp8Config, # Auto-detect, defaults to FP8
"modelopt_fp8": ModelOptFp8Config,
"modelopt_fp4": ModelOptFp4Config,
"per_token_nvfp4": PerTokenNvFp4Config,
"modelopt_mixed": ModelOptMixedPrecisionConfig,
"w8a8_int8": W8A8Int8Config,
"w8a8_fp8": W8A8Fp8Config,
Expand Down
46 changes: 41 additions & 5 deletions python/sglang/srt/layers/quantization/modelopt_quant.py

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we try to move it into a new file, for fp4 online quantization. Since this file is getting very long now

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 930908d8d. I moved the implementation-heavy ModelOptPerTokenNvFp4FusedMoEMethod into modelopt_nvfp4_online.py; PerTokenNvFp4Config now lazily imports it only for the FusedMoE online FP4 path. I also re-ran the focused registered test on the B200 devbox: TestFlashinferTrtllmGenMoeBackendOnlinePerTokenNVFP4 passed with GSM8K score 0.945.

Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ def __init__(
self.packed_modules_mapping = packed_modules_mapping
self.exclude_modules = exclude_modules or []
self.kv_cache_quant_algo = kv_cache_quant_algo
self.use_per_token_activation = False

def _get_quant_method(
self,
Expand Down Expand Up @@ -1171,6 +1172,7 @@ def __init__(
group_size: int = None,
exclude_modules: List[str] = None,
packed_modules_mapping: Optional[Dict[str, List[str]]] = None,
use_per_token_activation: Optional[bool] = None,
) -> None:
super().__init__(kv_cache_quant_algo, exclude_modules, packed_modules_mapping)
self.is_checkpoint_nvfp4_serialized = is_checkpoint_nvfp4_serialized
Expand All @@ -1180,6 +1182,10 @@ def __init__(
"format is experimental and subject to change."
)
self.group_size = group_size
self.use_per_token_activation = (
use_per_token_activation
or envs.SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION.get()
)

@classmethod
def override_quantization_method(cls, hf_quant_config, user_quant):
Expand Down Expand Up @@ -1740,11 +1746,24 @@ def create_weights(
params_dtype: torch.dtype,
**extra_weight_attrs,
):
if not self.quant_config.is_checkpoint_nvfp4_serialized:
is_online_per_token_nvfp4 = getattr(
self.quant_config, "is_online_per_token_nvfp4", False
)
if (
not self.quant_config.is_checkpoint_nvfp4_serialized
and not is_online_per_token_nvfp4
):
raise ValueError(
"NVFP4 quantization was selected, "
" dynamic quantization is not supported."
)
if is_online_per_token_nvfp4:
if not self.enable_flashinfer_trtllm_moe:
raise ValueError(
"--quantization per_token_nvfp4 supports only "
"--moe-runner-backend flashinfer_trtllm or "
"flashinfer_trtllm_routed."
)

# TODO(ch-wan): check if this is needed
layer.intermediate_size_per_partition = intermediate_size_per_partition
Expand All @@ -1754,6 +1773,8 @@ def create_weights(
weight_dtype = torch.uint8
weight_scale_dtype = torch.float8_e4m3fn
weight_loader = extra_weight_attrs.get("weight_loader")
if is_online_per_token_nvfp4:
weight_loader = self.get_online_weight_loader(layer, weight_loader)
# GEMM 1
num_shards = 2 if layer.moe_runner_config.is_gated else 1

Expand Down Expand Up @@ -1852,6 +1873,23 @@ def create_weights(
)
layer.register_parameter("w2_weight_scale_2", w2_weight_scale_2)

if is_online_per_token_nvfp4 and self.quant_config.is_checkpoint_fp8_serialized:
# FP8 checkpoints usually store expert scales as weight_scale_inv.
# Online NVFP4 consumes them in the loader and writes the generated
# NVFP4 scales into w*_weight_scale / w*_weight_scale_2 instead.
w13_source_weight_scale_inv = PerTensorScaleParameter(
data=torch.empty(0, dtype=torch.float32),
weight_loader=weight_loader,
)
layer.register_parameter(
"w13_weight_scale_inv", w13_source_weight_scale_inv
)
w2_source_weight_scale_inv = PerTensorScaleParameter(
data=torch.empty(0, dtype=torch.float32),
weight_loader=weight_loader,
)
layer.register_parameter("w2_weight_scale_inv", w2_source_weight_scale_inv)

extra_weight_attrs.update(
{"quant_method": FusedMoeWeightScaleSupported.TENSOR.value}
)
Expand Down Expand Up @@ -1939,10 +1977,7 @@ def _slice_scale(w):
w13_input_scale = layer.w13_input_scale.max(dim=-1).values.to(torch.float32)
w2_input_scale = layer.w2_input_scale

if (
self.enable_flashinfer_trtllm_moe
and envs.SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION.get()
):
if self.quant_config.use_per_token_activation:
w13_input_scale = torch.ones_like(w13_input_scale, dtype=torch.float32)
w2_input_scale = torch.ones_like(w2_input_scale, dtype=torch.float32)

Expand Down Expand Up @@ -2267,6 +2302,7 @@ def apply(
local_num_experts=layer.num_local_experts,
intermediate_size_per_partition=layer.intermediate_size_per_partition,
routing_method_type=routing_method_type,
use_per_token_activation=self.quant_config.use_per_token_activation,
)

return self.runner.run(dispatch_output, quant_info)
Expand Down
Loading
Loading