Validate CuTe DSL online NVFP4 W4A16 configuration by zianglih · Pull Request #38215 · sgl-project/sglang

zianglih · 2026-09-06T11:23:19Z

Motivation

CuTe DSL W4A16 demonstrates strong performance in FlashInfer #4985: 1.28–2.61x faster than pure TRTLLM BF16 across its three measured configurations. The complete tables below motivate validating this SGLang configuration:

SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16=1 python3 -m sglang.launch_server \
  --model-path MODEL --quantization nvfp4_online \
  --moe-runner-backend flashinfer_cutedsl --dtype bfloat16

The implementation from #35120 already loads NVFP4 expert weights online and keeps activations in BF16. This small follow-up validates hardware/dtype/A2A constraints early, skips an unused activation-scale check, and documents the W4A16 opt-in. The FlashInfer results are local-MoE microbenchmarks, not an end-to-end SGLang speedup claim.

Modifications

Gates: check SM10X and resolved BF16 dtype in handle_moe_kernel_config; check supported W4A16 A2A backends (none/flashinfer) in handle_a2a_moe, after backend overrides.
Dispatch: use the existing W4A16 BF16-dispatch selection instead of adding a broad rejection of SGLANG_MOE_NVFP4_DISPATCH. Preserve the existing FlashInfer A2A-specific guard. Skip the unused raw activation-scale equality/scalarization for W4A16, since online scale buffers can contain uninitialized values before neutralization.
Reuse: preserve the existing quantizer, weight-only alphas, kernel selection, per-token-env precedence, and source-layer exclusions. No new kernel or quantization implementation.
Scope: retain the existing nightly E2E and remove its redundant =0 env overrides. Remove broad new behavioral test scaffolding in a separate cleanup commit; validation helpers remain outside the PR.

Accuracy Tests

Initial GPU validation: a938aec0f7f3439e056eb80a14bf595e617c4c70, base 938dc5621d30f6983d97b092015304a35591a7cf. The GPU results below predate review follow-up d8c8b0b6db7e892687eac0dde630ae9620125513 and test-cleanup head 8a7cd7b56a3ad112e4affd453169d72deb77d508. The review changes only relocate argument gates and skip unused W4A16 raw-scale validation; the full-model GPU tests were not rerun after that follow-up.
Image: lmsysorg/sglang:nightly-dev-cu13-20260906-09daea94; amd64 digest sha256:a0d0f199e15d12e14e18c3a331316cfda82df9d15e49873f960e7d05caf346cb.
Hardware: one bare C2 devbox with 8 NVIDIA B300 GPUs; model tests use GPUs 0–3.
Runtime: Python 3.12.3, PyTorch 2.13.0+cu130, CUDA 13.0, FlashInfer Python/Cubin/JIT cache 0.6.18/0.6.18/0.6.18+cu130, CuTe DSL 4.6.2, Transformers 5.12.1, NVIDIA driver 590.48.01. No package replacement or dependency fallback.

Review follow-up validation

Final source: 8a7cd7b56a3ad112e4affd453169d72deb77d508. The behavioral-test removal is a separate commit from the review fixes. A temporary CPU devbox uses the same explicit SGLang image listed above; no GPU is requested for this follow-up.

git clone https://github.com/zianglih/sglang.git
cd sglang
git checkout 8a7cd7b56a3ad112e4affd453169d72deb77d508
export PYTHONPATH="$PWD/python"
python3 -m pytest \
  test/registered/unit/server_args/test_server_args.py \
  test/registered/unit/test_model_overrides.py \
  test/registered/unit/layers/moe/test_flashinfer_cutedsl_dispatch.py \
  test/registered/unit/layers/quantization/test_modelopt_nvfp4_moe_scales.py -q

The registered suite passed 296 tests and 79 subtests, with one existing CPU-platform skip (mxfp4 quantization is not registered). Local helpers also passed 16 tests and 2 subtests, including the reviewer-requested DeepEP checks, W4A16 dispatch contracts, and a NaN-scale case through real online scale preparation. Those helpers are preserved outside the final PR tree; the separate cleanup commit removes 190 lines of broad behavioral tests.

Complete raw registered-test output

................................................................... [ 22%]
......................................................... [ 41%]
........................................................................ [ 65%]
.....................................s.................. [ 84%]
.............................................                                                         [100%]
=============================== warnings summary ===============================
../../../opt/sglang/lib/python3.12/site-packages/_pytest/config/__init__.py:1464
  /opt/sglang/lib/python3.12/site-packages/_pytest/config/__init__.py:1464: PytestConfigWarning: Unknown config option: asyncio_mode
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

../../../opt/sglang/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /opt/sglang/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

python/sglang/srt/layers/quantization/awq/awq.py:53
  /sgl-workspace/nvfp4-w4a16-review/sglang/python/sglang/srt/layers/quantization/awq/awq.py:53: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
    warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

python/sglang/srt/layers/quantization/gguf.py:69
  /sgl-workspace/nvfp4-w4a16-review/sglang/python/sglang/srt/layers/quantization/gguf.py:69: UserWarning: Only CUDA, MUSA and NPU support GGUF quantization currently.
    warnings.warn(f"Only CUDA, MUSA and NPU support GGUF quantization currently.")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
296 passed, 1 skipped, 17 warnings, 79 subtests passed in 24.24s

Complete raw local validation output

The argument/dispatch helpers are recoverable from review commit d8c8b0b6db7e892687eac0dde630ae9620125513; the local NaN case calls the actual online MoE scale preparation with NaN input scales and MOE_NVFP4_DISPATCH=True, stopping at dispatcher configuration before GPU layout conversion. It verifies unit activation scales, weight-only alphas, and input_global_scale=None.

................                                                       [100%]
=============================== warnings summary ===============================
../../../opt/sglang/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /opt/sglang/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

python/sglang/srt/layers/quantization/awq/awq.py:53
  /sgl-workspace/nvfp4-w4a16-review/sglang/python/sglang/srt/layers/quantization/awq/awq.py:53: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
    warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

python/sglang/srt/layers/quantization/gguf.py:69
  /sgl-workspace/nvfp4-w4a16-review/sglang/python/sglang/srt/layers/quantization/gguf.py:69: UserWarning: Only CUDA, MUSA and NPU support GGUF quantization currently.
    warnings.warn(f"Only CUDA, MUSA and NPU support GGUF quantization currently.")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
16 passed, 16 warnings, 2 subtests passed in 9.35s

Pre-commit, Python compilation, and diff checks passed on the final head. No new full-model or performance benchmark was run for this review follow-up.

Initial GPU validation

Full-model validation uses nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 (snapshot 744b1880a37996c5d56bf454ae164dfd74d77c4e), online expert-weight conversion, TP4/DP4/EP4, DP attention/LM head, EAGLE with 3 steps/top-k 1/4 draft tokens, decode CUDA graphs, and prefill graphs disabled. Each arm runs the existing 200 GSM8K examples at temperature 1.0/top-p 0.95/max-tokens 16000; thresholds remain score > 0.90 and speculative acceptance > 2.5.

The existing nightly case uses FlashInfer A2A. A temporary subclass removes only --moe-a2a-backend flashinfer to validate the same W4A16 configuration with no A2A. Neither case sets the per-token activation or NVFP4-dispatch env to zero; the W4A16 opt-in is sufficient.

Reproduction commands in the image/check-out above:

export CUDA_VISIBLE_DEVICES=0,1,2,3
export HF_HOME=/sgl-workspace/nvfp4-online-w4a16/hf-cache
python3 test/registered/backends/test_flashinfer_nvfp4_online_moe_backend.py \
  TestFlashinferCuteDSLMoeBackendNvFp4OnlineW4A16.test_gsm8k -v

python3 - <<'PYTEST'
import runpy
import unittest

ns = runpy.run_path("test/registered/backends/test_flashinfer_nvfp4_online_moe_backend.py")
case = ns["TestFlashinferCuteDSLMoeBackendNvFp4OnlineW4A16"]
args = list(case.extra_args)
i = args.index("--moe-a2a-backend")
assert args[i + 1] == "flashinfer"
del args[i:i + 2]
case.extra_args = args
result = unittest.TextTestRunner(verbosity=2).run(
    unittest.defaultTestLoader.loadTestsFromTestCase(case)
)
raise SystemExit(not result.wasSuccessful())
PYTEST

Raw results, in order FlashInfer A2A, then no A2A:

metrics={'score:std': np.float64(0.1837797594948911), 'score': np.float64(0.965), 'latency': 50.38775707385503, 'output_throughput': 1567.42440200777}
avg_spec_accept_length=3.0968036529680365
metrics={'score:std': np.float64(0.1705872210923198), 'score': np.float64(0.97), 'latency': 52.299501925939694, 'output_throughput': 1537.5289828547482}
avg_spec_accept_length=3.1409861325115562
Ran 2 tests in 547.440s
OK

Both runs passed startup, online conversion, CUDA graph capture, 200 requests, and the unchanged accuracy/acceptance thresholds. Scores are stochastic quality checks, not a numerical-parity claim.

Runtime proof of the selected wrapper and BF16 activation contract

A temporary sitecustomize.py probe outside the PR checks the actual FlashInfer constructor and each wrapper's first successful run. All four workers in each arm selected w4a16 and passed BF16 inputs with no activation scales. Local validation also checks the online-method payload and rejects either activation quantizer being called, including when the per-token env is enabled. This helper is retained outside the PR after removing the broad behavioral test scaffolding.

PROBE pid=2964 constructor quant_mode=w4a16
PROBE pid=2899 constructor quant_mode=w4a16
PROBE pid=2850 constructor quant_mode=w4a16
PROBE pid=2802 constructor quant_mode=w4a16
PROBE pid=2964 first_run quant_mode=w4a16 x_dtype=torch.bfloat16 x_shape=(192, 1024) x_sf=None per_token_scale=None fc2_input_scale=None
PROBE pid=2802 first_run quant_mode=w4a16 x_dtype=torch.bfloat16 x_shape=(192, 1024) x_sf=None per_token_scale=None fc2_input_scale=None
PROBE pid=2899 first_run quant_mode=w4a16 x_dtype=torch.bfloat16 x_shape=(192, 1024) x_sf=None per_token_scale=None fc2_input_scale=None
PROBE pid=2850 first_run quant_mode=w4a16 x_dtype=torch.bfloat16 x_shape=(192, 1024) x_sf=None per_token_scale=None fc2_input_scale=None
PROBE pid=5286 constructor quant_mode=w4a16
PROBE pid=5123 constructor quant_mode=w4a16
PROBE pid=5219 constructor quant_mode=w4a16
PROBE pid=5171 constructor quant_mode=w4a16
PROBE pid=5286 first_run quant_mode=w4a16 x_dtype=torch.bfloat16 x_shape=(192, 1024) x_sf=None per_token_scale=None fc2_input_scale=None
PROBE pid=5123 first_run quant_mode=w4a16 x_dtype=torch.bfloat16 x_shape=(192, 1024) x_sf=None per_token_scale=None fc2_input_scale=None
PROBE pid=5219 first_run quant_mode=w4a16 x_dtype=torch.bfloat16 x_shape=(192, 1024) x_sf=None per_token_scale=None fc2_input_scale=None
PROBE pid=5171 first_run quant_mode=w4a16 x_dtype=torch.bfloat16 x_shape=(192, 1024) x_sf=None per_token_scale=None fc2_input_scale=None

To reproduce the instrumentation, put the following file at /tmp/w4a16-probe/sitecustomize.py and prepend /tmp/w4a16-probe to PYTHONPATH before running the commands above:

"""Temporary validation probe; inactive unless online CuTe DSL W4A16 is enabled.

No torch or FlashInfer import occurs at interpreter startup. The import hook
waits for SGLang's existing CuteDslMoEWrapper import, patches that real class,
then removes itself. This instrumentation is for correctness evidence, not
performance measurement.
"""

import builtins
import os


def _install():
    original_import = builtins.__import__

    def import_with_probe(name, globals=None, locals=None, fromlist=(), level=0):
        module = original_import(name, globals, locals, fromlist, level)
        if name == "flashinfer" and "CuteDslMoEWrapper" in (fromlist or ()):
            # Restore normal imports before instrumenting the already imported
            # class, including its constructor's own imports.
            builtins.__import__ = original_import
            _patch_wrapper(module.CuteDslMoEWrapper)
        return module

    builtins.__import__ = import_with_probe


def _patch_wrapper(wrapper_class):
    from functools import wraps

    original_init = wrapper_class.__init__
    original_run = wrapper_class.run
    reported_init = False
    reported_run = False

    @wraps(original_init)
    def checked_init(self, *args, **kwargs):
        nonlocal reported_init
        assert kwargs.get("quant_mode") == "w4a16", (
            f"PROBE expected CuTe DSL quant_mode=w4a16, got {kwargs.get('quant_mode')!r}"
        )
        original_init(self, *args, **kwargs)
        assert self.quant_mode == "w4a16", self.quant_mode
        self._online_w4a16_probe_checked = False
        if not reported_init:
            print(f"PROBE pid={os.getpid()} constructor quant_mode=w4a16", flush=True)
            reported_init = True

    @wraps(original_run)
    def checked_run(self, *args, **kwargs):
        nonlocal reported_run
        first_run = not self._online_w4a16_probe_checked
        if first_run:
            x = kwargs["x"]
            assert self.quant_mode == "w4a16", self.quant_mode
            assert str(x.dtype) == "torch.bfloat16", x.dtype
            for name in ("x_sf", "per_token_scale", "fc2_input_scale"):
                assert name in kwargs and kwargs[name] is None, (
                    f"PROBE W4A16 requires {name}=None"
                )
        result = original_run(self, *args, **kwargs)
        if first_run:
            self._online_w4a16_probe_checked = True
            if not reported_run:
                print(
                    f"PROBE pid={os.getpid()} first_run quant_mode=w4a16 "
                    f"x_dtype={x.dtype} x_shape={tuple(x.shape)} "
                    "x_sf=None per_token_scale=None fc2_input_scale=None",
                    flush=True,
                )
                reported_run = True
        return result

    wrapper_class.__init__ = checked_init
    wrapper_class.run = checked_run


if os.environ.get("SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16") == "1":
    _install()

Static checks: all changed-file pre-commit hooks passed; Python compilation and git diff --check passed.
Boundaries: this run covers SM103, Nemotron's ReLU2 experts, and its existing same-model EAGLE configuration. It does not revalidate SM100 hardware, independent target/draft configurations, serialized checkpoint reloads, or dense W4A16 numerics. The kernel and quantization implementations are unchanged; the follow-up changes argument validation and unused scale processing.

Speed Tests and Profiling

Motivation: FlashInfer W4A16 performance

FlashInfer PR #4985 reports strong CuTe DSL W4A16 performance: 1.28–2.61x over pure TRTLLM BF16 across all 39 configuration/token-count comparisons. Against TRTLLM NVFP4, the maximum W4A16 speedups are 1.56x, 1.66x, and 1.73x for inference per-tensor, inference per-token, and deterministic RL, respectively; the tables also preserve the slower cases. This motivates validating that --quantization nvfp4_online, --moe-runner-backend flashinfer_cutedsl, and SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16=1 reliably select the existing W4A16 path with BF16 activations.

Source: all three tables below are reproduced unchanged from the linked PR's derived summary. Each configuration contains three fresh-process sweeps, all measured at FlashInfer 1ac2233d43df747af455ed4a8406d7439d686d9f (0.6.18). They predate later benchmark refactoring; the source PR contains all nine complete raw result tables and setup/reproduction commands.
Hardware/image: one NVIDIA B300 SXM6 AC (SM103, 148 SMs, 267.7 GiB) on an eight-B300 host; driver 590.48.01. Image nvcr.io/nvidia/pytorch:26.05-py3, digest nvcr.io/nvidia/pytorch@sha256:222d8b18e671be5c3ef91cb41727a2572a0b23f59ded6c39f373a96946f6f2ba; CUDA 13.2 (nvcc V13.2.78), PyTorch 2.12.0a0+5aff3928d8.nv26.05, Python 3.12.3, CuTe DSL [cu13]==4.7.0.
Workload/timing: DeepSeek-V3 MoE shapes H=7168, I=2048, E=256, top-k=8; TP1 and 32 local experts simulate an EP8 shard on one GPU. Shared source BF16 inputs/logits per row; 8 groups/top-4 groups, routed scale 2.5, bias scale 0.01, seed 42. CUDA graphs, CUPTI, autotuning, cold L2, 10 warmups and 100 iterations per case. Timed work includes routing, expert computation, finalize, and initial FP4 activation quantization; W4A16 and pure BF16 consume BF16 directly. Weight preparation, compilation, and autotuning are excluded.
Inference per-tensor: global-scale FP4 activation quantization, 4over6 disabled, quantization fast math enabled; CuTe fused atomic finalize. Includes CUTLASS NVFP4.
Inference per-token: per-token FP4 activation scaling, 4over6 disabled, quantization fast math enabled; CuTe fused atomic finalize. CUTLASS is omitted because it does not consume per-token scales.
Deterministic RL: per-token 4over6 MSE, E4M3 max 256, error fast math enabled, FP4 quantization fast math disabled; CuTe two-stage finalize. This preserves the source configuration name and is not a full-model determinism claim. Both TRTLLM arms retain native finalize in all three configurations. The activation quantization flags do not quantize W4A16/BF16 activations.

Latencies are medians of three fresh-process runs, calculated from unrounded captures and displayed in milliseconds to three decimal places. Each W4A16 speedup is the named TRTLLM baseline's median latency divided by the CuTe DSL W4A16 median latency at that token count. These are ratios of medians; values above 1 mean W4A16 is faster; values below 1 mean W4A16 is slower. Both NVFP4 and pure BF16 comparisons are explicit.

Inference per-tensor — median of three runs

Tokens	CuTe W4A4 (ms)	CuTe W4A16 (ms)	CUTLASS NVFP4 (ms)	TRTLLM NVFP4 (ms)	TRTLLM BF16 (ms)	W4A16 speedup over TRTLLM NVFP4	W4A16 speedup over TRTLLM BF16
1	0.031	0.040	0.048	0.026	0.051	0.65x	1.29x
2	0.040	0.046	0.058	0.036	0.079	0.79x	1.72x
4	0.057	0.063	0.070	0.051	0.117	0.81x	1.85x
8	0.073	0.079	0.086	0.064	0.157	0.81x	1.99x
16	0.083	0.085	0.097	0.076	0.192	0.89x	2.25x
32	0.114	0.118	0.127	0.106	0.285	0.89x	2.41x
64	0.147	0.151	0.159	0.140	0.388	0.92x	2.56x
128	0.150	0.158	0.164	0.144	0.399	0.91x	2.53x
256	0.158	0.164	0.173	0.154	0.428	0.93x	2.60x
512	0.160	0.169	0.179	0.164	0.435	0.97x	2.58x
1024	0.165	0.190	0.197	0.295	0.443	1.56x	2.34x
2048	0.177	0.233	0.223	0.299	0.476	1.28x	2.04x
4096	0.217	0.363	0.281	0.309	0.529	0.85x	1.46x

Inference per-token — median of three runs

Tokens	CuTe W4A4 (ms)	CuTe W4A16 (ms)	TRTLLM NVFP4 (ms)	TRTLLM BF16 (ms)	W4A16 speedup over TRTLLM NVFP4	W4A16 speedup over TRTLLM BF16
1	0.048	0.040	0.034	0.051	0.87x	1.28x
2	0.053	0.046	0.046	0.079	1.01x	1.73x
4	0.072	0.062	0.060	0.116	0.97x	1.88x
8	0.088	0.078	0.074	0.156	0.95x	2.00x
16	0.097	0.087	0.086	0.190	1.00x	2.20x
32	0.132	0.120	0.116	0.287	0.97x	2.39x
64	0.165	0.153	0.149	0.387	0.97x	2.52x
128	0.170	0.157	0.154	0.398	0.98x	2.53x
256	0.178	0.164	0.164	0.427	1.00x	2.60x
512	0.182	0.169	0.174	0.436	1.03x	2.58x
1024	0.192	0.190	0.316	0.443	1.66x	2.33x
2048	0.213	0.233	0.326	0.477	1.40x	2.05x
4096	0.268	0.362	0.338	0.529	0.94x	1.46x

Deterministic RL — median of three runs

Tokens	CuTe W4A4 (ms)	CuTe W4A16 (ms)	TRTLLM NVFP4 (ms)	TRTLLM BF16 (ms)	W4A16 speedup over TRTLLM NVFP4	W4A16 speedup over TRTLLM BF16
1	0.048	0.039	0.036	0.053	0.91x	1.34x
2	0.059	0.046	0.047	0.079	1.03x	1.71x
4	0.077	0.060	0.061	0.117	1.01x	1.93x
8	0.093	0.075	0.075	0.157	1.00x	2.09x
16	0.104	0.083	0.087	0.190	1.04x	2.28x
32	0.138	0.117	0.117	0.287	1.00x	2.46x
64	0.175	0.150	0.150	0.386	1.00x	2.57x
128	0.180	0.155	0.156	0.398	1.01x	2.57x
256	0.189	0.167	0.166	0.427	0.99x	2.55x
512	0.194	0.167	0.177	0.436	1.06x	2.61x
1024	0.208	0.185	0.320	0.443	1.73x	2.40x
2048	0.238	0.224	0.333	0.477	1.49x	2.13x
4096	0.311	0.328	0.350	0.528	1.07x	1.61x

Inference per-tensor: W4A16 speedup 0.65–1.56x over TRTLLM NVFP4 (2/13 faster); 1.29–2.60x over TRTLLM BF16 (13/13 faster).
Inference per-token: W4A16 speedup 0.87–1.66x over TRTLLM NVFP4 (4/13 faster); 1.28–2.60x over TRTLLM BF16 (13/13 faster).
Deterministic RL: W4A16 speedup 0.91–1.73x over TRTLLM NVFP4 (9/13 faster); 1.34–2.61x over TRTLLM BF16 (13/13 faster).

Scope of this evidence: these are historical FlashInfer single-GPU local-expert microbenchmarks that motivate validating the SGLang configuration; this PR does not claim a measured SGLang end-to-end speedup. EP dispatch/combine, collectives, network, and the rest of the model are excluded. Native TRTLLM logits routing stores selected weights in BF16 while CuTe route weights are FP32, so these results do not establish numerical parity. Separate inference sweeps remeasure the unaffected W4A16/BF16 paths; differences between those sweeps cannot be attributed solely to activation scaling. Coverage is B300/SM103 and the stated shapes.

No SGLang performance benchmark was run for this follow-up. Throughput printed by the accuracy harness is incidental; those runs used temporary runtime assertions.

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #34044380251
Latest PR Test (Extra): ❌ Run #34044380578
Latest PR Test (AMD ROCm 7.2): ❌ Run #34044380101

mmangkad · 2026-09-06T12:33:15Z

+            if envs.SGLANG_MOE_NVFP4_DISPATCH.get():
+                raise ValueError(
+                    "CuTe DSL NVFP4 W4A16 requires BF16 MoE dispatch; "
+                    "unset SGLANG_MOE_NVFP4_DISPATCH."
+                )


I think this can go, since use_dispatch_fp4 already excludes W4A16

mmangkad · 2026-09-06T12:33:41Z

+            if view.moe_a2a_backend not in ("none", "flashinfer"):
+                raise ValueError(
+                    "CuTe DSL NVFP4 W4A16 requires "
+                    "moe_a2a_backend='none' or 'flashinfer'."
+                )


Maybe move this into handle_a2a_moe

mmangkad · 2026-09-06T12:34:11Z

 import os
 from typing import Any

+import torch


This should be function-local

mmangkad · 2026-09-06T12:34:45Z

-        if view.moe_a2a_backend == "deepep" and (
+        if envs.SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16.get():
+            if not get_platform().is_sm100:
+                raise ValueError("CuTe DSL NVFP4 W4A16 requires SM100/SM103 GPUs.")


Nit

Suggested change

raise ValueError("CuTe DSL NVFP4 W4A16 requires SM100/SM103 GPUs.")

raise ValueError("CuTe DSL NVFP4 W4A16 requires SM10X GPUs.")

mmangkad · 2026-09-06T12:35:10Z

+        for name in (
+            "SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION",
+            "SGLANG_MOE_NVFP4_DISPATCH",
+        ):
+            os.environ.pop(name, None)


Could these be envs.X.override(...) instead

mmangkad · 2026-09-06T12:35:31Z

+        with self.assertRaisesRegex(ValueError, "use --dtype bfloat16"):
+            handle_moe_kernel_config(args)
+
+    def test_online_w4a4_still_accepts_per_token_activation(self):


Maybe add moe_a2a_backend="deepep" here too, that's the branch the elif refactor actually touches

zianglih added 2 commits September 6, 2026 04:08

Validate CuTe DSL online NVFP4 W4A16 configuration

a938aec

Keep argument test setup compatible with Python 3.10

8cd85ec

zianglih requested review from Alisehen, AniZpZ, BBuf, Edwardf0t1, FlamingoPg, HaiShaw, OrangeRedeng, b8zhong, ch-wan and mmangkad as code owners September 6, 2026 11:23

github-actions Bot added the blackwell SM100/SM120 label Sep 6, 2026

mmangkad reviewed Sep 6, 2026

View reviewed changes

zianglih marked this pull request as draft September 6, 2026 15:54

zianglih added 2 commits September 6, 2026 09:02

Address W4A16 configuration review feedback

d8c8b0b

Remove broad W4A16 behavioral test scaffolding

8a7cd7b

github-actions Bot added the quant LLM Quantization label Sep 6, 2026

Navigation Menu

Uh oh!

Validate CuTe DSL online NVFP4 W4A16 configuration - #38215

Validate CuTe DSL online NVFP4 W4A16 configuration#38215
zianglih wants to merge 4 commits into
sgl-project:mainfrom
zianglih:fix/nvfp4-online-cutedsl-w4a16

zianglih commented Sep 6, 2026 •

edited

Loading

Uh oh!

mmangkad Sep 6, 2026

Uh oh!

mmangkad Sep 6, 2026

Uh oh!

mmangkad Sep 6, 2026

Uh oh!

mmangkad Sep 6, 2026

Uh oh!

mmangkad Sep 6, 2026

Uh oh!

mmangkad Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	raise ValueError("CuTe DSL NVFP4 W4A16 requires SM100/SM103 GPUs.")
	raise ValueError("CuTe DSL NVFP4 W4A16 requires SM10X GPUs.")

Uh oh!

Conversation

zianglih commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Review follow-up validation

Initial GPU validation

Speed Tests and Profiling

Motivation: FlashInfer W4A16 performance

Checklist

Review and Merge Process

CI States

Uh oh!

mmangkad Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

mmangkad Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

mmangkad Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

mmangkad Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

mmangkad Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

mmangkad Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zianglih commented Sep 6, 2026 •

edited

Loading