[Perf] Fuse prefill norm/act quantization for NVFP4 W4A4 hybrid models (Qwen3.5 family)

[BBuf]

Rebased onto main after #34859 merged: the SiLU+FP4-quant base fusion landed with the model support, so this PR's net diff contains the three remaining changes (attention-layer SiLU coverage, post-LN+FP4-quant, and gated-norm+FP8-quant).

Motivation

For Qwen3.5-family hybrid (GDN + attention) models served with a mixed ModelOpt NVFP4(W4A4)+FP8 checkpoint, prefill spends a measurable share of GPU time in standalone norm / activation / quantization glue kernels: every projection input is normalized, written to HBM in bf16, then re-read and quantized by a separate kernel. Kernel-level tracing against vLLM 0.27.1 on the same model showed the GEMM/attention/GDN families at parity while SGLang lost time in this glue (vLLM fuses norm+quant and act+quant via torch.compile).

This PR removes the extra round-trips with three targeted fusions (all bit-equivalent up to FP4/FP8 rounding ties, each with a kill switch):

1. SiLU+mul + NVFP4 quant for dense MLPs: replaces act_and_mul + the standalone per-block FP4 quantize with FlashInfer's silu_and_mul_scaled_nvfp4_experts_quantize (single-group form), handing down_proj a prequantized (fp4, swizzled_scale) tuple through the existing _accepts_prequantized_fp4 path. Enabled for both GDN-layer and attention-layer MLPs.
2. Post-attention GemmaRMSNorm + NVFP4 quant: FlashInfer add_rmsnorm_fp4quant fuses the residual-add, the norm, and the gate_up input quant into one kernel. Gemma's (1 + w) scaling is folded into a precomputed weight; the swizzled block-scale is exposed as the 2D (m_padded, K/16) view so the cuDNN mm_fp4 backend's shape validation also passes.
3. GDN gated RMSNorm + static-FP8 quant: adds an fp8-store epilogue to the fla layernorm_gated Triton kernel (skips the bf16 intermediate write entirely) and teaches ModelOptFp8LinearMethod to accept a prequantized (fp8, scale) tuple for out_proj (explicit opt-in attribute, mirroring the fp4 path).

Kill switches: SGLANG_DISABLE_SILU_FP4_QUANT_FUSION=1, SGLANG_DISABLE_POST_LN_FP4_QUANT_FUSION=1, SGLANG_DISABLE_GATED_NORM_FP8_QUANT_FUSION=1.

The fusions auto-enable only when the touched projections use the matching ModelOpt linear method (NVFP4 W4A4 for 1-2, per-tensor static FP8 for 3), so other checkpoints and quant modes keep their current control flow.

Benchmark

Re-benchmarked on 2026-08-20 against the current PR base and head. Both lanes ran back-to-back on the same otherwise-idle physical GPU with the same container, model snapshot, server flags, prompt seed, and client command.

Environment and revisions

Item	Value
GPU	NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120), physical GPU 2
Driver	580.159.03
Container	lmsysorg/sglang:qwen38
PyTorch / CUDA	2.13.0+cu130 / CUDA 13.0
FlashInfer	0.6.18
Model	RadixArk/Qwen3.8-27B-NVFP4
Model snapshot	554ebba9b5f1b79dc11246341960360e6ef05ef4
main/base	a4ffb996db3e823708a53637f851e2677d4455a8
PR head	a3facf77a9bf07464d87299d21dede0837df8e9d
merge base	a4ffb996db3e823708a53637f851e2677d4455a8

Because the merge base equals the tested main revision, this comparison isolates the PR's net diff.

Commands

The commands below were run inside the existing sglang_bbuf container. For the main lane, SGLANG_TREE=/work/pr34934-rebench-main; for the PR lane, SGLANG_TREE=/work/pr34934-rebench-head.

Server:

SGLANG_TREE=/work/pr34934-rebench-head

CUDA_VISIBLE_DEVICES=2 \
PYTHONPATH="${SGLANG_TREE}/python" \
python -m sglang.launch_server \
  --model-path RadixArk/Qwen3.8-27B-NVFP4 \
  --host 0.0.0.0 \
  --port 31002 \
  --tp-size 1 \
  --chunked-prefill-size 8192 \
  --max-prefill-tokens 8192 \
  --kv-cache-dtype fp8_e4m3 \
  --mamba-ssm-dtype bfloat16 \
  --disable-radix-cache

Benchmark client:

SGLANG_TREE=/work/pr34934-rebench-head
LANE=pr
ROUND=1

CUDA_VISIBLE_DEVICES= \
PYTHONPATH="${SGLANG_TREE}/python" \
python -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 31002 \
  --dataset-name random \
  --model RadixArk/Qwen3.8-27B-NVFP4 \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --random-range-ratio 1 \
  --num-prompts 5 \
  --request-rate inf \
  --max-concurrency 1 \
  --seed 1234 \
  --warmup-requests 1 \
  --disable-tqdm \
  --output-details \
  --output-file "/work/pr34934-rebench-artifacts/${LANE}-round-${ROUND}.jsonl"

For each revision, one complete five-request run was discarded as a pre-run, then the recorded command was run three times (five requests per round, 15 measured requests per revision). Every invocation also ran the client's one built-in warmup request. GPU 2 had 0 MiB allocated before the first server was started and after the final server was stopped.

Per-round results

Revision	Round	Mean TTFT (ms)	Median TTFT (ms)	Output tok/s	Median ITL (ms)
main	1	655.48	642.42	64.66	14.94
main	2	655.66	649.62	64.22	14.96
main	3	653.87	638.01	64.41	14.96
PR	1	638.52	626.78	64.32	14.95
PR	2	640.99	627.77	64.42	14.95
PR	3	640.33	628.54	64.53	14.95

Aggregate result

TTFT aggregates all 15 per-request observations. Output throughput is total generated tokens divided by total benchmark duration across the three recorded rounds.

Metric	main	main + this PR	Delta
Mean TTFT	655.00 ms	639.95 ms	-15.06 ms (-2.30%)
Median TTFT	642.42 ms	627.77 ms	-14.65 ms (-2.28%)
Output throughput	64.43 tok/s	64.42 tok/s	-0.02%
Median ITL	14.957 ms	14.949 ms	-0.008 ms

For the paired TTFT observations, the mean gain was 15.06 ms and the approximate 95% confidence interval was 10.79-19.32 ms. The current result supports a stable ~2.3% prefill TTFT improvement; decode throughput and ITL are unchanged within run-to-run noise.

Startup logs confirmed that the optimized paths were active for the expected layers:

• post-LN + FP4 quant: 64 layers
• SiLU + FP4 quant: 64 layers
• gated RMSNorm + FP8 quant: 48 GDN layers

Both server logs were checked for traceback, CUDA error, OOM, and exception signatures; none were found.

Correctness validation

Correctness was not rerun as part of the 2026-08-20 performance pass. The earlier full GSM8K validation (1319 examples) reported:

Configuration	Score
fusions off	0.9635
fusions on	0.9658

The fused kernels were also previously checked for identical residual propagation relative to fused_add_rmsnorm.

Data provenance

The performance tables above come from the current PR head and its exact merge base, using three saved bench_serving --output-details JSONL results per lane. They replace the older 5-request/FlashInfer-0.6.17 table, which was measured against an earlier base and was not reproducible as a current-main/current-PR comparison. The previous H200 FP8-dynamic table was removed because these W4A4-specific fusions are inactive for that checkpoint and therefore it did not measure this PR's performance effect.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ❌ Run #32368351888
Latest PR Test (Extra): ❌ Run #32368351907
Latest PR Test (AMD ROCm 7.2): ❌ Run #32368351823

[BBuf]

Retargeting: these fusions now stack into #34859 directly via yhyang201#5 (base = qwen38-27b-model-support), so the model-support PR carries them when it merges. That cross-fork PR also contains the main-conflict resolution for #34859 (merge of 8922bb9). All benchmark/accuracy data from this description applies unchanged.

[JustinTong0323]

Verified this PR on a DGX Spark (GB10) — summary: correctness checks out, perf is neutral on this chip (no gain, no regression).

Environment

Item	Value
GPU	NVIDIA GB10 (SM121, 121GB unified memory), DGX Spark
PyTorch / CUDA	2.13.0+cu130
FlashInfer	0.6.17
Model	RadixArk/Qwen3.8-27B-NVFP4, snapshot 554ebba9b5f1b79dc11246341960360e6ef05ef4 (same as PR)
base	a4ffb996db3e823708a53637f851e2677d4455a8 (= merge base)
head	a3facf77a9bf07464d87299d21dede0837df8e9d

Server flags and bench client command were copied verbatim from the PR description (8K input / 1K output, 5 prompts, --max-concurrency 1, seed 1234, PYTHONPATH-overlaid trees).

Correctness ✅

• Startup logs confirm all three fusions active with the expected layer counts: fused SiLU+FP4-quant ×64, fused post-LN GemmaRMSNorm+FP4-quant ×64, gated RMSNorm+FP8-quant ×48 GDN layers.
• 5 fixed prompts (GSM8K-style + short-form), greedy decoding (temperature=0), 256 max new tokens: head outputs are byte-identical to base.

Performance ⚠️ (neutral on GB10)

First pass (3 base rounds, then 3 head rounds) suggested the head lane was slightly slower, but absolute numbers drifted upward over the session (TPOT 75 → 80 ms) — GB10 thermal throttling. Re-ran interleaved with GPU temperature logged:

Lane	Round	GPU temp	Mean TTFT (ms)	Median TTFT (ms)	TPOT (ms)
head	4	51°C	4241.7	4206.5	80.0
base	4	59°C	4153.9	4119.6	80.2
head	5	60°C	4143.3	4114.8	80.6
base	5	60°C	4144.8	4114.3	80.4

Same-temperature adjacent pair (head r5 vs base r5, both at 60°C) is a dead heat, and decode TPOT is identical within noise across all interleaved rounds. On GB10 the fusions land within ±1% run-to-run noise — the RTX PRO 6000's ~2.3% TTFT gain does not reproduce here. Plausibly expected: with GB10's low memory bandwidth, prefill is dominated by GEMM/attention rather than the norm/act/quant HBM round-trips this PR removes.

No tracebacks, CUDA errors, or OOMs in any of the 12 server runs (6 bench lanes × server + 2 correctness + warmups). No objections from the GB10 side — functionally correct, perf-neutral here.