[mgoin]

Overview

On Hopper with DeepGEMM installed, is_deep_gemm_e8m0_used() is true by default for every layer. This forces two costs on the Triton FP8 MoE path, which takes float32 scales:

• Power-of-two activation scales. It rounds both of its activation quants' scales to powers of two, which adds error and buys nothing.
• Unfused SiLU+quant. It skips the fused silu_and_mul_per_block_quant, so SiLU+Mul and quant run as two kernels.

This PR makes the scale format a per-layer property of the selected MoE backend, and runs the fused kernel in every case it can serve.

It supersedes #64906 (same goal for the activation scales) and #43399 (UE8M0 option for the fused kernel). It also covers interactions those PRs predate: the sorted-A TMA path from #64760, the batched Triton experts, and the compile fusion.

Depends on #65791: the first commit here is #65791, which fixes the fused kernel's prefill performance. Without it, enabling the fused path by default costs -3.8% prefill. Only the last two commits are new.

Claims

• Lower error. Triton FP8 MoE layers use float32 activation scales. At the GLM-5.3 TP8 shape, relative error vs a bf16 reference drops from 0.069 to 0.062 at M=1 and from 0.066 to 0.063 at M >= 64, measured on main 4527515cd7.
• One fewer launch per layer at decode. SiLU+Mul+quant runs as one kernel by default. Qwen3-30B-A3B-FP8 H200 TPOT improves 2.5-3.5% at concurrency 1/8/32, with prefill unchanged.
• Faster at every M for GLM-5.3. At the TP8 shape (where [GLM 5.3 Perf] Sorted-A TMA for FP8 MoE down projection on H200, 6.0%~ 7.6% E2E Throughput Improvement #64760's sorted-A path is active), the layer is 0.7-5% faster than main at every M from 1 to 4096.
• No change for DeepGEMM backends (they keep UE8M0) or for checkpoints with scale_fmt: ue8m0.

Validation

Qwen/Qwen3-30B-A3B-FP8, H200 TP1, --moe-backend auto (Triton), default VLLM_USE_DEEP_GEMM_E8M0=1.

• Workloads, all vllm bench serve --dataset-name random --ignore-eos: prefill 4096/1 at concurrency 64, mixed 1024/512 at concurrency 128, decode 1024/256.
• Two runs per arm on two GPUs (swapped), best shown.
• Measured on 34d5de18cd with these changes; the code is identical after the rebase, apart from [GLM 5.3 Perf] Sorted-A TMA for FP8 MoE down projection on H200, 6.0%~ 7.6% E2E Throughput Improvement #64760 interplay, which doesn't affect this model.

	prefill tok/s	mixed out tok/s	TPOT c1	TPOT c8	TPOT c32	gsm8k strict / flex
main	66,596	7,669	4.0 ms	5.8 ms	8.6 ms	0.8878 / 0.8529
this PR	66,480	7,741	3.9 ms	5.6 ms	8.3 ms	0.8901 / 0.8688

gsm8k is 5-shot over 1319 questions, with stderr ~0.009. With --moe-backend deep_gemm the score is 0.8939 / 0.8582. On TP2 + EP (FlashInfer CUTLASS, which quantizes internally, so the compute is unchanged) it is 0.8893 / 0.8575 vs 0.8946 / 0.8628.

GLM-5.3 TP8 shape (H=6144, I/TP=256, E=256, top-8), on main 4527515cd7, Triton layer time, CUDA graph, p50, two runs within 0.3%:

M	1	8	64	256	1024	4096
main	38.8 µs	94.9	283.2	340.1	442.7	954.6
this PR	36.9	92.4	280.9	337.8	438.0	947.7

Dense Qwen/Qwen3-8B-FP8: perf and gsm8k are unchanged (59.7k vs 59.6k prefill tok/s; 0.8840 / 0.8772 both). On Hopper its FP8 linear quantizes inside the FlashInfer DeepGEMM op, so the compile fusion does not match.

Tests (H200, E8M0 on and off; every failure listed also fails on main). Confirmed on this exact branch (see the comment below for the run):

• tests/kernels/moe/test_block_fp8.py:
  • E8M0 on: 179 passed vs 176 on main, with the same 3 near-tolerance failures (N=4608, K=7168).
  • E8M0 off: 1 tolerance failure and 128 deep_gemm_fused_moe "WorkspaceManager not initialized" errors. On main the same setting fails those 128 errors plus 18 tolerance cases and test_sorted_a_tma_moe_matches_route_major.
  • test_sorted_a_tma_moe_matches_route_major (bitwise, now parametrized over UE8M0 and fp32 scales) passes with E8M0 on and off.
• tests/kernels/core/test_fused_silu_mul_block_quant.py: 371 passed. The new cases cover the UE8M0 and clamp options with opcheck, and vectorized-vs-scalar bitwise checks.
• tests/kernels/moe/test_deepgemm.py passes, including the new test_e8m0_act_scales_only_for_deepgemm_backend.
• test_batched_moe.py -k fused_moe_batched_experts: 288 passed, with E8M0 on and off.
• tests/compile/passes/test_silu_mul_quant{,_manual}_fusion.py: 126 passed.

Details

• Per-layer scale format. FusedMoEQuantConfig.use_ue8m0_act_scales is set by make_fp8_moe_quant_config from the selected backend: True for DeepGEMM backends or a scale_fmt=ue8m0 checkpoint, False otherwise, None (global behavior) when there is no vLLM config. It is read by the no-DP and naive-DP prepare steps, moe_kernel_quantize_input, the Triton experts' quant calls, the fused kernel and the sorted-A kernel.
  • [Perf][MoE] Keep fp32 activation scales on the Triton FP8 experts and re-enable their fused SiLU+quant path #64906 instead declares the preference on the experts class and copies it into the config in the modular kernel, and also covers DeepEP/NIXL prepare. Those prepare steps are not changed here: they keep the global setting, which is mathematically fine since mixed scale formats are valid.
• Fused kernel. silu_and_mul_per_block_quant gains use_ue8m0 and clamp_limit, so the E8M0 and SwiGLU-clamp cases no longer fall back. Remaining fallbacks log warning_once with the reason: non-SiLU, LoRA (it needs the bf16 activation) and group sizes other than 64/128.
• [GLM 5.3 Perf] Sorted-A TMA for FP8 MoE down projection on H200, 6.0%~ 7.6% E2E Throughput Improvement #64760 sorted-A path. Its Triton SiLU+Mul+quant kernel now uses the CUDA kernel's arithmetic: fp32 SiLU*up without intermediate bf16 rounding, max(amax/448, 1/(448*512)), then x/scale. It also takes the layer's scale format, so the two layouts stay bitwise identical and the path no longer requires global E8M0.
• Batched experts. BatchedTritonExperts uses persistent_masked_m_silu_mul_quant for SiLU block-128 instead of activation followed by a per-expert quant loop with .item() syncs. It still falls back with a SwiGLU clamp, since that kernel has no clamp.
• Compile fusion. The SiluMul+BlockQuant fusion now forwards the UE8M0 flag; it previously replaced a UE8M0 quant pattern with an fp32-scale fused op. This overlaps [Bugfix] Stop fusing scale_ue8m0=True group quant into ue8m0-unaware fused kernels #51477.
• Reference fix. test_block_fp8's fp32-SiLU reference now applies on CUDA too (it was ROCm-only), since the fused path now runs there by default.
• Not covered: the compile fusion's TMA-aligned scale layout ([Bugfix][Kernel][Perf] Honor outer_scale_stride in silu_and_mul_per_block_quant for TMA-aligned transposed scales #62386).

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mgoin]

Confirmation run on this exact branch (9c234b8b05), H200, VLLM_USE_DEEP_GEMM_E8M0=1 unless noted:

test	result
test_block_fp8.py::test_sorted_a_tma_moe_matches_route_major (E8M0=1 / E8M0=0)	2 passed / 2 passed
tests/kernels/moe/test_deepgemm.py	27 passed, 3 skipped
tests/kernels/core/test_fused_silu_mul_block_quant.py	371 passed, 8 skipped
tests/kernels/moe/test_block_fp8.py	179 passed, 3 failed (the same 3 test_w8a8_block_fp8_fused_moe N=4608/K=7168 tolerance cases fail on main)
test_batched_moe.py -k fused_moe_batched_experts	288 passed
tests/compile/passes/test_silu_mul_quant{,_manual}_fusion.py	126 passed

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mgoin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork