[qianlihuang]

Overview

vLLM has no MegaMoE that runs on Hopper today:

• the DeepGEMM MegaMoE (deep_gemm_mega_moe) requires SM100;
• the FlashInfer MoE-EP megakernels (flashinfer_moe_ep_cutedsl, flashinfer_moe_ep_mega_deep_gemm) require SM100

This PR runs block-FP8 MoE checkpoints on Hopper with FlashInfer's SM90 FP8 MegaMoE, which ships in FlashInfer 0.7.2. The kernel fuses dispatch, both expert GEMMs and combine. It goes through the existing --moe-backend flashinfer_moe_ep_cutedsl path with no model code, so any block-FP8 MoE can use it.

Related:

• [MoE] DeepSeek-V4: add Hopper (SM90) FP8 FlashInfer moe_ep mega backend #58540: the same kernel, but for DeepSeek-V4 FP4 checkpoints converted to FP8, and on files that [feat] FlashInfer CuteDSL MegaMoE integration  #54049 removed.
• [Beam] Enable CuTeDSL NVFP4 MegaMoE #62073: Beam on SM100. It contains the same idle-rank change as this PR.
• [Perf][GLM-5.3-Flash] DeepGEMM FP8 MegaMoE with fused shared experts #60895: DeepGEMM MegaMoE for GLM-5.3-Flash, SM100 only.
• [DSV4][SM90] add dsv4 fp8 weight use mega_moe backend #53527: DeepSeek-V4 through a DeepGEMM fork.

Claims

• GLM-5.3-Flash decode on 8x H200 (DP8/EP8, 128K/512, MTP3): +11% to +18% output throughput and -6% to -14% TPOT p50, at concurrency 8 to 192.
• GLM-5.3-Flash prefill on 4x H200 (TP2/DP2/EP4, MTP3, ISL 2048 / OSL 1): +23% saturated throughput, and TTFT p50 -14% to -54% at 15 to 21 req/s.
• One MoE layer at EP8: -7% at 8 tokens/rank, up to -48% at 8192 tokens/rank.
• GSM8K is unchanged in both configurations.
• The backend is opt-in. Default backends do not change.

Validation

Setup.

• Source: main fa75d34.
• Client: vllm-rs bench serve (0.32.0rc1.dev284) with token-ID prompts.
• Runs are serial A/B/A on the same GPUs. "Base" is the mean of the two baseline arms.

Decode.

• 8x H200, DP8/EP8, DecodeBenchConnector, ISL 128K / OSL 512, MTP3 with synthetic acceptance [0.8, 0.6, 0.5], N = 2C prompts, two passes per arm.
• These runs used the first commit. The later review fixes do not change this path under DP. One extra pass on the second commit gave +8% to +24% throughput over the same base.

C	8	32	64	128	192
output tok/s, base	911	2895	5204	7977	10074
output tok/s, this PR	1009 (+10.8%)	3402 (+17.5%)	5893 (+13.2%)	9326 (+16.9%)	11917 (+18.3%)
TPOT p50 ms, base → this PR	7.60 → 6.86	9.20 → 7.96	9.27 → 8.72	10.81 → 9.85	11.58 → 10.87

Prefill.

• 4x H200, TP2 × DP2 + EP4, natural MTP3, ISL 2048 / OSL 1. Both arms run sequence-parallel MoE.
• Saturation: 64 warm-up requests, then 3 × 1000 requests at an unlimited rate. Poisson: 60 s per rate.
• These runs used the current head's files.

	base	this PR
saturated req/s, median of 3	21.44	26.35 (+22.9%)
TTFT p50 ms at 11 / 15 / 19 / 21 req/s	370 / 488 / 787 / 1365	409 / 418 / 515 / 630
TTFT p99 ms at 11 / 15 / 19 / 21 req/s	838 / 939 / 1438 / 2699	737 / 808 / 820 / 1059

The base saturates at 21.4 req/s. At 23 and 25 req/s its TTFT p50 is 3.8 and 4.5 s, while this PR serves 22.7 and 24.7 req/s at 610 and 831 ms. At 11 req/s the median TTFT is 10% higher (base arms 351 and 389 ms).

MoE layer, EP8. GLM-5.3-Flash expert shapes, milliseconds per layer:

tokens/rank	8	64	128	512	1024	8192
baseline	0.265	0.381	0.446	0.834	1.378	9.204
this PR	0.247	0.294	0.310	0.461	0.754	4.798

GSM8K. 1319 questions, 5-shot, natural MTP3:

• DEP8: 0.917 → 0.919.
• TP2/DP2: 0.918 and 0.920 → 0.917 and 0.920. MTP acceptance 0.804 → 0.802.

Tests.

pytest tests/models/test_deepseek_v4_fi_moe_ep.py tests/kernels/moe/test_flashinfer_moe_ep.py
# CPU: 29 passed. SM90 GPU: 8 passed (4 expert shapes x EP2/EP4).

The SM90 GPU tests need at least two Hopper GPUs and nvshmem4py, so CI does not run them.

Commands

--moe-backend flashinfer_moe_ep_cutedsl is omitted for the baseline. Both servers run with VLLM_USE_V2_MODEL_RUNNER=1.

# Decode
vllm serve <GLM-5.3-Flash FP8> --data-parallel-size 8 --enable-expert-parallel \
  --max-num-seqs 64 --gpu-memory-utilization 0.9 --no-enable-prefix-caching --use-replayssm \
  --limit-mm-per-prompt '{"image":0,"video":0}' \
  --kv-transfer-config '{"kv_connector":"DecodeBenchConnector","kv_role":"kv_both"}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_rates":[0.8,0.6,0.5]}' \
  --moe-backend flashinfer_moe_ep_cutedsl

vllm-rs bench serve --dataset-name random --random-input-len 131072 --random-output-len 512 \
  --prompt-token-ids --ignore-eos --request-rate inf --max-concurrency $C --num-prompts $((2 * C)) --seed 0

# Prefill
vllm serve <GLM-5.3-Flash FP8> --tensor-parallel-size 2 --data-parallel-size 2 --enable-expert-parallel \
  --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --moe-backend flashinfer_moe_ep_cutedsl

# warm-up: --num-prompts 64 --seed 100; saturation: --seed 102, 104, 106
vllm-rs bench serve --dataset-name random --random-input-len 2048 --random-output-len 1 \
  --prompt-token-ids --ignore-eos --request-rate inf --num-prompts 1000 --seed 102
# Poisson
vllm-rs bench serve --dataset-name random --random-input-len 2048 --random-output-len 1 \
  --prompt-token-ids --ignore-eos --request-rate $R --num-prompts $((60 * R)) --seed $((200 + R))

# GSM8K: the decode server without the KV connector and synthetic acceptance, or the prefill server
python tests/evals/gsm8k/gsm8k_eval.py --port $PORT

Details

• Kernel.
  • FlashInfer Sm90PullFp8MegaMoeConfig, with FP32 128x128 weight scales and a SwiGLU clamp.
  • FlashInfer's push-style SM90 backend has no SwiGLU clamp, so it cannot serve swiglu_limit models.
• Weights.
  • The kernel wants FC1 rows interleaved as [pair, {gate, up}, 8] under one 128x128 scale. Each scale block therefore covers rows from two checkpoint blocks, and both use max(s_gate, s_up).
  • This is exact for power-of-two scales, which GLM-5.3-Flash has. Other checkpoints get one extra E4M3 rounding on half of the FC1 rows.
• Workspaces.
  • The kernel always processes its whole token buffer, with a launch geometry picked for that size. With a single buffer sized at max_num_batched_tokens, decode is slower than the baseline.
  • The adapter therefore adds 64-, 256- and 1024-token workspaces. Each call uses the smallest one that fits the largest batch across DP ranks. Without DP-wide token counts it uses the default workspace, because every rank must pick the same buffer.
• Loaders.
  • The fp8, compressed-tensors, online FP8 and Quark FP8 MoE methods now call the experts' process_weights_after_loading. It is a no-op for other experts.
  • The SM90 path is chosen from the quant config, not the weight dtype. FP8 weights keep their dtype, so a rebuild from preprocessed weights picks the same path.
• Idle ranks. With pass-through stages, the experts own the EP collective. A rank without tokens therefore still enters them. The change is identical to the one in [Beam] Enable CuTeDSL NVFP4 MegaMoE #62073.
• Limitations.
  • The backend needs expert parallelism and nvshmem4py.
  • It rejects DBO, runtime weight transfer and apply_router_weight_on_input.
  • Shared experts are not fused.
  • FlashInfer's SM90 input staging is torch-composed, about 1.2 ms per step at C128.

FlashInfer upstream

• [flashinfer-ai/flashinfer#4113](feat(moe_ep): SM90 (Hopper) pull-style FP8 mega-MoE backend flashinfer-ai/flashinfer#4113): Introduces the SM90 pull-style FP8 MegaMoE kernel (Sm90PullFp8MegaMoeConfig) used by this PR through FlashInfer's MoEEpLayer API.
• [flashinfer-ai/flashinfer#6999](feat(moe_ep): add SM90 NVFP4 x FP8 CuTe DSL MegaMoE backend flashinfer-ai/flashinfer#6999): Ongoing work to add an SM90 NVFP4-weight / FP8-activation (W4A8) MegaMoE backend to the same MoE-EP API. Support for this backend is outside the scope of this PR and can be added as a follow-up.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[kabishou11]

We are tracking this from an offline DeepInfer Hopper tree, not a vLLM serving stack.

On SM90 we already run DeepEP V2 + humming MXFP4 (float8e4m3-gs32, humming-kernels==0.1.13) and explicitly reject mega-MoE / NVFP4 backend names. We will not vendor FlashInfer moe_ep / CuTeDSL CUDA. After this PR merges we can re-test on H200 against that humming baseline, but only if FlashInfer is an importable dependency and the checkpoint under test is actually block-FP8 (stock DeepSeek-V4.1-Flash here is MXFP4 experts).

Question: does the Hopper moe_ep path still require block-FP8 weights, or is there a planned MXFP4-resident entry that would apply to an unmodified DeepSeek-V4.1-Flash checkpoint?

[qianlihuang]

@kabishou11 This PR focuses on the SM90 block-FP8 path, and FP8 achieved better serving throughput than NVFP4 in my H200 benchmarks.

For DeepSeek MXFP4 checkpoints, I think you could try FlashInfer's SM90 FP4 MegaMoE implementation and SM90 FP4 MoE-EP implementation.

[kabishou11]

Thanks @qianlihuang — that clarifies it. We'll keep treating this PR as the SM90 block-FP8 path, and park stock MXFP4 DeepSeek-V4.1-Flash against FlashInfer #6343 as a watch item rather than expecting an MXFP4-resident entry here. Appreciate the H200 FP8 vs NVFP4 pointer.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @qianlihuang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork