[qianlihuang]

Overview

--moe-backend flashinfer_moe_ep_cutedsl now runs MXFP4 MoE checkpoints, such as DeepSeek-V4.1-Flash, on Hopper. It uses FlashInfer's SM90 MXFP4 x FP8 MegaMoE (flashinfer-ai/flashinfer#4843) through the existing FlashInfer MoE-EP adapter. The packed MXFP4 weights go to the kernel as loaded, without a bf16 round trip.

• Stacked on [Perf][MoE] Enable SM90 block-FP8 MegaMoE via FlashInfer MoE-EP #62603 (SM90 block-FP8 MegaMoE through the same adapter). Only the last commit, db751d5, is new here.
• Draft until FlashInfer ships #4843 in a release. It is not in 0.7.2.post1, main's current pin.
• This answers the question in [Perf][MoE] Enable SM90 block-FP8 MegaMoE via FlashInfer MoE-EP #62603 (comment): it is an MXFP4-resident Hopper moe_ep entry, and it applies to an unmodified DeepSeek-V4.1-Flash checkpoint.

Related:

• [Kernel][Perf] Run DeepGEMM Mega MoE on SM90 with MXFP4 experts #64061: the same goal through DeepGEMM's SM90 FP8 x FP4 Mega MoE (Add SM90 FP8 x FP4 Mega MoE kernel DeepGEMM#44). It is measured head to head below. The two kernels trade wins: DeepGEMM is faster in decode and this PR is faster in prefill.
• [Kernel] Add a FlashInfer SM90 MXFP4 x FP8 fused MoE backend #54032: FlashInfer's CUTLASS Humming grouped-GEMM MoE for the same quantization. It is not a megakernel.

Claims

All results are DeepSeek-V4.1-Flash on 8x H200, compared against main's default MXFP4 backend (HUMMING with block-FP8 activations and allgather_reducescatter):

• Decode (DP8/EP8, 128K/512, DSpark3 with synthetic acceptance): +8% to +18% output throughput and -6% to -13% TPOT p50 at concurrency 32 to 192. No resolved change at concurrency 8.
• Prefill (DP4/EP4 at the full 1M context, 2048/1, natural DSpark3): +3.5% saturated throughput and lower TTFT p50 at every offered rate.
• [Kernel][Perf] Run DeepGEMM Mega MoE on SM90 with MXFP4 experts #64061, on the same workload: +12% to +21% in decode, but -8.7% in prefill.
• GSM8K shows no resolved difference among the three backends.
• The backend is opt-in. Default backends do not change.

Validation

Setup.

• vLLM main 77f58e5, on the native build of the 1cb31be nightly wheel. Between those two commits, csrc, cmake, setup and requirements are identical.
• All arms use FlashInfer nightly 0.7.2.dev20261027, torch 2.14.0+cu130, and DeepGEMM#44 at f4685f201e.
• Arms:
  • main: default backend.
  • main + [Kernel][Perf] Run DeepGEMM Mega MoE on SM90 with MXFP4 experts #64061 (335cff3): --moe-backend deep_gemm_mega_moe.
  • main + [Perf][MoE] Enable SM90 block-FP8 MegaMoE via FlashInfer MoE-EP #62603 + this PR: --moe-backend flashinfer_moe_ep_cutedsl.
  • The trees differ only in each PR's files. The FlashInfer arm's 12 files hash-match this branch.
• Order is serial on the same GPUs: main A1, [Kernel][Perf] Run DeepGEMM Mega MoE on SM90 with MXFP4 experts #64061, this PR, main A2. "main" is the mean of the A1 and A2 medians.
• Client: vllm-rs bench serve with token-ID prompts.

Decode. DP8/EP8, DecodeBenchConnector, ISL 128K / OSL 512, DSpark with 3 draft tokens and synthetic acceptance [0.8, 0.6, 0.5], N = 2C prompts. Each arm runs a warm-up pass, then three passes; cells are three-pass medians.

C	8	32	64	128	192
output tok/s, main	1338	3641	5806	8823	10130
output tok/s, #64061	1507 (+12.6%)	4336 (+19.1%)	6986 (+20.3%)	9879 (+12.0%)	12237 (+20.8%)
output tok/s, this PR	1371 (+2.4%)	3950 (+8.5%)	6425 (+10.7%)	9575 (+8.5%)	11981 (+18.3%)
TPOT p50 ms, main	5.00	7.03	8.35	9.58	12.05
TPOT p50 ms, #64061	4.37	5.50	6.58	8.35	9.04
TPOT p50 ms, this PR	4.96	6.50	7.61	9.02	10.48

The two main arms differ by -5.8% at C=8 and by at most 3.3% elsewhere. The +2.4% at C=8 is inside that spread.

Decode output tok/s, median [min, max] of the three passes per arm

C	main A1	#64061	this PR	main A2
8	1378 [1346, 1420]	1507 [1473, 1570]	1371 [1345, 1425]	1298 [1253, 1316]
32	3702 [3700, 3735]	4336 [4326, 4397]	3949 [3879, 4004]	3580 [3571, 3584]
64	5763 [5721, 5855]	6986 [6940, 6987]	6425 [6318, 6501]	5850 [5751, 5895]
128	8894 [8850, 8972]	9879 [9524, 9908]	9575 [9528, 9657]	8751 [7969, 8888]
192	10220 [10214, 10242]	12237 [12130, 12241]	11981 [11912, 12171]	10040 [9883, 10309]

Prefill.

• DP4/EP4 on 4x H200 at the default (full, 1M) context length, ISL 2048 / OSL 1, natural DSpark3.
• Saturation: 64 warm-up requests, then 3 x 1000 requests at an unlimited rate.
• Poisson sweep: 60 s of load at each rate, at 0.5 to 1.1 times main A1's saturation.

	main (A1 / A2)	#64061	this PR
saturated req/s, three runs	40.47 / 40.68 / 40.47; 40.21 / 40.54 / 40.30	36.44 / 36.89 / 36.90	41.80 / 41.84 / 41.81
saturated req/s vs main	40.39	36.89 (-8.7%)	41.81 (+3.5%)
TTFT p50 ms at 20.2 / 32.4 / 40.5 / 44.5 req/s	569 / 802 / 1166 / 4553	600 / 943 / 2635 / 7684	521 / 778 / 1072 / 3539
TTFT p99 ms at the same rates	1080 / 1201 / 2249 / 6037	1126 / 1336 / 6053 / 12693	1009 / 1189 / 1821 / 4334

At 44.5 req/s, main serves 40.5 req/s, #64061 serves 36.7 req/s and this PR serves 41.7 req/s.

GSM8K. 1319 questions, 5-shot, tests/evals/gsm8k/gsm8k_eval.py against the prefill servers with natural DSpark3.

main A1	main A2	#64061	this PR
89.8%	90.5%	90.4%, 89.9%	90.4%, 91.1%

One standard error is about 0.8 points, so no difference is resolved.

Tests. On H200 with FlashInfer 0.7.2.dev20261027:

pytest tests/models/test_deepseek_v4_fi_moe_ep.py   # 23 passed
pytest tests/kernels/moe/test_flashinfer_moe_ep.py   # CPU: 9 passed
pytest tests/kernels/moe/test_flashinfer_moe_ep.py -k sm90_init_warmup
# SM90 GPU: 16 passed (MXFP4 and block FP8, 4 expert shapes x EP2/EP4)

• The SM90 GPU test is exact on grid inputs. It covers the GLM-5.3-Flash, DeepSeek-V4.1-Flash, Kimi-K3 and Qwen3.8-Flash-Next expert shapes.
• For MXFP4 it runs with DeepSeek-V4 routing, which exercises the output view, the workspaces, an idle rank and a CUDA graph replay.
• CI does not run the GPU cases. They need at least two Hopper GPUs, nvshmem4py, and a FlashInfer build with #4843; without one they skip.

Commands

Every server runs with VLLM_USE_V2_MODEL_RUNNER=1. The main arm omits --moe-backend; the #64061 arm passes --moe-backend deep_gemm_mega_moe.

The loader is safetensors with prefetch for all arms. fastsafetensors OOMs here: it allocates a 94.6 GiB transient device buffer on top of 51.8 GiB of parameters.

# Decode
vllm serve <DeepSeek-V4.1-Flash> --tensor-parallel-size 1 --data-parallel-size 8 --enable-expert-parallel \
  --max-num-seqs 64 --gpu-memory-utilization 0.9 --no-enable-prefix-caching --limit-mm-per-prompt '{"image":0}' \
  --load-format safetensors --safetensors-load-strategy prefetch \
  --kv-transfer-config '{"kv_connector":"DecodeBenchConnector","kv_role":"kv_both"}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_rates":[0.8,0.6,0.5]}' \
  --moe-backend flashinfer_moe_ep_cutedsl

vllm-rs bench serve --dataset-name random --random-input-len 131072 --random-output-len 512 \
  --prompt-token-ids --ignore-eos --request-rate inf --max-concurrency $C --num-prompts $((2 * C)) --seed 0

# Prefill
vllm serve <DeepSeek-V4.1-Flash> --tensor-parallel-size 1 --data-parallel-size 4 --enable-expert-parallel \
  --no-enable-prefix-caching --limit-mm-per-prompt '{"image":0}' \
  --load-format safetensors --safetensors-load-strategy prefetch \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}' \
  --moe-backend flashinfer_moe_ep_cutedsl

# warm-up: --num-prompts 64 --seed 100; saturation: --seed 102, 104, 106
vllm-rs bench serve --dataset-name random --random-input-len 2048 --random-output-len 1 \
  --prompt-token-ids --ignore-eos --request-rate inf --num-prompts 1000 --seed 102
# Poisson: R in 20.2 28.3 32.4 36.4 40.5 44.5, N = round(60 * R), seeds 200 to 205
vllm-rs bench serve --dataset-name random --random-input-len 2048 --random-output-len 1 \
  --prompt-token-ids --ignore-eos --request-rate $R --num-prompts $N --seed $SEED

# GSM8K, on each prefill server
python tests/evals/gsm8k/gsm8k_eval.py --port $PORT

Details

• Weights.
  • The MXFP4 oracle keeps the checkpoint's packed E2M1 bytes and E8M0 scales on Hopper. The adapter hands them to FlashInfer as PrequantizedMoEWeights.
  • FlashInfer interleaves gate/up rows and folds the scales into its kernel layout, an 11-exponent window per expert; it does not requantize.
  • On Blackwell nothing changes: the CuTeDSL kernel still gets bf16 and requantizes to NVFP4.
• Activations.
  • The kernel quantizes activations to E4M3 with one scale per token, and its fc2 input with one scale per 64 channels.
  • main's default uses block-FP8 (1x128) activations, so the arms are not bit-identical. GSM8K shows no resolved difference.
• Adapter.
  • FlashInferMoeEp takes the Hopper weight format ("fp8_block" or "mxfp4") instead of a block-FP8 flag.
  • The MXFP4 path reuses [Perf][MoE] Enable SM90 block-FP8 MegaMoE via FlashInfer MoE-EP #62603's 64/256/1024-token workspaces. With DeepSeek-V4 routing they now combine with FlashInfer's output view.
  • The kernel requires int64 expert ids, so the adapter converts top-k ids to the kernel's index dtype. This is one device-side cast per layer; nothing new reads back to the host.
• Limitations.
  • EPLB is rejected for MXFP4 on Hopper, because FlashInfer keeps a per-expert FP32 scale of these weights outside the layer's parameters.
  • Input-side router weights are rejected, as for block FP8: the kernel weights the activation before fc2.
  • The DeepGEMM MoE-EP backend stays Blackwell-only.
  • With a FlashInfer that lacks #4843, MXFP4 on Hopper fails at weight processing with a message that names the FlashInfer PR.
• Versus [Kernel][Perf] Run DeepGEMM Mega MoE on SM90 with MXFP4 experts #64061. DeepGEMM's kernel is faster in decode at every tested concurrency; the gap is 2% to 10%, smallest at C=192. This kernel is faster in prefill (41.8 vs 36.9 saturated req/s). The two can coexist as backend choices, for example per prefill/decode role.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[kabishou11]

Thanks for stacking MXFP4 on the FlashInfer MoE-EP adapter (#62603 / flashinfer#4843).

From an offline DeepInfer Hopper fork (kabishou11/deepinfer-hopper) we are watching this as the FlashInfer SM90 MXFP4 MegaMoE path next to the DeepGEMM route in #64061 + vllm-project/DeepGEMM#44. No GPU on our side yet — docs only, no copied tok/s — so this is just a port-note interest, not a serving claim. Happy to re-test once FlashInfer ships #4843 in a release (same pin gap you called out vs 0.7.2.post1).

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @qianlihuang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork