Bump Flashinfer Version and Re-enable DeepSeek NVFP4 AR+Norm Fusion by wzhao18 · Pull Request #34899 · vllm-project/vllm

wzhao18 · 2026-02-19T17:07:38Z

Purpose

The patch for fixing the Deepseek V3 accuracy issue with AR+rms+fp4 fusion (#34395) is included in flashinfer 0.6.4. This PR bumps flashinfer version and re-enables the fusion pass by default.

Test Plan

serve nvidia/DeepSeek-V3.1-NVFP4 -tp=4 -cc.pass_config.fuse_allreduce_rms=True

Test Result

lm_eval --model local-completions --model_args "base_url=http://0.0.0.0:8000/v1/completions,max_length=8192,tokenized_requests=False,tokenizer_backend=None,num_concurrent=32" --tasks gsm8k --num_fewshot 5

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9538|±  |0.0058|
|     |       |strict-match    |     5|exact_match|↑  |0.9492|±  |0.0060|

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

gemini-code-assist

Code Review

This pull request correctly bumps the flashinfer version to 0.6.4 across several configuration and dependency files. This update incorporates a fix for an accuracy issue with AR+rms+fp4 fusion in Deepseek V3 models. As a result, the workaround that previously disabled this fusion pass has been removed from vllm/model_executor/models/config.py. The changes are consistent, well-justified, and should improve performance by re-enabling the fusion pass. The provided test results support the correctness of this change.

ProExpertProg

Thanks for the quick turnaround, very clean. cc @mgoin any blockers for upgrading the FI version?

wzhao18 · 2026-02-19T17:14:01Z

@ProExpertProg Can we run full CI for this PR? Not sure by default it performs a wide enough sweep of tests.

mgoin · 2026-02-19T17:14:16Z

Yeah this should be good! I will like to enable a lot of extra testing before merge to make sure we catch the moe tests

ProExpertProg · 2026-02-19T17:14:48Z

Okay enabled all CI tests!

wzhao18 · 2026-02-19T17:47:23Z

CI fails because flashinfer-jit-cache 0.6.4 is not yet available on https://flashinfer.ai/whl/cu129/flashinfer-jit-cache/. Will check about this.

EDIT: seems there is an issue in flashinfer's release build: https://github.com/flashinfer-ai/flashinfer/actions/runs/22168423112

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>

wzhao18 · 2026-02-20T15:22:54Z

Took a look at the failed tests. Some tests aborted without running. Only one looking suspicious is:

FAILED evals/gsm8k/test_gsm8k_correctness.py::test_gsm8k_correctness[DeepSeek-V2-Lite-Instruct-FP8] - AssertionError: GSM8K metric too low: 0.0114 < 0.7200 - 0.0800 = 0.6400
--
assert np.float64(0.011372251705837756) >= (0.72 - 0.08)
==================================================== 1 failed, 6 passed, 17 warnings in 1504.50s (0:25:04) =====================================================

But I ran again locally and it is passing.

vadiklyutiy · 2026-02-20T16:03:22Z

May we rerun CI?

mgoin · 2026-02-20T18:39:36Z

@vadiklyutiy don't worry about re-running, I've validated the failures are unrelated. I just want to wait for a few other PRs to merge first so main CI has good signal

dosubot · 2026-02-20T21:37:42Z

Related Documentation

Checked 0 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: joezuo <qianzhou.zuo@gmail.com>

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

This test aims to cover: vllm-project#34899 Not sure where this test would be put in the tests folder, open to suggestions from reviewers. Signed-off-by: Rishi Puri <riship@nvidia.com>

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Add comprehensive performance analysis for MiniMax-M2.5-REAP-139B-A10B-NVFP4-GB10: Architecture confirmed: - Attention IS NVFP4 in this model (ignore list = only lm_head + MoE gates) - 3 MTP modules present (layers 62-64) — biggest performance lever available - Per-step weight load: ~6.15 GB → 36–44 tok/s theoretical ceiling on GB10 Performance gap analysis: - Current: 24 tok/s on Strix Halo (AMD); GB10 expected similar baseline - vLLM is 1.78x slower than SGLang at BS=1 for NVFP4 MoE (documented gap) - Gap sources: activation quant overhead, kernel launch overhead, no fused shuffle+reduce in MoE, generic CUTLASS configs Key new PRs to integrate: - vllm-project#35041 (OPEN): MTP+NVFP4 weight shape mismatch — required for MTP+NVFP4 - vllm-project#35442 (OPEN): Non-blocking MTP token copy — 6ms→200µs CPU-GPU sync - vllm-project#33303 (OPEN): MiniMax PP+DP for multi-Spark scaling Already-merged PRs confirmed in HEAD: - vllm-project#34718 (act_quant_fusion.py): SiLU+FP4 fusion - vllm-project#34899 (allreduce_rms_fusion.py): NVFP4 AR+Norm fusion - vllm-project#30885: 8x4 SF tiling (not yet effective on GB10 — TRTLLM backend blocked)

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Add comprehensive performance analysis for MiniMax-M2.5-REAP-139B-A10B-NVFP4-GB10: Architecture confirmed: - Attention IS NVFP4 in this model (ignore list = only lm_head + MoE gates) - 3 MTP modules present (layers 62-64) — biggest performance lever available - Per-step weight load: ~6.15 GB → 36–44 tok/s theoretical ceiling on GB10 Performance gap analysis: - Current: 24 tok/s on Strix Halo (AMD); GB10 expected similar baseline - vLLM is 1.78x slower than SGLang at BS=1 for NVFP4 MoE (documented gap) - Gap sources: activation quant overhead, kernel launch overhead, no fused shuffle+reduce in MoE, generic CUTLASS configs Key new PRs to integrate: - vllm-project#35041 (OPEN): MTP+NVFP4 weight shape mismatch — required for MTP+NVFP4 - vllm-project#35442 (OPEN): Non-blocking MTP token copy — 6ms→200µs CPU-GPU sync - vllm-project#33303 (OPEN): MiniMax PP+DP for multi-Spark scaling Already-merged PRs confirmed in HEAD: - vllm-project#34718 (act_quant_fusion.py): SiLU+FP4 fusion - vllm-project#34899 (allreduce_rms_fusion.py): NVFP4 AR+Norm fusion - vllm-project#30885: 8x4 SF tiling (not yet effective on GB10 — TRTLLM backend blocked)

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: Andrii Skliar <askliar@nvidia.com>

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

mergify bot added ci/build deepseek Related to DeepSeek models nvidia labels Feb 19, 2026

github-project-automation bot added this to NVIDIA Feb 19, 2026

gemini-code-assist bot reviewed Feb 19, 2026

View reviewed changes

ProExpertProg approved these changes Feb 19, 2026

View reviewed changes

github-project-automation bot moved this to Ready in NVIDIA Feb 19, 2026

ProExpertProg added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 19, 2026

ProExpertProg added the ready-run-all-tests Trigger CI with all tests for wide-ranging PRs label Feb 19, 2026

wzhao18 force-pushed the wzhao/bump-flashinfer-version branch from 9abeaa8 to e41b6a1 Compare February 19, 2026 17:19

mgoin self-requested a review February 19, 2026 18:25

wzhao18 force-pushed the wzhao/bump-flashinfer-version branch from e41b6a1 to 6afd78b Compare February 19, 2026 21:23

wzhao18 added 2 commits February 19, 2026 20:23

Bump flashinfer version

5a079a6

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>

Re-enable AR+Norm fusion for DS NVFP4

42b4213

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>

wzhao18 force-pushed the wzhao/bump-flashinfer-version branch from 6afd78b to 42b4213 Compare February 20, 2026 01:23

Merge branch 'main' into wzhao/bump-flashinfer-version

917127a

vadiklyutiy mentioned this pull request Feb 20, 2026

[Build] Bump FlashInfer version from v0.6.3 to v0.6.4 #34962

Closed

vllm-bot merged commit ea5f903 into vllm-project:main Feb 20, 2026
143 of 151 checks passed

github-project-automation bot moved this from Ready to Done in NVIDIA Feb 20, 2026

jmamou pushed a commit to jmamou/vllm that referenced this pull request Feb 23, 2026

Bump Flashinfer Version and Re-enable DeepSeek NVFP4 AR+Norm Fusion (v…

32c0270

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

puririshi98 mentioned this pull request Feb 25, 2026

Create test_deepseek_v32_config.py #35247

Closed

ProExpertProg mentioned this pull request Feb 26, 2026

[Bug]: AR+rms+fp4 fusion results in total accuracy collapse for DSV3-fp4 #34395

Closed

1 task

ProExpertProg linked an issue Feb 26, 2026 that may be closed by this pull request

[Bug]: AR+rms+fp4 fusion results in total accuracy collapse for DSV3-fp4 #34395

Closed

1 task

llsj14 pushed a commit to llsj14/vllm that referenced this pull request Mar 1, 2026

Bump Flashinfer Version and Re-enable DeepSeek NVFP4 AR+Norm Fusion (v…

d780e9b

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Copilot AI pushed a commit to machov/vllm that referenced this pull request Mar 10, 2026

Bump Flashinfer Version and Re-enable DeepSeek NVFP4 AR+Norm Fusion (v…

987ebb2

…llm-project#34899) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bump Flashinfer Version and Re-enable DeepSeek NVFP4 AR+Norm Fusion#34899

Bump Flashinfer Version and Re-enable DeepSeek NVFP4 AR+Norm Fusion#34899
vllm-bot merged 3 commits intovllm-project:mainfrom
wzhao18:wzhao/bump-flashinfer-version

wzhao18 commented Feb 19, 2026 •

edited by github-actions bot

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

ProExpertProg left a comment

Uh oh!

wzhao18 commented Feb 19, 2026

Uh oh!

mgoin commented Feb 19, 2026

Uh oh!

ProExpertProg commented Feb 19, 2026

Uh oh!

wzhao18 commented Feb 19, 2026 •

edited

Loading

Uh oh!

wzhao18 commented Feb 20, 2026

Uh oh!

vadiklyutiy commented Feb 20, 2026

Uh oh!

mgoin commented Feb 20, 2026

Uh oh!

Uh oh!

dosubot bot commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

wzhao18 commented Feb 19, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

ProExpertProg left a comment

Choose a reason for hiding this comment

Uh oh!

wzhao18 commented Feb 19, 2026

Uh oh!

mgoin commented Feb 19, 2026

Uh oh!

ProExpertProg commented Feb 19, 2026

Uh oh!

wzhao18 commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wzhao18 commented Feb 20, 2026

Uh oh!

vadiklyutiy commented Feb 20, 2026

Uh oh!

mgoin commented Feb 20, 2026

Uh oh!

Uh oh!

dosubot bot commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wzhao18 commented Feb 19, 2026 •

edited by github-actions bot

Loading

wzhao18 commented Feb 19, 2026 •

edited

Loading