Skip to content

[Bugfix] fix qwen3-omni performance regression#3575

Merged
hsliuustc0106 merged 4 commits into
vllm-project:mainfrom
R2-Y:perf_regression_bugfix
Jun 2, 2026
Merged

[Bugfix] fix qwen3-omni performance regression#3575
hsliuustc0106 merged 4 commits into
vllm-project:mainfrom
R2-Y:perf_regression_bugfix

Conversation

@R2-Y

@R2-Y R2-Y commented May 13, 2026

Copy link
Copy Markdown
Contributor

To avoid performance degradation when using FlashInfer CUTLASS Unquantized MoE backend, Qwen3-Omni sets VLLM_USE_FLASHINFER_MOE_FP16=0 by default (Using TRITON Unquantized MoE backend).

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

to solve #3556
before: start a non async chunk Qwen3-omni server with FlashInfer CUTLASS Unquantized MoE backend
2dc3a03c755aa2a8814b1a2c33872ac0

vllm bench serve --omni \
  --host 127.0.0.1 \
  --port 8666 \
  --backend openai-chat-omni \
  --endpoint /v1/chat/completions \
  --dataset-name random \
  --num-prompts 4 \
  --max-concurrency 1 \
  --request-rate inf \
  --random-input-len 2500 \
  --random-output-len 900 \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration
image

now: start a non async chunk Qwen3-omni server with Triton Unquantized MoE backend

eaa4a1a29dbddd54ea6f8eba1d6c314a image

TTFP change from 46525.73 to 29900.31 ms on same bench test.

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
  • The test results. Please paste the results comparison before and after, or the e2e results.
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
  • (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)
@ZeldaHuang @amy-why-3459 @tzhouam @Gaohan123 @hsliuustc0106

@gcanlin

gcanlin commented May 13, 2026

Copy link
Copy Markdown
Collaborator

cc @Isotr0py @ZJY0516

@amy-why-3459

amy-why-3459 commented May 13, 2026

Copy link
Copy Markdown
Contributor

VLLM_USE_FLASHINFER_MOE_FP16 seems to default to 0.

# Allow use of FlashInfer BF16 MoE kernels for fused moe ops.
    "VLLM_USE_FLASHINFER_MOE_FP16": lambda: bool(
        int(os.getenv("VLLM_USE_FLASHINFER_MOE_FP16", "0"))
    ),

However, it will check whether the VLLM_USE_FLASHINFER_MOE_FP16 environment variable is set.

# Handle explicit FlashInfer FP16 configuration.
    if envs.is_set("VLLM_USE_FLASHINFER_MOE_FP16"):
        if not envs.VLLM_USE_FLASHINFER_MOE_FP16:
            if UnquantizedMoeBackend.FLASHINFER_TRTLLM in AVAILABLE_BACKENDS:
                AVAILABLE_BACKENDS.remove(UnquantizedMoeBackend.FLASHINFER_TRTLLM)
            if UnquantizedMoeBackend.FLASHINFER_CUTLASS in AVAILABLE_BACKENDS:
                AVAILABLE_BACKENDS.remove(UnquantizedMoeBackend.FLASHINFER_CUTLASS)

@Gaohan123 Gaohan123 added this to the v0.22.0 milestone May 14, 2026
@gcanlin

gcanlin commented May 14, 2026

Copy link
Copy Markdown
Collaborator

Would it happen regression for thinker-only when VLLM_USE_FLASHINFER_MOE_FP16=0?

@R2-Y

R2-Y commented May 14, 2026

Copy link
Copy Markdown
Contributor Author

Would it happen regression for thinker-only when VLLM_USE_FLASHINFER_MOE_FP16=0?

regression for both thinker & talker when VLLM_USE_FLASHINFER_MOE_FP16=1,
VLLM_USE_FLASHINFER_MOE_FP16 = 0 -> triton
VLLM_USE_FLASHINFER_MOE_FP16 = 1 -> flashinfer

@amy-why-3459

Copy link
Copy Markdown
Contributor

Would it happen regression for thinker-only when VLLM_USE_FLASHINFER_MOE_FP16=0?

regression for both thinker & talker when VLLM_USE_FLASHINFER_MOE_FP16=1, VLLM_USE_FLASHINFER_MOE_FP16 = 0 -> triton VLLM_USE_FLASHINFER_MOE_FP16 = 1 -> flashinfer

Even if the environment variable VLLM_USE_FLASHINFER_MOE_FP16 is not set, the flash infer backend will still be used.

@ZJY0516 ZJY0516 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a todo in comment to remove this later. It's a vllm moe performance regression

R2-Y added 2 commits May 14, 2026 09:38
…e regression

Signed-off-by: rein yang <ruiruyang2@gmail.com>
Signed-off-by: rein yang <ruiruyang2@gmail.com>
@R2-Y R2-Y force-pushed the perf_regression_bugfix branch from b0f3589 to 3d27e2b Compare May 14, 2026 09:41
@R2-Y

R2-Y commented May 14, 2026

Copy link
Copy Markdown
Contributor Author

Please add a todo in comment to remove this later. It's a vllm moe performance regression

DONE

@gcanlin gcanlin added the ready label to trigger buildkite CI label May 14, 2026
@hsliuustc0106 hsliuustc0106 changed the title [Bugfix] fix qwen3-omni performance regression [do not merge][Bugfix] fix qwen3-omni performance regression May 14, 2026
@R2-Y R2-Y closed this May 18, 2026
@R2-Y R2-Y reopened this May 30, 2026
@R2-Y R2-Y changed the title [do not merge][Bugfix] fix qwen3-omni performance regression [Bugfix] fix qwen3-omni performance regression May 30, 2026
@R2-Y

R2-Y commented May 30, 2026

Copy link
Copy Markdown
Contributor Author

reopen because regression issues still exist in the main branch, the environment setting is only introduced in CI.

@hsliuustc0106 hsliuustc0106 merged commit 3a7c7f1 into vllm-project:main Jun 2, 2026
7 of 8 checks passed
@R2-Y R2-Y deleted the perf_regression_bugfix branch June 2, 2026 15:08
86MaxCao pushed a commit to 86MaxCao/vllm-omni that referenced this pull request Jun 4, 2026
Signed-off-by: rein yang <ruiruyang2@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
akshatvishu pushed a commit to akshatvishu/vllm-omni that referenced this pull request Jun 13, 2026
Signed-off-by: rein yang <ruiruyang2@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants