Skip to content

[Test] Add FP8 KV Cache Testing for MLA Backends#34473

Merged
LucasWilkinson merged 5 commits intovllm-project:mainfrom
wzhao18:wzhao/flashinfer-mla-fp8-tests
Feb 20, 2026
Merged

[Test] Add FP8 KV Cache Testing for MLA Backends#34473
LucasWilkinson merged 5 commits intovllm-project:mainfrom
wzhao18:wzhao/flashinfer-mla-fp8-tests

Conversation

@wzhao18
Copy link
Contributor

@wzhao18 wzhao18 commented Feb 12, 2026 •
edited by github-actions bot
Loading

Purpose

This PR improves the MLA backend test coverage to include fp8 kv cache testing.

Test Plan

Tested tests/v1/attention/

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@mergify mergify bot added the v1 label Feb 12, 2026
gemini-code-assist[bot]
gemini-code-assist bot reviewed Feb 12, 2026
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the MLA backend test coverage by adding support for FP8 KV cache testing. The changes correctly parameterize the tests for different kv_cache_dtype values and generalize the test logic to handle various FP8 formats. My review found one critical issue where a safety check was removed, which could lead to a runtime error. I've provided a suggestion to fix it.

@wzhao18 wzhao18 force-pushed the wzhao/flashinfer-mla-fp8-tests branch from fc4170b to aafb941 Compare February 13, 2026 04:14
@wzhao18
Copy link
Contributor Author

wzhao18 commented Feb 13, 2026

@pavanimajety Could you help review this PR?

MatthewBonanni
MatthewBonanni approved these changes Feb 17, 2026
View reviewed changes
Copy link
Collaborator

@MatthewBonanni MatthewBonanni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the contribution!

mgoin
mgoin approved these changes Feb 17, 2026
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@mgoin mgoin added ready ONLY add when PR is ready to merge/full CI is needed ci/build labels Feb 17, 2026
pavanimajety
pavanimajety approved these changes Feb 17, 2026
View reviewed changes
Copy link
Collaborator

@pavanimajety pavanimajety left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution @wzhao18, LGTM!

For a future self note: We need to add tests for chunked prefill when it uses FP8 KV Cache + MHA kernels

@wzhao18
Copy link
Contributor Author

wzhao18 commented Feb 17, 2026

@pavanimajety Please feel free to pin me and Xin for assistance on improving the test coverage.

@wzhao18 wzhao18 force-pushed the wzhao/flashinfer-mla-fp8-tests branch from 2cd12a0 to 29aecb2 Compare February 18, 2026 17:39
@github-project-automation github-project-automation bot moved this to Ready in NVIDIA Feb 18, 2026
@mergify mergify bot added the cpu Related to CPU backends label Feb 18, 2026
@mergify mergify bot added structured-output tpu Related to Google TPUs labels Feb 18, 2026
@github-project-automation github-project-automation bot moved this from To Triage to Ready in gpt-oss Issues & Enhancements Feb 18, 2026
@wzhao18 wzhao18 marked this pull request as draft February 18, 2026 17:40
@mergify mergify bot added the kv-connector label Feb 18, 2026
@mergify
Copy link

mergify bot commented Feb 18, 2026

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wzhao18.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Feb 18, 2026
@wzhao18 wzhao18 force-pushed the wzhao/flashinfer-mla-fp8-tests branch from 29aecb2 to e456a25 Compare February 18, 2026 17:46
@mergify mergify bot removed the tpu Related to Google TPUs label Feb 18, 2026
@wzhao18 wzhao18 force-pushed the wzhao/flashinfer-mla-fp8-tests branch from e456a25 to 50c031d Compare February 18, 2026 17:46
@wzhao18
Test Fp8 flashinfer MLA backend
5800947 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
@wzhao18
Fix comments
8ef6a0b 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
@wzhao18
Fix format
4f1ac6b 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
@wzhao18
fix hopper test on flashMLA
1622bad 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
@wzhao18 wzhao18 force-pushed the wzhao/flashinfer-mla-fp8-tests branch from 50c031d to 1622bad Compare February 18, 2026 17:47
@wzhao18 wzhao18 marked this pull request as ready for review February 18, 2026 17:48
@mergify mergify bot removed the needs-rebase label Feb 18, 2026
yewentao256
yewentao256 approved these changes Feb 18, 2026
View reviewed changes
Copy link
Member

@yewentao256 yewentao256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the work!

@wzhao18
Copy link
Contributor Author

wzhao18 commented Feb 18, 2026

@mgoin Relevant CI tests are passing. I believe the failing ones also fail on main.

yewentao256
yewentao256 reviewed Feb 19, 2026
View reviewed changes
Copy link
Member

@yewentao256 yewentao256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the issue could be fixed by #34913

LucasWilkinson
LucasWilkinson approved these changes Feb 19, 2026
View reviewed changes
Copy link
Collaborator

@LucasWilkinson LucasWilkinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@LucasWilkinson LucasWilkinson enabled auto-merge (squash) February 19, 2026 21:51
@LucasWilkinson LucasWilkinson merged commit f24b2de into vllm-project:main Feb 20, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgoin mgoin mgoin approved these changes

@LucasWilkinson LucasWilkinson LucasWilkinson approved these changes

@MatthewBonanni MatthewBonanni MatthewBonanni approved these changes

@pavanimajety pavanimajety pavanimajety approved these changes

@yewentao256 yewentao256 yewentao256 approved these changes

@noooop noooop Awaiting requested review from noooop

@tjtanaa tjtanaa Awaiting requested review from tjtanaa

@patrickvonplaten patrickvonplaten Awaiting requested review from patrickvonplaten

@sighingnow sighingnow Awaiting requested review from sighingnow

@bigPYJ1151 bigPYJ1151 Awaiting requested review from bigPYJ1151

@hmellor hmellor Awaiting requested review from hmellor

@markmc markmc Awaiting requested review from markmc

@ApostaC ApostaC Awaiting requested review from ApostaC

@orozery orozery Awaiting requested review from orozery

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat

@njhill njhill Awaiting requested review from njhill

@ywang96 ywang96 Awaiting requested review from ywang96

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat

@heheda12345 heheda12345 Awaiting requested review from heheda12345

@russellb russellb Awaiting requested review from russellb

@aarnphm aarnphm Awaiting requested review from aarnphm

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth

@NickLucche NickLucche Awaiting requested review from NickLucche

@22quinn 22quinn Awaiting requested review from 22quinn

@houseroad houseroad Awaiting requested review from houseroad

@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123

@youkaichao youkaichao Awaiting requested review from youkaichao

@chaunceyjiang chaunceyjiang Awaiting requested review from chaunceyjiang

@ProExpertProg ProExpertProg Awaiting requested review from ProExpertProg

@tdoublep tdoublep Awaiting requested review from tdoublep

@jeejeelee jeejeelee Awaiting requested review from jeejeelee

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build cpu Related to CPU backends documentation Improvements or additions to documentation frontend gpt-oss Related to GPT-OSS models kv-connector llama Related to Llama models multi-modality Related to multi-modality (#4194) new-model Requests to new models nvidia performance Performance-related issues qwen Related to Qwen models ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm structured-output v1

Projects

AMD
Status: Done
NVIDIA
Status: Done
Structured Output
Status: Done
gpt-oss Issues & Enhancements
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@wzhao18 @mgoin @LucasWilkinson @MatthewBonanni @pavanimajety @yewentao256