Repository navigation
Conversation
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Andy Lo <andy@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Andy Lo <andy@mistral.ai>
andylolu2
force-pushed
the
andylolu2/zero-flashinfer-mla-kv-blocks
branch
from
September 27, 2026 15:58
f2ebde8 to
85262f0
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
FlashInfer MLA kernels can consume values from the unwritten suffix of a KV-cache block. When a recycled block still contains non-finite data from its previous owner, masking is insufficient (
0 * NaNremains NaN), which can corrupt attention outputs.vLLM already has a scheduler-to-worker path for zeroing newly allocated KV blocks, but uniform-precision attention models do not currently enable it. This change adds a generic attention-backend capability and opts in only FlashInfer MLA dense/sparse today.
Summary
AttentionBackend.requires_kv_cache_zeroingcapability and publish it on the baseAttentionSpec.Reviewer guide
vllm/v1/attention/backend.pyandvllm/v1/kv_cache_interface.pycontain the core logic: the generic backend opt-in, base attention-spec metadata, merge propagation, and zeroing decision.vllm/v1/core/kv_cache_utils.pypreserves the capability when aUniformTypeKVCacheSpecsgroup is collapsed for the scheduler.vllm/v1/attention/backends/composite.pyORs the requirement from either child backend.Duplicate check
This is not a duplicate of the closest related work:
AttentionSpec#51749 generalized the existing worker zeroer to allAttentionSpecsubclasses, but did not enable zeroing for uniform-precision MLA configurations.Searches performed:
Test plan
Model evaluation
Not run in this CPU-only development environment because FlashInfer and a CUDA accelerator are unavailable. GPU reproduction of the incident and an output-correctness check are required before this draft is marked ready for review.
AI assistance
AI assistance was used to investigate the relevant backend/cache-spec paths, implement the change, and draft this description. The submitter must review every changed line and validate the fix on the affected GPU configuration before merge.