Skip to content

[Bugfix][Attention] Zero reused FlashInfer MLA KV blocks - #58716

Draft
andylolu2 wants to merge 2 commits into
vllm-project:mainfrom
andylolu2:andylolu2/zero-flashinfer-mla-kv-blocks
Draft

andylolu2 wants to merge 2 commits into
vllm-project:mainfrom
andylolu2:andylolu2/zero-flashinfer-mla-kv-blocks

Conversation

@andylolu2

@andylolu2 andylolu2 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Why

FlashInfer MLA kernels can consume values from the unwritten suffix of a KV-cache block. When a recycled block still contains non-finite data from its previous owner, masking is insufficient (0 * NaN remains NaN), which can corrupt attention outputs.

vLLM already has a scheduler-to-worker path for zeroing newly allocated KV blocks, but uniform-precision attention models do not currently enable it. This change adds a generic attention-backend capability and opts in only FlashInfer MLA dense/sparse today.

Summary

  • Add a generic AttentionBackend.requires_kv_cache_zeroing capability and publish it on the base AttentionSpec.
  • Preserve the capability across attention-spec merges, uniform-type scheduler projection, and composite backends.
  • Enable it for FlashInfer MLA dense and all registered FlashInfer MLA sparse variants, including DeepSeek V4/V4.1 compressed and sliding-window caches.
  • Leave FlashMLA dense/sparse and all other attention backends unchanged.

Reviewer guide

  • vllm/v1/attention/backend.py and vllm/v1/kv_cache_interface.py contain the core logic: the generic backend opt-in, base attention-spec metadata, merge propagation, and zeroing decision.
  • vllm/v1/core/kv_cache_utils.py preserves the capability when a UniformTypeKVCacheSpecs group is collapsed for the scheduler.
  • vllm/v1/attention/backends/composite.py ORs the requirement from either child backend.
  • The FlashInfer backend files only opt the relevant dense/sparse backend classes into the generic capability.
  • The tests cover backend scoping and propagation through base/MLA, sliding-window, uniform-type, and composite paths.
  • No kernel or block-zeroer implementation changes are needed; this uses the existing newly-allocated-block zeroing pipeline.

Duplicate check

This is not a duplicate of the closest related work:

Searches performed:

gh pr list --repo vllm-project/vllm --state open --search "FlashInfer MLA KV cache zeroing"
gh pr list --repo vllm-project/vllm --state open --search "KV cache corruption FlashInfer"
gh pr list --repo vllm-project/vllm --state open --search "flashinfer mla zero OR stale OR reused"

Test plan

.venv/bin/python -m pytest -q tests/v1/core/test_kv_cache_utils.py -k attention_kv_cache_zeroing
# 16 passed

.venv/bin/python -m pytest -q tests/v1/attention/test_mm_prefix.py -k composite_backend_preserves_kv_cache_zeroing_requirement
# 1 passed

.venv/bin/python -m pytest -q tests/v1/attention/test_mla_backends.py -k only_flashinfer_mla_backends_require_kv_cache_zeroing
# 1 skipped locally because FlashInfer is not installed

# Directly imported and exercised customize_spec for all seven sparse FlashInfer MLA backend classes.
# verified sparse customize_spec paths

.venv/bin/pre-commit run --files <changed files>
# all applicable hooks passed, including Ruff and mypy

Model evaluation

Not run in this CPU-only development environment because FlashInfer and a CUDA accelerator are unavailable. GPU reproduction of the incident and an output-correctness check are required before this draft is marked ready for review.

AI assistance

AI assistance was used to investigate the relevant backend/cache-spec paths, implement the change, and draft this description. The submitter must review every changed line and validate the fix on the affected GPU configuration before merge.

@mergify mergify Bot added deepseek Related to DeepSeek models DSv4 DSv4.1 Related to DeepSeek-V4.1 models nvidia bug Something isn't working labels Sep 25, 2026
andylolu2 and others added 2 commits September 27, 2026 15:48
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Andy Lo <andy@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Andy Lo <andy@mistral.ai>
@andylolu2
andylolu2 force-pushed the andylolu2/zero-flashinfer-mla-kv-blocks branch from f2ebde8 to 85262f0 Compare September 27, 2026 15:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working deepseek Related to DeepSeek models DSv4 DSv4.1 Related to DeepSeek-V4.1 models kv-cache-manager nvidia

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant