Conversation
What / 问题描述: SM120 FlashInfer sparse MLA decode could not reliably serve GLM/Deepseek V3.2 FP8 KV cache paths with released FlashInfer wheels. Why / 根本原因: The SM120 FlashInfer decode API requires the packed fp8_ds_mla KV layout and kv_scale_format support, while the indexer still selected DeepGEMM paged MQA paths on CUDA platforms where that kernel is not supported. How / 修复内容: Canonicalize SM120 sparse MLA FP8 cache requests to fp8_ds_mla, validate the FlashInfer decode signature, pass packed KV metadata to FlashInfer, and add an FP8 torch indexer fallback when DeepGEMM paged MQA is unavailable. Details / 技术细节: The SM120 path now passes seq_lens and kv_scale_format into FlashInfer, keeps BF16 query inputs for the packed path, disables cudagraph support for the CUDA fallback indexer path, and updates CUDA requirements to flashinfer-python 0.6.14 with cu13 JIT cache instead of the unpublished flashinfer-cubin 0.6.14. Impact / 影响范围: This affects SM120 sparse MLA FP8 decode and indexer selection. FP4 indexer cache still requires DeepGEMM paged MQA support, and older FlashInfer builds without kv_scale_format support are rejected explicitly. Assisted-by: Codex Signed-off-by: wangxc <wangxc_a_bj@si-tech.com.cn>
ChamHerry
marked this pull request as ready for review
July 3, 2026 06:40
ChamHerry
requested review from
LucasWilkinson,
MatthewBonanni,
pavanimajety and
zyongye
as code owners
July 3, 2026 06:40
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
jasl
added a commit
to jasl/vllm
that referenced
this pull request
Jul 4, 2026
forward_mqa passed seq_lens=None to the trtllm-gen sparse decode, so the kernel walked all topk_tokens block-table entries and masked the -1 padding that appears whenever a request's context < topk_tokens. Every sibling sparse-MLA backend (flashinfer_mla_sparse, flashattn_mla_sparse, flashmla_sparse) passes the per-token valid count instead, and upstream PR vllm-project#47527 proposes the same for this SM120 path. triton_convert_req_index_to_global_index already computes that valid count in the same kernel pass at no extra cost; request it (return_valid_counts=True) and forward it as seq_lens so the kernel reads only each request's valid prefix. Bit-identical to seq_lens=None on the long-context path (context >= topk_tokens => valid_count == topk_tokens); at short context it skips the padding tail instead of masking it. Primary value is convergence with the sibling backends and vllm-project#47527; perf-neutral on GB10 (interconnect-bound decode, long-context benchmarks carry no padding). Validated on GB10 2-node TP=2 MTP2 (mml 524288): arthur 434k conc1 2/2 (long, no padding), GSM8K 5-shot/200 = 0.96 (short, padding present -> new prefix-skip path), serve-side error scan clean.
|
+1 |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
allenh1
pushed a commit
to allenh1/vllm
that referenced
this pull request
Aug 15, 2026
forward_mqa passed seq_lens=None to the trtllm-gen sparse decode, so the kernel walked all topk_tokens block-table entries and masked the -1 padding that appears whenever a request's context < topk_tokens. Every sibling sparse-MLA backend (flashinfer_mla_sparse, flashattn_mla_sparse, flashmla_sparse) passes the per-token valid count instead, and upstream PR vllm-project#47527 proposes the same for this SM120 path. triton_convert_req_index_to_global_index already computes that valid count in the same kernel pass at no extra cost; request it (return_valid_counts=True) and forward it as seq_lens so the kernel reads only each request's valid prefix. Bit-identical to seq_lens=None on the long-context path (context >= topk_tokens => valid_count == topk_tokens); at short context it skips the padding tail instead of masking it. Primary value is convergence with the sibling backends and vllm-project#47527; perf-neutral on GB10 (interconnect-bound decode, long-context benchmarks carry no padding). Validated on GB10 2-node TP=2 MTP2 (mml 524288): arthur 434k conc1 2/2 (long, no padding), GSM8K 5-shot/200 = 0.96 (short, padding present -> new prefix-skip path), serve-side error scan clean.
InitialLight01
pushed a commit
to InitialLight01/vllm
that referenced
this pull request
Aug 17, 2026
forward_mqa passed seq_lens=None to the trtllm-gen sparse decode, so the kernel walked all topk_tokens block-table entries and masked the -1 padding that appears whenever a request's context < topk_tokens. Every sibling sparse-MLA backend (flashinfer_mla_sparse, flashattn_mla_sparse, flashmla_sparse) passes the per-token valid count instead, and upstream PR vllm-project#47527 proposes the same for this SM120 path. triton_convert_req_index_to_global_index already computes that valid count in the same kernel pass at no extra cost; request it (return_valid_counts=True) and forward it as seq_lens so the kernel reads only each request's valid prefix. Bit-identical to seq_lens=None on the long-context path (context >= topk_tokens => valid_count == topk_tokens); at short context it skips the padding tail instead of masking it. Primary value is convergence with the sibling backends and vllm-project#47527; perf-neutral on GB10 (interconnect-bound decode, long-context benchmarks carry no padding). Validated on GB10 2-node TP=2 MTP2 (mml 524288): arthur 434k conc1 2/2 (long, no padding), GSM8K 5-shot/200 = 0.96 (short, padding present -> new prefix-skip path), serve-side error scan clean.
allenh1
pushed a commit
to allenh1/vllm
that referenced
this pull request
Aug 26, 2026
forward_mqa passed seq_lens=None to the trtllm-gen sparse decode, so the kernel walked all topk_tokens block-table entries and masked the -1 padding that appears whenever a request's context < topk_tokens. Every sibling sparse-MLA backend (flashinfer_mla_sparse, flashattn_mla_sparse, flashmla_sparse) passes the per-token valid count instead, and upstream PR vllm-project#47527 proposes the same for this SM120 path. triton_convert_req_index_to_global_index already computes that valid count in the same kernel pass at no extra cost; request it (return_valid_counts=True) and forward it as seq_lens so the kernel reads only each request's valid prefix. Bit-identical to seq_lens=None on the long-context path (context >= topk_tokens => valid_count == topk_tokens); at short context it skips the padding tail instead of masking it. Primary value is convergence with the sibling backends and vllm-project#47527; perf-neutral on GB10 (interconnect-bound decode, long-context benchmarks carry no padding). Validated on GB10 2-node TP=2 MTP2 (mml 524288): arthur 434k conc1 2/2 (long, no padding), GSM8K 5-shot/200 = 0.96 (short, padding present -> new prefix-skip path), serve-side error scan clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR fixes the released-wheel SM120 FlashInfer sparse MLA path used by GLM / Deepseek V3.2 style sparse MLA models with FP8 KV cache.
The current path can select the SM120 FlashInfer sparse MLA backend, but the decode API needs the packed
fp8_ds_mlaKV layout andkv_scale_formatsupport. Separately, the sparse attention indexer only checked for the presence of DeepGEMM, so CUDA platforms where DeepGEMM paged MQA logits are not supported could still enter the wrong path or try to capture the torch fallback in CUDA graphs.Changes
flashinfer-python[cu13]==0.6.14and document that CUDA 13 images should useflashinfer-jit-cachebecauseflashinfer-cubin 0.6.14is not published.fp8_ds_mla.kv_scale_formatand reject older FlashInfer builds explicitly.seq_lens,bmm1_scale,bmm2_scale,sparse_mla_top_k, andkv_scale_formatto FlashInfer decode.DeepseekV32IndexerBackendwhen the CUDA fallback path is active.Duplicate-work check
I checked for related open PRs before opening this draft:
Nearest related PRs found:
fp8_ds_mlaFlashInfer decode arguments, FlashInfer 0.6.14 dependency, or DeepGEMM paged MQA indexer fallback.I did not find an open PR that addresses this specific FlashInfer 0.6.14 packed sparse MLA decode and SM120 indexer fallback combination.
Test Plan
Local formatting/static checks:
Remote SM120 Docker tests in an image rebuilt from
vllm/vllm-openai:latest-0701with this branch overlaid and FlashInfer dependencies updated:Remote image dependency validation:
Remote GLM-5.2-NVFP4 service smoke test:
Validated endpoints:
Test Result
Passed:
The same model with its default
max_model_len=1048576did not fit in available KV cache on the test machine. Reducing--max-model-lento131072gaveGPU KV cache size: 319,424 tokensandMaximum concurrency for 131,072 tokens per request: 2.44x, after which the service-level smoke test passed.AI Assistance Disclosure
AI assistance was used to investigate the runtime failures, implement this patch, run remote validation, and draft this PR description. This PR is opened as a draft; a human submitter should review every changed line and be prepared to defend the change end-to-end before marking it ready for review.