Repository navigation
Conversation
The SM90 sparse decode kernel bounded index reads by topk_length only for
the DeepSeek-V4 layout and asserted topk_length == nullptr for V3.2. SM100
already honors it for V3.2.
With topk_length, the V3.2 kernel now takes each request's block count
from it (matching get_decoding_sched_meta) and treats index positions at
or past it as invalid. The checks are compiled into two new
v32_persistent_h{64,128}_topklen instantiations only, so the existing V3.2
kernels are unchanged.
The sparse decoding test now also generates V3.2 cases with topk_length.
Assisted-by: Claude
Signed-off-by: LoongPei <3136347099@qq.com>
This was referenced Sep 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The SM90 sparse decode kernel honors
topk_lengthonly for the DeepSeek-V4 layout; for V3.2 it assertstopk_length == nullptr. SM100 already honors it for V3.2. This makes SM90 consistent:topk_length, rounded up to 64 with at least one block, which matches whatget_decoding_sched_metaplans.topk_lengthare treated as invalid, as in the V4 path.v32_persistent_h{64,128}_topklen.cu, selected whentopk_lengthis given (the same scheme as the SM90 sparse prefill kernel). The existing V3.2 kernels are unchanged.The same kernel change is proposed upstream in deepseek-ai#216 (open, not reviewed yet), also as a compile-time specialization. This PR is for the fork, whose source layout differs, because vLLM needs it for DCP (vllm-project/vllm#58980). If deepseek-ai#216 lands upstream first, a later sync can replace this.
Motivation
vLLM runs V3.2 with decode context parallelism, where each rank owns about 1/dcp of every 2048-wide top-k row. Today the kernel runs on the full row with the other ranks' slots masked as
-1. Compacting the rank's slots and passing their count astopk_length(one batch entry per token) skips the masked slots.Kernel time on one DCP4 rank (H20, h_q=64, top-k 2048, ~512 owned slots per row):
The lengths differ per layer, so vLLM plans every call; the last column includes that planner run. A companion PR, opened at the same time, makes the planner 2.1-3.6x faster at these batch sizes.
End to end in vLLM (8x H20, TP8 + DCP4, vllm-project/vllm#58980), the prefill time of fresh prompts drops by 24-28%, and the decode step time by 3.9-8.3% at 16-32 concurrent requests. At 8 requests the per-layer planning cancels the gain out until the planner is faster.
Tests
tests/test_flash_mla_sparse_decoding.pynow also generates V3.2 cases withtopk_length: 180 new correctness cases. The harness poisons every KV row not selected withintopk_lengthwith NaN, so any read past the length fails.topk_length).topk_length. FlashMLA's SM90 decode perf cases before/after, best of two runs (us):V4 cases are within ±0.9% as well.
Not tested on SM100; this PR doesn't touch SM100 code.
AI assistance (Claude) was used for this change. I reviewed all of it and ran the tests above.