Repository navigation
[refactor][model] Kimi-K3 DSpark MLA common base class - #60054
Draft
gronsti-amd wants to merge 7 commits into
Draft
gronsti-amd wants to merge 7 commits into
gronsti-amd wants to merge 7 commits into
Conversation
2 of 7 tasks
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Keep the NVIDIA module as a re-export so the registry path is unchanged. Signed-off-by: Stig-Arne Grönroos <stig-arne.gronroos@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
NVIDIA keeps MultiHeadLatentAttention as the cache owner. A platform override can point context-KV insert and draft layer names at another module. Signed-off-by: Stig-Arne Grönroos <stig-arne.gronroos@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The defaults keep today's concat and cache write. A subclass can replace the decode query and the KV rows it writes without a new custom-op argument. Signed-off-by: Stig-Arne Grönroos <stig-arne.gronroos@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The common module is a new file, so pre-commit lints it in full. The (list, tuple) form was already on main and fails UP038 once the file moves. Signed-off-by: Stig-Arne Grönroos <stig-arne.gronroos@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The file was not listed in Buildkite, so the refactor would not be collected on either platform. Signed-off-by: Stig-Arne Grönroos <stig-arne.gronroos@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
DeepseekV32Attention already assigns a concrete RotaryEmbedding to that name. Typing the base attribute as optional made mypy reject its cache reads. Signed-off-by: Stig-Arne Grönroos <stig-arne.gronroos@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Replay reads the captured positions and slot-mapping allocations, so a replacement tensor freezes the pointer. Lock the NVIDIA binding class and the default hook returns. Signed-off-by: Stig-Arne Grönroos <stig-arne.gronroos@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
gronsti-amd
force-pushed
the
sgronroo/refactor-kimi-k3-dspark
branch
from
October 9, 2026 12:19
d19390b to
e1b1905
Compare
This was referenced Oct 9, 2026
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Currently, the Kimi-K3 DSpark draft model has only one NVIDIA implementation,
and the dispatch is hard-wired to it on every platform. A ROCm module that copies that file drifts.
This PR moves the draft model into
vllm/models/kimi_k3/common/dspark_mla.pyand leaves NVIDIA as a thin binding.NVIDIA behavior stays the same: the same
MultiHeadLatentAttention, the same layer name, and the same kernels.The registry still loads
vllm.models.kimi_k3.nvidia.dspark_mla:removing the hard-wired dispatch is left for a follow-up PR (#60857).
What changes
common/dspark_mla.py. That file imports neitheramd/nornvidia/.build_self_attnandbuild_mlp.kv_cache_layerdefaults to the attention module, so context-KV insert andget_draft_kv_cache_layer_names()still useself_attn.Validation
pytest tests/models/test_dspark_mla.pypasses.vllm.models.kimi_k3.common.dspark_mla.MultiHeadLatentAttention, thatkv_cache_layerreturns the attention module, that a wrapper subclass'smla_attn_clsis constructed with aprefixending in.attn, and that the default hooks return their inputs.Details
AMD and NVIDIA attention and cache architecture for MLA differ: NVIDIA
MultiHeadLatentAttentionowns the KV cache, while the AMD stack isKimiK3MultiHeadLatentAttentionWrapperaround a genericMLAAttentionwhoselayer_nameisprefix + ".attn". Geometry, norms, and context RoPE already sit onself_attnon both sides. This PR refactors so that the three reads that differ (cache, dtype and scale, layer name) go throughkv_cache_layer.common/does not construct the NVIDIA attention class. A default there would importnvidia.mla, which is also the NVIDIA target K3 attention.#58814pointer refresh is in the shared path:_has_uniform_block_layoutkeys the cache ondata_ptr()and drops_context_cache_ptrswhen the allocation changes.enable_confidence_headbuildsDSparkConfidenceHead, keeps those checkpoint weights, andcompute_confidenceserves adaptive verification. The head staysNonewhen the flag is off.MLAAttentionstores positions as_rope_positions(assigned every forward, includingNone). The wrapper takesmla_attn_cls. Three hooks default to today's behavior:_prepare_kv_cache_update,_prepare_mha_inputs, and_form_decode_q. Custom-op schemas stay as they are. Positions are module state the ops read.A planned fusion of QK-RoPE plus cache insert (fused_qk_rope_concat_and_cache_mla) requires an epilogue to replace query formation and the KV write.
The fusion cannot live only in the draft model: Absorb runs inside
MLAAttention.forward_impl, andunified_mla_kv_cache_updatecannot takepositions.This PR adds three defaulted hooks as an extension point for this decode epilogue, to avoid needing a second refactor immediately after this. The defaults do not read
_rope_positions.The Kimi-K3 DSpark MLA is under active development, causing a naive duplicate-and-modify approach to quickly drift, as shown by #57942:
_kv_cache_ptrs_key, drop_context_cache_ptrswhen the cache is rebound).'Refactoring reduces the drift, allowing both platforms to gain new features in a single change.
Out of scope
fused_qk_rope_concat_and_cache_mla([ROCm] Fuse Kimi-K3 DSpark QK RoPE with the MLA cache write #60864).Suggested review order
Review by commit. The aggregate diff looks like a rewrite of
nvidia/dspark_mla.pybecause that path stays as the public binding, so Git does not record a rename.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.