[II] Select backend-owned Kimi dense MLA DCP - #360
voipmonitor wants to merge 1 commit into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
The behavior implemented by this pull request is preserved in vLLM #387 as commit |
Behavior
Kimi-K3 detects attention implementations that declare
owns_decode_dcp_collectives=True. For those implementations, the model passes rank-local queries directly to the backend and accepts the backend's combined output. It does not allocate or invokeMLADCPManagerfor decode query gathering or output combination.B12xMLAImpldeclares that contract.B12xMLABackendaccepts supported DCP geometries when prefill context parallelism is disabled, the gathered head count satisfies the eight-head kernel tile, and the largest local interleaved KV shard does not exceed 1,048,576 tokens. Kimi RoPE decode is permitted because the B12X backend gathers already-rotated rank-local queries.Backends without the declaration retain the existing
MLADCPManagerpath.Dependencies
Depends on #359. TP16 serving also requires #338 for world-size-16 B12X transport, #339 for FP32 LSE transport, and #340 for bounded transport fallback.
Validation
Status: implemented and unit-qualified; composed serving qualification is pending.
ruff format,ruff check, and Python compilation: pass.tests/v1/attention/test_b12x_mla.pyplustests/models/kimi_k3/test_sequence_parallel.py: 47 passed in the Kimi-K3 CUDA 13.3 / PyTorch 2.13 source-locked runtime image.