Skip to content

[DCP][Kimi-k3] Decode on the gathered head width - #2060

Merged
zejunchen-zejun merged 2 commits into
mainfrom
hexwang/mla_nhead96
Sep 1, 2026
Merged

[DCP][Kimi-k3] Decode on the gathered head width#2060
zejunchen-zejun merged 2 commits into
mainfrom
hexwang/mla_nhead96

Conversation

@whx-sjtu

@whx-sjtu whx-sjtu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

After ROCm/aiter#4964, we can run q_head 96 mla cprr directly without padding to 128.

Two changes, the first two on the Kimi-K3 tp8 + dcp8 decode path.

Drop the MLA decode head padding (attention_mla.py). mla_dcp_kernel_num_heads rounded the gathered width (num_heads * dcp) up to one of (16, 32, 64, 128), on the premise that aiter's fold onto the 16-head kernel desynchronises the DCP round-robin causal mask. That premise no longer holds — aiter now rebuilds qo_indptr and the kv indptrs as part of the fold, so every multiple of 16 is correct. A persistent decode now dispatches on the gathered width as-is, which is what lets K3 tp8+dcp8 (12 × 8 = 96) reach the 96-head kernel instead of padding to 128: 183.6us vs 190.8us at fp8 / ctx 32k / bs 64 / qlen 4.

Fuse add + RMSNorm in the K3 DSpark draft. Thread residual through the draft layers so the norms take their fused add form, in both the native and vLLM model files.

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2060 --add-label <label>

@whx-sjtu whx-sjtu added ci:full and removed ci:full labels Aug 27, 2026
@whx-sjtu
whx-sjtu marked this pull request as draft August 28, 2026 06:35
@zufayu
zufayu requested review from yhl-amd and removed request for yhl-amd August 29, 2026 09:54
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
@whx-sjtu
whx-sjtu force-pushed the hexwang/mla_nhead96 branch from cf6580d to 5a8c47f Compare August 31, 2026 07:34
@whx-sjtu whx-sjtu changed the title [DCP] Decode on the gathered head width, fix K3's missing QREP q_proj override [DCP][Kimi-k3] Decode on the gathered head width Aug 31, 2026
@whx-sjtu
whx-sjtu marked this pull request as ready for review August 31, 2026 07:37
@whx-sjtu
whx-sjtu requested review from ZhangLirong-amd and valarLip and removed request for valarLip August 31, 2026 10:42
@zejunchen-zejun
zejunchen-zejun merged commit f11f883 into main Sep 1, 2026
112 of 147 checks passed
@zejunchen-zejun
zejunchen-zejun deleted the hexwang/mla_nhead96 branch September 1, 2026 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants