Skip to content

[DSpark][DCP] Allow speculative decode with DCP on gfx1250's PS1 MLA kernel - #2760

Open
gbyu-amd wants to merge 1 commit into
gyu/tp-across-nodesfrom
gyu/dspark-dcp-gfx1250
Open

gbyu-amd wants to merge 1 commit into
gyu/tp-across-nodesfrom
gyu/dspark-dcp-gfx1250

Conversation

@gbyu-amd

Copy link
Copy Markdown
Contributor

Stacked on #2647 (base gyu/tp-across-nodes); retarget to main once #2647 merges. This change needs #2647's dcp_persistent_supported(), which is true on gfx1250 when AITER_MLA_DECODE_PS1_FLYDSL=1. On main it is still gfx950-only.

Problem

Speculative decode with DCP is gated to gfx950. A q > 1 verify pass over a round-robin DCP KV shard needs a persistent MLA kernel that applies the causal mask on global positions. On gfx1250, --method dspark with --decode-context-parallel-size 8 stops at startup with Speculative decode + DCP is only supported on gfx950.

gfx1250's PS1 kernel (AITER_MLA_DECODE_PS1_FLYDSL=1) takes g_kv_indptr and applies that mask (mla_a8w8_qh96_1tg_16mx6_64nx1_lse_cprr_ps, ROCm/aiter#5844). But it takes a multi-token query only at 16 or 96 gathered heads, and aiter does not fold other widths onto it. Under TP8 + DCP8, Kimi-K3's DSpark draft gathers 8 × 8 = 64 heads, so its block pass fell through to the page-64 path.

Changes

  • atom/config.py: gate speculative decode + DCP on dcp_persistent_supported() instead of gfx == "gfx950".
  • atom/model_ops/attention_mla.py: mla_dcp_kernel_num_heads(..., multi_token=True) pads a multi-token decode on gfx1250 to the next PS1 width, 16 or 96. The draft's 64 becomes 96; the target's 96 is unchanged. MLAAttention passes multi_token whenever a speculative config is set, for the target's verify pass and the draft's block pass alike.
  • atom/model_ops/attentions/aiter_mla.py: the metadata builder plans the same width.

Validation

Kimi-K3 on two MI455X nodes (4 GPUs each): TP8 + DCP8, MoE TP8, DSpark with 7 draft tokens (Inferact/Kimi-K3-DSpark), no forced acceptance. GSM8K 5-shot, all 1319:

ATOM under test All-reduce between the nodes strict-match Draft tokens accepted Tokens per forward
this commit on gyu/k3-455-tp8 (2026-10-19) RCCL 0.9545 ± 0.0057 53.6% 4.75
this commit on the ATOM of image rocm/fw-bringup:gfx1250-atom-k3-20261027 (main 481c2bf + #2647 + #2737 + #2557 + #2554; 2026-10-27) aiter custom all-reduce (symm_mem) 0.9530 ± 0.0058 53.2% 4.73

…kernel

Speculative decode with DCP was gated to gfx950: a q > 1 verify pass over
a round-robin DCP KV shard needs a persistent kernel that applies the
causal mask on global positions. gfx1250's PS1 kernel
(AITER_MLA_DECODE_PS1_FLYDSL=1) takes g_kv_indptr and does that, through
mla_a8w8_qh96_..._lse_cprr_ps. Gate on dcp_persistent_supported() instead.

PS1 takes a multi-token query only at 16 or 96 gathered heads, and aiter
does not fold other widths onto it. Kimi-K3's DSpark draft gathers
8 x 8 = 64 heads under TP8 + DCP8, so its block pass fell through to the
page-64 path. mla_dcp_kernel_num_heads now pads a multi-token decode to
96 on gfx1250 (the target's 96 is unchanged), and the metadata builder
plans the same width.

Kimi-K3 on two MI455X trays, TP8 + DCP8, MoE TP8, DSpark with 7 draft
tokens and no forced acceptance: GSM8K 5-shot strict-match
0.9545 +- 0.0057, against 0.9621 +- 0.0053 without DSpark; acceptance
53.6%, 4.75 tokens per forward. Capturing the q_len 8 graphs with RCCL
over the scale-up fabric needed NCCL_GRAPH_REGISTER=0.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants