Repository navigation
Conversation
…kernel Speculative decode with DCP was gated to gfx950: a q > 1 verify pass over a round-robin DCP KV shard needs a persistent kernel that applies the causal mask on global positions. gfx1250's PS1 kernel (AITER_MLA_DECODE_PS1_FLYDSL=1) takes g_kv_indptr and does that, through mla_a8w8_qh96_..._lse_cprr_ps. Gate on dcp_persistent_supported() instead. PS1 takes a multi-token query only at 16 or 96 gathered heads, and aiter does not fold other widths onto it. Kimi-K3's DSpark draft gathers 8 x 8 = 64 heads under TP8 + DCP8, so its block pass fell through to the page-64 path. mla_dcp_kernel_num_heads now pads a multi-token decode to 96 on gfx1250 (the target's 96 is unchanged), and the metadata builder plans the same width. Kimi-K3 on two MI455X trays, TP8 + DCP8, MoE TP8, DSpark with 7 draft tokens and no forced acceptance: GSM8K 5-shot strict-match 0.9545 +- 0.0057, against 0.9621 +- 0.0053 without DSpark; acceptance 53.6%, 4.75 tokens per forward. Capturing the q_len 8 graphs with RCCL over the scale-up fabric needed NCCL_GRAPH_REGISTER=0. Co-authored-by: Cursor <cursoragent@cursor.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2647 (base
gyu/tp-across-nodes); retarget tomainonce #2647 merges. This change needs #2647'sdcp_persistent_supported(), which is true on gfx1250 whenAITER_MLA_DECODE_PS1_FLYDSL=1. Onmainit is still gfx950-only.Problem
Speculative decode with DCP is gated to gfx950. A q > 1 verify pass over a round-robin DCP KV shard needs a persistent MLA kernel that applies the causal mask on global positions. On gfx1250,
--method dsparkwith--decode-context-parallel-size 8stops at startup withSpeculative decode + DCP is only supported on gfx950.gfx1250's PS1 kernel (
AITER_MLA_DECODE_PS1_FLYDSL=1) takesg_kv_indptrand applies that mask (mla_a8w8_qh96_1tg_16mx6_64nx1_lse_cprr_ps, ROCm/aiter#5844). But it takes a multi-token query only at 16 or 96 gathered heads, and aiter does not fold other widths onto it. Under TP8 + DCP8, Kimi-K3's DSpark draft gathers 8 × 8 = 64 heads, so its block pass fell through to the page-64 path.Changes
atom/config.py: gate speculative decode + DCP ondcp_persistent_supported()instead ofgfx == "gfx950".atom/model_ops/attention_mla.py:mla_dcp_kernel_num_heads(..., multi_token=True)pads a multi-token decode on gfx1250 to the next PS1 width, 16 or 96. The draft's 64 becomes 96; the target's 96 is unchanged.MLAAttentionpassesmulti_tokenwhenever a speculative config is set, for the target's verify pass and the draft's block pass alike.atom/model_ops/attentions/aiter_mla.py: the metadata builder plans the same width.Validation
Kimi-K3 on two MI455X nodes (4 GPUs each): TP8 + DCP8, MoE TP8, DSpark with 7 draft tokens (
Inferact/Kimi-K3-DSpark), no forced acceptance. GSM8K 5-shot, all 1319:gyu/k3-455-tp8(2026-10-19)rocm/fw-bringup:gfx1250-atom-k3-20261027(main481c2bf + #2647 + #2737 + #2557 + #2554; 2026-10-27)cc8d4ff12(the grouped-MoE K-tail fit of [FlyDSL] [MoE] Fit the gfx1250 grouped GEMM k-tile to K, and refuse one that does not divide it aiter#6706); the second usedmain7061cbd88, which has [FlyDSL] [gfx1250] Fix the dropped K tail and the bias column offset in the grouped MoE TDM GEMM aiter#6663.NCCL_GRAPH_REGISTER=0.