Repository navigation
[PD] Transfer DFlash draft KV separately from DCP target indices - #40192
Closed
chromecast56 wants to merge 10 commits into
Closed
chromecast56 wants to merge 10 commits into
chromecast56 wants to merge 10 commits into
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 22, 2026
chromecast56
force-pushed
the
codex/dflash-pd-layout-upstream
branch
from
September 22, 2026 21:36
b2415b2 to
d7ab34d
Compare
chromecast56
marked this pull request as ready for review
September 22, 2026 21:37
Collaborator
Author
|
/tag-and-rerun-ci |
chromecast56
requested review from
ByronHsu,
Duyi-Wang,
Fridge003,
HaiShaw,
Qiaolin-Yu,
ShangmingCai,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
sogalin,
xiezhq-hermann and
yizhang2077
as code owners
September 22, 2026 21:37
chromecast56
force-pushed
the
codex/dflash-pd-layout-upstream
branch
from
September 22, 2026 23:10
d7ab34d to
b603249
Compare
chromecast56
requested review from
JustinTong0323 and
zijiexia
as code owners
September 22, 2026 23:10
K3 exposed only set_dspark_layers_to_capture, so configure_aux_hidden_state_capture rejected --speculative-algorithm DFLASH at startup. K3's DSPARK taps already capture layer outputs, which is what DFLASH target_layer_ids name, so the new set_dflash_layers_to_capture hooks on the LM and the multimodal wrapper reuse them without the +1 shift that layer-input targets apply. The Kimi K3 cookbook enables the DFLASH option with the public modal-labs/Kimi-K3-DFlash draft at block size 8, gated off where the recipe is pipelined, runs DP attention, or is an NPU recipe, and on Hopper and AMD until it has been validated there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under DCP with the unified Mamba pool, the allocator size also counts Mamba state bytes as tokens. Compute logical capacity as the configured per-rank rows times attn_dcp_size in KVCacheConfigurator (draft sizes are already widened; SWA allocators never widen), divide sglang:num_pages by the widened allocator page size, and use the 3.10-safe enter_scope helper in the test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the externally assigned cpu_retraction_draft_pool with DFlashWorkerV2.full_draft_kv_pool, which accepts only a non-compact MHATokenToKVPool (MLA pools localize token ids to DCP rows), and BaseTokenToKVPoolAllocator.register_full_draft_kv_pool. Trim the round-trip matrix to the DCP1 and DCP8 cases that fail without the fix and use the 3.10-safe enter_scope helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_draft_kv_pool, which tagged the draft pool with private attributes, has no caller: the scheduler hands the PD queues primary_draft_kv_pool. Resolve a DFlashDraftTransfer once per PD queue from the allocator's registered full draft pool (sgl-project#39749), and only under DCP on NIXL, so non-DCP deployments keep the draft in the target KV entries. Take the tail window only when every draft layer slides, start the payload at the decode prefix, append the state component last, share one payload helper between prefill and decode, check TP equality at peer registration (DFLASH already requires pp_size 1 at startup), and cover the payload through NixlKVManager.maybe_send_extra. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither PD side knows the peer's DCP size when it registers: prefill learns it at the handshake and decode never does. A DCP>1 decode therefore keeps the draft among its KV entries, for a DCP=1 prefill's relayout plan, and also registers the trailing DFLASH_KV component that a same-DCP prefill sends instead. NIXL registers each buffer once when a state component reuses a KV region. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chromecast56
force-pushed
the
codex/dflash-pd-layout-upstream
branch
from
September 22, 2026 23:13
b603249 to
5374823
Compare
Collaborator
Author
|
Closing. On main, a DCP8 → DCP8 PD transfer already delivers the DFlash draft KV correctly: the DCP draft pool is built with the widened page size (page_size * loc_space_scale, kv_cache_configurator.py:409-415), so one page-index vector addresses the same logical rows in the target and draft pools. A GPU check on Kimi K3 1P1D (prefill TP8 DCP8 → decode TP8 DCP8, DFLASH full draft, NIXL) without this PR gave GSM8K 95.2% and average accept length 4.97, matching DCP1 → DCP1 (95.7% / 4.99). The layout problem this PR worked around only exists on our fork. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PD + DCP + DFlash PR stack (merge bottom-up; each PR is based on
mainand includes the PRs below it)Review only the last 3 commits; the earlier commits belong to the PRs below.
Motivation
Under DCP the target MLA KV moves as rank-local rows, while a full (non-compact) DFlash draft pool is replicated and indexed by the target allocator's logical token ids. With the same DCP size on prefill and decode, NIXL's
send_kvcacheappends the draft buffers to the target KV entries and applies the target's rank-local page indices to them, so decode receives the wrong draft rows. The target verifies every drafted token, so the cost is acceptance rate.Modifications
DFlashDraftTransfer(newdisaggregation/dflash_kv.py) is resolved once per PD queue from the full draft pool that Preserve full DFlash draft KV during PD CPU retraction #39749 registers on the allocator. It applies only under DCP on NIXL. Without DCP the target indices are already logical, and the draft keeps riding the target KV entries (num_draft_entries).DFLASH_KVstate component, appended last so other components keep their positions when only decode registers a draft. Its buffers are described in target wire-page units, and both sides index them with logical token ids fromreq_to_token.DFLASH_KV, which a same-DCP prefill sends instead. NIXL registers each buffer once.sliding_attention; otherwise it sends the whole suffix.pp_size == 1at startup.Registering the draft both ways on decode is a new transfer shape: a same-DCP prefill sends fewer KV regions than decode registered, as decode-only speculative decoding already does on main. If you'd rather negotiate this at handshake time, I'm happy to restructure.
Tests
test_dflash_logical_transfer.py(5 CPU tests, 11 subtests):DFLASH_KVis registered only for a registered full draft under DCP on NIXL; non-DCP deployments keep the existing target-entry path.sliding_windowalone would truncate mixed and full-attention drafts, whose configs can still carry it.NixlKVManager.maybe_send_extrawrites the logical rows into the decode draft buffers.CI States
Latest PR Test (Base): ⏳ Run #35796399951
Latest PR Test (Extra): ❌ Run #35796399956
Latest PR Test (AMD ROCm 10): ⏳ Run #35796400143