Skip to content

[PD] Transfer DFlash draft KV separately from DCP target indices - #40192

Closed
chromecast56 wants to merge 10 commits into
sgl-project:mainfrom
chromecast56:codex/dflash-pd-layout-upstream
Closed

chromecast56 wants to merge 10 commits into
sgl-project:mainfrom
chromecast56:codex/dflash-pd-layout-upstream

Conversation

@chromecast56

@chromecast56 chromecast56 commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

PD + DCP + DFlash PR stack (merge bottom-up; each PR is based on main and includes the PRs below it)

  1. Fix PD retraction checkpoint boundaries with overlap decoding #39743
  2. [Spec] Support DFLASH for Kimi K3 #40794
  3. [DCP] Use logical token capacity for PD admission and load reporting #39731
  4. Preserve full DFlash draft KV during PD CPU retraction #39749
  5. [PD] Transfer DFlash draft KV separately from DCP target indices #40192 ← this PR
  6. [PD] Count transferred state bytes per component #40164

Review only the last 3 commits; the earlier commits belong to the PRs below.


Motivation

Under DCP the target MLA KV moves as rank-local rows, while a full (non-compact) DFlash draft pool is replicated and indexed by the target allocator's logical token ids. With the same DCP size on prefill and decode, NIXL's send_kvcache appends the draft buffers to the target KV entries and applies the target's rank-local page indices to them, so decode receives the wrong draft rows. The target verifies every drafted token, so the cost is acceptance rate.

Modifications

  • DFlashDraftTransfer (new disaggregation/dflash_kv.py) is resolved once per PD queue from the full draft pool that Preserve full DFlash draft KV during PD CPU retraction #39749 registers on the allocator. It applies only under DCP on NIXL. Without DCP the target indices are already logical, and the draft keeps riding the target KV entries (num_draft_entries).
  • The draft is sent as its own DFLASH_KV state component, appended last so other components keep their positions when only decode registers a draft. Its buffers are described in target wire-page units, and both sides index them with logical token ids from req_to_token.
  • Neither side knows the peer's DCP size when it registers, so a DCP>1 decode registers the draft both ways: among its KV entries, for a DCP=1 prefill's existing relayout plan, and as DFLASH_KV, which a same-DCP prefill sends instead. NIXL registers each buffer once.
  • The payload starts at the decode prefix, which already holds its draft KV (same as the SWA payload). It sends only the sliding-window tail when every draft layer is sliding_attention; otherwise it sends the whole suffix.
  • Prefill and decode share one payload helper. NIXL rejects prefill/decode attention-TP mismatch for this component when the peer registers. DFLASH already requires pp_size == 1 at startup.
  • Compact draft caches and the Mooncake path are unchanged.

Registering the draft both ways on decode is a new transfer shape: a same-DCP prefill sends fewer KV regions than decode registered, as decode-only speculative decoding already does on main. If you'd rather negotiate this at handshake time, I'm happy to restructure.

Tests

test_dflash_logical_transfer.py (5 CPU tests, 11 subtests):

  • DFLASH_KV is registered only for a registered full draft under DCP on NIXL; non-DCP deployments keep the existing target-entry path.
  • The tail window applies only when every draft layer slides. Reading sliding_window alone would truncate mixed and full-attention drafts, whose configs can still carry it.
  • A DCP8 request's payload skips the decode prefix, and NixlKVManager.maybe_send_extra writes the logical rows into the decode draft buffers.
  • The window tail starts on a page boundary.
  • Prefill and decode layouts agree for P1->D8, P8->D8 and P1->D1: decode has at least prefill's KV regions, prefill's state list is a prefix of decode's, the relayout geometry check passes, and decode registers no buffer twice. P1->D8 failed the region check when decode dropped its draft entries.

CI States

Latest PR Test (Base): ⏳ Run #35796399951
Latest PR Test (Extra): ❌ Run #35796399956
Latest PR Test (AMD ROCm 10): ⏳ Run #35796400143

@chromecast56

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 22, 2026
chromecast56 and others added 8 commits September 22, 2026 23:13
K3 exposed only set_dspark_layers_to_capture, so configure_aux_hidden_state_capture
rejected --speculative-algorithm DFLASH at startup. K3's DSPARK taps already
capture layer outputs, which is what DFLASH target_layer_ids name, so the new
set_dflash_layers_to_capture hooks on the LM and the multimodal wrapper reuse
them without the +1 shift that layer-input targets apply.

The Kimi K3 cookbook enables the DFLASH option with the public
modal-labs/Kimi-K3-DFlash draft at block size 8, gated off where the recipe is
pipelined, runs DP attention, or is an NPU recipe, and on Hopper and AMD
until it has been validated there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under DCP with the unified Mamba pool, the allocator size also counts
Mamba state bytes as tokens. Compute logical capacity as the configured
per-rank rows times attn_dcp_size in KVCacheConfigurator (draft sizes are
already widened; SWA allocators never widen), divide sglang:num_pages by
the widened allocator page size, and use the 3.10-safe enter_scope helper
in the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the externally assigned cpu_retraction_draft_pool with
DFlashWorkerV2.full_draft_kv_pool, which accepts only a non-compact
MHATokenToKVPool (MLA pools localize token ids to DCP rows), and
BaseTokenToKVPoolAllocator.register_full_draft_kv_pool. Trim the
round-trip matrix to the DCP1 and DCP8 cases that fail without the fix
and use the 3.10-safe enter_scope helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_draft_kv_pool, which tagged the draft pool with private attributes, has
no caller: the scheduler hands the PD queues primary_draft_kv_pool. Resolve
a DFlashDraftTransfer once per PD queue from the allocator's registered full
draft pool (sgl-project#39749), and only under DCP on NIXL, so non-DCP deployments keep
the draft in the target KV entries. Take the tail window only when every
draft layer slides, start the payload at the decode prefix, append the state
component last, share one payload helper between prefill and decode, check
TP equality at peer registration (DFLASH already requires pp_size 1 at
startup), and cover the payload through NixlKVManager.maybe_send_extra.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither PD side knows the peer's DCP size when it registers: prefill learns
it at the handshake and decode never does. A DCP>1 decode therefore keeps
the draft among its KV entries, for a DCP=1 prefill's relayout plan, and
also registers the trailing DFLASH_KV component that a same-DCP prefill
sends instead. NIXL registers each buffer once when a state component
reuses a KV region.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chromecast56
chromecast56 force-pushed the codex/dflash-pd-layout-upstream branch from b603249 to 5374823 Compare September 22, 2026 23:13
@chromecast56

Copy link
Copy Markdown
Collaborator Author

Closing. On main, a DCP8 → DCP8 PD transfer already delivers the DFlash draft KV correctly: the DCP draft pool is built with the widened page size (page_size * loc_space_scale, kv_cache_configurator.py:409-415), so one page-index vector addresses the same logical rows in the target and draft pools. A GPU check on Kimi K3 1P1D (prefill TP8 DCP8 → decode TP8 DCP8, DFLASH full draft, NIXL) without this PR gave GSM8K 95.2% and average accept length 4.97, matching DCP1 → DCP1 (95.7% / 4.99). The layout problem this PR worked around only exists on our fork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation memory-pool run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant