Repository navigation
[DCP] [HiCache] Support role-local DCP storage in prefill/decode serving (upstream #41695) - #19
Merged
Merged
Conversation
github-actions
Bot
added
documentation
Venkat2811
force-pushed
the
venkat/v0522-4-hybrid-dcp
branch
from
October 19, 2026 19:25
7bb3ec6 to
df33fe9
Compare
…nt state Reuse upstream Mamba admission with reserved checkpoint slots, decline unregistered prefetch hits before advertising them, and keep incoming recurrent state separate from cache restoration. Require wait-complete prefetch and reject unqualified decode offload for DCP L3. (cherry picked from commit 9994a5f)
Use upstream restore locks for pending capacity and GPU assertions. Drop the obsolete pre-gate failure test already covered by upstream gated receiver tests. (cherry picked from commit 7f48a80)
(cherry picked from commit c1fa6d7)
(cherry picked from commit 10e0de8)
Pass the selected policy through both controller layers instead of rechecking the startup default. Exercise real controller validation in the attach regression; timeout-to-wait_complete and retained-policy reattach fail before this change. (cherry picked from commit ed85005)
Venkat2811
force-pushed
the
venkat/v0522-5-pd-dcp
branch
from
October 19, 2026 19:26
b43300f to
3e932ec
Compare
Venkat2811
changed the base branch from
venkat/v0522-4-hybrid-dcp
to
kimi-k3-v0.5.22
October 19, 2026 19:26
Venkat2811
added a commit
that referenced
this pull request
Oct 27, 2026
…e slots (upstream sgl-project#41695) Layer 12 of the Kimi-K3 nightly line on upstream main 5884e4f. The prefill/decode part of upstream sgl-project#41695 that 5884e4f does not have. A decode server with --disaggregation-decode-enable-radix-cache and HiCache restores the cached prefix locally while prefill sends the rest: - The transfer-side load-back handed the request to init_load_back, so for a hybrid (Mamba/KDA) model it also copied the host checkpoint at the prefix end into the request's own state slot. That slot receives the live state prefill computes for the whole prompt; whichever copy lands last wins. InitLoadBackParams.restore_request_state=False restores the canonical cache slot only. - A hybrid restore needs one canonical state slot besides the request's live slot. Admission keeps a slot free for each queued restore (ensure_mamba_capacity(..., reserved_slots=)) and promises prefill only the device-resident prefix when no slot can be reclaimed; before, the promise went out and the restore failed after prefill had skipped those tokens. - Under DCP, a decode server with an L3 backend requires --hicache-storage-prefetch-policy wait_complete (decode promises the probed L3 span before the read completes) and refuses --disaggregation-decode-enable-offload-kvcache, as sgl-project#41695 did. Checked at startup (resolve_hicache_dcp_compatibility) and on runtime attach, including the same-backend policy update (StorageAttachment.attach; HiCacheController.attach_storage_backend never sees the effective policy). Source: 91ecb13 on the kimi-k3-nightly-3c560089 line, which equals upstream sgl-project#41695's own commits over sgl-project#41694 (head ed85005). Dropped because 5884e4f has it: starting the L3 prefetch before preallocation and promising only the span it took (sgl-project#43548). The sgl-project#41693/sgl-project#41694 ports that 91ecb13 sat on are superseded by sgl-project#41187 (no P/D refusal, no decode-offload case), so the guards are re-expressed on upstream's structure and the tests are rewritten on 5884e4f's fixtures. No change for a decode server without the decode radix cache, which is every K3 recipe today. Tests (CPU), this commit's test files on the parent: test_server_args.py TestHiCacheArgs::test_dcp_decode_storage_requires_wait_complete_without_offload 3 subtests fail (NotImplementedError not raised for timeout, best_effort and decode offload); test_hicache_dcp_host_pool.py TestDcpDecodeStorageAttach 6 subtests fail (attach accepted an early-stopping policy, attached or not); test_mamba_capacity.py test_pd_restore_promise_keeps_a_canonical_state_slot 2 subtests fail (full and pending_restore promised 6 tokens, not 2). With this commit all pass. Tests (GPU, one B300): test_unified_radix_cache_load_back.py Test_FULL_MAMBA_ps1::test_pd_decode_restore_keeps_the_live_request_state fails on the parent (the live slot ends with the checkpoint's marker 11, not the live 37) and passes with this commit. Fork PR (v0.5.22 line): #19 Upstream: sgl-project#41695 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Venkat2811
added a commit
that referenced
this pull request
Oct 27, 2026
…e slots (upstream sgl-project#41695) Layer 12 of the Kimi-K3 nightly line on upstream main 5884e4f. The prefill/decode part of upstream sgl-project#41695 that 5884e4f does not have. A decode server with --disaggregation-decode-enable-radix-cache and HiCache restores the cached prefix locally while prefill sends the rest: - The transfer-side load-back handed the request to init_load_back, so for a hybrid (Mamba/KDA) model it also copied the host checkpoint at the prefix end into the request's own state slot. That slot receives the live state prefill computes for the whole prompt; whichever copy lands last wins. InitLoadBackParams.restore_request_state=False restores the canonical cache slot only. - A hybrid restore needs one canonical state slot besides the request's live slot. Admission keeps a slot free for each queued restore (ensure_mamba_capacity(..., reserved_slots=)) and promises prefill only the device-resident prefix when no slot can be reclaimed; before, the promise went out and the restore failed after prefill had skipped those tokens. - Under DCP, a decode server with an L3 backend requires --hicache-storage-prefetch-policy wait_complete (decode promises the probed L3 span before the read completes) and refuses --disaggregation-decode-enable-offload-kvcache, as sgl-project#41695 did. Checked at startup (resolve_hicache_dcp_compatibility) and on runtime attach, including the same-backend policy update (StorageAttachment.attach; HiCacheController.attach_storage_backend never sees the effective policy). Source: 91ecb13 on the kimi-k3-nightly-3c560089 line, which equals upstream sgl-project#41695's own commits over sgl-project#41694 (head ed85005). Dropped because 5884e4f has it: starting the L3 prefetch before preallocation and promising only the span it took (sgl-project#43548). The sgl-project#41693/sgl-project#41694 ports that 91ecb13 sat on are superseded by sgl-project#41187 (no P/D refusal, no decode-offload case), so the guards are re-expressed on upstream's structure and the tests are rewritten on 5884e4f's fixtures. No change for a decode server without the decode radix cache, which is every K3 recipe today. Tests (CPU), this commit's test files on the parent: test_server_args.py TestHiCacheArgs::test_dcp_decode_storage_requires_wait_complete_without_offload 3 subtests fail (NotImplementedError not raised for timeout, best_effort and decode offload); test_hicache_dcp_host_pool.py TestDcpDecodeStorageAttach 6 subtests fail (attach accepted an early-stopping policy, attached or not); test_mamba_capacity.py test_pd_restore_promise_keeps_a_canonical_state_slot 2 subtests fail (full and pending_restore promised 6 tokens, not 2). With this commit all pass. Tests (GPU, one B300): test_unified_radix_cache_load_back.py Test_FULL_MAMBA_ps1::test_pd_decode_restore_keeps_the_live_request_state fails on the parent (the live slot ends with the checkpoint's marker 11, not the live 37) and passes with this commit. Fork PR (v0.5.22 line): #19 Upstream: sgl-project#41695 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 6, the head of the
kimi-k3-v0.5.22stack. Mirror of sgl-project#41695: its five commits (9994a5f322 .. ed850058f8, the 16 Oct PR head) cherry-picked onto layer 5 without conflict.Role-local DCP storage in prefill/decode serving: a decode with its own L3 keeps its cache promises and the live recurrent state it already received, tracks pending restores through the current cache locks, and validates a runtime attach with the effective prefetch policy. v0.5.22 already has the decode Mamba pre-allocation check (sgl-project#41927); the rest is still needed for DCP L3 inside P/D (long-ctx recipe with a TP16 prefill; any future decode L3). Neither live copy uses it today; it is carried so one release serves every recipe.
Upstream: sgl-project#41695
lmsysorg/sglang:v0.5.22-cu130)--controller,--hybrid: all PASStest_hybrid_l3_restore.py(1 GPU, both tree cores, cancellation, live state)Kimi-K3 on this head (full model, one HGX B300 node per role, the v0.5.22 image with this head checked out in place, Rust tree core):
kimi-k3-agg-b300shape (TP8 x DCP8, host tier 160 GB per rank, Mooncake L3, Triton KDA)Two things seen on the way, neither from this stack: v0.5.22 does not refresh
sglang:hicache_host_used_tokenswhile the engine is idle (it reads stale after/flush_cacheuntil the next batch), and on the P/D prefill the first L3 lookup after/flush_cacheusually finds nothing (5 of 6 runs with layer 1 alone, 7 of 7 on this head; later lookups restore).🤖 Generated with Claude Code
CI States
Latest PR Test (Base): 🚫 Run #39311072013
Latest PR Test (Extra): 🚫 Run #39311071568
Latest PR Test (AMD ROCm 10): 🚫 Run #39311071477