[DCP] [HiCache] Support role-local DCP storage in prefill/decode serving

docs/docs/advanced_features/hicache_best_practices.mdx

@@ -63,6 +63,67 @@ Guidelines:
 - For MHA models with Mooncake and `page_head` layout, HiCache will split head shards based on `tp_lcm_size` to make keys reusable across heterogeneous TP deployments.
 - If all clusters use the same TP size, this option is not needed.
 
+### Hybrid model file checkpoints
+
+Hybrid MLA models share attention KV across TP ranks, but their recurrent state
+is rank-local. File storage separates these checkpoints by TP rank, TP size and
+host tensor schema. Readers must use a compatible topology and state layout;
+host cache capacity may differ.
+
+Legacy rank-less `.mamba` files are intentionally cold misses because they may
+contain another rank's state. Ordinary KV filenames are unchanged. Very long
+model namespaces are compacted to fit filesystem filename limits and also start
+with a cold cache. With file eviction enabled, each rank manages its own state
+files; the MLA KV writer also manages the shared KV files. The `max_size` limit
+applies to each rank's owned files, not to the total shared-directory size.
+If you change TP size or the namespace is compacted, old state files no longer
+count toward any active rank's cap and are not evicted; use a new storage
+directory or remove the obsolete files.
+
+### Decode context parallelism and storage
+
+For MLA models, `--dcp-size > 1` supports HiCache L3 through the `file` and
+`mooncake` backends. Use the same TP, DCP and PP sizes, physical page size,
+KV dtype, host layout and model identity for writers and readers of a cache
+namespace. `tp_lcm_size` does not convert persisted DCP shards between layouts.
+Other L3 backends remain rejected under DCP, including at runtime attach.
+
+Each logical page spans `--page-size * --dcp-size` tokens. A cache hit requires
+every DCP shard. For a hybrid model with recurrent state, the checkpoint also
+requires every TP state shard at the same legal prefix boundary. Missing state
+or KV shortens reuse to a complete checkpoint; it cannot be treated as a full hit.
+
+The supported host-pool shape is one materialized MLA pool, optionally with one
+independent materialized Mamba/KDA pool. Packed draft pools, additional side
+pools and separate KV scale buffers are outside this storage support. The
+general DCP and model-specific attention-backend restrictions still apply.
+Use `--hicache-host-memory-mode cache`; DCP L3 rejects `buffer_only` and
+speculative decoding. DSPARK with DCP remains supported at L1/L2 without L3.
+
+In P/D deployments, keep persisted cache namespaces role-local when the roles
+use different layouts. Use separate file directories or Mooncake
+`extra_backend_tag` values. Live P-to-D transfer remains responsible for moving
+the request's KV and current recurrent state; L3 does not convert the stored
+objects between the roles' topologies. Decode-side radix reuse requires
+`--disaggregation-decode-enable-radix-cache`.
+For decode with DCP L3, also set
+`--hicache-storage-prefetch-policy wait_complete`. The default `timeout` and
+`best_effort` policies can stop before fulfilling the prefix promised to
+prefill, so startup and runtime attachment reject them on decode. Prefill and
+aggregated servers may use all three policies. Decode KV offload
+(`--disaggregation-decode-enable-offload-kvcache`) is not supported with DCP L3.
+
+DCP keys include shard and layout identity. Recurrent file keys also preserve
+TP ownership independently of MLA replication, including when long filenames
+are compacted. Older incompatible recurrent files become cache misses and are
+recomputed, including at DCP1. DCP1 MLA KV keys keep their format except for
+long namespaces compacted to fit the filesystem filename limit; those objects
+also become cold misses.
+
+L2 remains host DRAM. To use local SSD as L3, select `file` and place its storage
+directory on an SSD filesystem. Mooncake's own SSD offload is configured in the
+Mooncake store service; selecting `mooncake` in SGLang alone does not enable it.
+
 ### Prefetch Policies
 
 ```bash Command

python/sglang/srt/arg_groups/hicache_hook.py

@@ -110,16 +110,13 @@ def handle_hicache_ratio_default(server_args: Any):
 def resolve_hicache_dcp_compatibility(server_args: Any):
 
     cfg = resolving_view(server_args)
-    if cfg.dcp_size <= 1 or not cfg.enable_hierarchical_cache:
+    if cfg.dcp_size <= 1 or not (
+        cfg.enable_hierarchical_cache
+        or cfg.disaggregation_decode_enable_offload_kvcache
+    ):
         return
     if cfg.hicache_storage_backend is not None:
-        raise NotImplementedError(
-            "--hicache-storage-backend (L3) with --dcp-size > 1 is not "
-            "supported yet: under DCP each rank holds a distinct "
-            "interleaved MLA KV shard, so the rank-0-only replicated-MLA "
-            "backup and the storage keys must become dcp_rank-aware "
-            "first. Run HiCache+DCP with L1/L2 only."
-        )
+        validate_hicache_dcp_storage(server_args)
     if cfg.speculative_algorithm not in (None, "DSPARK"):
         raise NotImplementedError(
             "HiCache with --dcp-size > 1 only supports DSPARK speculative "
@@ -143,13 +140,51 @@ def resolve_hicache_dcp_compatibility(server_args: Any):
             "MHA host pool has none."
         )
     logger.info(
-        "HiCache + DCP enabled (L1/L2 only): host pool uses widened "
+        "HiCache + DCP enabled: host pool uses widened "
         "logical slot accounting with per-rank physical translation at "
         "the transfer boundary (dcp_size=%d).",
         cfg.dcp_size,
     )
 
 
+def validate_hicache_dcp_storage(
+    server_args: Any, *, storage_backend=None, prefetch_policy=None
+):
+    """Backends implementing fixed-topology MLA DCP shard storage."""
+    cfg = resolving_view(server_args)
+    if cfg.dcp_size <= 1:
+        return
+    if not use_mla_backend(server_args):
+        raise NotImplementedError("HiCache L3 with DCP requires MLA.")
+    if cfg.hicache_host_memory_mode == "buffer_only":
+        # Buffer-mode staging budgets still count physical host rows.
+        raise NotImplementedError("HiCache L3 with DCP requires host cache mode.")
+    if cfg.speculative_algorithm is not None:
+        raise NotImplementedError(
+            "HiCache L3 with DCP does not support speculative draft storage."
+        )
+    if cfg.disaggregation_mode == "decode":
+        if cfg.disaggregation_decode_enable_offload_kvcache:
+            raise NotImplementedError(
+                "HiCache L3 with DCP does not support decode offload."
+            )
+        # Decode promises the probed L3 span to prefill before the read completes.
+        policy = (
+            cfg.hicache_storage_prefetch_policy
+            if prefetch_policy is None
+            else prefetch_policy
+        )
+        if policy != "wait_complete":
+            raise NotImplementedError(
+                "Decode HiCache L3 with DCP requires "
+                "--hicache-storage-prefetch-policy wait_complete."
+            )
+    if (storage_backend or cfg.hicache_storage_backend) not in ("file", "mooncake"):
+        raise NotImplementedError(
+            "HiCache L3 with DCP requires file or Mooncake storage."
+        )
+
+
 def resolve_layout_io_compatibility(server_args: Any):
     cfg = resolving_view(server_args)
     if (

python/sglang/srt/disaggregation/decode.py

@@ -1350,6 +1350,18 @@ def pop_preallocated(
                 retractable_tokens=retractable_tokens, count_retracted=True
             )
         reserved_restore_tokens = self._hicache_pending_restore_tokens()
+        reserved_restore_states = 0
+        if (
+            getattr(self.req_to_token_pool, "mamba_allocator", None) is not None
+            and self.scheduler.enable_decode_hicache
+        ):
+            reserved_restore_states = sum(
+                dr.prefix_match.needs_local_restore
+                for dr in self.transfer_queue.queue
+                if dr.prefix_match is not None
+                and dr.hicache_restore_status == HiCacheRestoreResult.PENDING
+                and dr.hicache_restore_lock is None
+            )
         full_allocatable_tokens -= reserved_restore_tokens
         # Sort by priority before any index-based bookkeeping so that both the
         # abort-scan loop and the preallocation loop operate on the same order.
@@ -1452,10 +1464,12 @@ def pop_preallocated(
             if self.req_to_token_pool.available_size() <= 0:
                 break
 
-            # Reserve enough Mamba capacity for radix COW and tracking buffers
-            # before prefix matching can bind state or lock cached entries.
+            # Pending HiCache restores also need canonical recurrent slots.
             if not ensure_mamba_capacity(
-                self.req_to_token_pool, [decode_req.req], self.tree_cache
+                self.req_to_token_pool,
+                [decode_req.req],
+                self.tree_cache,
+                reserved_slots=reserved_restore_states,
             ):
                 break
 
@@ -1473,6 +1487,17 @@ def pop_preallocated(
             if use_decode_radix_cache:
                 # Match prefix against decode's radix cache.
                 prefix_match = self._match_prefix_and_lock(decode_req.req)
+                # A hybrid restore materializes one canonical checkpoint in
+                # addition to the request's live state received from prefill.
+                # If it cannot fit, promise only the device-resident prefix.
+                if prefix_match.needs_local_restore and not ensure_mamba_capacity(
+                    self.req_to_token_pool,
+                    [decode_req.req],
+                    self.tree_cache,
+                    reserved_slots=reserved_restore_states + 1,
+                ):
+                    prefix_match.l2_host_hit_length = 0
+                    prefix_match.l3_storage_hit_length = 0
                 prefix_indices = prefix_match.prefix_indices
                 # prefix_len: tokens already on device (L1 hit).
                 # total_prefix_len: full prefix promised to prefill
@@ -1574,20 +1599,28 @@ def pop_preallocated(
                     failed_reqs.append(decode_req)
                     indices_to_remove.add(i)
                     continue
+            if self.scheduler.enable_decode_hicache:
+                self._start_hicache_prefetch(decode_req.req, prefix_match)
+                if prefix_match is not None:
+                    # A probe is only a candidate hit. Admission may decline
+                    # prefetch, so allocate/advertise the resulting prefix.
+                    # Preserve any smaller SWA transfer cap selected above.
+                    total_prefix_len = min(
+                        total_prefix_len, prefix_match.decode_prefix_len
+                    )
             dst_kv_indices = self._pre_alloc(
                 decode_req.req,
                 prefix_indices,
                 prefix_len,
                 total_prefix_len,
             )
             decode_req.prefix_match = prefix_match
-            if self.scheduler.enable_decode_hicache:
-                self._start_hicache_prefetch(decode_req.req, prefix_match)
             hisparse_req_budget -= 1
             # Recompute from actual pool state for the next queue entry.
             # This accounts for page rounding and newly locked evictable cache.
             if prefix_match is not None:
                 reserved_restore_tokens += prefix_match.restore_token_count
+                reserved_restore_states += int(prefix_match.needs_local_restore)
             full_allocatable_tokens = self._allocatable_token_budgets(
                 retractable_tokens=retractable_tokens,
                 count_retracted=True,

python/sglang/srt/disaggregation/decode_hicache_mixin.py

@@ -143,6 +143,8 @@ def _start_hicache_prefetch(
             prefix_match.prefetch_registered = self.tree_cache.has_ongoing_prefetch(
                 req.cache_request_handle
             )
+            if not prefix_match.prefetch_registered:
+                prefix_match.l3_storage_hit_length = 0
         except Exception as e:
             logger.warning(
                 "HiCache L3 prefetch failed for rid=%s: %s; falling back to L2-only LoadingBack",
@@ -223,6 +225,7 @@ def _try_hicache_queue_load_back(self, dr: DecodeRequest) -> bool:
                 best_match_node=rematch.best_match_node,
                 host_hit_length=rematch.host_hit_length,
                 req=dr.req,
+                restore_request_state=False,
             )
         )
         # The rematch repointed req.last_node to feed init_load_back's device