Fix Mooncake HiCache registration for DeepSeek-V4 logical KV anchor by Li-brua · Pull Request #26649 · sgl-project/sglang

Li-brua · 2026-05-29T03:26:19Z

Motivation

DeepSeek-V4-Flash uses a hybrid HiCache layout with a logical KV anchor pool. The anchor pool owns page indices but does not own a real KV buffer; the actual data is stored in hybrid side pools such as SWA, compressed KV, indexer, and state pools.
When enabling Mooncake as the HiCache storage backend, Mooncake previously assumed the primary host pool always had a physical kv_buffer and a Mooncake-supported memory layout. For DeepSeek-V4, this caused startup to fail during host pool registration because the logical anchor has kv_buffer=None and layout=layer_first.

Verify fix works. Launch server as follows:

Example Commands

CUDA_VISIBLE_DEVICES="0,1,2,3" \
MOONCAKE_MASTER="0.0.0.0:50058" \
MOONCAKE_DEVICE="mlx5_0,mlx5_1,mlx5_2,mlx5_3" \
MOONCAKE_PROTOCOL="rdma" \
MOONCAKE_GLOBAL_SEGMENT_SIZE="0" \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
python -m sglang.launch_server \
    --served-model-name deepseek-v4-flash \
    --model-path /mnt/models/deepseek-ai/DeepSeek-V4-Flash \
    --host 0.0.0.0 \
    --port 30000 \
    --page-size 64 \
    --enable-hierarchical-cache \
    --hicache-ratio 1.7 \
    --hicache-mem-layout page_first \
    --hicache-io-backend kernel \
    --hicache-storage-backend mooncake \
    --hicache-write-policy write_through \
    --hicache-storage-prefetch-policy wait_complete \
    --trust-remote-code \
    --tensor-parallel-size 4 \
    --mem-fraction-static 0.8 \
    --enable-metrics \
    --enable-cache-report \
    --moe-runner-backend marlin \
    --max-running-requests 100 \
    --dist-init-addr "0.0.0.0:29500"

This PR fixes Mooncake storage registration for DeepSeek-V4 hybrid HiCache. #26647

Modifications

Detect logical KV anchor pools during Mooncake host pool registration.
Skip physical buffer registration and layout validation for logical KV anchors.
Use lightweight marker keys to track logical KV page existence in Mooncake.
Keep existing v2 registration and zero-copy I/O paths unchanged for real hybrid side pools.

Accuracy Tests

This change does not modify model forward logic, kernels, sampling, or numerical computation. No accuracy impact is expected.

Speed Tests and Profiling

This change only affects Mooncake HiCache storage registration and logical KV page existence tracking for DeepSeek-V4 hybrid HiCache.

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #27398986088
Latest PR Test (Extra): ❌ Run #27398985992

gemini-code-assist · 2026-05-29T03:26:23Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

stmatengss

Review: 0 critical, 4 informational. Correct approach for skipping physical buffer registration on logical KV anchors.

stmatengss · 2026-06-08T17:33:52Z

@@ -597,6 +598,17 @@ def warmup(self):

    def register_mem_pool_host(self, mem_pool_host: HostKVCache):
        super().register_mem_pool_host(mem_pool_host)
+        self._logical_kv_anchor = (
+            getattr(self.mem_pool_host, "kv_buffer", None) is None
+            and getattr(self.mem_pool_host, "entries", None) is not None


[INFO] (8/10) Duck-typing heuristic: kv_buffer is None and entries is not None. Works for the current DeepSeek-V4 anchor shape, but if a future pool has kv_buffer=None for a different reason (e.g. lazy allocation), this would misclassify it. Consider checking an explicit attribute like is_logical_anchor on the pool if one exists, or adding one.

Good point. I replaced the kv_buffer/entries heuristic with an explicit is_logical_anchor marker on LogicalHostPool, propagated through HostPoolGroup.

stmatengss · 2026-06-08T17:33:52Z

+        marker = b"1"
+        for i, key in enumerate(key_strs):
+            if results[i]:
+                continue


[INFO] (7/10) Sequential self.store.put() in a loop for each non-existing key. If many logical pages are set simultaneously, this is N individual RPCs to Mooncake instead of a batch write. Fine for small page counts; could become a bottleneck if the anchor pool is large. Consider batching if a batch-put API exists.

Keeping this as-is for now. These are 1-byte marker values, while the available batch API here is the zero-copy batch_put_from(ptr, size) path for registered buffers. We can optimize if Mooncake exposes a batch bytes put API.

stmatengss · 2026-06-08T17:33:52Z

@@ -652,6 +683,10 @@ def _get_hybrid_page_component_keys(
            suffixes = [f"{base_suffix}_temporal"] + [
                f"{base_suffix}_conv_{i}" for i in range(conv_num)
            ]
+        elif name in getattr(self, "registered_pools", {}):
+            # DeepSeek-V4 and other hybrid side pools store one object per
+            # logical page. Use the pool name as the storage-key suffix.


[INFO] (7/10) getattr(self, "registered_pools", {}) silently returns {} if the attribute doesn't exist, making this branch dead code in that case. If registered_pools is guaranteed to be set by register_mem_host_pool_v2 before this method runs, consider using self.registered_pools directly so a missing attribute fails loudly rather than silently skipping the DeepSeek-V4 path.

Agreed. Changed this to use self.registered_pools directly so a missing initialization fails loudly.

stmatengss · 2026-06-08T17:33:52Z

@@ -843,6 +878,9 @@ def batch_get_v1(
        # Apply extra_backend_tag prefix if available
        keys = self._tag_keys(keys)

+        if getattr(self, "_logical_kv_anchor", False):


[INFO] (6/10) getattr(self, "_logical_kv_anchor", False) is used here, in batch_set_v1, and in batch_exists, but _logical_kv_anchor is always initialized in __init__. The defensive getattr is harmless but inconsistent with the direct self._logical_kv_anchor access in register_mem_pool_host. Pick one pattern.

Agreed. _logical_kv_anchor is initialized in init, so I switched the call sites to direct self._logical_kv_anchor access.

Li-brua · 2026-06-09T07:29:21Z

Update after rebasing on latest main:

The latest main has already fixed the startup-side part of this issue by allowing Mooncake registration to skip physical KV buffer registration for logical/hybrid anchor pools, and it also includes more complete v2 handling for hybrid side pools.

This PR is still useful as a semantic follow-up: it makes logical KV anchors explicit via is_logical_anchor and tracks logical KV page existence with lightweight marker keys. This keeps the v1 batch_set / batch_get / batch_exists path consistent for logical anchors, instead of treating the logical KV anchor as always present and relying only on side-pool existence checks.

So the PR is no longer the sole startup fix, but it tightens Mooncake’s logical-anchor registration and existence-tracking semantics for DeepSeek-V4-style hybrid HiCache.

stmatengss · 2026-06-10T17:44:22Z

/tag-and-rerun-ci

stmatengss · 2026-06-11T17:05:22Z

/rerun-failed-ci

stmatengss · 2026-06-11T17:19:15Z

+        assert self.mem_pool_host.layout in [
+            "page_first",
+            "page_first_direct",
+            "page_head",
+            "page_first_kv_split",
+        ], "mooncake store storage backend only support page first, page first direct, page head and  page_first_kv_split layout"


Should remove to parameter checking phases?

Removed the extra layout assert from register_mem_pool_host. Backend/layout compatibility should stay in the argument/config normalization path, while this PR only needs to handle the logical-anchor
registration path.

stmatengss · 2026-06-11T17:24:41Z

+        if self.mem_pool_host.kv_buffer is None:
+            # Non-KV logical anchors carry data through v2 side pools only.
+            return [True] * len(keys)


Don't need to change the position.

Adjusted the branch order so the non-logical kv_buffer=None fast path stays before key tagging. Logical KV anchors still fall through to the tagged marker-key path.

stmatengss · 2026-06-12T17:43:07Z

/rerun-failed-ci

stmatengss · 2026-06-15T12:00:24Z

/rerun-failed-ci

Li-brua requested review from Ying1123, hanming-lu, hnyls2002, hzh0425, ispobock, merrymercy, xiezhq-hermann and yizhang2077 as code owners May 29, 2026 03:26

Li-brua force-pushed the librua branch 2 times, most recently from 7f8b837 to 1549941 Compare May 29, 2026 03:50

Li-brua requested review from JustinTong0323, wisclmy0611 and zijiexia as code owners May 29, 2026 03:50

github-actions Bot added the documentation Improvements or additions to documentation label May 29, 2026

Li-brua force-pushed the librua branch from 1549941 to b285657 Compare May 29, 2026 04:06

alphabetc1 assigned stmatengss and huangtingwei9988 Jun 2, 2026

This was referenced Jun 2, 2026

[CI Monitor] Daily Report - 2026-06-02 bingxche/sglang-ci-bot#91

Open

[CI Monitor] Daily Report - 2026-06-05 bingxche/sglang-ci-bot#94

Open

stmatengss reviewed Jun 8, 2026

View reviewed changes

Li-brua force-pushed the librua branch from b285657 to 421d6b1 Compare June 9, 2026 07:23

Li-brua requested a review from alphabetc1 as a code owner June 9, 2026 07:23

github-actions Bot added the run-ci label Jun 10, 2026

stmatengss reviewed Jun 11, 2026

View reviewed changes

Comment thread python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_store.py

stmatengss reviewed Jun 11, 2026

View reviewed changes

Li-brua force-pushed the librua branch 2 times, most recently from 22d2b89 to abbeb27 Compare June 12, 2026 06:20

Fix Mooncake logical KV anchor registration

ac6c583

Li-brua force-pushed the librua branch from abbeb27 to ac6c583 Compare June 12, 2026 06:32

amd-bot mentioned this pull request Jun 14, 2026

[CI Monitor] Daily Report - 2026-06-14 bingxche/sglang-ci-bot#104

Open

stmatengss mentioned this pull request Jul 1, 2026

[HiCache] Unified Mooncake Registration for Logical Anchors and Draft Pools #29859

Open

5 tasks

benmyles mentioned this pull request Jul 27, 2026

Harden DeepSeek V4 DSpark DP2 serving shiftup-ai/sglang#1

Draft

21 tasks

Uh oh!

Conversation

Li-brua commented May 29, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Example Commands

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

gemini-code-assist Bot commented May 29, 2026

Uh oh!

stmatengss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Li-brua commented Jun 9, 2026

Uh oh!

stmatengss commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stmatengss commented Jun 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stmatengss commented Jun 12, 2026

Uh oh!

stmatengss commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Li-brua commented May 29, 2026 •

edited by github-actions Bot

Loading

stmatengss commented Jun 10, 2026 •

edited

Loading