[Unified Cache][4/N]: Add Mooncake backend for external linker by huangtingwei9988 · Pull Request #37205 · sgl-project/sglang

huangtingwei9988 · 2026-08-31T06:16:24Z

Motivation

This is the fourth PR split from #35687.

Add Mooncake backend for external linker

Previous splits: #37091 and #37098 and #37151

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): 🚫 Run #33383052227
Latest PR Test (Extra): ✅ Run #33383051768
Latest PR Test (AMD ROCm 7.2): ❌ Run #33383052171

Co-authored-by: hzh0425 <hzh0425@apache.org>

huangtingwei9988 · 2026-08-31T06:19:19Z

/tag-and-rerun-ci

huangtingwei9988 · 2026-08-31T06:24:27Z

/rerun-test test/registered/unit/mem_cache/test_unified_cache_linker.py

huangtingwei9988 · 2026-08-31T06:24:38Z

/rerun-group radix_cache/unified_radix_tree

github-actions · 2026-08-31T06:24:51Z

Results for /rerun-test test/registered/unit/mem_cache/test_unified_cache_linker.py:

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/mem_cache/test_unified_cache_linker.py

github-actions · 2026-08-31T06:24:59Z

Results for /rerun-group radix_cache/unified_radix_tree:

🚀 4-gpu-h100 (4 tests): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_hicache_pp_kl.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_cp.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mamba.py

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dcp.py

🚀 8-gpu-h200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4_pp.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_glm52.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mimo.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_swa.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_hybrid_bitexact.py

hzh0425 · 2026-08-31T06:57:28Z

/rerun-group hicache

github-actions · 2026-08-31T06:57:54Z

Results for /rerun-group hicache:

🚀 1-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/hicache/test_hicache_spec_file_storage.py
cd test/ && python3 registered/hicache/test_hicache_variants.py

🚀 2-gpu-h100 (5 tests): ✅ View workflow run

cd test/ && python3 registered/hicache/test_hicache_spec_mooncake_storage.py
cd test/ && python3 registered/hicache/test_hicache_storage_3fs_backend.py
cd test/ && python3 registered/hicache/test_hicache_storage_file_backend.py
cd test/ && python3 registered/hicache/test_hicache_storage_mooncake_backend.py
cd test/ && python3 registered/hicache/test_hicache_storage_runtime_attach_detach.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/hicache/test_hicache_storage.py

🚀 4-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/hicache/test_pp_with_hicache.py
cd test/ && python3 registered/hicache/test_qwen35_hicache.py

⛔ registered/hicache/test_hicache_storage_umbp_backend.py: test/registered/hicache/test_hicache_storage_umbp_backend.py is registered for AMD (suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash), not for CUDA or CPU; rerun-test.yml has no AMD job. Rerun it with /rerun-failed-ci, or dispatch the AMD workflow manually.

huangtingwei9988 · 2026-08-31T08:05:36Z

/rerun-test test/registered/unit/mem_cache/test_mooncake_direct_linker.py

huangtingwei9988 · 2026-08-31T08:06:10Z

/rerun-group radix_cache/unified_radix_tree

github-actions · 2026-08-31T08:06:10Z

Results for /rerun-test test/registered/unit/mem_cache/test_mooncake_direct_linker.py:

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/mem_cache/test_mooncake_direct_linker.py

github-actions · 2026-08-31T08:06:33Z

Results for /rerun-group radix_cache/unified_radix_tree:

🚀 4-gpu-h100 (4 tests): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_hicache_pp_kl.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_cp.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mamba.py

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dcp.py

🚀 8-gpu-h200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4_pp.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_glm52.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mimo.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_swa.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_hybrid_bitexact.py

huangtingwei9988 · 2026-09-01T01:34:59Z

/rerun-failed-ci

hzh0425 · 2026-09-01T01:53:34Z

/rerun-test registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4.py test_unified_radix_cache_kl_dsv4_pp.py

github-actions · 2026-09-01T01:53:58Z

Results for /rerun-test registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4.py test_unified_radix_cache_kl_dsv4_pp.py:

🚀 4-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4_pp.py

Wires the Mooncake direct external linker (PR sgl-project#37205 backend, sgl-project#37307 translator passthrough) into the serving stack as an opt-in direct-L3 mode for UnifiedRadixCache, with no host cache tier: - server_args: --enable-unified-cache-external-linker (default FALSE, runtime stays off unless explicitly enabled) + --unified-cache-external-linker-backend {mooncake}. - arg_groups/hicache_hook: mutual-exclusion guard against --enable-hierarchical-cache and --hicache-storage-backend; linker mode skips hicache normalization entirely. - registry: route to _create_unified_radix_cache under the flag and attach MooncakeDirectLinker via init_cache_linker, registering its layer-done counter with the kv cache and tp worker. - scheduler: consolidate aborted-request cache-state release into _release_aborted_request (fires for hicache storage or the external linker); poll linker events, and gate NO_TOKEN / hicache-consumer batch behavior under the linker flag. - kv_cache_builder: forward attn_cp_rank / attn_cp_size into CacheInitParams (consumed by the storage config suffix). - kv_cache_configurator: keep the DSA indexer-K cache under the linker. - full_component: honor the PREPARE phase of ExternalLinkerLoadPhase (return the transfer unmodified; assert COMMIT after it). - tests: MooncakeTestServices lifecycle helper, DSV4-Flash and GLM-5.2 direct-linker KL E2E suites (registered as extra-b CUDA CI), plus the unit-test fixtures for the new scheduler/registry fields. Applied cleanly over the local branch (no textual conflicts; scheduler abort consolidation coexists with the local chunked-prefill abort re-dispatch and fail-soft changes). Default-off runtime behavior is preserved per the port-but-don't-switch decision. (cherry picked from commit d2ade53, PR sgl-project#37381)

…roject#37205) Co-authored-by: hzh0425 <hzh0425@apache.org>

Add Mooncake backend for external linker

af420d9

Co-authored-by: hzh0425 <hzh0425@apache.org>

github-actions Bot added the hicache Hierarchical Caching for SGLang label Aug 31, 2026

github-actions Bot added the run-ci label Aug 31, 2026

huangtingwei9988 added run-ci-extra unified-radix-cache labels Aug 31, 2026

add ci test & fix

209722c

huangtingwei9988 requested review from Ying1123, alphabetc1, hanming-lu, hnyls2002, hzh0425, ispobock, merrymercy, xiezhq-hermann and yizhang2077 as code owners August 31, 2026 08:04

huangtingwei9988 added 3 commits August 31, 2026 16:33

fix lint

44caf54

fix ci

ebe666d

revert

3f478be

Jialin mentioned this pull request Sep 1, 2026

[Rust TreeCore] Support external cache linker #37306

Merged

5 tasks

hzh0425 merged commit b21000a into sgl-project:main Sep 1, 2026
282 of 351 checks passed

This was referenced Sep 1, 2026

[Unified Cache][5/N]: Integrate external linker mode end to end #37381

Merged

[Unified Cache]: Add optional external-cache linker mode to Unified Radix Cache #35687

Open

maning00 mentioned this pull request Sep 2, 2026

[Unified Cache][6/N]: Add UMBP external linker #37578

Merged

5 tasks

huangtingwei9988 mentioned this pull request Sep 4, 2026

[Unified Cache][7/N] Support MTP, EAGLE, and DSpark draft KV caches in the external linker #37914

Merged

5 tasks

StevenChenSE pushed a commit to StevenChenSE/sglang that referenced this pull request Sep 6, 2026

[Unified Cache][4/N]: Add Mooncake backend for external linker (sgl-p…

c850c8e

…roject#37205) Co-authored-by: hzh0425 <hzh0425@apache.org>

Uh oh!

Conversation

huangtingwei9988 commented Aug 31, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

huangtingwei9988 commented Aug 31, 2026

Uh oh!

huangtingwei9988 commented Aug 31, 2026

Uh oh!

huangtingwei9988 commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hzh0425 commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huangtingwei9988 commented Aug 31, 2026

Uh oh!

huangtingwei9988 commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huangtingwei9988 commented Sep 1, 2026

Uh oh!

hzh0425 commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

huangtingwei9988 commented Aug 31, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Aug 31, 2026 •

edited

Loading

github-actions Bot commented Sep 1, 2026 •

edited

Loading