Skip to content

[HiCache] Make LayerSplit MLA cache backup CP-aware - #37234

Draft
dblate wants to merge 2 commits into
sgl-project:mainfrom
dblate:glm52-cp-mooncake
Draft

dblate wants to merge 2 commits into
sgl-project:mainfrom
dblate:glm52-cp-mooncake

Conversation

@dblate

@dblate dblate commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Motivation

Make GLM-5.2 LayerSplit context-parallel cache backup compatible with page-first HiCache storage and Mooncake L3. CP ranks own distinct token shards, so each rank must publish its cache under rank-scoped keys.

Depends on #37231 for the per-layer LF-to-PF transfer operator.

Modifications

  • Keep MLA storage backup enabled on every CP rank while preserving the single-writer behavior without CP.
  • Add CP rank/size suffixes to Mooncake object keys and group IDs.
  • Use the per-layer LF-to-PF fallback for layer-sharded MLA and DSA page-first write-back.
  • Add CPU unit coverage for backup selection, Mooncake key isolation, and fallback dispatch.

Accuracy Tests

  • H20, CUDA 12.9: LayerSplit MLA page-first fallback -> 1 passed.
  • H20, CUDA 12.9: DSA indexer page-first fallback -> 1 passed.
  • H20 development environment: CP backup, Mooncake key isolation, and HiCache dispatch -> 24 passed, 1 skipped.
  • git diff --check and Python AST parsing passed for changed files.

Speed Tests and Profiling

No end-to-end serving benchmark was run for this runtime-only change. The underlying LF-to-PF kernel microbenchmark is reported in #37231.

Checklist

  • Format code with pre-commit.
  • Add unit tests according to the contribution guide.
  • Update documentation according to the contribution guide.
  • Provide targeted NVIDIA GPU validation.
  • Run GLM-5.2 CP + HiCache + Mooncake end-to-end validation.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #33382072906
Latest PR Test (Extra): ❌ Run #33382072655
Latest PR Test (AMD ROCm 7.2): ❌ Run #33382072894

@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Aug 31, 2026
@dblate
dblate marked this pull request as ready for review August 31, 2026 09:46
@dblate

dblate commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

@dblate

dblate commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

补充验证结果:在 H20(CUDA 12.9)开发空间运行 LayerSplit MLA page-first fallback,结果为 1 passed;DSA indexer fallback,结果为 1 passed。CP backup、Mooncake key isolation 和 HiCache dispatch 定向单测合计 24 passed, 1 skipped

当前 CI 仍未开始真实测试,原因是缺少上游要求的 run-ci 标签;请维护者按贡献指南执行 /tag-and-rerun-ci

@dblate
dblate force-pushed the glm52-cp-mooncake branch from 85d886f to cff50f6 Compare August 31, 2026 10:21
@dblate
dblate marked this pull request as draft September 1, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant