[HZY-Wade]

Motivation

Co-author: @Shunkangz

Connect GLM DSA/MTP serving to page-sharded KV caches so prefill, P/D transfer and CPU HiCache preserve the same ownership for target/draft latent KV and DSA indexer keys/scales.

Dependencies and review scope

Base: upstream main 5987ec82ed. Current head: 346068aaa329cf1692fbb6031f60469fdac77dcb.

DSA attention/indexer support (#44280), MTP pool/scratch support (#40929), and the control-plane/pool/basic-PD prerequisites are already merged.

The remaining unmerged prerequisite is #43778, pinned at 6d1e510b2429a36fe499943ff075472d995d8457. Its three implementation commits are rebased onto this main base with Shunkang's author attribution preserved. The dependency snapshot is 152e93348ad825449c484fbee2068fc614116c67.

Review the integration increment: HZY-Wade/sglang@codex/kvshard-glm-dependencies-20261026-clean...codex/kvshard-dsa-hicache

GitHub's main-based Files changed view also includes #43778. Rust support is separate in #46445; this PR contains no Rust implementation and selects Python for sharded launches when the tree backend is unset.

Modifications

• Select page-sharded DSA target and single-layer GLM NextN draft pools. Validate matching cache geometry and budget latent/indexer scratch for both runners.
• Preserve the shared sender lifecycle. Pair source/destination DSA pages before owner filtering, transfer every owner's target/draft state, retain empty-final-chunk completion, and account for physical transfer bytes.
• Compose a standalone sharded allocation facade with native host pools. Reuse HybridCacheController and L2TransferEngine; resolve owner-local indices through existing preparation hooks before the H2D start event. Preserve logical queue indices and ACK counts.
• Scope scratch bookkeeping to sharding, cache the controller's sharding flag, retain ordinary transfer descriptor builders, and remove the redundant FULL eviction retry.
• Pass start_owner to backups only when the tree node has a shard owner. Preserve owner zero and the non-sharded controller's existing call signature.
• Add a four-GPU GLM P2/D2 E2E with CP2, EAGLE MTP and CPU HiCache. Use fabric NVLink with Mooncake's exportable KV allocator, matching the existing P2/D2 test on the same GB300 runner. CPU tests follow the registered CustomTestCase convention.

Supported configuration: CUDA PD-prefill, interleave CP > 1, PP1, TRTLLM DSA prefill/decode, FP8 E4M3/page64, Mooncake, and uncompressed index_kpool=1. Optional MTP is single-layer GLM NextN EAGLE with topk=1. CPU HiCache uses cache mode, kernel I/O and layer_first/page_first. L3, external linkers, DSA layer splitting, HiSparse and two-batch overlap are outside this combination. Decode remains unsharded.

Configuration requirements, paired launch example and validation guidance

Accuracy Tests

• Runtime regression on f01f901a6a (runtime source unchanged by the following documentation/fixture commits): 200 tests and 140 subtests passed, 2 skipped, 1 deselected. This covers owner-aware PD transfer, allocator/rotation invariants, scratch sizing, scheduler admission, host-transfer dispatch, launch compatibility and controller event ordering.
• Both new CPU test files also pass when executed directly with fail-fast, as CI invokes them: 4 PD transfer tests and 2 controller tests.
• The unsharded host-prefix backup setup independently reproduces the published head's unexpected keyword argument 'start_owner' failure and passes with 8620dbdaa1.
• Local CPU runs bypass only the Mac Torch/MPS version preflight. A platform-specific NPU host-pool test is deselected because the local host-memory reserve cannot be met; GPU-dependent tests are skipped. These runs do not validate CUDA/NCCL or physical host transfers.
• Pre-commit and git diff --check pass. Mintlify 4.2.559 reports no broken links with anchor/redirect checks. Both documented launch commands pass the CLI parser and their shell block passes bash -n.

Hardware E2E status

Passed on current head 346068aaa329cf1692fbb6031f60469fdac77dcb, Base attempt 2.

• Four-GB300 hardware job: test_dsa_glm52_pd_kv_shard_hicache.py actually ran and passed, with a 273-second suite duration. The test method completed with Ran 1 test ... OK; this was not a collection-only or skipped result.
• The test retains P2/D2, CP2, EAGLE MTP and all cold/device-hit/host-reloaded answer, speculative-verification, host-hit and rank-specific H2D assertions. Two 6,144-token pressure requests exceed the 8,192 logical GPU slots while fitting with the target prefix in the larger host cache.
• CPU partition 17: all 52 files passed, including test_decode_hicache_l2.py (36 seconds), which previously exposed the non-sharded backup signature regression.
• The existing AARCH64 P2/D2 accuracy test also passed in the same GB300 job (89 seconds). This fixture uses its supported fabric configuration: MC_FORCE_MNNVL=true and SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK. The NVLink allocator supplies exportable KV buffers; the runner's wheel does not provide nvlink_intra.

The launch example has separate prefill/decode commands with one option per line, addressing review comment 4379290548. CLI arguments are unchanged; both examples match the E2E transport settings. Shell syntax, CLI parsing, allocator selection equivalence, pre-commit and Mintlify anchor/link checks pass.

The separate existing GLM-5.2 layer-split accuracy rerun passed at 0.9422632794457275, above the unchanged 0.935 GSM8K threshold; the full job passed 2/2 test files. The previous attempt scored 0.9345650500384911. Both attempts tested merge e79b8b8c52735038eb60edcacff7a3e6f24ea69e with KV sharding disabled, and no code or threshold changes were made between them. The successful rerun does not establish the cause of the score variation.

On current head, Base attempt 3 and Extra attempt 3 are both successful. Arm64 and Xeon retain separately investigated failures in unchanged CPU-operator/parser paths; the complete platform matrix is not all green.

Speed Tests and Profiling

A controlled production replay is still pending; no throughput or latency speedup is claimed.

For the requested comparison, use the same model revision, hardware, runtime, P/D topology, memory budget, request order, concurrency and warm-up policy. Report device/host hit tokens, TTFT percentiles, throughput, peak memory and answer correctness for replicated/deduplicated KV and KV sharding. CPU unit-test counts and increased logical capacity are not performance measurements.

Checklist

• Format and pre-commit checks.
• Focused regression coverage and CI registration.
• Configuration, launch and validation documentation.
• Preserve non-sharded call paths and follow composition/preparation-hook review feedback.
• Pass the corrected full-model P/D + MTP + HiCache E2E on hardware.
• Provide controlled production accuracy/performance replay results.
• Pass the separate GLM layer-split accuracy evaluation at the unchanged 0.935 threshold.
• Resolve the remaining [Feat] KVShard: Support HiCache with KV Cache Sharding #43778 dependency and required CI/maintainer approvals.

---

CI States

Latest PR Test (Base): ✅ Run #40354128180
Latest PR Test (Extra): ✅ Run #40354126514
Latest PR Test (AMD ROCm 10): ⏳ Run #40354127190

[mintlify]

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 346068a.

Approve preview

[HZY-Wade]

/rerun-failed-ci

[hzh0425]

should optimize this command's style, like:

DeepSeek Multi-Node

# prefill 0
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-ib-device ${device_name} \
  --disaggregation-mode prefill \
  --host ${local_ip} \
  --port 30000 \
  --trust-remote-code \
  --dist-init-addr ${prefill_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 0 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8
# prefill 1
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-ib-device ${device_name} \
  --disaggregation-mode prefill \
  --host ${local_ip} \
  --port 30000 \
  --trust-remote-code \
  --dist-init-addr ${prefill_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 1 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8

[HZY-Wade]

/rerun-failed-ci

[HZY-Wade]

/rerun-failed-ci

[HZY-Wade]

Additional GLM-5.3 accuracy evaluation results from my run, using the RadixArk/GLM-5.3-NVFP4 weights:

Benchmark	Score
agentic3-TB-2-1	88.76
aime_AIME2025	95.56

[hzh0425]

/rerun-test test/registered/unit/disaggregation/test_dsa_sharded_pd_transfer.py

[hzh0425]

[github-actions]

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/disaggregation/test_dsa_sharded_pd_transfer.py

[hzh0425]

/rerun-test -c

[github-actions]

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_dsa_glm52_pd_kv_shard_hicache.py

🚀 ubuntu-latest (6 tests): ✅ View workflow run

cd test/ && python3 registered/unit/disaggregation/test_dsa_sharded_pd_transfer.py
cd test/ && python3 registered/unit/managers/test_prefill_adder.py
cd test/ && python3 registered/unit/mem_cache/test_hicache_staged_write_back_dispatch.py
cd test/ && python3 registered/unit/mem_cache/test_page_interleave_shard.py
cd test/ && python3 registered/unit/mem_cache/test_sharded_dsa_hicache.py
cd test/ && python3 registered/unit/server_args/test_server_args.py

[hzh0425]

thanks @HZY-Wade
need to add a accuracy hicache + sharding e2e test in next pr