[Test] KVShard: Extend HiCache E2E with GLM accuracy coverage

docs/docs.json

@@ -955,6 +955,7 @@
               "docs/advanced_features/reasoning_aware_compression",
               "docs/advanced_features/dp_dpa_smg_guide",
               "docs/advanced_features/prefill_cp",
+              "docs/advanced_features/kv_cache_sharing",
               "docs/advanced_features/dcp",
               "docs/advanced_features/expert_parallelism",
               "docs/advanced_features/paged_experts",

docs/docs/advanced_features/kv_cache_sharing.mdx

@@ -0,0 +1,197 @@
+---
+title: "KV Cache Sharing"
+metatags:
+    description: "Page-based KV cache sharding for prefill context parallelism, CPU HiCache, and P/D serving."
+---
+
+KV cache sharing reduces duplicate KV storage across GPUs in a prefill worker. It keeps one persistent copy of each cache page within the shard group, allowing the same GPU memory budget to retain more unique tokens.
+
+[Prefill context parallelism (CP)](/docs/advanced_features/prefill_cp) distributes prompt computation, but can still leave the cached history replicated across ranks. KV cache sharing distributes those stored pages as well. This is useful for long-context and multi-turn workloads that reuse large prefixes.
+
+## How it works
+
+### Shared logical tree, distributed pages
+
+Every rank keeps the same logical radix tree and request-to-token mapping. A prefix match therefore identifies the same cached tokens on all ranks, while each GPU stores only the pages it owns.
+
+For example, a CP4 group can distribute eight logical pages as follows:
+
+| Rank | Stored pages |
+| --- | --- |
+| 0 | A, E |
+| 1 | B, F |
+| 2 | C, G |
+| 3 | D, H |
+
+A request that reuses A–D matches the same logical prefix on every rank. New pages continue the cached path's ownership sequence, and prefixes remain reusable at the original page size.
+
+### Gather one layer at a time
+
+Before attention reads a cached prefix, each rank contributes its owned pages to an all-gather. Attention reads the assembled prefix from a temporary layer buffer, using a page-table mapping to preserve the logical token order.
+
+Two buffers alternate across layers so the next gather can overlap with current-layer computation. Their memory is included in the cache budget. A prefix hit saves recomputation, but still incurs the cost of gathering the distributed KV.
+
+### HiCache and MTP integration
+
+[HiCache](/docs/advanced_features/hicache) preserves page ownership during GPU-to-host backup and host-to-GPU reload. Each rank backs up and restores its own pages. For each layer, execution follows this order:
+
+```text Read order
+Restore local pages → Gather the prefix → Run attention
+```
+
+With GLM MTP, the target and draft KV/indexer state follow the same ownership plan and are restored together. During [P/D transfer](/docs/advanced_features/pd_disaggregation), each prefill rank sends the pages it owns; decode receives complete, unsharded cache entries.
+
+## Usage example
+
+Enable KV cache sharing with `--enable-kv-cache-sharding` on the prefill worker only.
+
+The following example uses Linux with NVIDIA CUDA on two GB300 nodes, with four GPUs per worker. Prefill uses TP4/CP4 and CPU HiCache; decode uses TP4/EP4 with attention DP4. Both workers use the same GLM-5.3 NVFP4 checkpoint and model revision.
+
+### Environment variables
+
+Create `kv-shard.env` on both nodes. Replace the example IP addresses and RDMA NIC names with values for your environment. This example uses Mooncake RDMA for P/D transfer.
+
+```bash Environment variables
+cat > kv-shard.env <<'EOF'
+export MODEL_PATH=RadixArk/GLM-5.3-NVFP4
+export MODEL_REVISION=6e389189d564d9d3e4ea7fa284fe91f136d8ae2a
+export PREFILL_HOST=192.0.2.10
+export DECODE_HOST=192.0.2.20
+export PREFILL_IB_DEVICES=mlx5_0,mlx5_1,mlx5_2,mlx5_3
+export DECODE_IB_DEVICES=mlx5_0,mlx5_1,mlx5_2,mlx5_3
+export CUDA_VISIBLE_DEVICES=0,1,2,3
+export SGLANG_DISAGG_STAGING_BUFFER=0
+export MC_TCP_ENABLE_CONNECTION_POOL=true
+unset MC_FORCE_MNNVL SGLANG_MOONCAKE_CUSTOM_MEM_POOL
+EOF
+```
+
+### Prefill worker: TP4/CP4
+
+Run this command on the prefill node. It enables interleave CP, page-based KV sharding, CPU HiCache, and EAGLE MTP.
+
+```bash Prefill worker
+source ./kv-shard.env
+export SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python
+python -m sglang.launch_server \
+  --model-path "$MODEL_PATH" \
+  --revision "$MODEL_REVISION" \
+  --served-model-name glm-kv-shard \
+  --trust-remote-code \
+  --host 0.0.0.0 \
+  --port 30001 \
+  --disaggregation-mode prefill \
+  --disaggregation-transfer-backend mooncake \
+  --disaggregation-bootstrap-port 30200 \
+  --disaggregation-ib-device "$PREFILL_IB_DEVICES" \
+  --nccl-port 29501 \
+  --tp 4 \
+  --base-gpu-id 0 \
+  --enable-prefill-cp \
+  --attn-cp-size 4 \
+  --cp-strategy interleave \
+  --enable-kv-cache-sharding \
+  --dsa-prefill-backend trtllm \
+  --dsa-decode-backend trtllm \
+  --kv-cache-dtype fp8_e4m3 \
+  --page-size 64 \
+  --mem-fraction-static 0.9 \
+  --chunked-prefill-size 16384 \
+  --max-running-requests 32 \
+  --enable-hierarchical-cache \
+  --hicache-size 240 \
+  --hicache-host-memory-mode cache \
+  --hicache-write-policy write_through \
+  --hicache-io-backend kernel \
+  --hicache-mem-layout page_first \
+  --bf16-gemm-backend cutedsl \
+  --moe-runner-backend flashinfer_cutlass \
+  --disable-shared-experts-fusion \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 3 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 4 \
+  --enable-metrics \
+  --enable-metrics-for-all-schedulers \
+  --enable-cache-report
+```
+
+### Decode worker: TP4/EP4
+
+Run this command on the decode node. The target uses attention DP4 and FlashInfer A2A with EP4; the EAGLE draft uses the TP-only path. Decode does not enable KV cache sharding.
+
+```bash Decode worker
+source ./kv-shard.env
+python -m sglang.launch_server \
+  --model-path "$MODEL_PATH" \
+  --revision "$MODEL_REVISION" \
+  --served-model-name glm-kv-shard \
+  --trust-remote-code \
+  --host 0.0.0.0 \
+  --port 30003 \
+  --disaggregation-mode decode \
+  --disaggregation-transfer-backend mooncake \
+  --disaggregation-bootstrap-port 30200 \
+  --disaggregation-ib-device "$DECODE_IB_DEVICES" \
+  --nccl-port 31000 \
+  --tp 4 \
+  --base-gpu-id 0 \
+  --attn-dp-size 4 \
+  --ep-size 4 \
+  --enable-dp-lm-head \
+  --enable-dp-attention-local-control-broadcast \
+  --dsa-prefill-backend trtllm \
+  --dsa-decode-backend trtllm \
+  --kv-cache-dtype fp8_e4m3 \
+  --page-size 64 \
+  --mem-fraction-static 0.9 \
+  --max-running-requests 512 \
+  --disable-radix-cache \
+  --bf16-gemm-backend cutedsl \
+  --moe-a2a-backend flashinfer \
+  --moe-runner-backend flashinfer_trtllm_routed \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 3 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 4 \
+  --speculative-moe-a2a-backend none \
+  --speculative-moe-runner-backend flashinfer_trtllm \
+  --enable-metrics \
+  --enable-metrics-for-all-schedulers \
+  --enable-cache-report
+```
+
+### Router
+
+After both workers are healthy, start the router on a host that can reach them. The worker HTTP endpoints, prefill bootstrap port, and RDMA transport must be reachable between nodes.
+
+```bash Router
+source ./kv-shard.env
+curl --fail "http://${PREFILL_HOST}:30001/health"
+curl --fail "http://${DECODE_HOST}:30003/health"
+
+python -m sglang_router.launch_router \
+  --pd-disaggregation \
+  --prefill "http://${PREFILL_HOST}:30001" 30200 \
+  --decode "http://${DECODE_HOST}:30003" \
+  --policy cache_aware \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+### Send a request
+
+From the router host, send a request using the shared served model name:
+
+```bash Request
+curl --fail http://127.0.0.1:30000/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"glm-kv-shard","messages":[{"role":"user","content":"What is 17 + 25?"}],"temperature":0,"max_tokens":128}'
+```
+
+## Related documentation
+
+- [Prefill context parallelism](/docs/advanced_features/prefill_cp)
+- [HiCache design](/docs/advanced_features/hicache_design)
+- [P/D disaggregation](/docs/advanced_features/pd_disaggregation)
+- [GLM-5.3 cookbook](/cookbook/autoregressive/GLM/GLM-5.3)

docs/docs/advanced_features/pd_disaggregation.mdx

@@ -74,132 +74,6 @@ destination registration. Older decode workers do not enforce this check. Keep
 `early` on both sides until every worker has been upgraded, then enable the new
 policy on both sides.
 
-## GLM DSA KV sharding with CPU HiCache
-
-On prefill workers, `--enable-kv-cache-sharding` distributes cached pages across
-the attention-CP group. Each owner stores and transfers its target/draft latent
-KV and DSA indexer pages. Optional CPU HiCache retains the same page ownership
-during backup and reload. Decode workers receive complete caches and do not
-enable KV sharding.
-
-Use this combination on Linux with NVIDIA CUDA and hardware supported by the
-TRTLLM DSA backends. The `nvidia/GLM-5.2-NVFP4` example requires Blackwell GPUs
-with enough memory for both target and draft weights on each side.
-
-| Setting | Supported configuration |
-| --- | --- |
-| Prefill parallelism | Interleave CP greater than one; pipeline parallel size one |
-| Attention and KV | TRTLLM DSA prefill/decode, `fp8_e4m3`, page size 64 |
-| DSA indexer | Uncompressed `index_kpool=1`; no cache layer splitting |
-| P/D transfer | Mooncake, with staging buffers disabled |
-| Optional MTP | GLM DSA single-layer NextN, EAGLE, top-k one |
-| Optional CPU HiCache | `cache` mode, `kernel` I/O, `layer_first` or `page_first` |
-
-KV-sharded serving selects the Python radix tree when no tree backend is
-explicitly configured. L3 storage (including runtime attachment), external cache
-linkers, HiSparse and two-batch overlap are unsupported. Prefill CUDA graphs,
-P-side speculative graphs and generic FlashInfer autotuning are disabled for
-this path. Scratch buffers for target and draft are charged against the KV
-memory budget, so the usable cache capacity also depends on context length.
-
-The following single-host GB300/NVL72 example uses GPUs 0–1 for prefill and 2–3
-for decode. Install the [Mooncake transfer engine](#requirements) first. Both
-workers use fabric NVLink with Mooncake's NVLink allocator, which makes their
-KV buffers exportable. Enabling fabric transport alone does not make ordinary
-CUDA allocations remotely accessible. For other hardware, configure the
-appropriate [RDMA or NVLink transport](#nvlink-transport-configuration).
-
-Run prefill and decode in separate terminals on the same host.
-
-### Prefill worker (GPUs 0–1)
-
-```bash
-export SGLANG_DISAGG_STAGING_BUFFER=0
-export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK
-export MC_FORCE_MNNVL=true
-
-python3 -m sglang.launch_server \
-  --model-path nvidia/GLM-5.2-NVFP4 \
-  --trust-remote-code \
-  --tp-size 2 \
-  --context-length 16384 \
-  --max-running-requests 1 \
-  --mem-fraction-static 0.85 \
-  --page-size 64 \
-  --kv-cache-dtype fp8_e4m3 \
-  --dsa-prefill-backend trtllm \
-  --dsa-decode-backend trtllm \
-  --speculative-algorithm EAGLE \
-  --speculative-num-steps 3 \
-  --speculative-eagle-topk 1 \
-  --speculative-num-draft-tokens 4 \
-  --disaggregation-transfer-backend mooncake \
-  --disaggregation-bootstrap-port 8998 \
-  --enable-metrics \
-  --enable-cache-report \
-  --disaggregation-mode prefill \
-  --port 30001 \
-  --nccl-port 29501 \
-  --enable-kv-cache-sharding \
-  --enable-prefill-cp \
-  --attn-cp-size 2 \
-  --cp-strategy interleave \
-  --chunked-prefill-size 2048 \
-  --max-total-tokens 4096 \
-  --enable-hierarchical-cache \
-  --hicache-ratio 4 \
-  --hicache-write-policy write_through \
-  --hicache-io-backend kernel \
-  --hicache-mem-layout page_first
-```
-
-### Decode worker (GPUs 2–3)
-
-```bash
-export SGLANG_DISAGG_STAGING_BUFFER=0
-export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK
-export MC_FORCE_MNNVL=true
-
-python3 -m sglang.launch_server \
-  --model-path nvidia/GLM-5.2-NVFP4 \
-  --trust-remote-code \
-  --tp-size 2 \
-  --context-length 16384 \
-  --max-running-requests 1 \
-  --mem-fraction-static 0.85 \
-  --page-size 64 \
-  --kv-cache-dtype fp8_e4m3 \
-  --dsa-prefill-backend trtllm \
-  --dsa-decode-backend trtllm \
-  --speculative-algorithm EAGLE \
-  --speculative-num-steps 3 \
-  --speculative-eagle-topk 1 \
-  --speculative-num-draft-tokens 4 \
-  --disaggregation-transfer-backend mooncake \
-  --disaggregation-bootstrap-port 8998 \
-  --enable-metrics \
-  --enable-cache-report \
-  --disaggregation-mode decode \
-  --port 30002 \
-  --nccl-port 29502 \
-  --base-gpu-id 2 \
-  --max-total-tokens 16384 \
-  --cuda-graph-max-bs-decode 1
-```
-
-Connect the two endpoints through the [P/D router](#router-integration).
-To check CPU-cache reuse, repeat a prompt after enough unrelated requests have
-evicted its GPU prefix: `cached_tokens_details.host` and the prefill
-`sglang:load_back_tokens_total` counter must increase. A device-cache hit alone
-does not exercise the host reload path.
-
-When comparing with replicated or deduplicated KV, keep the model revision,
-hardware, P/D topology, memory budget, request order, concurrency and warm-up
-policy fixed. Report device/host cache-hit tokens, TTFT percentiles, throughput,
-peak memory and answer correctness. Additional cache capacity alone does not
-establish a latency or throughput gain; use a representative replay and the
-[benchmarking guide](/docs/developer_guide/benchmark_and_profiling).
-
 ## Profiling in PD Disaggregation Mode
 
 When you need to profile prefill or decode workers in PD disaggregation mode, please refer to the [Profile In PD Disaggregation Mode](../developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode) section in the Benchmark and Profiling guide. Due to torch profiler limitations, prefill and decode workers must be profiled separately using dedicated command-line options.

docs/docs/advanced_features/server_arguments.mdx

@@ -2927,7 +2927,7 @@ Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `dec
     </tr>
     <tr>
       <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-kv-cache-sharding`</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Shard at-rest KV cache storage at page granularity across the group that currently replicates it (attention-CP ranks under prefill CP; attention-TP ranks for MLA models without prefill CP), multiplying the group's unique KV capacity. Prefill workers of PD disaggregation only. See <a href="/docs/advanced_features/pd_disaggregation#glm-dsa-kv-sharding-with-cpu-hicache">GLM DSA KV sharding with CPU HiCache</a> for DSA/MTP requirements and a launch example. With NCCL 2.31.2 or newer and supported NVIDIA CUDA hardware, eligible single-node shard groups request zero-SM copy-engine prefix all-gather on a dedicated NCCL communicator. Older NCCL versions keep the SM-kernel all-gather. This path needs cuMem-backed NCCL memory, so on single-node deployments the server sets <code>NCCL_CUMEM_ENABLE=1</code> for every NCCL communicator of all server processes unless <code>NCCL_CUMEM_ENABLE</code> is already set; it does not under <code>--enable-memory-saver</code> or <code>SGLANG_MOONCAKE_CUSTOM_MEM_POOL</code>, which keep the SM-kernel all-gather. Multi-node deployments keep NCCL's setting (MNNVL fabrics default it to 1); export <code>NCCL_CUMEM_ENABLE=1</code> there to use this path for shard groups within one node. Set <code>SGLANG_ENABLE_KV_SHARD_ZERO_SM_GATHER=0</code> to disable this path. Leave <code>NCCL_CTA_POLICY</code> unset: a global value overrides both the gather communicator's zero-SM policy and the default policy of forward collectives.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Shard KV cache pages across GPUs to reduce duplication and increase cache capacity. Only supported on P/D prefill workers. See <a href="/docs/advanced_features/kv_cache_sharing">KV cache sharing</a> for requirements and examples.</td>
       <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
       <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
     </tr>