[Jzz1943]

Motivation

Enable HiSparse for hybrid DSA/linear-attention models in PD serving. DSA KV uses CPU backing storage and a GPU hot buffer, while KDA state remains on the GPU.

In our PD serving tests with GLM-5.3-Flash on 8 × H200, enabling HiSparse with host_to_device_ratio=2 increased KV capacity by 89% compared with HiSparse disabled. At concurrency 48 with long-context requests, it raises observed D-side concurrency from 34 to 48, improves output throughput by 9.1%, and reduces median TTFT by 84.9%, with a 23.6% increase in mean TPOT.

This PR covers non-speculative decoding. A follow-up PR adding PD disaggregation + HiSparse + MTP support is coming soon.

Modifications

• Connect HiSparse DSA backing pools to hybrid linear KV pools, preserving KDA state access and DSA layer mapping.
• Support KPool tail tokens and TileLang FP8 swap-in, and report logical KV occupancy consistently.
• Defer HiSparse buffer release until transfer drain is confirmed across ranks, preventing reuse while writes may still be in flight.
• Add an opt-in same-node shared-memory D2H path for HiSparse CPU KV with SGLANG_DISAGGREGATION_NIXL_LOCAL_SHM=1 on both P and D. GPU state and notifications remain on NIXL; existing transfer behavior is retained by default.

The shared-memory path initially supports Linux/CUDA, matching TP, PP1/DCP1, and flat MLA/DSA host KV pools without staging or draft KV.

Accuracy Tests

Full GSM8K on GLM-5.3-Flash, using 5-shot, temperature 0 and reasoning_effort=low, without MTP:

Configuration	Accuracy
HiSparse disabled baseline	95.89%
HiSparse enabled, shared-memory KV transfer	95.80%

HiSparse achieves accuracy comparable to the baseline.

166 focused regression tests passed, covering transfer correctness, slot reuse, drain-before-release, peer isolation, cleanup and default-backend compatibility. BF16/FP8 two-process GPU transfers and pre-commit checks passed.

Speed Tests and Profiling

GLM-5.3-Flash on 8 × H200, same-node PD (P: GPUs 0–3; D: GPUs 4–7), TP4/EP4 per endpoint, FP8 TileLang KV, Triton MoE, no MTP. Both configurations use the same GPU memory budget, Prefill BCG and Decode CUDA Graphs. HiSparse uses top_k=2048, a 4096-token device buffer and host_to_device_ratio=2; CPU KV uses shared-memory D2H, while the baseline keeps full KV on GPU via NIXL.

HiSparse increases full KV capacity by 1.89× at similar GPU memory usage, with 50 GB of CPU backing storage per rank:

D configuration	Full KV capacity (tokens per TP group)	Observed maximum running requests
HiSparse disabled	4,707,584	34
HiSparse enabled	8,877,824	48

Workload: 136.5K input / 1K output tokens, with a shared prefix on P and client concurrency 48. Results are medians of three runs after warmup; all requests succeeded.

Metric	HiSparse disabled	HiSparse enabled	Change
Estimated output throughput (tok/s)	2023.59	2207.00	+9.1%
TTFT P50 (ms)	7101.58	1073.18	−84.9%
TTFT P95 (ms)	12513.45	1743.57	−86.1%
Mean TPOT (ms)	16.02	19.80	+23.6%

HiSparse benefits KV-capacity-limited workloads: at C48, it improves throughput by 9.1% and reduces TTFT P50 by 84.9% by alleviating D-side queueing caused by limited KV capacity. The 23.6% higher TPOT may reflect both KV transfer overhead and larger decode batches. At C16/C32, where both configurations fit in GPU KV capacity, HiSparse reduces throughput by roughly 9%.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #41795537367
Latest PR Test (Extra): ❌ Run #41795537022
Latest PR Test (AMD ROCm 10): ❌ Run #41795537087

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

[Jzz1943]

Could you also share end-to-end results comparing local SHM against the default NIXL path with HiSparse on? That would show what the SHM path actually gains.

The current HiSparse off/on results don’t isolate SHM’s contribution. I’ll add a comparison with HiSparse enabled in both cases, toggling only local SHM, and document the NIXL transport used.