[PD] Bound cached-prefix DCP transfers by pack capacity by kpham-sgl · Pull Request #40376 · sgl-project/sglang

kpham-sgl · 2026-09-19T19:50:11Z

Motivation

A prefill radix hit can leave very little compute while the decode worker still needs a long KV prefix. DCP pack buffers introduced in #35762 are sized from the prefill compute-chunk bound, but this cached-prefix send can be much larger. It then falls back to per-token RDMA and loses most of the packing benefit.

Split these sends to fit the existing allocation. This extends the scheduler grid-splitting mechanism from #30545 to DCP packing, where it was previously bypassed unless hetero-TP staging was enabled.

Modifications

Publish the page-aligned token capacity used to allocate DCP pack buffers through the sender only for requests whose peers require DCP relayout, and split oversized scheduler sends at that capacity. Plain TP requests remain unsplit even when the same prefill worker serves DCP decoders. Preserve token/page offsets, partial final pages, and final-segment-only state/metadata.
Check NIXL's per-DCP-rank region capacity in addition to the total allocation, so a gather cannot overwrite an adjacent rank's packed region.
Add scheduler/pack regression tests, warm-prefill/cold-decode generation parity coverage, and a standalone cached-prefix TTFT benchmark client.

No new flags. The existing allocation formula is retained; on the measured configuration it remains 4 × 31.5 MiB per prefill rank.

Accuracy Tests

GPU and performance measurements used eight B200s with lmsysorg/sglang:dev, base 993d1fccbaafe3e79d91567d2fc1d665cc94fa50, and candidate 55d10370c7. The latest peer-scoping and test changes were validated with the CPU suite and mutation checks below; the GPU matrix was not rerun.

75 CPU tests + 11 subtests passed on the latest revision with CUDA disabled in the dev image: DCP packing, NIXL basic, Mooncake transfer batching, and staging draft KV slots. The existing cached-prefix test now exercises the real allocation-size calculation, sender limit lookup and send bookkeeping. It checks bounded DCP sends, unsplit TP sends on the same manager, nonzero decode prefixes, page offsets, and final-state delivery. The mixed-peer cases fail on the previous PR revision (8/9 sends instead of 1). Removing capacity publication makes the DCP cases fail. The existing region test now calls through NIXL; removing its capacity argument makes the overflow assertion fail. No new test cases or files are added.
NIXL: all 6 Kimi Linear GPU acceptance tests passed, including GSM8K 0.91 over 200 examples, page/chunk boundary parity, graph/eager batches, and 65,536-token needle prompts at three depths. The added cached-prefix case clears only decode's cache, confirms a large prefill cache hit, then compares generated tokens and logprobs against a monolithic reference with up to 16 output tokens.
Mooncake over TCP: all 6 acceptance tests passed with the same 65,536-token cached-prefix coverage. The initial RDMA run hit NIC-to-NIC transport retry-counter failures and was interrupted; this PR does not claim a passing Mooncake RDMA run on this devbox.
All applicable pre-commit hooks passed.

Speed Tests and Profiling

Kimi Linear 48B-A3B Instruct on 8 × B200, NIXL; prefill TP4 on GPUs 0–3, decode TP4 on GPUs 4–7. DCP condition uses decode EP4/DCP4; controls use EP1/DCP1 and EP4/DCP1. FP8 KV, BF16 weights, 64-token pages, 8192-token prefill chunks, tokenspeed_mla, prefill graphs disabled, decode graph max batch 32, max running requests 32. Decode radix caching is disabled in every performance condition.

Warm shapes, flush both worker caches, warm the common prefix, then measure unique suffixes with OSL=1. C1: 12 requests; C8: 32 requests. Arithmetic means of 5 candidate repeats and 2 repeats per control. Each cell is requests/s / mean TTFT (ms).

Cached prefix + unique suffix	Concurrency	Main DCP4	This PR, DCP4	TP4 → TP4	TP4 → TEP4
83,264 + 6,720	1	1.36 / 735.5	10.27 / 97.0	10.40 / 95.7	11.24 / 88.5
83,264 + 6,720	8	1.37 / 5548.8	14.89 / 497.4	15.66 / 470.6	16.43 / 449.8
163,840 + 6,720	1	0.69 / 1455.9	7.33 / 136.0	8.04 / 123.6	7.92 / 125.6
163,840 + 6,720	8	0.70 / 10925.8	11.54 / 641.3	12.07 / 611.0	12.43 / 593.8
0 + 32,768	1	3.80 / 263.2	4.09 / 251.3	4.35 / 229.7	4.20 / 237.7
0 + 32,768	8	4.16 / 1731.9	4.58 / 1568.8	4.51 / 1592.5	4.49 / 1601.5

Zero per-token fallback warnings across all candidate runs. Cached-prefix C8 throughput improves about 10.9× / 16.6× over main and reaches 95.0% / 95.6% of TP4 → TP4. This is near parity on this proxy, not an exact-parity or K3/DCP8 claim. Cold C1 varied substantially between runs, so its small differences should not be treated as a reliable speedup or regression.

The allocation-only change from #40336 was also tested. It increases headroom, but these prefixes with C=8192/D=4 still exceed it: C8 remained at 1.42 / 0.70 requests/s with fallback. This does not contradict its reported improvement for a prefix that fits the larger allocation; bounded sends remove the prompt-length threshold.

A separate instrumented run measured roughly 0.25 ms per gather for index upload, stream setup, helper/launch, and synchronization, excluding initial JIT outliers. Preparing/enqueueing an 11-segment send averaged about 1 ms. Instrumented client timings are excluded from the table.

Reproduction

Start a TP4 prefill worker and the desired TP4 decode configuration behind the PD router. Common worker flags used here:

--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--trust-remote-code --tp-size 4 --attention-backend tokenspeed_mla \
--kv-cache-dtype fp8_e4m3 --dtype bfloat16 --random-seed 0 \
--page-size 64 --chunked-prefill-size 8192 \
--cuda-graph-backend-prefill disabled --cuda-graph-max-bs-decode 32 \
--mem-fraction-static 0.80 --max-running-requests 32 \
--disaggregation-transfer-backend nixl

Prefill: --disaggregation-mode prefill --port 30001 --disaggregation-bootstrap-port 8998 --nccl-port 31001. Decode: --disaggregation-mode decode --port 30002 --base-gpu-id 4 --nccl-port 31002 --ep-size 4 --dcp-size 4. For the controls, use decode EP1/DCP1 or EP4/DCP1.

python -m sglang_router.launch_router --pd-disaggregation --mini-lb \
  --prefill http://127.0.0.1:30001 --decode http://127.0.0.1:30002 --port 30000
python benchmark/disaggregation/bench_cached_prefix.py \
  --prefix 83264 --unique 6720 --concurrency 8 --requests 32 --result cached-c8.json

The client preserves every response, cache-hit count and TTFT. Change prefix to 163840 for the longer-prefix case, or prefix=0/unique=32768 for cold prefill. Set concurrency=1/requests=12 for C1.

GPU correctness: run test/registered/disaggregation/test_kimi_linear_pd_dcp4.py with SGLANG_TEST_PD_LONG_CONTEXT_TOKENS=65536 and SGLANG_TEST_PD_DISAGG_BACKEND=nixl or mooncake.

Checklist

Pre-commit formatting and lint checks.
Unit and GPU regression coverage.
Accuracy and speed measurements.
Code style followed.
Documentation: benchmark usage above; no public configuration changes.

CI States

Latest PR Test (Base): ⏳ Run #35517946934
Latest PR Test (Extra): ❌ Run #35517946812
Latest PR Test (AMD ROCm 10): ⏳ Run #35517946956

kpham-sgl · 2026-09-20T08:30:08Z

/rerun-test test/registered/disaggregation/test_kimi_linear_pd_dcp4.py

github-actions · 2026-09-20T08:30:32Z

Results for /rerun-test test/registered/disaggregation/test_kimi_linear_pd_dcp4.py:

🚀 8-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_kimi_linear_pd_dcp4.py

ShangmingCai

Looks good. Nice fix.

ShangmingCai · 2026-09-20T11:31:36Z

/tag-and-rerun-ci

ShangmingCai · 2026-09-20T12:17:57Z

/rerun-group disaggregagtion

github-actions · 2026-09-20T12:18:54Z

⛔ disaggregagtion: Unknown test group disaggregagtion.

Known groups: accuracy, amd, attention, backends, bench_fn, chunked_prefill, constrained_decoding, core, cp, cpu, cuda_graph, dcp, debug_utils, disaggregation, dllm, dp_attn, dp_engine, e2e, ep, eplb, expert_pack, function_call, gb300, gemm, hicache, input_embedding, kernel, kernels, kv_canary, layers, lora, mem_cache, mla, mlx, mock_model, model_loading, moe, musa, npu, observability, openai_server, ops, page_major, perf, pp, prefill_only, profiling, quant, radix_cache, reasoning, rl, rotary, rust, rust-server, sampling, scheduler, scripted_runtime, sessions, spec, stress, tokenizer, unified-memory, unit, utils, vlm, xpu

ShangmingCai · 2026-09-20T14:55:08Z

/rerun-test disaggregation

github-actions · 2026-09-20T14:55:33Z

Results for /rerun-test disaggregation:

⛔ disaggregation: Ambiguous filename disaggregation — matched 5 files:

test/registered/amd/disaggregation
test/registered/disaggregation
test/registered/e2e/disaggregation
test/registered/kernels/ops/disaggregation
test/registered/unit/disaggregation

Please provide the full path, e.g. /rerun-test test/registered/disaggregation

ShangmingCai · 2026-09-20T15:16:31Z

/rerun-group disaggregation

github-actions · 2026-09-20T15:16:53Z

Results for /rerun-group disaggregation:

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_aarch64.py

🚀 2-gpu-h100 (6 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py
cd test/ && python3 registered/disaggregation/test_disaggregation_chunked_prefill_abort.py
cd test/ && python3 registered/disaggregation/test_disaggregation_decode_offload.py
cd test/ && python3 registered/disaggregation/test_disaggregation_optimistic_prefill.py
cd test/ && python3 registered/disaggregation/test_disaggregation_rust_server.py
cd test/ && python3 registered/disaggregation/test_disaggregation_unified_memory.py

🚀 8-gpu-h20 (5 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_decode_radix_cache.py
cd test/ && python3 registered/disaggregation/test_disaggregation_different_tp.py
cd test/ && python3 registered/disaggregation/test_disaggregation_dp_attention.py
cd test/ && python3 registered/disaggregation/test_disaggregation_nixl.py
cd test/ && python3 registered/disaggregation/test_disaggregation_pp.py

🚀 8-gpu-h200 (4 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_decode_radix_cache_swa.py
cd test/ && python3 registered/disaggregation/test_disaggregation_dsv4.py
cd test/ && python3 registered/disaggregation/test_disaggregation_hisparse.py
cd test/ && python3 registered/disaggregation/test_disaggregation_hybrid_attention.py

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dwdp_gpt_oss.py
cd test/ && python3 registered/disaggregation/test_disaggregation_inkling_mxfp8.py

🚀 4-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_kimi_linear.py
cd test/ && python3 registered/disaggregation/test_epd_disaggregation.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_xpu.py

🚀 8-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_kimi_linear_pd_dcp4.py

Fix oversized cached-prefix DCP transfers with bounded sends

55d1037

kpham-sgl requested review from ByronHsu, Duyi-Wang, HaiShaw, ShangmingCai, hnyls2002 and sogalin as code owners September 19, 2026 19:50

kpham-sgl added run-ci CI: run the baseline test suite on this PR and removed run-ci CI: run the baseline test suite on this PR labels Sep 19, 2026

Consolidate DCP regression coverage [skip ci]

cbf4167

zijiexia mentioned this pull request Sep 19, 2026

[PD] Size the DCP pack buffer per rank, not per rank-shard #40336

Closed

5 tasks done

kpham-sgl added 2 commits September 19, 2026 18:50

Simplify DCP pack capacity selection [skip ci]

da555f7

Scope DCP pack limits to request peers and test real callers [skip ci]

b23a75f

kpham-sgl added 2 commits September 20, 2026 01:30

Clarify prefill cache hit transfer comment [skip ci]

5ccc919

Run lint checks for DCP transfer changes

451ceaf

ShangmingCai approved these changes Sep 20, 2026

View reviewed changes

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 20, 2026

Merge branch 'main' into fix/dcp-cached-prefix-chunks

639f8b9

ShangmingCai merged commit b3e4d19 into sgl-project:main Sep 20, 2026
143 of 159 checks passed

This was referenced Sep 20, 2026

[CI Monitor] Daily Report - 2026-09-20 bingxche/sglang-ci-bot#201

Open

[CI Monitor] Daily Report - 2026-09-21 bingxche/sglang-ci-bot#202

Open

kpham-sgl mentioned this pull request Sep 22, 2026

[DO NOT MERGE] disagg/mooncake: slice-pack oversized PD DCP transfers instead of per-token RDMA fallback #39992

Closed

Navigation Menu

Repository navigation

[PD] Bound cached-prefix DCP transfers by pack capacity - #40376

[PD] Bound cached-prefix DCP transfers by pack capacity#40376
ShangmingCai merged 7 commits into
sgl-project:mainfrom
kpham-sgl:fix/dcp-cached-prefix-chunks

kpham-sgl commented Sep 19, 2026 •

edited by github-actions Bot

Loading

Uh oh!

kpham-sgl commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

ShangmingCai left a comment

Uh oh!

ShangmingCai commented Sep 20, 2026

Uh oh!

ShangmingCai commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026

Uh oh!

ShangmingCai commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

ShangmingCai commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kpham-sgl commented Sep 19, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

CI States

Uh oh!

kpham-sgl commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShangmingCai left a comment

Choose a reason for hiding this comment

Uh oh!

ShangmingCai commented Sep 20, 2026

Uh oh!

ShangmingCai commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026

Uh oh!

ShangmingCai commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShangmingCai commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kpham-sgl commented Sep 19, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Sep 20, 2026 •

edited

Loading

github-actions Bot commented Sep 20, 2026 •

edited

Loading

github-actions Bot commented Sep 20, 2026 •

edited

Loading