[PP+PD] Fix #38206 Reduce bootstrap consensus latency by stepinto · Pull Request #38959 · sgl-project/sglang

stepinto · 2026-09-10T23:51:19Z

Motivation

In current implementation, PP+PD bootstrap ready takes a full round-trip from PP0 to PP(n-1). This contributes to TTFT in large pipeline deployement, e.g. pp_size=8.

This has been reported in #34815 and analyzed in depth in #38206. PR #36613 was a different solution to the issue.

This PR proposes an alternative approach. It introduces PPConsensusStore, which is simply a Dict shared among PP ranks. All PP ranks update their bootstrap status into the map. PP0 collects others bootstrap status and determine bootstrap readiness. PP0 broadcast its decision to other PP ranks using pp_sync. This completely removes two-round consensus for bootstrap readiness.

This PR is working in progress. Here are limitations:

~~Only works with mooncake transfer engine. Other transfer engine is not fixed yet.~~
~~TCPStore lacks of batch and non-blocking operation. We may need to implement our own store.~~
~~Only bootstrap consensus is removed. Other consensus processes are kept unchanged.~~. This will be done in follow-up PRs.

Discussion and advices are welcome.

Modifications

Add PPConsensusStore, which helps PP0 to gather boostrap ready events from other PP ranks.
Add MooncakeKVSender.poll_pp_consensus. This function collects bootstrap events of other PP ranks, and
In pop_bootstrapped, PP0 calls poll_pp_consensus to collect bootstrap status from others, and then propagate decision to other PP ranks.
Remove two-round bootstrap consensus at prefill side.

Accuracy Tests

Tested SWE bench verfieid with DSv4 flash, pd disaggregation, P pp_size=4 tp_size=1, D pp_size=1 tp_size=4.

Baseline: 387/500
This PR: 385/500

Details

# router
nohup env HF_HUB_OFFLINE=1 \
  python -m sglang_router.launch_router \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --pd-disaggregation \
  --prefill http://127.0.0.1:30000 \
  --decode http://127.0.0.1:30001 \
  --host 0.0.0.0 \
  --port 9000 > router.log 2>&1 &

# prefill
nohup env CUDA_VISIBLE_DEVICES=0,1,2,3 \
HF_HUB_OFFLINE=1 \
sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --pp-size=4 \
  --tp-size=1 \
  --moe-runner-backend marlin \
  --host 0.0.0.0 \
  --port 30000 \
  --reasoning-parser deepseek-v4 \
  --tool-call-parser deepseekv4 \
  --chunked-prefill-size=8192 \
  --enable-metrics \
  --disaggregation-mode=prefill \
  --enable-metrics \
  --log-level=debug > prefill.log 2>&1 &


# decode
nohup env CUDA_VISIBLE_DEVICES=4,5,6,7 \
HF_HUB_OFFLINE=1 \
sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp-size=4 \
  --reasoning-parser deepseek-v4 \
  --tool-call-parser deepseekv4 \
  --moe-runner-backend marlin \
  --host 0.0.0.0 \
  --port 30001 \
  --enable-metrics \
  --disaggregation-mode=decode > decode.log 2>&1 &

# run swebench
env HF_HUB_OFFLINE=1 \
    OPENAI_API_KEY=dummy \
    OPENAI_BASE_URL=http://x.x.x.x:9000/v1  \
    MSWEA_COST_TRACKING=ignore_errors \
uv run python -m minisweagent.run.benchmarks.swebench \
    --subset verified \
    --split test \
    --shuffle \
    -c swebench.yaml \
    -c environment.container_timeout=72h \
    -c model.model_kwargs.timeout=3600 \
    -o /root/swe_test/pd_tp_baseline_20260915 \
    -m "openai/deepseek-ai/DeepSeek-V4-Flash-0731" \
    -w 64

# read results
cd /root/projects/SWE-bench
uv pip install -e .
env http_proxy=socks5://127.0.0.1:1080 https_proxy=socks5://127.0.0.1:1080 \
python -m swebench.harness.run_evaluation \
    --predictions_path /root/swe_test/pd_pp_baseline_20260913/preds.json \
    --dataset_name princeton-nlp/SWE-Bench_Verified \
    --split test \
    --max_workers 32 \
    --run_id test-mini-swe-agent-$(date +%s) \
    --namespace=mirrors-ssl.aliyuncs.com/swebench

Speed Tests and Profiling

I tested dsv4 flash with PD and no HiCache. The benchmark shows reduciton on TTFT by a single round of prefill (~1.4s).

Details

export HF_ENDPOINT=https://hf-mirror.com

# router
nohup env HF_HUB_OFFLINE=1 \
  python -m sglang_router.launch_router \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --pd-disaggregation \
  --prefill http://127.0.0.1:30000 \
  --decode http://127.0.0.1:30001 \
  --host 0.0.0.0 \
  --port 9000 > router.log 2>&1 &

# prefill
nohup env CUDA_VISIBLE_DEVICES=0,1,2,3 \
HF_HUB_OFFLINE=1 \
sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --pp-size=4 \
  --tp-size=1 \
  --moe-runner-backend marlin \
  --host 0.0.0.0 \
  --port 30000 \
  --chunked-prefill-size=8192 \
  --enable-metrics \
  --disaggregation-mode=prefill \
  --enable-metrics \
  --log-level=debug > prefill.log 2>&1 &


# decode
nohup env CUDA_VISIBLE_DEVICES=4,5,6,7 \
HF_HUB_OFFLINE=1 \
sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp-size=4 \
  --moe-runner-backend marlin \
  --host 0.0.0.0 \
  --port 30001 \
  --enable-metrics \
  --disaggregation-mode=decode > decode.log 2>&1 &

# test script
set -e
for c in 1 4 16; do
        for i in $(seq 1 5); do
            python -m sglang.bench_serving  \
                --backend sglang-oai \
                --base-url http://127.0.0.1:9000 \
                --flush-cache \
                --dataset-name random \
                --num-prompts 256 \
                --random-input 4096 \
                --random-output 1 \
                --random-range-ratio 1 \
                --model=deepseek-ai/DeepSeek-V4-Flash-0731 \
                --max-concurrency=$c
        done
done

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #37272668112
Latest PR Test (Extra): ⚠️ Not enabled -- add run-ci-extra label to opt in.
Latest PR Test (AMD ROCm 10): ❌ Run #37272668001

stepinto · 2026-09-11T02:30:09Z

cc @ShangmingCai

stepinto · 2026-09-11T10:41:29Z

/tag-and-rerun-ci

stepinto · 2026-09-12T02:40:59Z

/rerun-failed-ci

YAMY1234 · 2026-09-16T06:33:43Z

+            # chunk can still be pending, and concluding now would drop it.
+            if (
+                status == KVPoll.Success
+                and self.kv_mgr._staging_outstanding.get(self.bootstrap_room, 0) > 0


Thanks for these changes, the logic reads much cleaner this way.

One thing I'm not sure about: _staging_outstanding here is PP0-local, while check_status_pp_consensus is deciding on the replicated request_status of all ranks. If some rank k>0 publishes Success with a deferred staging chunk still pending, PP0 has no way to see that, right?

It doesn't bite today since nothing is sent before pop_bootstrapped, but I think it will once transfer consensus moves onto the store. Maybe the transfer thread should hold update_status(Success) until _staging_outstanding drops to 0, so request_status alone is the source of truth? That would also let poll and poll_pp_consensus share one body. Curious what you think.

Maybe the transfer thread should hold update_status(Success) until _staging_outstanding drops to 0, so request_status alone is the source of truth?

Thanks for your feedback.

That is exactly what I tought. With the change you mentioned, i.e. making request_status the single source of truth, it will make code cleaner.

I think we can do that in a follow-up PR. This PR focuses on only improving TTFT by removing bootstrap consensus.

stepinto · 2026-09-23T08:34:01Z

Resolved conflicts against master.

ShangmingCai · 2026-09-26T05:11:58Z

Begin (19/53):
python3 /home/runner/work/sglang/sglang/test/registered/unit/managers/test_scheduler_hicache_events.py
.
.

/home/runner/work/sglang/sglang/python/sglang/srt/layers/quantization/awq/awq.py:53: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")
/home/runner/work/sglang/sglang/python/sglang/srt/layers/quantization/gguf.py:69: UserWarning: Only CUDA, MUSA and NPU support GGUF quantization currently.
  warnings.warn(f"Only CUDA, MUSA and NPU support GGUF quantization currently.")
..E
======================================================================
ERROR: test_pp_prefill_drains_before_admission (__main__.TestSchedulerHiCacheEvents)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/sglang/sglang/test/registered/unit/managers/test_scheduler_hicache_events.py", line 134, in test_pp_prefill_drains_before_admission
    s.event_loop_pp_disagg_prefill()
  File "/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/home/runner/work/sglang/sglang/python/sglang/srt/managers/scheduler_pp_mixin.py", line 266, in event_loop_pp_disagg_prefill
    self.process_bootstrapped_queue()
  File "/home/runner/work/sglang/sglang/python/sglang/srt/managers/scheduler_pp_mixin.py", line 599, in process_bootstrapped_queue
    reqs = self.disagg_prefill_bootstrap_queue.pop_bootstrapped()
AttributeError: 'Scheduler' object has no attribute 'disagg_prefill_bootstrap_queue'

----------------------------------------------------------------------
Ran 3 tests in 0.053s

FAILED (errors=1)
.
.
End (19/53):
filename='/home/runner/work/sglang/sglang/test/registered/unit/managers/test_scheduler_hicache_events.py', elapsed=9, estimated_time=2.0
.

stepinto · 2026-09-28T02:11:51Z

Begin (19/53):
python3 /home/runner/work/sglang/sglang/test/registered/unit/managers/test_scheduler_hicache_events.py
.
.

/home/runner/work/sglang/sglang/python/sglang/srt/layers/quantization/awq/awq.py:53: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")
/home/runner/work/sglang/sglang/python/sglang/srt/layers/quantization/gguf.py:69: UserWarning: Only CUDA, MUSA and NPU support GGUF quantization currently.
  warnings.warn(f"Only CUDA, MUSA and NPU support GGUF quantization currently.")
..E
======================================================================
ERROR: test_pp_prefill_drains_before_admission (__main__.TestSchedulerHiCacheEvents)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/sglang/sglang/test/registered/unit/managers/test_scheduler_hicache_events.py", line 134, in test_pp_prefill_drains_before_admission
    s.event_loop_pp_disagg_prefill()
  File "/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/home/runner/work/sglang/sglang/python/sglang/srt/managers/scheduler_pp_mixin.py", line 266, in event_loop_pp_disagg_prefill
    self.process_bootstrapped_queue()
  File "/home/runner/work/sglang/sglang/python/sglang/srt/managers/scheduler_pp_mixin.py", line 599, in process_bootstrapped_queue
    reqs = self.disagg_prefill_bootstrap_queue.pop_bootstrapped()
AttributeError: 'Scheduler' object has no attribute 'disagg_prefill_bootstrap_queue'

----------------------------------------------------------------------
Ran 3 tests in 0.053s

FAILED (errors=1)
.
.
End (19/53):
filename='/home/runner/work/sglang/sglang/test/registered/unit/managers/test_scheduler_hicache_events.py', elapsed=9, estimated_time=2.0
.

OK. Fixed.

ShangmingCai · 2026-09-28T10:58:06Z

+            assert get_disagg().optimistic_prefill_attempts == 0, (
+                "PP prefill is incompatible with optimistic prefill"
+            )


No need, just need to know optimistic prefill is not on your roadmap, and then it is fine. Since this is in the hot path, let us remove this assertion.

OK. Not in roadmap.

ShangmingCai · 2026-09-28T10:59:04Z

/tag-and-rerun-ci

ShangmingCai · 2026-10-04T10:39:14Z

/rerun-failed-ci

Resolve conflicts: - common/conn.py: keep both the PPConsensusStore import and main's common.bootstrap imports. - mooncake/conn.py: keep the PR's poll()/_poll_with_status() split and carry main's deferred-decode-allocation Success-hold condition into _poll_with_status() so poll_pp_consensus() honors it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Lzy17

Thanks for the PR. Requested changes:

Don't rely on queue symmetry across PP ranks. pop_bootstrapped applies PP0's polls to the local self.queue by position, and returns early on an empty queue before pp_sync_polls. If queues diverge (e.g. continue after ensure_metadata_buffer / finalize_bootstrap fails on one rank), polls attach to the wrong request or recv_object blocks. Send rids with the polls, or at least assert len(polls) == len(self.queue).
Replace pickle in PPConsensusStore with msgspec. The payload is only ints, and the PULL socket is TCP-bound and unauthenticated.
Add a scheduler-level test for abort/failure with PP>1.

ShangmingCai · 2026-10-09T07:53:58Z

Thanks for the PR. Requested changes:

Don't rely on queue symmetry across PP ranks. pop_bootstrapped applies PP0's polls to the local self.queue by position, and returns early on an empty queue before pp_sync_polls. If queues diverge (e.g. continue after ensure_metadata_buffer / finalize_bootstrap fails on one rank), polls attach to the wrong request or recv_object blocks. Send rids with the polls, or at least assert len(polls) == len(self.queue).

Replace pickle in PPConsensusStore with msgspec. The payload is only ints, and the PULL socket is TCP-bound and unauthenticated.

Add a scheduler-level test for abort/failure with PP>1.

Every divergence source is symmetric. It should be fine.
2-3. I think they could be addressed in a following PR. Let me draft a PR now.

stepinto · 2026-10-09T08:50:00Z

Thanks for the PR. Requested changes:

Don't rely on queue symmetry across PP ranks. pop_bootstrapped applies PP0's polls to the local self.queue by position, and returns early on an empty queue before pp_sync_polls. If queues diverge (e.g. continue after ensure_metadata_buffer / finalize_bootstrap fails on one rank), polls attach to the wrong request or recv_object blocks. Send rids with the polls, or at least assert len(polls) == len(self.queue).

Replace pickle in PPConsensusStore with msgspec. The payload is only ints, and the PULL socket is TCP-bound and unauthenticated.

Add a scheduler-level test for abort/failure with PP>1.

Every divergence source is symmetric. It should be fine.
2-3. I think they could be addressed in a following PR. Let me draft a PR now.

OK. I see. Thanks for reviewing, Shangming and Lzy17.

PP+PD: Fix sgl-project#38206 Refine bootstrap consensus

b9d1c44

stepinto requested review from ByronHsu, Duyi-Wang, HaiShaw, ShangmingCai, ch-wan, hnyls2002, merrymercy, sogalin and yizhang2077 as code owners September 10, 2026 23:51

stepinto added 2 commits September 11, 2026 11:25

Make linter happy

41cb7d3

Refine PPConsensusStore

6ac08da

huangtingwei9988 self-assigned this Sep 11, 2026

Refine pop_bootstrapped

8242c4e

stepinto changed the title ~~[POC][PP+PD] Fix #38206 Reduce bootstrap consensus latency~~ [WIP][PP+PD] Fix #38206 Reduce bootstrap consensus latency Sep 11, 2026

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 11, 2026

stepinto mentioned this pull request Sep 12, 2026

PP8 disaggregated prefill has a load-independent ~30 s TTFT floor on Kimi-K3 #34815

Open

YAMY1234 reviewed Sep 16, 2026

View reviewed changes

bingxche mentioned this pull request Sep 16, 2026

[CI Monitor] Daily Report - 2026-09-16 bingxche/sglang-ci-bot#197

Open

stepinto added 5 commits September 18, 2026 14:03

Add poll_pp_consensus to NIXL backend

b599c41

Refine mooncake path, removing duplicated code

9c6f773

Fix typing

9a12ec7

Simplify pop_bootstrapped

0a9c6d3

Fix typo

b2eaa04

stepinto changed the title ~~[WIP][PP+PD] Fix #38206 Reduce bootstrap consensus latency~~ [PP+PD] Fix #38206 Reduce bootstrap consensus latency Sep 20, 2026

Merge remote-tracking branch 'origin/main' into pp_consensus_0910

5cd82d5

bingxche mentioned this pull request Sep 26, 2026

[CI Monitor] Daily Report - 2026-09-26 bingxche/sglang-ci-bot#207

Open

stepinto added 4 commits September 28, 2026 13:57

Fix failed UT

fcc985a

Rename poll_and_all_reduce_pp2 to poll_and_all_reduce_prefill_pp

d4d1d2d

Make linter happy

cdda187

Add assertion to pop_bootstrapped to guard from optimistic prefill

37a2bab

github-actions Bot added the hicache Hierarchical Caching for SGLang label Sep 28, 2026

ShangmingCai reviewed Sep 28, 2026

View reviewed changes

revert assert

e136b09

ShangmingCai approved these changes Sep 28, 2026

View reviewed changes

Merge branch 'main' into pp_consensus_0910

454cbaf

ShangmingCai added bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) bypass-maintenance labels Oct 4, 2026

ShangmingCai requested a review from Lzy17 as a code owner October 5, 2026 06:29

Lzy17 reviewed Oct 8, 2026

View reviewed changes

Lzy17 mentioned this pull request Oct 8, 2026

[PD] Consolidate sender and receiver implementations #41140

Open

3 of 4 tasks

ShangmingCai added the highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed label Oct 9, 2026

ShangmingCai mentioned this pull request Oct 9, 2026

[PP+PD] Harden PPConsensusStore: msgspec wire format, drop malformed messages #43329

Merged

3 of 5 tasks

ShangmingCai merged commit 348689d into sgl-project:main Oct 9, 2026
272 of 297 checks passed

This was referenced Oct 9, 2026

[CI Monitor] Daily Report - 2026-10-09 bingxche/sglang-ci-bot#220

Open

[CI Monitor] Daily Report - 2026-10-10 bingxche/sglang-ci-bot#221

Open

[CI Monitor] Daily Report - 2026-10-11 bingxche/sglang-ci-bot#222

Open

This was referenced Oct 11, 2026

[Roadmap] Prefill-Decode Disaggregation Roadmap (2026 Q4) #43699

Open

[Roadmap] Pipeline Parallelism Roadmap (2026 Q4) #43885

Open

Conversation

stepinto commented Sep 10, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

stepinto commented Sep 11, 2026

Uh oh!

stepinto commented Sep 11, 2026

Uh oh!

stepinto commented Sep 12, 2026

Uh oh!

YAMY1234 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

stepinto Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

stepinto commented Sep 23, 2026

Uh oh!

ShangmingCai commented Sep 26, 2026

Uh oh!

stepinto commented Sep 28, 2026

Uh oh!

ShangmingCai Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

stepinto Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

ShangmingCai commented Sep 28, 2026

Uh oh!

ShangmingCai commented Oct 4, 2026

Uh oh!

Lzy17 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShangmingCai commented Oct 9, 2026

Uh oh!

stepinto commented Oct 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

stepinto commented Sep 10, 2026 •

edited by github-actions Bot

Loading

Lzy17 left a comment •

edited

Loading