Repository navigation
[Fix] eplb: Warm default-group P2P transports on multi-node layouts - #46592
Merged
Merged
Conversation
weireweire
requested review from
ch-wan,
merrymercy and
yizhang2077
as code owners
October 26, 2026 12:14
Root cause: sgl-project#42026 materializes EPLB's default-group NCCL P2P transports before KV-cache sizing only when nnodes == 1. On multi-node full-EP layouts the first rebalance still connects the peers it exchanges with lazily (in our runs, all of them). By then the KV pool has been sized to the free memory, so the transport buffers land on a full pool and the first rebalance stalls for the connection setup. Fix: drop the single-node gate. Also run one 4 MiB all-reduce on the default group, which the rebalance uses for the recorded expert counts. This warms only the NCCL algorithm and protocol picked at 4 MiB; the rebalance all-reduces the whole logical_count buffer (over 100 MB with a 3000-pass window), which may pick another one. On GB300 MNNVL it covered what the rebalance used; IB-only fabrics are not validated. The all-reduce runs on single-node layouts too, where it adds 384 MiB to the warmup. The warmup's KV cost grows with world size (~0.45 GiB per peer on MNNVL), which stock pays later at the first rebalance. The warmup posts all peers in one batch, unlike the chunked rebalance P2P; it has only run at W <= 8. Validation: GB300, DeepSeek-V4.1-Flash MegaMoE decode with DSpark and --enable-eplb --ep-num-redundant-experts 32, SGLang main at 6563312, InferenceX AgentX trace. 2-node DEP8 decode (rebalance every 3000 passes, c512, 14 rebalances per arm), stock -> patched: - first rebalance 3.21 s -> 1.06 s (later ones 1.01-1.04 s stock, 1.01-1.06 s patched) - free memory lost at the first rebalance 4.38 GiB -> 1.14 GiB per rank (the 1.14 GiB is the staging cache every rebalance grows in both arms) - warmup 1.65 s and 3.24 GiB per rank, now taken before KV sizing; KV pool -2.0% per rank - no OOM in either arm at --mem-fraction-static 0.85 1-node DEP4 decode (rebalance every 1000 passes, c256, 5-6 rebalances), stock (sgl-project#42026 warmup) -> patched: - warmup 0.93 s / 1396 MiB -> 1.08 s / 1780 MiB per rank - KV pool 42.99M -> 42.79M tokens per rank (-0.5%) - first rebalance 2.06 s -> 1.80 s, free memory lost 2.45 -> 2.08 GiB Stock 4-node DEP16: the first rebalance takes 6.6-6.7 s vs 0.85 s steady, and loses 7.0 GiB per rank after KV sizing. Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
weireweire
force-pushed
the
eplb-prewarm-multinode
branch
from
October 27, 2026 01:22
574e7bc to
0ab98a5
Compare
nvpohanh
approved these changes
Oct 27, 2026
Collaborator
|
/rerun-test test/registered/ep/test_eplb_no_a2a.py test/registered/model_loading/test_weight_cache_daemon.py test/registered/e2e/models_large/test_nvidia_nemotron_3_super_nvfp4.py |
Contributor
|
🚀 🚀 🚀 |
weireweire
added a commit
to weireweire/sglang
that referenced
this pull request
Oct 27, 2026
… pass Root cause: sgl-project#41593 keeps waiting requests' device prefixes warm by touching them in the radix LRU, but on a PD prefill worker it walks only `waiting_queue`. Requests still in `disagg_prefill_bootstrap_queue`, waiting for decode to pre-allocate their slots, are never touched. When decode is full, the bootstrap queue is the deep one, and those prefixes age out while the requests wait. With DEP2 prefill (9 workers) + DEP16 decode serving an agentic multi-turn trace at 7480 concurrent sessions, the bootstrap queue held ~300 requests per rank for ~100 s. Device-radix hits fell from ~65 % to ~22 % of prompt tokens, and the extra external (Mooncake L3) reloads slowed prefill further, so prefill collapsed after the client's turn burst and did not recover. Fix: `calc_priority` takes the PD prefill bootstrap queue, and on the same refresh pass `_touch_waiting_prefixes` touches up to WAITING_PREFIX_REFRESH_BOOTSTRAP_MAX (512, a module constant next to the existing waiting-queue bounds) bootstrap entries before the waiting queue, in reverse order. SGLANG_ENABLE_WAITING_PREFIX_REFRESH=0 still turns the whole refresh off. The existing gate is unchanged: cache-agnostic policy, LRU eviction, not decode, no fast match, prefix sharing supported. Bootstrap entries are refreshed only on passes that reach `calc_priority`. The caller returns earlier when the waiting queue is empty or no request rows are free, so the bootstrap queue is not refreshed on its own. With more than 128 waiting entries, the entries past the first 128 are still not touched and can end up older than the refreshed bootstrap entries. Validation: GB300, DSv4.1-Flash, 9 DEP2 prefill + DEP16 decode, 7480 concurrent sessions, 1200 s window, SGLang main at 6563312 plus open PRs sgl-project#46497/sgl-project#46593/sgl-project#46592/sgl-project#46404 on both arms: without: 361,815 tok/s/GPU @ p90 interactivity 145.0, TTFT p90 135.9 s; L3 73 % / device 22 % of prompt tokens in minutes 8-20. with: 560,412 tok/s/GPU @ p90 interactivity 161.0, TTFT p90 58.9 s; L3 21 % / device 74 %. test_scheduler_held_row_admission's Scheduler stub gains `disagg_prefill_bootstrap_queue = None` for the new argument.
Collaborator
|
@ch-wan could you review this small change? thanks! |
kpham-sgl
approved these changes
Oct 29, 2026
Collaborator
|
/rerun-test test_eplb_no_a2a.py |
Contributor
|
🚀 |
This was referenced Oct 29, 2026
Yuxingwang-intel
pushed a commit
to Yuxingwang-intel/sglang
that referenced
this pull request
Nov 6, 2026
…gl-project#46592) Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com> Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
#42026 warms EPLB's default-group NCCL P2P transports before KV-cache sizing, but only on a single node. On multi-node full-EP layouts, the first rebalance still creates the cross-node connections to the peers it exchanges with lazily (in our runs, all of them). By then the KV pool has already been sized to the free memory, so the transport buffers come out of memory the pool was counting on. The first rebalance also stalls while it connects.
Measured on GB300, DeepSeek-V4.1-Flash MegaMoE decode with DSpark and
--enable-eplb --ep-num-redundant-experts 32, stock main:That memory comes out after KV sizing, so a high
--mem-fraction-staticrisks an OOM at the first rebalance, the failure #42026 addressed for single-node layouts. We have not hit that OOM in any run at 0.85; on 1-node DEP4 the ranks still had 52-57 GiB free before each rebalance. This is a stability fix, not a throughput change.Modifications
nnodes == 1gate from the EPLB P2P warmup. The other gates are unchanged: CUDA with NCCL,--enable-eplbwithout an elastic EP backend, PP=1, and EP=TP>1.logical_countbuffer (over 100 MB with a 3000-pass window), which NCCL may run with another algorithm. On GB300 MNNVL the warmup covered what the rebalance used (the first-rebalance loss drops to the staging cache, below); IB-only fabrics are not validated.The all-reduce also runs on single-node layouts, so single-node memory accounting changes slightly (numbers below).
The warmup's KV cost grows with world size (~0.45 GiB per peer, measured on MNNVL); stock pays the same transports later, at the first rebalance. The warmup posts all peers in one batch, unlike the rebalance P2P, which is chunked; it has only run at W <= 8.
Accuracy Tests
Not applicable: a startup-only change with no effect on model outputs.
Speed Tests and Profiling
GB300, SGLang main at 6563312, InferenceX AgentX trace, MegaMoE decode with DSpark and dynamic EPLB (+32 redundant experts).
2-node DEP8 decode, rebalance every 3000 passes, c512, 14 rebalances per arm:
The remaining 1.14 GiB is the staging cache that every rebalance grows in both arms.
End-to-end throughput was not compared, and none is expected: the PR saves about 2 s once per server lifetime (the first rebalance at DEP8), and later rebalances take the same time in both arms. The 0.5-2.0% smaller KV pool could cost slightly when the KV pool is the limit.
1-node DEP4 decode, where #42026 already warms the P2P transports and this PR adds only the all-reduce. Rebalance every 1000 passes, c256, 5-6 rebalances per arm:
Not validated:
CI States
Latest PR Test (Base): ❌ Run #40477015885
Latest PR Test (Extra): ❌ Run #40477015876
Latest PR Test (AMD ROCm 10): ❌ Run #40477015774