Repository navigation
Conversation
…f CUDA and under prefill CP The graph tail block for decoder SWA bounded replay is only built by the CUDA dsv4 backend, and its metadata covers a rank's full rows. Reject the breakable prefill graph + bounded replay combination in the validator when the platform is not CUDA or prefill context parallelism is on, instead of failing at graph capture; assert in the model that a tail inside a breakable graph carries the captured scatter indices. Eager prefill keeps CP and HIP as before. Follow-up to sgl-project#43736.
jyizheng
changed the title
This was referenced Oct 15, 2026
Closed
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 20, 2026
… correctness) - core batch [10-15]: sgl-project#43376 long-ctx indexer OOM bound, sgl-project#42368 FlashInfer-0.7 decode dispatch, sgl-project#39914 extra-KV 64-token page split + candidate fallback, sgl-project#42714 graph-safe page-split buffers, sgl-project#32464 SM120 sparse-prefill test gate, sgl-project#34027 int32->int64 page-split overflow guard. - Group C [16-23]: sgl-project#42717 TP-rank verify-budget, sgl-project#43225/sgl-project#43374 compact-verify seq_lens + padding renorm, sgl-project#42718 packed-verify token->request map, sgl-project#42713 forced-uniform capture metadata, sgl-project#45301 padded-query graph lifetime, sgl-project#44141 bounded-replay guard, sgl-project#45185 V4.1-Flash-1017 FP8 QDQ (env-gated, default OFF). sgl-project#42710 skipped (superseded upstream: step_logits already fp32).
4 of 5 tasks
SAXEM1997
added a commit
to SAXEM1997/sglang
that referenced
this pull request
Oct 21, 2026
…ill CP gather with DSpark When prefill CP, decoder SWA bounded replay and DSpark run together, the late layers return only each rank's SWA tail rows (16 of 32768), while _eager_runner._execute_extend_cp gathers with the full-batch CP layout (4096 rows per rank). The capture_dspark guard skipped the tail scatter, so the first large prefill aborted in interleave.py: RuntimeError: Interleave CP gather received an unexpected local token count: rank=3, got=16, expected_at_least=4096, total=32768, cp_size=8 Add _needs_cp_tail_scatter() for the CUDA/CP gate and scatter hidden_states, pre_hc_head and the tail-layer DSpark aux tensors back into the rank's full layout before the CP gather. Non-tail rows are zero-filled (defensive; they are never read). Scoped to CUDA so it composes with the HIP counterpart in sgl-project#43465. This covers eager prefill only: the breakable prefill CUDA graph with prefill CP is guarded by sgl-project#44141. Tested on 8x H20-3e (SM90), sglang v0.5.22: 5/5 greedy outputs match the non-CP production baseline; cold TTFT 128K/256K/512K = 3.37/6.86/15.95 s vs 5.41/11.71/28.66 s with replay off.
SAXEM1997
added a commit
to SAXEM1997/sglang
that referenced
this pull request
Oct 21, 2026
…ill CP gather with DSpark When prefill CP, decoder SWA bounded replay and DSpark run together, the late layers return only each rank's SWA tail rows (16 of 32768), while _eager_runner._execute_extend_cp gathers with the full-batch CP layout (4096 rows per rank). The capture_dspark guard skipped the tail scatter, so the first large prefill aborted in interleave.py: RuntimeError: Interleave CP gather received an unexpected local token count: rank=3, got=16, expected_at_least=4096, total=32768, cp_size=8 Add _needs_cp_tail_scatter() for the CUDA/CP gate and scatter hidden_states, pre_hc_head and the tail-layer DSpark aux tensors back into the rank's full layout before the CP gather. Non-tail rows are zero-filled (defensive; they are never read). Scoped to CUDA so it composes with the HIP counterpart in sgl-project#43465. This covers eager prefill only: the breakable prefill CUDA graph with prefill CP is guarded by sgl-project#44141. Tested on 8x H20-3e (SM90), sglang v0.5.22: 5/5 greedy outputs match the non-CP production baseline; cold TTFT 128K/256K/512K = 3.37/6.86/15.95 s vs 5.41/11.71/28.66 s with replay off.
This was referenced Oct 28, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Follow-up to #43736. Decoder SWA bounded replay inside the breakable prefill CUDA graph relies on a graph tail block that only the CUDA dsv4 backend builds at capture, and whose metadata covers a rank's full rows. Off CUDA, or with prefill context parallelism, the combination currently fails at graph capture instead of at argument validation.
Modifications
deepseek_v4_hook.py: with--enable-decoder-swa-bounded-replay, reject the breakable prefill graph on a platform other than CUDA and with prefill CP (--enable-prefill-cporattn_cp_size > 1). Eager prefill keeps CP and HIP as before; the breakable graph without bounded replay is untouched.deepseek_v4.py: assert that a tail inside a breakable graph carries the capturedscatter_indices, so an eager tail can never be gathered live under a replay.test/registered/unit/server_args/test_dsv41_bounded_replay_graph_guards.py(3 cases: non-CUDA rejected, prefill CP rejected, eager prefill / no bounded replay still admitted).Accuracy Tests
N/A (validation-only change).
Benchmarking and Profiling
N/A.
Checklist
CI States
Latest PR Test (Base): ❌ Run #38477600845
Latest PR Test (Extra): ❌ Run #38477600884
Latest PR Test (AMD ROCm 10): ❌ Run #38477600870