Repository navigation
[Refactor] Record what a bound stage path does instead of reading it back - #43453
Merged
Merged
Conversation
ch-wan
requested review from
BBuf,
Edwardf0t1,
Fridge003,
HaiShaw,
Ying1123,
ispobock and
merrymercy
as code owners
October 10, 2026 00:46
This was referenced Oct 10, 2026
Every arrival a stage boundary builds carries one known update capability, so drop the capability tuple, the per-update dispatcher behind it, and the tests that exercised only that dispatcher. Also remove the unused fetch_hidden_states path on the attention inputs.
Entry selection now returns the facts it already knows: whether the input move gathers over attention TP with the stage's own gather, whether that move's output is storage a capture may keep, and which fused kernels the step tries for the attention-TP sum. The exit move records its attention-TP gather. Construction checks, the residual-preservation certificate and aux capture read these fields instead of inspecting the bound partials. Every bound path and capture result stays the same.
ch-wan
force-pushed
the
cheng/refactor/stage-boundary-bound-paths
branch
from
October 10, 2026 19:40
1b1a779 to
08e64b9
Compare
Shunkangz
added a commit
to Shunkangz/sglang
that referenced
this pull request
Oct 12, 2026
Main's sgl-project#42044 (GLM-5.3-Flash prefill CP) added declare_attn(tp_group=) and an ATTN_CP entry for a head-parallel mixer on full rows: the head partition this branch declared as sum_group. The resolution uses main's names and keeps what zigzag needs: - The GDN declares tp_group plus the branch's in_token_order. Under sharing its sum is ATTN_CP, the same coordinator as TP. - The token-order take-back is a move in main's ATTN_CP entry (ops.cp_take_back_in_token_order: complete the sum, then take this rank's zigzag shard). Main's CP moves are rank-major and would take back the wrong rows. prepare._cp_take_back_update_read is gone. - _select_entry_step returns EntryStep (sgl-project#43453); the token-order gather is its input_move. - _bind_stage exempts both DSA/MLA CP and CP-TP group sharing. - _qwen4_exp_stage_facts (sgl-project#43460) declares the GDN options; the linear layer's stage_facts passes linear_attention=True. _ffn_exit_rows went with ExitRows (sgl-project#42936). - Hook: both models registered. The EP/a2a/waterfill/PP/decode rejections and the PD prefill allowance are Qwen4-Exp's; GLM keeps main's disaggregation rejection. Qwen4-Exp keeps the plain speculative rejection message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 15, 2026
…ject#4 Conflict resolutions (35 files), preserving stack behavior: srt/utils/common.py: make_layers — folded our per-layer empty_cache body INTO upstream's stage-binding build() (layer_stack refactor sgl-project#43453-43460). Upstream removed is_building_neighbour_layer/building_neighbour_layer entirely (our earlier sgl-project#42481 pick); only user was qwen4_exp. srt/models/qwen4_exp.py: import unions (cached_property/partial/TYPE_CHECKING, linear_attn_parallel_group); dropped dead is_building_neighbour_layer import; prefetch_stream keeps our _graph_prefetch_buffer line with upstream's ple_offload_embedding conditional; our PLE host-staging design kept (_offloads_ple guard moot under new stage facts). srt/managers/cache_controller.py: upstream sgl-project#43550 absorbed our sgl-project#39830 prefetch-degrade fixes; took upstream restructure, deleted auto-merge's stale duplicate tails (double-ack/deadlock hazard), kept our rationale comments. Fenced start_writing + load_fence_stream survive. srt/mem_cache/hybrid_cache/hybrid_cache_controller.py: same pattern; sidecar degrade comment updated ('upstream adopted unconditionally'). model_executor/pool_configurator.py: UNION draft-cell sizing — our FP4 _compute_eagle_fp4_draft_cell_size branch + upstream HiSparse draft_loc_space_scale else-branch. b05a7c1 None layer_info guard intact. srt/layers/attention/qsa/sparse_attn.py: kept OUR FP8-descale chunk loop (superset; upstream sgl-project#41933 only re-commented); took upstream _pack_qsa_prefill_kv + pack_qsa_prefill_kv; our ck interface keeps k_scale/v_scale/max_q; __all__ union. qsa/mqa.py: import functools + min_scored_len kwarg into our triton-first _resolve_mqa_backend dispatch; upstream ctas_per_row rework kept. qsa/qsa_indexer.py: batch-invariance gate moved INTO upstream's shared use_fast_topk predicate. qwen_sparse_attn_backend.py: imports union; gather path UNION — upstream pack_qsa_prefill_kv used when scales are unit (QSA stores unscaled K/V), our index_select fallback when checkpoint k/v descales != 1.0. managers/schedule_policy.py: add_chunked_req UNION — upstream slot-reserve gate first, then our prefill_chunk_alignment truncation (monotone-safe); all 3 alignment sites (1177/1340/1608) intact. managers/tokenizer_manager.py: import unions (PositionalEmbedsTransport + RequestPreprocessor + watermarking fns). entrypoints/openai/serving_base.py: kept OUR request_preprocessor.run path. model_executor/runner/eager_runner.py: took upstream TCPCG eager-fallback removal (sgl-project#42287); kept our shared cp_extend_forward call shape (our runner_utils/cp_extend.py is byte-equivalent to upstream's _execute_extend_cp — single source of truth). runner/flashinfer_autotune.py: import union (can_run_dummy_draft_extend + cdiv/ceil_align + upstream get_eager_max_batch_size sgl-project#43789). arg_groups/model_hook.py: Gemma4 accepted_backends union fa4 + flashinfer (NVFP4-KV enablement). multimodal_gen/runtime/layers/lora/linear.py: KEPT OUR sgl-project#37460 pipeline structure; upstream runtime_lora_scale==0 early-return ordering adopted; _add_lora_delta/_add_lora_output_offset wired into pipeline; kept UPSTREAM's quality-tiered _scale_lora_delta (deleted our simple variant). DEBT: upstream RowParallel bias_in_gemm lossless opt not ported (perf-only). multimodal_gen/runtime/layers/attention/layer.py: _gather_sharded_sequence took upstream sgl-project#43793 .contiguous() fix (subsumes our 55aaa02 guard); our _forward_with_kv_gather k/v guards kept (separate path, not covered). configs/pipeline_configs/base.py + configs/sample/minimax_h3.py + envs.py: unions (supports_continuous_batching / teacache_params+fields / SGLANG_TEACACHE_CALIBRATE). models/dits/ltx_2.py mask-fill: took upstream (identical to our fix). ltx_2/denoising.py: took upstream pass_specs (sgl-project#43667) — subsumes our legacy one-stage branch; our _ltx2_model_forward_context helpers already upstreamed (entry-142 sgl-project#42928 CONFIRMED SUPERSEDED). ltx_2/decoding_av.py: KEPT OUR split-stage AV decode (independent pool scaling) over upstream single-class restructure; adopted upstream chunked pinned-memory _postprocess_video; TODO(merge-4): upstream EncodeWhileDecoding streaming + rank-0 retain gating not ported. kernels/ops/diffusion/activation/silu_mul_bitexact.py: our out= column-slice support + XPU guard UNION with upstream @replay_launches (sgl-project#43859). kernels/ops/diffusion/README.md + docs performance-optimization.mdx: unions. rust/sglang-renderer: ACCEPTED upstream crate removal (sampling.rs git rm); our sgl-project#42779 top_k vocab-bound port moves to sglang-server message/sampling.rs (follow-up). Tests: anthropic test_serving — UNION fake-class exception param + our request_preprocessor stubs, took upstream's _FakeNonStreamingOpenAI removal (68 pass). test_server_args — union both new tests (pre-existing GPU-only collection ENV fail unchanged). test_lora_inference_mode — took upstream sgl-project#43028 runtime_scale+quality-tier test (supersedes our adapted variant; our kept quality-tiered _scale_lora_delta satisfies it). test_fused_moe_triton_config — kept our int4-filename + RTX-PRO-6000 JSON tests despite upstream sgl-project#43503 test-drop. test_hicache_file_lru_unit — union our corrupt-page-degrade class + upstream sgl-project#42077 uncached-batch test (40 pass). test_hicache_host_register — upstream alias host_common. test_cuda_graph_setup — took upstream rewrite of our override-decode-runner test into draft-local-capture-sizes test (sgl-project#43303 API). Verified auto-merged stack markers survived: flashinfer_backend KVLocPlan/ prefill_wrappers_ragged/NVFP4-KV forwarding, memory_pool set_kv_buffer_prefix_valid, fp8 grouped_moe kwarg, fp8_utils reshape fix 3642490, SGLANG_QWEN4_PLE_FILE_REUSE, schedule_batch freed_rows fix, prefill_chunk_alignment sites, sgl-project#42736 autotune default-on, PCIe-IPC vocab gathers, flashinfer 0.7.1 pin taken (NOT installed). Co-Authored-By: Claude Opus 5.5 (1M context)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack (oldest at bottom):
Motivation
Several checks of a bound stage boundary path (construction checks, the residual-preservation certificate, auxiliary capture) found out what the path does by inspecting the bound partials:
partial.func is Xand their keywords. Wrapping a step, or binding it differently, silently changes what those checks see. The arriving capability was also carried as a tuple with a per-update dispatcher, although every arrival has exactly one.Modifications
EdgeContract.arriving_plain_addis a single value; the capability dispatcher, the multi-capability branch ofbind_entry, and the tests that only exercised them are removed, together with the unusedfetch_hidden_statespath on the attention inputs.input_gather_declared), whether a capture may keep that move's output without copying (input_retainable), and which fused kernels the step tries (ffn_fusions); the exit move records its attention-TP gather (StagePath.output_gathers_attn_tp). The checks read these fields.No bound path or capture result changes.
Accuracy Tests
How the stack is checked:
test_aiter_allreduce_fusion_amdcases that fail on NVIDIA on main too; the test files that changed most recently pass on every commit. The fulltest/registered/unitsuite (926 files, one process per file) shows no failure on the stack's head that main does not have.516fd1c1b5unless marked as on the merge candidate; rebasing onto current main had no conflicts, and the CPU checks were run again after it.961bcf481d, H200, asymmetric dummy weights unless noted): DeepSeek-V2-Lite input-scattered PP2 is bit-identical to PP1 (main crashes on it); Nemotron-H cut between a mixer and its FFN differs from PP1 only at the rounding level of the FFN-to-mixer cuts (main: 0.93 and 2.21); Llama, Qwen3-MoE, GPT-OSS and DeepSeek-V2-Lite are bit-identical to main at PP1 and PP2; GSM8K (real DeepSeek-V2-Lite weights, 250 questions, TP4 with attention DP2 and the dense FFN fully data-parallel) gives 0.360 at PP2, 0.364 at PP1 and 0.368 on main at PP1, medians of three runs whose run-to-run spread is 0.012–0.016. The last revision of the stack changed only comments, docstrings, an equivalent condition and explicitly set attributes after these runs; its bound plans are identical.For this PR: the plan comparison against main shows no difference in any scenario, and over the whole configuration matrix the new fields equal what the checks used to infer from the partials.
Speed Tests and Profiling
No kernel, collective or forward-path change.
Checklist
CI States
Latest PR Test (Base): 🚫 Run #38080721135
Latest PR Test (Extra): 🚫 Run #38080720881
Latest PR Test (AMD ROCm 10): ❌ Run #38080721057