Skip to content

[Refactor] Record what a bound stage path does instead of reading it back - #43453

Merged
ch-wan merged 2 commits into
mainfrom
cheng/refactor/stage-boundary-bound-paths
Oct 10, 2026
Merged

ch-wan merged 2 commits into
mainfrom
cheng/refactor/stage-boundary-bound-paths

Conversation

@ch-wan

@ch-wan ch-wan commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

Several checks of a bound stage boundary path (construction checks, the residual-preservation certificate, auxiliary capture) found out what the path does by inspecting the bound partials: partial.func is X and their keywords. Wrapping a step, or binding it differently, silently changes what those checks see. The arriving capability was also carried as a tuple with a per-update dispatcher, although every arrival has exactly one.

Modifications

  • Bind a single arriving capability on stage boundaries. EdgeContract.arriving_plain_add is a single value; the capability dispatcher, the multi-capability branch of bind_entry, and the tests that only exercised them are removed, together with the unused fetch_hidden_states path on the attention inputs.
  • Record what a bound stage boundary path does instead of reading it back. Entry selection returns what it already knows: whether the input move gathers over attention TP with the stage's own gather (input_gather_declared), whether a capture may keep that move's output without copying (input_retainable), and which fused kernels the step tries (ffn_fusions); the exit move records its attention-TP gather (StagePath.output_gathers_attn_tp). The checks read these fields.

No bound path or capture result changes.

Accuracy Tests

How the stack is checked:

  • Unit tests: the 39 layer-boundary test files pass at the head of every PR in the stack, apart from the two AMD-only test_aiter_allreduce_fusion_amd cases that fail on NVIDIA on main too; the test files that changed most recently pass on every commit. The full test/registered/unit suite (926 files, one process per file) shows no failure on the stack's head that main does not have.
  • Plan comparison: an out-of-tree harness binds every stage plan for seven stack shapes (decoder layers with and without an output transform, MHC, Nemotron-H patterns, Kimi K3, Qwen4-exp, NemotronNAS-style attention-only layers, single-stage appends) over 23 parallel configurations, four boundary reductions and one or two pipeline ranks (3,928 scenarios), and compares each bound plan leaf by leaf with the parent commit's.
  • GPU: dummy weights are seeded per TP rank and scaled (0.05, norms 1.0), so that a sum taken twice or a mixture of rank partials changes the output; the default dummy weights give every rank the same shard and would hide both.
  • GPU results below were measured on these commits as based on main 516fd1c1b5 unless marked as on the merge candidate; rebasing onto current main had no conflicts, and the CPU checks were run again after it.
  • On the merge candidate (head of this stack against main 961bcf481d, H200, asymmetric dummy weights unless noted): DeepSeek-V2-Lite input-scattered PP2 is bit-identical to PP1 (main crashes on it); Nemotron-H cut between a mixer and its FFN differs from PP1 only at the rounding level of the FFN-to-mixer cuts (main: 0.93 and 2.21); Llama, Qwen3-MoE, GPT-OSS and DeepSeek-V2-Lite are bit-identical to main at PP1 and PP2; GSM8K (real DeepSeek-V2-Lite weights, 250 questions, TP4 with attention DP2 and the dense FFN fully data-parallel) gives 0.360 at PP2, 0.364 at PP1 and 0.368 on main at PP1, medians of three runs whose run-to-run spread is 0.012–0.016. The last revision of the stack changed only comments, docstrings, an equivalent condition and explicitly set attributes after these runs; its bound plans are identical.

For this PR: the plan comparison against main shows no difference in any scenario, and over the whole configuration matrix the new fields equal what the checks used to infer from the partials.

Speed Tests and Profiling

No kernel, collective or forward-path change.

Checklist


CI States

Latest PR Test (Base): 🚫 Run #38080721135
Latest PR Test (Extra): 🚫 Run #38080720881
Latest PR Test (AMD ROCm 10): ❌ Run #38080721057

Every arrival a stage boundary builds carries one known update capability,
so drop the capability tuple, the per-update dispatcher behind it, and the
tests that exercised only that dispatcher. Also remove the unused
fetch_hidden_states path on the attention inputs.
Entry selection now returns the facts it already knows: whether the input
move gathers over attention TP with the stage's own gather, whether that
move's output is storage a capture may keep, and which fused kernels the
step tries for the attention-TP sum. The exit move records its
attention-TP gather. Construction checks, the residual-preservation
certificate and aux capture read these fields instead of inspecting the
bound partials. Every bound path and capture result stays the same.
@ch-wan
ch-wan force-pushed the cheng/refactor/stage-boundary-bound-paths branch from 1b1a779 to 08e64b9 Compare October 10, 2026 19:40
@ch-wan
ch-wan merged commit 8bd7f2d into main Oct 10, 2026
19 of 26 checks passed
@ch-wan
ch-wan deleted the cheng/refactor/stage-boundary-bound-paths branch October 10, 2026 19:40
Shunkangz added a commit to Shunkangz/sglang that referenced this pull request Oct 12, 2026
Main's sgl-project#42044 (GLM-5.3-Flash prefill CP) added declare_attn(tp_group=)
and an ATTN_CP entry for a head-parallel mixer on full rows: the head
partition this branch declared as sum_group. The resolution uses main's
names and keeps what zigzag needs:

- The GDN declares tp_group plus the branch's in_token_order. Under
  sharing its sum is ATTN_CP, the same coordinator as TP.
- The token-order take-back is a move in main's ATTN_CP entry
  (ops.cp_take_back_in_token_order: complete the sum, then take this
  rank's zigzag shard). Main's CP moves are rank-major and would take
  back the wrong rows. prepare._cp_take_back_update_read is gone.
- _select_entry_step returns EntryStep (sgl-project#43453); the token-order gather
  is its input_move.
- _bind_stage exempts both DSA/MLA CP and CP-TP group sharing.
- _qwen4_exp_stage_facts (sgl-project#43460) declares the GDN options; the linear
  layer's stage_facts passes linear_attention=True. _ffn_exit_rows went
  with ExitRows (sgl-project#42936).
- Hook: both models registered. The EP/a2a/waterfill/PP/decode
  rejections and the PD prefill allowance are Qwen4-Exp's; GLM keeps
  main's disaggregation rejection. Qwen4-Exp keeps the plain speculative
  rejection message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
detain added a commit to detain/sglang that referenced this pull request Oct 15, 2026
…ject#4

Conflict resolutions (35 files), preserving stack behavior:

srt/utils/common.py: make_layers — folded our per-layer empty_cache body
  INTO upstream's stage-binding build() (layer_stack refactor sgl-project#43453-43460).
  Upstream removed is_building_neighbour_layer/building_neighbour_layer
  entirely (our earlier sgl-project#42481 pick); only user was qwen4_exp.
srt/models/qwen4_exp.py: import unions (cached_property/partial/TYPE_CHECKING,
  linear_attn_parallel_group); dropped dead is_building_neighbour_layer import;
  prefetch_stream keeps our _graph_prefetch_buffer line with upstream's
  ple_offload_embedding conditional; our PLE host-staging design kept
  (_offloads_ple guard moot under new stage facts).
srt/managers/cache_controller.py: upstream sgl-project#43550 absorbed our sgl-project#39830
  prefetch-degrade fixes; took upstream restructure, deleted auto-merge's
  stale duplicate tails (double-ack/deadlock hazard), kept our rationale
  comments. Fenced start_writing + load_fence_stream survive.
srt/mem_cache/hybrid_cache/hybrid_cache_controller.py: same pattern;
  sidecar degrade comment updated ('upstream adopted unconditionally').
model_executor/pool_configurator.py: UNION draft-cell sizing — our FP4
  _compute_eagle_fp4_draft_cell_size branch + upstream HiSparse
  draft_loc_space_scale else-branch. b05a7c1 None layer_info guard intact.
srt/layers/attention/qsa/sparse_attn.py: kept OUR FP8-descale chunk loop
  (superset; upstream sgl-project#41933 only re-commented); took upstream
  _pack_qsa_prefill_kv + pack_qsa_prefill_kv; our ck interface keeps
  k_scale/v_scale/max_q; __all__ union.
qsa/mqa.py: import functools + min_scored_len kwarg into our triton-first
  _resolve_mqa_backend dispatch; upstream ctas_per_row rework kept.
qsa/qsa_indexer.py: batch-invariance gate moved INTO upstream's shared
  use_fast_topk predicate.
qwen_sparse_attn_backend.py: imports union; gather path UNION — upstream
  pack_qsa_prefill_kv used when scales are unit (QSA stores unscaled K/V),
  our index_select fallback when checkpoint k/v descales != 1.0.
managers/schedule_policy.py: add_chunked_req UNION — upstream slot-reserve
  gate first, then our prefill_chunk_alignment truncation (monotone-safe);
  all 3 alignment sites (1177/1340/1608) intact.
managers/tokenizer_manager.py: import unions (PositionalEmbedsTransport +
  RequestPreprocessor + watermarking fns).
entrypoints/openai/serving_base.py: kept OUR request_preprocessor.run path.
model_executor/runner/eager_runner.py: took upstream TCPCG eager-fallback
  removal (sgl-project#42287); kept our shared cp_extend_forward call shape (our
  runner_utils/cp_extend.py is byte-equivalent to upstream's
  _execute_extend_cp — single source of truth).
runner/flashinfer_autotune.py: import union (can_run_dummy_draft_extend +
  cdiv/ceil_align + upstream get_eager_max_batch_size sgl-project#43789).
arg_groups/model_hook.py: Gemma4 accepted_backends union fa4 + flashinfer
  (NVFP4-KV enablement).
multimodal_gen/runtime/layers/lora/linear.py: KEPT OUR sgl-project#37460 pipeline
  structure; upstream runtime_lora_scale==0 early-return ordering adopted;
  _add_lora_delta/_add_lora_output_offset wired into pipeline; kept
  UPSTREAM's quality-tiered _scale_lora_delta (deleted our simple variant).
  DEBT: upstream RowParallel bias_in_gemm lossless opt not ported (perf-only).
multimodal_gen/runtime/layers/attention/layer.py: _gather_sharded_sequence
  took upstream sgl-project#43793 .contiguous() fix (subsumes our 55aaa02 guard);
  our _forward_with_kv_gather k/v guards kept (separate path, not covered).
configs/pipeline_configs/base.py + configs/sample/minimax_h3.py + envs.py:
  unions (supports_continuous_batching / teacache_params+fields /
  SGLANG_TEACACHE_CALIBRATE).
models/dits/ltx_2.py mask-fill: took upstream (identical to our fix).
ltx_2/denoising.py: took upstream pass_specs (sgl-project#43667) — subsumes our
  legacy one-stage branch; our _ltx2_model_forward_context helpers already
  upstreamed (entry-142 sgl-project#42928 CONFIRMED SUPERSEDED).
ltx_2/decoding_av.py: KEPT OUR split-stage AV decode (independent pool
  scaling) over upstream single-class restructure; adopted upstream
  chunked pinned-memory _postprocess_video; TODO(merge-4): upstream
  EncodeWhileDecoding streaming + rank-0 retain gating not ported.
kernels/ops/diffusion/activation/silu_mul_bitexact.py: our out= column-slice
  support + XPU guard UNION with upstream @replay_launches (sgl-project#43859).
kernels/ops/diffusion/README.md + docs performance-optimization.mdx: unions.
rust/sglang-renderer: ACCEPTED upstream crate removal (sampling.rs git rm);
  our sgl-project#42779 top_k vocab-bound port moves to sglang-server message/sampling.rs
  (follow-up).
Tests: anthropic test_serving — UNION fake-class exception param + our
  request_preprocessor stubs, took upstream's _FakeNonStreamingOpenAI removal
  (68 pass). test_server_args — union both new tests (pre-existing GPU-only
  collection ENV fail unchanged). test_lora_inference_mode — took upstream
  sgl-project#43028 runtime_scale+quality-tier test (supersedes our adapted variant;
  our kept quality-tiered _scale_lora_delta satisfies it).
  test_fused_moe_triton_config — kept our int4-filename + RTX-PRO-6000 JSON
  tests despite upstream sgl-project#43503 test-drop. test_hicache_file_lru_unit —
  union our corrupt-page-degrade class + upstream sgl-project#42077 uncached-batch test
  (40 pass). test_hicache_host_register — upstream alias host_common.
  test_cuda_graph_setup — took upstream rewrite of our override-decode-runner
  test into draft-local-capture-sizes test (sgl-project#43303 API).

Verified auto-merged stack markers survived: flashinfer_backend KVLocPlan/
prefill_wrappers_ragged/NVFP4-KV forwarding, memory_pool set_kv_buffer_prefix_valid,
fp8 grouped_moe kwarg, fp8_utils reshape fix 3642490, SGLANG_QWEN4_PLE_FILE_REUSE,
schedule_batch freed_rows fix, prefill_chunk_alignment sites, sgl-project#42736 autotune
default-on, PCIe-IPC vocab gathers, flashinfer 0.7.1 pin taken (NOT installed).

Co-Authored-By: Claude Opus 5.5 (1M context)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant