[Refactor] Build seven more decoders from stage boundaries

[ch-wan]

This PR is part of a stack (oldest at bottom):

• -> [Refactor] Build seven more decoders from stage boundaries #41779
• [Refactor] Simplify layer boundary internals #41778
• [Fix] Step-3.5: build the shared expert without TP under all-to-all MoE backends #41754
• [Refactor] Send producer-written residual streams through to_pp #41753
• [Fix] Reduce the Sarvam dense MLP through its row-parallel projection #41752
• [Refactor] Leave decoder stacks through final_norm(skip_empty=True) #41751
• [Refactor] Rename layer boundary protocols, contracts and helpers #41750
• [Fix] Release consumed residual contributions #41749

Motivation

Apertus, Spark 2.5, Mixtral, Arcee, Kimi-Linear, EXAONE 4.0 and Nanbeige still thread their residual tensor through each decoder layer by hand, and send both tensors to the next pipeline rank themselves. Building them from stage boundaries gives them the same attention and FFN boundaries as the other decoders: the pipeline handoff, the attention-DP rows and the aux hidden-state captures.

Modifications

• finish_complete_output() moves an FFN output that compute has already reduced. On two paths the move bound for the exit also completes the sum, with a reduce-scatter onto the attention-TP slices of the residual or a CP reduce-scatter, so such an output was summed twice. The non-summing counterpart of those moves is now bound at construction and used there.
• residual/post_norm.py: a plain read and a post-norm add (residual + norm(output)), for decoders that read the residual without a norm and normalize each sublayer's output before adding it. The plain read rejects a quantized input format and a post-residual addition, with or without a residual to update.
• One commit per model:
  • The attention output projection leaves its TP sum to the FFN input (reduce_results=False), which runs the same all-reduce and fused add + norm as before.
  • The MLP or MoE still completes its own sum and finishes through finish_complete_output(). No sum is left to the next layer.
  • The model enters and leaves the stack through residual_batch (start, from_pp, to_pp, final_norm). EAGLE3 and DFlash captures come from the boundary; Kimi-Linear's DSpark captures read the stream through residual_batch.snapshot() and still travel to the next pipeline rank.
  • Kimi-Linear declares per layer whether its FFN is sparse, and Mixtral declares every FFN sparse.
  • EXAONE 4.0 is post-LN and uses the post-norm operations. The layer writes the MLP's post-norm add itself, so the stream it hands on is already written.
  • Nanbeige folds the residual back in between loops, applies the loop norm when the config asks for it, and starts a new stream for the next pass over the layers.
• Behavior changes:
  • EXAONE 4.0 loops over its own layers only. It used to call every layer, including the placeholders of other pipeline ranks, which made --pp-size 2 fail at launch. Its split-prefill path now ends with the same final norm as forward() instead of norming the output added to itself.
  • Spark 2.5 already builds its attention on the attention-TP group. Under attention DP the boundaries now also bring its MLP the gathered rows.
  • With an explicit --enable-flashinfer-allreduce-fusion or --enable-quant-communications, the attention all-reduce of these models now takes the fused or quantized path. None of these architectures enables the fusion automatically, so the default configuration runs the same kernels as before.

Accuracy Tests

Each model on a real checkpoint, the tree before this PR against this PR. TP2 is python -m sglang.benchmark.one_batch --correctness-test (prefill logits and three generations), run twice on the parent. The server runs greedy-decode four prompts with input and output logprobs and the top-5 logprobs.

Model	GPU	TP2 one_batch	Server
swiss-ai/Apertus-8B-2509	H200	identical	PP2 identical
XHToken/Spark-X2.5-4B	H200	identical	PP2 identical
mistralai/Mixtral-8x7B-Instruct-v0.1	H200	identical	PP2 identical
arcee-ai/AFM-4.5B-Base	H200	identical	PP2 identical
moonshotai/Kimi-Linear-48B-A3B-Instruct	B200	identical	PP2 identical (see below)
LGAI-EXAONE/EXAONE-4.0-1.2B	H200	identical	TP1 identical
LGAI-EXAONE/EXAONE-4.0-32B	H200		TP1 identical; PP2 of this PR identical to its TP1 (the parent fails to launch with PP2)
Nanbeige/Nanbeige4.2-3B (two loops)	H200	identical	TP1 identical

• Kimi-Linear PP2: the parent's first launch differs from its own second launch by up to 0.177 in logprob, from first-launch kernel tuning. The parent's second run and both runs of this PR are bitwise identical.
• 109 affected unit test files (2,697 tests) pass at the top of this stack, the same as on main.

Not addressed here: EXAONE 4.0 checkpoints with tied embeddings (the 1.2B) still cannot run with PP, because the last pipeline rank has no lm_head.

Not tested: Nanbeige with PP. As before this PR, each pipeline rank runs every loop over its own layers only, so looped checkpoints (num_loops > 1) are not claimed to work with PP.

Speed Tests and Profiling

Not applicable: the same kernels and collectives run in the default configuration.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): 🚫 Run #36668115383
Latest PR Test (Extra): 🚫 Run #36668115221
Latest PR Test (AMD ROCm 10): ⏳ Run #36668115633

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
sglang-doc	🟢 Ready	View Preview	Sep 30, 2026, 4:18 AM

💡 Tip: Enable Automations to automatically generate PRs for you.