[PP] Batch independent chunks for small-chunk disaggregated prefill

[YAMY1234]

Motivation

Scope correction: This draft improves the case where each request is capped at a 2K chunk but the aggregate model-forward budget is increased to approximately 32K. It does not improve a model forward whose aggregate batch is fixed at 2K, so it is not a solution to the fixed-small-batch PP bubble problem. The corrected fixed-2K launch/control-overhead work is proposed separately in #36248; the measurements below should be read only as an independent-chunk aggregation experiment.

In PP disaggregated prefill, chunked_prefill_size currently serves two different purposes: the per-request chunk boundary and the aggregate token budget of the entire forward batch. With chunked_prefill_size=2048, this limits every PP prefill forward to approximately 2K tokens even when many independent requests are waiting, so increasing client concurrency cannot recover GPU utilization and pipeline efficiency.

This PR separates those two limits for an opt-in PP disaggregated-prefill path: each request remains capped by chunked_prefill_size, while the batch can aggregate independent chunks up to max_prefill_tokens. For the evaluated 2K configuration, one forward can therefore combine up to approximately 16 independent chunks instead of launching one approximately 2K-token forward at a time.

Modifications

This PR is organized as five dependency commits:

1. Add a per-request chunk limit to PrefillAdder and use max_prefill_tokens as the aggregate budget, allowing multiple independent middle chunks in one PP prefill batch.
2. Track, cache, transfer, and requeue all batched middle chunks without losing their committed ChunkCache progress.
3. Apply the same per-request chunk limit to the ignore_eos admission path so it cannot consume the aggregate budget as one request.
4. Let requeued middle chunks reuse the request-pool slots they already own when the pool is saturated.
5. Prioritize those slot-owning continuations whenever slotless requests would exhaust the remaining request slots before all continuations are visited, preventing both full-pool and partial-capacity head-of-line blocking.

The first item is the throughput mechanism. Items 2-5 are correctness and liveness requirements introduced by batching multiple independent middle chunks; they are not unrelated scheduler fixes. The new path is guarded by SGLANG_PP_BATCH_INDEPENDENT_CHUNKS and currently requires PP disaggregated prefill, max_prefill_tokens > chunked_prefill_size, overlap scheduling disabled, and ChunkCache (--disable-radix-cache). The default path is unchanged.

The patch intentionally excludes one-page tail merging, pure-middle-chunk PP output skipping, and 2K prefill BCG because their controlled peak contributions were small, inconsistent, or negative.

Accuracy Tests

Matched GSM8K A/B used the same Qwen3.5 NVFP4 model, 4-GPU TP1 PP4 prefill, 4-GPU TP4 decode, and 2K per-request chunks. The protocol was 200 questions, five-shot chat, 128 threads, temperature=0.6, top_p=0.95, and top_k=20.

Arm	Independent chunks	Aggregate budget	GSM8K score	Correct
Before	off	2,048	0.980	196/200
After	on	33,792	0.995	199/200

Both runs completed 200/200 requests without errors, and their captured request payloads were byte-identical. Since the protocol uses stochastic sampling without a per-request seed, this is an accuracy non-regression result rather than an accuracy-improvement claim.

Scheduler correctness tests: 6/6 passed. git diff --check, seven-file py_compile, and lint also passed.

Speed Tests and Profiling

Setup

Field	Value
Model	nvidia/Qwen3.5-397B-A17B-NVFP4-V2
Precision	NVFP4 weights, FP8 KV cache
Prefill topology	4 x GB300, TP1 PP4
Decode topology	4 x GB300, TP4 PP1
Serving	Dynamo disaggregated prefill/decode with Mooncake transfer
Workload	random data, ISL=8192, OSL=1, request rate=inf
Concurrency	16, 32, 64, 128, 256
Requests per point	10 x concurrency
Page size	64
Prefill settings	overlap disabled, radix cache disabled, dynamic chunking disabled, CUDA graph disabled, max running requests=128
Before 2K	chunked_prefill_size=2048, aggregate budget=2048
After 2K	chunked_prefill_size=2048, max_prefill_tokens=33792, aggregate FlashInfer dispatch budget=33792
32K reference	chunked_prefill_size=max_prefill_tokens=32768
Metric	aggregate input tokens/s

The controlled five-point performance run used the final no-tail/no-output-skip ablation source with runtime semantics equivalent to this PR. After diagnosing the C256 result, a same-configuration C128/C256 follow-up changed only the partial-capacity queue-ordering rule and source identity. The five commits were then cleaned and transplanted onto current upstream main; the final main-based branch reran the targeted correctness and formatting tests but did not repeat the GB300 sweep. Every benchmark point completed all requests with exact 8192/1 lengths and zero non-empty errors.

Baseline chunk-size sweep

Chunk size	C16	C32	C64	C128	C256	Peak concurrency	Peak input tok/s
2K	30,507	30,318	30,093	29,873	29,690	C16	30,507
4K	58,392	58,652	58,442	57,617	57,027	C32	58,652
8K	113,512	114,642	114,614	113,200	113,069	C32	114,642
16K	116,059	166,216	167,231	168,354	167,057	C128	168,354
32K	88,803	139,336	176,180	177,586	177,273	C128	177,586

The original 2K peak reached only 17.18% of the 32K peak. Raising client concurrency from 16 to 256 reduced 2K throughput by 2.68%, confirming that the aggregate forward shape, rather than request supply, was the limiting factor.

2K before and after

Concurrency	Original 2K	Batched 2K	Matched-concurrency speedup	32K reference	Batched 2K / 32K
16	30,507	60,417	1.98x	88,803	68.04%
32	30,318	84,924	2.80x	139,336	60.95%
64	30,093	135,633	4.51x	176,180	76.99%
128	29,873	170,652	5.71x	177,586	96.10%
256	29,690	171,489	5.78x	177,273	96.74%

Peak-to-peak, the final core path improves 2K throughput from 30,507 tok/s at C16 to 171,489 tok/s at C256: 5.621x / +462.12%. It reaches 96.74% of the matched-concurrency 32K reference while preserving a 2K per-request chunk boundary. C128 also remains non-regressed at 170,652 tok/s, and C256 is 0.49% above C128 rather than falling 20.53% below it.

The C256 follow-up directly isolates the partial-capacity queue-ordering change:

C256 metric	Before partial-capacity fix	After	Change
Input throughput	133,843 tok/s	171,489 tok/s	+28.13%
Benchmark duration	156.69 s	122.29 s	-21.95%
Mean TTFT	15,196.7 ms	11,718.7 ms	-22.89%
P99 TTFT	19,355.3 ms	13,827.6 ms	-28.56%
PP0 full 33,792-token batches	472 / 897 (52.62%)	614 / 640 (95.94%)	+43.32 pp
Mean PP0 new tokens per batch	23,445.5	32,860.6	+40.16%

Both C256 runs completed 2560/2560 requests with exact 8192/1 lengths and zero non-empty errors. Before the fix, a large waiting queue still produced repeated 2K multiples because a few newly available request slots were consumed by slotless requests before the scan reached slot-owning continuations. After the fix, the full-batch ratio rises to 95.94%, which explains the throughput recovery without changing the 2K per-request cap or any kernel.

An earlier batching-only prototype already reached 52,246 / 92,368 / 143,483 tok/s at C16/C32/C64, or 1.71x / 3.05x / 4.77x over the original 2K baseline, before request-slot saturation stopped C128. This isolates independent-chunk aggregation as the primary throughput mechanism; the slot reuse and queue-ordering commits make the new mechanism complete at C128/C256 rather than accelerating an individual GPU forward.

Excluded micro-optimizations

These increments were measured in the controlled sweep before the final partial-capacity queue follow-up and are not combined with the new C256 result.

Incremental change	C16	C32	C64	C128	C256	Peak effect
One-page tail merge over core	+8.45%	-4.29%	-2.84%	+0.43%	+6.10%	+0.43%
Pure-middle output skip over tail	+3.58%	+8.10%	+2.33%	+0.72%	-2.01%	+0.72%
2K prefill BCG	approximately -4%	approximately -4%	approximately -4%	not continued	not continued	negative

Tail merging plus output skipping increased peak throughput from 168,421 to 170,363 tok/s, only +1.15% in one run, while showing inconsistent effects across concurrency. They are intentionally left out of this PR so the upstream patch contains only the dominant batching mechanism and its required correctness/liveness support.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

[YAMY1234]

Closing this draft because it changes aggregate batch construction and does not address the genuine fixed-small-forward PP bubble. The corrected implementation and matched benchmarks are in #36248.