[qu0b]

Motivation

With a nonzero prefill/decode interval, a completed cached or short prefill starts another full decode-only hold. At interval 64, an already-waiting request can therefore miss 64 scheduler admission opportunities even though no unfinished prompt chunk remains.

Add an opt-in policy to protect decoding between unfinished prompt chunks while allowing admission immediately after a completed prefill. The default remains unchanged.

Modifications

SGLANG_PDI_CHUNKED_ONLY=1 arms the interval only when batch.chunked_req is present; completing a prefill clears any remaining hold. Data-parallel attention always retains the synchronized fixed cadence, using the existing global extend flag. This adds no rank-local timers or collectives.

Tests cover the default policy, completed/unfinished prefills, clearing a previous hold, decode/empty batches, and matching DP cadence on ranks with different local work. Developed with AI assistance.

Accuracy Tests

pytest -q test/registered/unit/managers/test_scheduler_prefill_decode_interval.py: 10 passed, 3 subtests passed, importing the current upstream scheduler. All pre-commit hooks pass on the three changed files.

This changes admission timing, with no model-forward or kernel changes. A prior downstream serving validation passed its 10-case quality gate and concurrent long-prompt recall checks; that is supplementary evidence, not a full current-main serving qualification.

Speed Tests and Profiling

Prior controlled measurement of this same policy on a v0.6.0-based serving build, two RTX PRO 6000 Blackwell Max-Q GPUs at 300 W, concurrency 2, interval 64: 576 replay requests across four sequential arms. Median cached TTFT changed from 726 to 380 ms (-47.7%; session-bootstrap 95% interval -58.1% to -22.1%). Completion median changed from 989 to 1,054 ms (inconclusive); p90 TTFT was essentially unchanged at 976 versus 987 ms. Long-prefill overlap decode remained approximately 114 tokens/s. This is workload-specific evidence, not a general throughput claim or a measurement of current upstream main.

Paired CUPTI/Python traces on that build identified the mechanism: the fixed policy made zero prefill-admission calls in 25 observed scheduler decisions while GPUs continued decoding; chunk-only protection admitted a mixed batch. This motivated the narrow scheduler change.

The added CPU tests provide a deterministic reproduction of the admission behavior with synthetic batches. Request payloads and raw deployment traces are not part of this contribution.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #38241381224
Latest PR Test (Extra): ❌ Run #38241380867
Latest PR Test (AMD ROCm 10): ❌ Run #38241381443