[ShangmingCai]

Motivation

--prefill-decode-interval N forces N decode-only rounds after every prefill, regardless of how many requests are running. That protects decode ITL once the batch is large, but while the batch is still small it slows its growth: each waiting request waits up to N rounds to be admitted. Throughput-bound deployments (e.g. DeepSeek-V4.1-Flash with --max-running-requests 48 --prefill-decode-interval 4, where TTFT matters less than TPS) want eager admission until the batch is reasonably full, and the interval only after that.

Modifications

• New flag --prefill-decode-interval-min-bs (default 0 = current behavior). The interval only applies while the running batch has at least this many requests.
• Checked in Scheduler._should_defer_prefill(running_batch), not when the interval is armed. Below the threshold, any pending interval is cleared and prefill is admitted. So a batch that shrinks below the threshold mid-interval (requests finishing) resumes admission immediately.
• Under DP attention, arming still uses the globally synchronized is_extend_in_batch, and the threshold is compared against each rank's own running batch (no batch size is synced across ranks). A rank below the threshold admits prefill every round. Because that prefill re-arms the interval on every rank, a rank that is still above the threshold keeps deferring until the small rank crosses the threshold or drains its queue, or until its own batch shrinks below the threshold. This is bounded and adds no collective, so no desync is introduced; with the default 0 nothing changes. Gating on a group-level batch size (e.g. a slot in the DP sync tensor) can be a follow-up if a deployment needs it.
• Validation: the new value must be non-negative.
• Docs: added a row for the new flag to server_arguments.mdx, and while there, a row for --prefill-decode-interval itself (it was still on the server_arguments_undocumented.txt allowlist; removed from the list).

Related: #43844 (chunk-only interval protection) also changes this code path. The two policies compose; whichever lands second needs a trivial rebase.

Accuracy Tests

No model-forward change, only scheduling admission timing.

Unit tests, rebased onto main e262b44eef1 (CPU-only run; the touched tests do not need a GPU):

• test_scheduler_prefill_decode_interval.py + test_scheduler_hicache_events.py: 11 passed, 7 subtests passed. New cases: below threshold admits and clears; at/above threshold defers; shrinking below threshold mid-interval ends it.
• test_server_args.py (TestPrepareServerArgs): 22 passed, 25 subtests passed, including the new negative-value rejection.
• test_server_args_docs_coverage.py: 4 passed. Red check: with --prefill-decode-interval left on the allowlist, test_undocumented_allowlist_is_current fails as expected.

Earlier run on an rx devbox at the pre-rebase head (main d8f85223485): test_scheduler_prefill_decode_interval.py, test_scheduler_hicache_events.py, test_scheduler_chunked_req_gate.py, test_scheduler_dp_converted_batch_full.py, test_server_args_docs_coverage.py, test_server_args.py, test_model_overrides.py: 430 passed, 310 subtests passed.

ruff check (pre-commit rule set) + ruff format + isort clean on all changed files.

Speed Tests and Profiling

Not yet benchmarked end to end. Expected effect: identical to today when the flag is unset. With it set, at low concurrency the effect is the same as interval 0; near the batch cap it matches the configured interval.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ⏳ Run #39730845141
Latest PR Test (Extra): ❌ Run #39730845015
Latest PR Test (AMD ROCm 10): ⏳ Run #39730845098

[ShangmingCai]

/tag-and-rerun-ci

[ShangmingCai]

TODO before this is ready to merge:

• rebase onto latest main. The branch is behind the MIN_BASE_SHA gate (ff8d964), so check-changes bails out and none of the suites actually run. Then /tag-and-rerun-ci again.
• fix the DP attention paragraph in the description. A rank that is still above the threshold gets re-armed by the small rank's prefill every step, so it isn't quite "same as interval 0" for that rank. Not a correctness problem, just inaccurate wording. Gating on a group-level batch size can be a follow-up if anyone needs it.
• while touching the docs, maybe add a row for --prefill-decode-interval itself, it's still in the undocumented list.

[ShangmingCai]

Done, all three items:

• Rebased onto main e262b44eef1, which is past the current MIN_BASE_SHA (6a39ddf1f2e). check-changes passes on 7c07efe52b5 and the base suites are running: https://github.com/sgl-project/sglang/actions/runs/39438087739
• Rewrote the DP attention paragraph in the description. A rank still above the threshold keeps deferring while a smaller rank prefills every round, until the small rank crosses the threshold or drains its queue, or its own batch shrinks below the threshold. Bounded, no new collective; a group-level gate can be a follow-up.
• --prefill-decode-interval now has its own row in server_arguments.mdx and is off the server_arguments_undocumented.txt allowlist.

Local check on the rebased tree: test_scheduler_prefill_decode_interval.py + test_scheduler_hicache_events.py 11 passed (7 subtests); test_server_args_docs_coverage.py 4 passed, and it goes red if the flag is left on the allowlist; TestPrepareServerArgs 22 passed.

For the record, the two red base-a-test-cpu partitions (1 and 7) on the previous run hit the 15-minute job timeout with no failing test; the other 12 partitions were green. Not related to this change.

[akhilg-nv]

[by Claude Code]

Thanks for this. We hit a related case and want to suggest a small extension, since it touches the same check.

The case. When nothing is decoding, there is no decode ITL for the interval to protect, yet with the default
--prefill-decode-interval-min-bs 0 the interval still applies. The interval counts scheduler loop iterations, not
decode rounds, and with no decode work those iterations are empty spins (recv_requests plus on_idle). So a long
prompt that arrives at an idle server waits about N empty iterations after each of its chunks.

We measured this end to end; numbers will come with the follow-up PR.

Suggestion. Treat "nothing is decoding" as below the threshold even at the default min_bs = 0, on the non-DP
path only:

def _should_defer_prefill(self, running_batch: ScheduleBatch) -> bool:
    if self._prefill_decode_interval_remaining == 0:
        return False
    # With no decode to protect, waiting only idles the GPU between chunks.
    # DP keeps the synchronized cadence because peer ranks may still be decoding.
    nothing_decodes = running_batch.is_empty() or running_batch.is_prefill_only
    if running_batch.batch_size() < self.prefill_decode_interval_min_bs or (
        nothing_decodes and not self.require_mlp_sync
    ):
        self._prefill_decode_interval_remaining = 0
        return False
    self._prefill_decode_interval_remaining -= 1
    return True

Why it is safe:

• It only fires when there is no decode work, so no ITL can get worse.
• It leaves DP attention unchanged.
• It makes the bug fix default-on while your flag still controls the small-batch policy.

min_bs = 1 already covers the non-DP part, but it is opt-in and also changes DP behavior.

We have three unit tests for exactly this, which pass with the change above:

• an empty running batch does not defer;
• a prefill-only running batch does not defer;
• DP still defers without local decode.

We are happy to push them to this branch or open a small follow-up after this lands, whichever you prefer.

[mintlify]

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit d4f8a52.

Approve preview

[ShangmingCai]

@akhilg-nv Thanks — agreed this is a real bug: the interval counts loop iterations, so with nothing decoding it just idles the GPU between chunks. It's independent of this PR (it reproduces on main with the flag unset), and this PR is meant to keep the default exactly unchanged, so I'd rather not fold a default-on change in here. Please open the small follow-up with your three tests and VR200 numbers once this lands; happy to review. In the meantime --prefill-decode-interval-min-bs 1 covers the non-DP empty-batch case as an opt-in.