Skip to content

perf: backport prefill interval batch gate - #25

Merged
knapcio merged 1 commit into
mainfrom
perf/prefill-interval-min-bs-20261022
Oct 22, 2026
Merged

knapcio merged 1 commit into
mainfrom
perf/prefill-interval-min-bs-20261022

Conversation

@knapcio

@knapcio knapcio commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Summary

  • backport the runtime part of SGLang #44001 to the pinned f80c91a4b tree with full upstream credit
  • keep the new --prefill-decode-interval-min-bs option at zero by default, preserving current scheduling
  • add zero-fuzz staging, semantic verification, build-time tests, CI coverage, and opt-in operator docs

Why

With --prefill-decode-interval 2, a small running batch can delay queued requests before the interval is useful. The new optional threshold admits prefills eagerly below a selected live batch size and restores the interval at/above it. This is a multi-user TTFT candidate, not a C1 decode-TPS claim.

Verification

  • fresh f80c91a4b GitHub tarball: #41303, #42901 and #44001 applied with patch --fuzz=0
  • all three semantic verifiers passed on the staged tree
  • offline regression gate: 54 tests passed
  • Python compile, shell syntax and git diff --check passed

No fleet, container, NUC, network or live serving state was touched. GB10 A/B remains required before enabling min-bs in a deployment.

Credits: Shangming Cai / sgl-project/sglang#44001.

@knapcio
knapcio merged commit bc425f1 into main Oct 22, 2026
2 checks passed
@knapcio
knapcio deleted the perf/prefill-interval-min-bs-20261022 branch October 22, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant