[minor] Make DEFAULT_FORCE_STREAM_INTERVAL configurable via SGLANG_FORCE_STREAM_INTERVAL - #23215
Merged
hnyls2002 merged 2 commits intoApr 22, 2026
Conversation
ByronHsu
requested review from
Ying1123,
hnyls2002,
merrymercy and
xiezhq-hermann
as code owners
April 20, 2026 05:57
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Register `SGLANG_FORCE_STREAM_INTERVAL` (default `50`, matching today's hardcoded value) and resolve `DEFAULT_FORCE_STREAM_INTERVAL` from it at module import. The constant continues to be a plain int at the call site so the per-token check stays a cheap integer comparison. Motivation: for non-streaming requests the scheduler only flushes an intermediate output batch every `DEFAULT_FORCE_STREAM_INTERVAL` decoded tokens, which is also when `set_first_token_time()` runs. With the default of 50, the engine's TTFT histogram is biased upward by roughly `(N-1) * ITL` tokens of decode time (~50x the true TTFT for short ITLs), which makes benchmarking TTFT under `stream=False` unreliable. Making this overridable lets operators / benchmarks dial the interval down (e.g. `SGLANG_FORCE_STREAM_INTERVAL=1`) for accurate TTFT measurements without changing engine defaults.
ByronHsu
force-pushed
the
byron/configurable-force-stream-interval
branch
from
April 20, 2026 06:03
2e043b1 to
19e00f6
Compare
ByronHsu
changed the title
hnyls2002
approved these changes
Apr 22, 2026
Collaborator
|
/tag-and-rerun-ci |
ByronHsu
added a commit
that referenced
this pull request
Apr 23, 2026
…RCE_STREAM_INTERVAL (#23215)
caitengwei
pushed a commit
to caitengwei/sglang
that referenced
this pull request
Jun 1, 2026
…RCE_STREAM_INTERVAL (sgl-project#23215)
Chronostasys
pushed a commit
to MindLab-Research/sglang
that referenced
this pull request
Aug 24, 2026
…RCE_STREAM_INTERVAL (sgl-project#23215)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
For non-streaming requests (
stream=False), the scheduler only flushes an intermediate output batch everyDEFAULT_FORCE_STREAM_INTERVALdecoded tokens (currently a hardcoded50). That same flush is whenset_first_token_time()runs, so the engine's TTFT histogram (sglang:time_to_first_token_seconds) is biased upward by roughly(N - 1) * ITL— i.e. reported TTFT can be ~50× the true TTFT.This makes benchmarking TTFT under
stream=Falsehard to interpret, and there is currently no way to override the interval without patching the source.Modifications
SGLANG_FORCE_STREAM_INTERVALtosglang/srt/environ.py(EnvInt(50), default unchanged).DEFAULT_FORCE_STREAM_INTERVALfrom the env var at module import inscheduler_output_processor_mixin.py, so the per-token check stays a plainint % int == 0comparison with no extra overhead.Fully backward-compatible. Typical usage for accurate TTFT benchmarks:
Checklist