Repository navigation
[Refactor] scheduler: Track running requests in an owner list - #43899
Merged
Merged
Conversation
hnyls2002
requested review from
Ying1123,
merrymercy and
xiezhq-hermann
as code owners
October 12, 2026 19:50
hnyls2002
force-pushed
the
lsyin/sched-pr04
branch
from
October 13, 2026 00:43
7a854ba to
a11cf1b
Compare
hnyls2002
requested review from
ByronHsu,
Duyi-Wang,
HaiShaw,
JustinTong0323,
ShangmingCai,
sogalin,
wisclmy0611,
yeahdongcn and
zijiexia
as code owners
October 13, 2026 00:43
hnyls2002
force-pushed
the
lsyin/sched-pr05
branch
from
October 13, 2026 00:44
dc44ace to
17a7300
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr04
branch
from
October 13, 2026 01:00
a11cf1b to
6d02e99
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr05
branch
from
October 13, 2026 01:00
17a7300 to
5617123
Compare
hnyls2002
marked this pull request as draft
October 13, 2026 02:01
hnyls2002
force-pushed
the
lsyin/sched-pr04
branch
from
October 30, 2026 03:14
6d02e99 to
2a1f5eb
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr05
branch
from
October 30, 2026 03:14
5617123 to
0601ed6
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr04
branch
from
October 30, 2026 08:30
2a1f5eb to
1e80c5f
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr05
branch
from
October 30, 2026 08:30
0601ed6 to
57360b0
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr04
branch
from
October 30, 2026 08:54
1e80c5f to
a1836b2
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr05
branch
from
October 30, 2026 08:54
57360b0 to
7ea29a0
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr04
branch
from
October 30, 2026 11:09
1bd78e3 to
a52c207
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr05
branch
from
October 30, 2026 11:09
08b5bcf to
482f93c
Compare
hnyls2002
force-pushed
the
lsyin/sched-pr04
branch
from
October 30, 2026 22:15
a52c207 to
824a3aa
Compare
…nk launches and leaves when it finishes, is retracted or preempted, or a pause retracts it; a debug check compares it with the batch-derived running set
hnyls2002
force-pushed
the
lsyin/sched-pr05
branch
from
October 30, 2026 23:07
482f93c to
93df528
Compare
…ing requests finish, instead of at the schedule filter
hnyls2002
marked this pull request as ready for review
October 30, 2026 23:23
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
…ts switch SGLANG_ENABLE_RUNNING_REQS_CHECK; keep RunningReqs in one ordered dict with join, leave and leave_finished, dropping clear and reqs(); the pause test fakes hash by identity like Req
…maybe_check_running_reqs); the scheduling test fakes get a checker
…for both the launch join and the check (prefill_launch_starts_running); define running in the RunningReqs docstring
…-cache check's CI default
…overlap-flow and retract tests, and keep the dp-converted test at main's shape
…d step's decode rows are running already, or finished by the result processed just before the launch
…a running row already dedups, and only a row finished by the result processed before the launch must not join
Yuxingwang-intel
pushed a commit
to Yuxingwang-intel/sglang
that referenced
this pull request
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The scheduler has no owner for "running": the running requests are spread over
running_batch,last_batch, the overlap result copies and the PP micro-batches, and a request just prefilled lives only inlast_batchuntil the next merge. This adds an owner list kept at state transitions. Nothing reads it yet; later PRs move the readers (abort targets, the running timeout, the weight-version inventory, the idle check) to it.RunningReqs(scheduler_components/running_reqs.py), owned by the scheduler asrunning_reqs: the running requests in join order (join,leave,leave_finished), rebound as a new dict on every change so iterating while requests leave is safe.run_batch, and the MLX overlap launch): the extend or mixed step's requests other than its chunked request and any finished row. A step composed before the previous result is processed (overlap off, e.g. a DP-converted decode step) can still carry a row that result finished; it must not join._prefill_launch_starts_running()says when a launch does not start a request running: a PD prefill request goes to the KV transfer, a PD decode request starts when the prebuilt batch is merged, a HiSparse one when_build_hisparse_decode_batchmerges it after staging, and a dLLM one never runs.process_batch_result: running requests finish only while a step is processed), whenretract_decoderetracts or aborts it, when priority preemption takes it, and when a retract-mode pause retracts it (under PP that pause covers only the current micro-batch).SGLANG_ENABLE_RUNNING_REQS_CHECK(on by default in CI, likeSGLANG_ENABLE_TREE_CACHE_SANITY_CHECK; off otherwise) makesSchedulerInvariantChecker.maybe_check_running_reqsrun at each schedule, after the merge and filter. It checks that the list equals the unfinished requests the batches hold (running_batchandlast_batch, every micro-batch under PP), apart from the chunked request. When a prefill launch does not start a request running, extend step batches are left out of that set.test_scheduler_running_reqs.py: launch joins through the realrun_batch(chunked, mixed, decode, and the modes that skip it), an overlap sequence through the realget_next_batch_to_runwith the check on, and the retract and preemption leaves.test_scheduler_dp_converted_batch_full.pyruns with the check on; unit fakes that reach the hooks get the list.No behavior change.
CI States
Latest PR Test (Base): 🚫 Run #41177278010
Latest PR Test (Extra): 🚫 Run #41177278023
Latest PR Test (AMD ROCm 10): 🚫 Run #41177277979