Skip to content

[Refactor] scheduler: Track running requests in an owner list - #43899

Merged
hnyls2002 merged 12 commits into
mainfrom
lsyin/sched-pr05
Oct 31, 2026
Merged

hnyls2002 merged 12 commits into
mainfrom
lsyin/sched-pr05

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

The scheduler has no owner for "running": the running requests are spread over running_batch, last_batch, the overlap result copies and the PP micro-batches, and a request just prefilled lives only in last_batch until the next merge. This adds an owner list kept at state transitions. Nothing reads it yet; later PRs move the readers (abort targets, the running timeout, the weight-version inventory, the idle check) to it.

  • Add RunningReqs (scheduler_components/running_reqs.py), owned by the scheduler as running_reqs: the running requests in join order (join, leave, leave_finished), rebound as a new dict on every change so iterating while requests leave is safe.
  • A request is running once its whole prompt is launched, until it finishes; the steps still computing for it are tracked apart. It joins when a launched step runs its last prefill chunk (run_batch, and the MLX overlap launch): the extend or mixed step's requests other than its chunked request and any finished row. A step composed before the previous result is processed (overlap off, e.g. a DP-converted decode step) can still carry a row that result finished; it must not join. _prefill_launch_starts_running() says when a launch does not start a request running: a PD prefill request goes to the KV transfer, a PD decode request starts when the prebuilt batch is merged, a HiSparse one when _build_hisparse_decode_batch merges it after staging, and a dLLM one never runs.
  • A request leaves when it finishes (at the end of process_batch_result: running requests finish only while a step is processed), when retract_decode retracts or aborts it, when priority preemption takes it, and when a retract-mode pause retracts it (under PP that pause covers only the current micro-batch).
  • PP keeps one list for all micro-batches.
  • SGLANG_ENABLE_RUNNING_REQS_CHECK (on by default in CI, like SGLANG_ENABLE_TREE_CACHE_SANITY_CHECK; off otherwise) makes SchedulerInvariantChecker.maybe_check_running_reqs run at each schedule, after the merge and filter. It checks that the list equals the unfinished requests the batches hold (running_batch and last_batch, every micro-batch under PP), apart from the chunked request. When a prefill launch does not start a request running, extend step batches are left out of that set.
  • Add test_scheduler_running_reqs.py: launch joins through the real run_batch (chunked, mixed, decode, and the modes that skip it), an overlap sequence through the real get_next_batch_to_run with the check on, and the retract and preemption leaves. test_scheduler_dp_converted_batch_full.py runs with the check on; unit fakes that reach the hooks get the list.

No behavior change.


CI States

Latest PR Test (Base): 🚫 Run #41177278010
Latest PR Test (Extra): 🚫 Run #41177278023
Latest PR Test (AMD ROCm 10): 🚫 Run #41177277979

@hnyls2002
hnyls2002 marked this pull request as draft October 13, 2026 02:01
@github-actions github-actions Bot added hicache Hierarchical Caching for SGLang apple-silicon labels Oct 30, 2026
@hnyls2002 hnyls2002 changed the title [Scheduler] Split next-batch planning into named stages [Refactor] scheduler: Drain pending finishes once per loop iteration Oct 30, 2026
@hnyls2002 hnyls2002 changed the title [Refactor] scheduler: Drain pending finishes once per loop iteration [Refactor] scheduler: Finish an aborted chunked request at the drain Oct 30, 2026
@hnyls2002 hnyls2002 added bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) parallel-stages CI: stages dispatch together instead of waiting on each other labels Oct 30, 2026
@hnyls2002 hnyls2002 added run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci) run-ci CI: run the baseline test suite on this PR labels Oct 30, 2026
@hnyls2002 hnyls2002 changed the title [Refactor] scheduler: Finish an aborted chunked request at the drain [Refactor] scheduler: Count the launched steps carrying each request Oct 30, 2026
Base automatically changed from lsyin/sched-pr04 to main October 30, 2026 22:56
…nk launches and leaves when it finishes, is retracted or preempted, or a pause retracts it; a debug check compares it with the batch-derived running set
@github-actions github-actions Bot added the lora label Oct 30, 2026
@hnyls2002 hnyls2002 changed the title [Refactor] scheduler: Count the launched steps carrying each request [Refactor] scheduler: Track running requests in an owner list Oct 30, 2026
…ing requests finish, instead of at the schedule filter
@hnyls2002
hnyls2002 marked this pull request as ready for review October 30, 2026 23:23

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

hnyls2002 and others added 10 commits October 30, 2026 16:33
…ts switch SGLANG_ENABLE_RUNNING_REQS_CHECK; keep RunningReqs in one ordered dict with join, leave and leave_finished, dropping clear and reqs(); the pause test fakes hash by identity like Req
…maybe_check_running_reqs); the scheduling test fakes get a checker
…for both the launch join and the check (prefill_launch_starts_running); define running in the RunningReqs docstring
…overlap-flow and retract tests, and keep the dp-converted test at main's shape
…d step's decode rows are running already, or finished by the result processed just before the launch
…a running row already dedups, and only a row finished by the result processed before the launch must not join
@hnyls2002
hnyls2002 merged commit b8d8635 into main Oct 31, 2026
70 of 101 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/sched-pr05 branch October 31, 2026 06:37
Yuxingwang-intel pushed a commit to Yuxingwang-intel/sglang that referenced this pull request Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

apple-silicon bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) hicache Hierarchical Caching for SGLang lora parallel-stages CI: stages dispatch together instead of waiting on each other run-ci CI: run the baseline test suite on this PR run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant