Skip to content

[Refactor] scheduler: Hold wrapped requests in request lists and abort through each list - #43908

Merged
hnyls2002 merged 12 commits into
mainfrom
lsyin/sched-pr14
Nov 3, 2026
Merged

hnyls2002 merged 12 commits into
mainfrom
lsyin/sched-pr14

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Request lists whose owners keep extra state with each request (a DecodeRequest, a release record, a HiSparse action, a deferred response) use the same _RequestList as the plain lists, and their aborts move onto the lists.

  • _RequestList takes req_of, which finds the request in a joined item: it iterates requests, keys leave / take / clear by request, and gives the owner its items through items().
  • The PD decode prealloc, transfer and releasing queues, HiSparse staging and deferred responses are _RequestLists. The prealloc, transfer and deferred aborts move from Scheduler.abort_request onto the lists; releasing and HiSparse staging keep having none. Poll loops take one snapshot per pass, as the queue order is all-reduced across ranks; the priority sort writes the new order back with clear and join.
  • uOCR keeps its held document windows in two lists, UOCR_PARKED (waiting for the previous window) and the new UOCR_RELEASED (waiting to be queued); their abort replaces drop_parked and the uOCR block in abort_request.
  • DecodeRequest.seqlen and .priority, DeferredResponses.held_reqs() and DocPrefetch.held_reqs() / drop() are removed: readers get the request from the list.
  • Test fakes build these queues with decode_queue / _RequestList and identity-hashed FakeReqs.

Behavior changes:

  • Within one abort, these lists are aborted in request_lists() order, so the abort outputs of different requests can be sent in a different order.
  • Fixes a uOCR follow-up window being lost: when two follow-ups waited on the same window, the second replaced the first, which was then never queued, aborted or reset. Both now stay parked, and both are released with the remaining budget when the window finishes.

Test plan

  • test_every_follow_up_of_a_window_is_released covers the uOCR fix.
  • test_decode_loads_count_prealloc_requests_waiting_for_input covers the prealloc ready count read through the wrapped queue.
  • The unit tests that build these queues or call the changed methods, and the unit tests under test/registered/unit/{managers,disaggregation,dllm,constrained,mem_cache,observability,session}, pass locally with SGLANG_IS_IN_CI=1, apart from failures the base has too.

CI States

Latest PR Test (Base): 🚫 Run #41604956583
Latest PR Test (Extra): ❌ Run #41604956672
Latest PR Test (AMD ROCm 10): 🚫 Run #41604956536

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@github-actions github-actions Bot added hicache Hierarchical Caching for SGLang Multi-modal multi-modal language model labels Oct 12, 2026
@hnyls2002
hnyls2002 marked this pull request as draft October 13, 2026 02:01
@hnyls2002
hnyls2002 force-pushed the lsyin/sched-pr13 branch 2 times, most recently from ab1ed7a to 12a039b Compare November 3, 2026 04:15
@hnyls2002 hnyls2002 changed the title [PP] Join and drop a slot's running rows at the ring commit [Refactor] scheduler: Hold wrapped requests in request lists and abort through each list Nov 3, 2026
Base automatically changed from lsyin/sched-pr13 to main November 3, 2026 06:32
@hnyls2002
hnyls2002 marked this pull request as ready for review November 3, 2026 06:35

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 3, 2026
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_decode_retraction_backup.py test_hisparse_unit.py test_uocr_doc_prefetch_cuda.py test_decode_prealloc_no_host_sync.py registered/unit/disaggregation/test_kv_checksum.py registered/disaggregation/test_disaggregation_basic.py test_disaggregation_chunked_prefill_abort.py test_unlimited_ocr_v2_server.py test_retract_decode_logprob.py test_disaggregation_decode_offload.py test_disaggregation_optimistic_prefill.py test_disaggregation_unified_memory.py test_prefill_complete_allocation.py test_retract_decode.py test_update_weights_from_tensor.py test_session_control.py test_disaggregation_hisparse.py test_disaggregation_kv_checksum.py test_disaggregation_decode_radix_linker.py registered/disaggregation/test_disaggregation_pp.py test_disaggregation_dp_attention.py test_disaggregation_lora.py

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-5090 (7 tests): ✅ View workflow run

cd test/ && python3 registered/unit/mem_cache/test_decode_retraction_backup.py
cd test/ && python3 registered/unit/managers/test_hisparse_unit.py
cd test/ && python3 registered/unit/session/test_uocr_doc_prefetch_cuda.py
cd test/ && python3 registered/unit/disaggregation/test_decode_prealloc_no_host_sync.py
cd test/ && python3 registered/scheduler/test_retract_decode_logprob.py
cd test/ && python3 registered/scheduler/test_retract_decode.py
cd test/ && python3 registered/rl/test_update_weights_from_tensor.py

🚀 1-gpu-h100 (3 tests): ❌ View workflow run

cd test/ && python3 registered/unit/disaggregation/test_kv_checksum.py
cd test/ && python3 registered/vlm/test_unlimited_ocr_v2_server.py
cd test/ && python3 registered/sessions/test_session_control.py

🚀 2-gpu-h100 (7 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py
cd test/ && python3 registered/disaggregation/test_disaggregation_chunked_prefill_abort.py
cd test/ && python3 registered/disaggregation/test_disaggregation_decode_offload.py
cd test/ && python3 registered/disaggregation/test_disaggregation_optimistic_prefill.py
cd test/ && python3 registered/disaggregation/test_disaggregation_unified_memory.py
cd test/ && python3 registered/e2e/disaggregation/test_prefill_complete_allocation.py
cd test/ && python3 registered/e2e/disaggregation/test_disaggregation_lora.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_hisparse.py

🚀 4-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_kv_checksum.py
cd test/ && python3 registered/disaggregation/test_disaggregation_decode_radix_linker.py

🚀 8-gpu-h20 (2 tests): ❌ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_pp.py
cd test/ && python3 registered/disaggregation/test_disaggregation_dp_attention.py

# Conflicts:
#	python/sglang/srt/managers/scheduler_components/deferred_responses.py
@hnyls2002
hnyls2002 merged commit 94aaad8 into main Nov 3, 2026
83 of 106 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/sched-pr14 branch November 3, 2026 08:45
Yuxingwang-intel pushed a commit to Yuxingwang-intel/sglang that referenced this pull request Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang Multi-modal multi-modal language model run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant