Repository navigation
[Refactor] scheduler: Hold wrapped requests in request lists and abort through each list - #43908
Conversation
2449c2c to
dae91d1
Compare
688f2b7 to
9f103f0
Compare
ab1ed7a to
12a039b
Compare
9f103f0 to
5e2e57d
Compare
5e2e57d to
01b52e8
Compare
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
This comment has been minimized.
This comment has been minimized.
…to request lists; fix the prealloc ready count
… through each list
This comment has been minimized.
This comment has been minimized.
|
/tag-and-rerun-ci |
…red duplicates by request
…sts as RequestList
|
/rerun-test test_decode_retraction_backup.py test_hisparse_unit.py test_uocr_doc_prefetch_cuda.py test_decode_prealloc_no_host_sync.py registered/unit/disaggregation/test_kv_checksum.py registered/disaggregation/test_disaggregation_basic.py test_disaggregation_chunked_prefill_abort.py test_unlimited_ocr_v2_server.py test_retract_decode_logprob.py test_disaggregation_decode_offload.py test_disaggregation_optimistic_prefill.py test_disaggregation_unified_memory.py test_prefill_complete_allocation.py test_retract_decode.py test_update_weights_from_tensor.py test_session_control.py test_disaggregation_hisparse.py test_disaggregation_kv_checksum.py test_disaggregation_decode_radix_linker.py registered/disaggregation/test_disaggregation_pp.py test_disaggregation_dp_attention.py test_disaggregation_lora.py |
|
🚀 🚀 🚀 🚀 🚀 🚀 |
# Conflicts: # python/sglang/srt/managers/scheduler_components/deferred_responses.py
…t through each list (sgl-project#43908)
Request lists whose owners keep extra state with each request (a
DecodeRequest, a release record, a HiSparse action, a deferred response) use the same_RequestListas the plain lists, and their aborts move onto the lists._RequestListtakesreq_of, which finds the request in a joined item: it iterates requests, keysleave/take/clearby request, and gives the owner its items throughitems()._RequestLists. The prealloc, transfer and deferred aborts move fromScheduler.abort_requestonto the lists; releasing and HiSparse staging keep having none. Poll loops take one snapshot per pass, as the queue order is all-reduced across ranks; the priority sort writes the new order back withclearandjoin.UOCR_PARKED(waiting for the previous window) and the newUOCR_RELEASED(waiting to be queued); their abort replacesdrop_parkedand the uOCR block inabort_request.DecodeRequest.seqlenand.priority,DeferredResponses.held_reqs()andDocPrefetch.held_reqs()/drop()are removed: readers get the request from the list.decode_queue/_RequestListand identity-hashedFakeReqs.Behavior changes:
request_lists()order, so the abort outputs of different requests can be sent in a different order.Test plan
test_every_follow_up_of_a_window_is_releasedcovers the uOCR fix.test_decode_loads_count_prealloc_requests_waiting_for_inputcovers the prealloc ready count read through the wrapped queue.test/registered/unit/{managers,disaggregation,dllm,constrained,mem_cache,observability,session}, pass locally withSGLANG_IS_IN_CI=1, apart from failures the base has too.CI States
Latest PR Test (Base): 🚫 Run #41604956583
Latest PR Test (Extra): ❌ Run #41604956672
Latest PR Test (AMD ROCm 10): 🚫 Run #41604956536
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: