Repository navigation
fix(scheduler): remove a request's queue entry once it stops waiting #43061
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
mateo-berri
merged 15 commits into
BerriAI:main
from
RachelHuangZW:litellm_scheduler_remove_admitted_requests
Oct 8, 2026
Merged
Changes from 3 commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
876fa55
fix(scheduler): remove a request's queue entry once it stops waiting
RachelHuangZW da4dc27
test(router): allowlist _wait_for_scheduler_turn in the router covera…
RachelHuangZW 629b9ef
fix(scheduler): admit healthy requests before reading the queue, and …
RachelHuangZW f5383a8
refactor(scheduler): move the wait loop into Scheduler.wait_for_turn
RachelHuangZW 471cc69
Merge remote-tracking branch 'upstream/main' into litellm_scheduler_r…
RachelHuangZW d4596aa
Merge remote-tracking branch 'origin/main' into litellm_scheduler_rem…
RachelHuangZW e7db391
test(scheduler): move scheduler tests into tests/unit
RachelHuangZW 0c9ab36
fix(scheduler): finish the queue removal when cancellation is deliver…
RachelHuangZW fe16915
Merge remote-tracking branch 'upstream/main' into litellm_scheduler_r…
RachelHuangZW f9b785a
Merge remote-tracking branch 'upstream/main' into litellm_scheduler_r…
RachelHuangZW 5de5644
Merge remote-tracking branch 'origin/main' into litellm_scheduler_rem…
mateo-berri e6cdc06
fix(scheduler): read queue entries back from redis as tuples
mateo-berri 1313444
fix(scheduler): admit a request whose queue entry vanished during coo…
mateo-berri c835b73
fix(scheduler): re-enqueue a request whose queue entry vanished durin…
mateo-berri 0be0db2
test(integration): cover priority scheduler queue cleanup across inst…
mateo-berri File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| from typing import Final | ||
|
|
||
| import pytest | ||
|
|
||
| from litellm.scheduler import FlowItem, Scheduler | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_poll_admits_request_missing_from_queue_while_a_deployment_is_healthy(): | ||
| scheduler: Final = Scheduler() | ||
|
|
||
| assert await scheduler.poll( | ||
| id="erased-by-concurrent-write", model_name="sched-model", health_deployments=[{"model_info": {"id": "a"}}] | ||
| ) | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_poll_during_cooldown_admits_only_the_head_of_the_queue(): | ||
| scheduler: Final = Scheduler() | ||
| await scheduler.add_request(FlowItem(priority=2, request_id="later", model_name="sched-model")) | ||
| await scheduler.add_request(FlowItem(priority=1, request_id="head", model_name="sched-model")) | ||
|
|
||
| assert not await scheduler.poll(id="later", model_name="sched-model", health_deployments=[]) | ||
| assert await scheduler.poll(id="head", model_name="sched-model", health_deployments=[]) | ||
| assert await scheduler.get_queue("sched-model") == [(2, "later")] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.