Repository navigation
[Elastic EP] Support Model Runner V2 #53934
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+171
−80
Merged
Changes from 2 commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
fb971a6
[Elastic EP] Support Model Runner V2
almogtavor 533aaf5
Merge branch 'main' into mrv2-elastic-ep
almogtavor c932a47
register eplb on elastic ep dummy load so new workers join the commun…
almogtavor f1670f5
skip the request level warmup when elastic ep re-warms after a scale
almogtavor ca0ca71
trim the model runner and elastic execute comments
almogtavor 3abc725
[Elastic EP] Isolate MRV2 warmup state instead of skipping the warmup
almogtavor 1e3378f
Merge branch 'main' into mrv2-elastic-ep
almogtavor f0fad46
Support elastic EP warmup with reserved slots
almogtavor 3b98fa0
Merge branch 'main' into mrv2-elastic-ep
almogtavor 4119d7b
refactor: improve warmup handling by removing reserved slot configura…
almogtavor 1755262
Merge branch 'main' into mrv2-elastic-ep
almogtavor fa23c77
refactor: enhance request handling and warmup conditions in GPU model…
almogtavor 6ef02bb
refactor: improve logging for request draining in AsyncLLM
almogtavor babb214
Drop unrelated whitespace, formatting and comment changes in worker a…
almogtavor 0b3f7dc
Explain the null block redirect flag on BlockTables
almogtavor 2921bd9
Move the dp property comment inside each property and remove dead war…
almogtavor 0097800
Move the MRV2 warmup context manager onto EPLBController and drop the…
almogtavor 8b5bfee
Drop the AsyncLLM admission lock and register parallel sampling child…
almogtavor c1f8042
Merge branch 'main' into mrv2-elastic-ep
almogtavor fbea15a
Move preserve_serving_state onto the MRV2 model runner
almogtavor ca7b124
Suppress EPLB stepping during Elastic EP warmup and drop mocked tests
almogtavor 812c0f2
[Elastic EP] Drain MRV2 requests with the engine-side wait pause
almogtavor c80da5d
Merge remote-tracking branch 'upstream/main' into mrv2-elastic-ep
almogtavor b4de0a5
[Elastic EP] Document the drain assumption of preserve_serving_state
almogtavor 2b339b0
Update vllm/v1/engine/core_client.py
almogtavor 8657192
Merge remote-tracking branch 'upstream/main' into mrv2-elastic-ep
almogtavor 85e9725
[Elastic EP] Reset the MRV2 speculator when releasing CUDA graphs
almogtavor a9ca522
Update vllm/v1/worker/gpu/model_runner.py
almogtavor 8db0a00
[Elastic EP] Move the MRV2 warmup isolation into eplb_utils
almogtavor 1029423
[Elastic EP] Type the model runner and model config args in eplb_utils
almogtavor c714a73
Merge branch 'main' into mrv2-elastic-ep
njhill 2dbd4cb
Merge branch 'main' into mrv2-elastic-ep
njhill 47b5df5
[Tests] Fix test_speculator_capture_preserves_decode_query_bounds for…
njhill 84c1899
Merge branch 'main' into mrv2-elastic-ep
njhill File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.