Skip to content

[core] add sleep and wake up endpoint and v1 support#12987

Merged
youkaichao merged 24 commits intovllm-project:mainfrom
youkaichao:sleep_apiserver
Feb 20, 2025
Merged

[core] add sleep and wake up endpoint and v1 support#12987
youkaichao merged 24 commits intovllm-project:mainfrom
youkaichao:sleep_apiserver

Conversation

@youkaichao
Copy link
Member

@youkaichao youkaichao commented Feb 9, 2025 •
edited
Loading

expose /sleep and /wake_up endpoint, and also support it in v1

@youkaichao
add sleep and wake up endpoint
3a7ecb4 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add tests
4fbb39e 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add ack
e74dd9a 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@github-actions
Copy link

github-actions bot commented Feb 9, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@youkaichao
add status code check
79ec652 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix socket
13f4bdf 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
use send_request_and_get_response
fa86994 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix
ef145d9 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix output
042c959 
Signed-off-by: youkaichao <youkaichao@gmail.com>
youkaichao
youkaichao commented Feb 9, 2025
View reviewed changes
youkaichao
youkaichao commented Feb 9, 2025
View reviewed changes
njhill
njhill reviewed Feb 9, 2025
View reviewed changes
@njhill
Copy link
Member

njhill commented Feb 9, 2025

@youkaichao I wonder if we could merge #12918 first which simplifies the v1 request handling a bit? (it should be ready now)

@mergify
Copy link

mergify bot commented Feb 10, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @youkaichao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Feb 10, 2025
@njhill njhill mentioned this pull request Feb 11, 2025
Closed
@mergify mergify bot removed the needs-rebase label Feb 12, 2025
cennn and others added 6 commits February 12, 2025 14:54
@cennn
support VLLM_USE_V1 && fix typo WAKE_UP
4010671 
Signed-off-by: cennn <2523403608@qq.com>
@youkaichao
Merge branch 'main' into sleep_apiserver
1f808d5
@youkaichao
add comments
283507d 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
minimize diff
e6d618d 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
minimize diff
7511795 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add v1 test
aa23022 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao youkaichao changed the title [core] add sleep and wake up endpoint [core] add sleep and wake up endpoint and v1 support Feb 13, 2025
@youkaichao
add timing
f7bcffa 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add v1 test
aaba2f4 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add v1 test
4b8bd93 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao youkaichao mentioned this pull request Feb 15, 2025
Merged
cennn and others added 4 commits February 19, 2025 23:41
@cennn
Merge branch 'main' of https://github.com/vllm-project/vllm into slee…
e23031f 
…p-youkaichao

Signed-off-by: cennn <2523403608@qq.com>
@cennn
rm time.sleep(30)
83813d8 
Signed-off-by: cennn <2523403608@qq.com>
@youkaichao
improve tests
c795115 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
improve tests
1657de0 
Signed-off-by: youkaichao <youkaichao@gmail.com>
youkaichao
youkaichao commented Feb 20, 2025
View reviewed changes
vllm/entrypoints/openai/serving_transcription.py
# TODO(rob): figure out a way to pipe streaming in.
# Non-streaming response.
try:
assert result_generator is not None
Copy link
Member Author

@youkaichao youkaichao Feb 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is to pass the linter.

@youkaichao youkaichao marked this pull request as ready for review February 20, 2025 02:21
@youkaichao youkaichao requested a review from njhill February 20, 2025 02:27
njhill
njhill approved these changes Feb 20, 2025
View reviewed changes
Copy link
Member

@njhill njhill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @youkaichao LGTM

vllm/entrypoints/openai/api_server.py Outdated
await engine_client(raw_request).sleep(int(level))
# when we return a response, the sleep command
# is sent but does not finish yet.
# TODO: we should wait for the sleep to finish and then return
Copy link
Member

@njhill njhill Feb 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's worth updating this comment since it does now wait in V1?

Copy link
Member Author

@youkaichao youkaichao Feb 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated in e8416d9 now.

@youkaichao
update comments
e8416d9 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao youkaichao merged commit ba81163 into vllm-project:main Feb 20, 2025
7 of 9 checks passed
@youkaichao youkaichao deleted the sleep_apiserver branch February 20, 2025 04:41
xjpang pushed a commit to xjpang/vllm that referenced this pull request Feb 20, 2025
@youkaichao @cennn
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
46b8c77 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
Akshat-Tripathi pushed a commit to krai/vllm that referenced this pull request Mar 3, 2025
@youkaichao @cennn @Akshat-Tripathi
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
5cde704 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
@sivanantha321 sivanantha321 mentioned this pull request Mar 4, 2025
Merged
9 tasks
lulmer pushed a commit to lulmer/vllm that referenced this pull request Apr 7, 2025
@youkaichao @cennn @lulmer
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
8d2dca1 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>
shreyankg pushed a commit to shreyankg/vllm that referenced this pull request May 3, 2025
@youkaichao @cennn @shreyankg
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
e34e54d 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@comaniac comaniac comaniac approved these changes

@njhill njhill njhill approved these changes

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

@ywang96 ywang96 Awaiting requested review from ywang96

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat

Assignees

No one assigned

Labels

frontend v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@youkaichao @njhill @comaniac @cennn