[serve] update to support latest vLLM#58945
Conversation
soodoshll
force-pushed
the
fix-vllm-args
branch
from
a5c6dce to
b657d20
Compare
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates ray-serve to support the latest version of vLLM, which has a modified API for its OpenAI-compatible server. The changes introduce backward compatibility by inspecting the signature of vLLM's init_app_state function at runtime and calling it with the appropriate arguments. My review focuses on improving the maintainability of this new compatibility logic. I've suggested a refactoring to reduce code duplication, making the code cleaner and easier to maintain.
| if "vllm_config" in inspect.signature(init_app_state).parameters: | ||
| await init_app_state( | ||
| self._engine_client, | ||
| vllm_config=vllm_engine_config, | ||
| state=state, | ||
| args=args, | ||
| ) | ||
| else: | ||
| await init_app_state( | ||
| self._engine_client, | ||
| state=state, | ||
| args=args, | ||
| ) |
There was a problem hiding this comment.
To improve maintainability and reduce code duplication, you can refactor this logic to build a dictionary of keyword arguments and then call init_app_state once. This avoids repeating the call and makes the code more concise and easier to read.
init_kwargs = {
"engine_client": self._engine_client,
"state": state,
"args": args,
}
if "vllm_config" in inspect.signature(init_app_state).parameters:
init_kwargs["vllm_config"] = vllm_engine_config
await init_app_state(**init_kwargs)
|
Hi @soodoshll, thanks for the PR! We're tracking the upgrade work here: #58026. I see that this PR maintains some backward compatibility with the previous version of vLLM. Is this a requirement on your end? |
|
Hi @eicherseiji. Thanks for pointing me to the right PR! We currently don't have a requirement for back compatibility. |
ray-gardener
bot
added
serve
eicherseiji
left a comment
eicherseiji
left a comment
There was a problem hiding this comment.
Moving forward with this one while #58026 is pending
## Description vLLM's interface of launching an OpenAI-compatible server has changed. This PR makes necessary changes to make ray-serve work with latest vLLM. ## Related issues ray-project#58937 --------- Signed-off-by: Qidong Su <qidongs@nvidia.com> Signed-off-by: peterxcli <peterxcli@gmail.com>
3 participants
Description
vLLM's interface of launching an OpenAI-compatible server has changed. This PR makes necessary changes to make ray-serve work with latest vLLM.
Related issues
#58937