[TPU] Use Ray for default distributed backend #8389
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
|
@WoosukKwon I'm curious of the reason not to use multiprocessing distributed backend for this? |
|
@njhill Good question. Actually, the MP backend would also work for TPUs. However, I think users such as GKE prefer Ray, because 1) they are interested in multi-host inference (which TPU is quite good at), and 2) they are already familiar with Ray. |
| @@ -869,6 +869,13 @@ def __init__( | |||
| f"distributed executor backend " | |||
| f"'{self.distributed_executor_backend}'.") | |||
|
|
|||
| if current_platform.is_tpu() and self.world_size > 1: | |||
| if self.distributed_executor_backend is None: | |||
| self.distributed_executor_backend = "ray" | |||
There was a problem hiding this comment.
I mean, I think this line should be enough to change the default backend to ray in tpu case.
There was a problem hiding this comment.
Oh the error is for those who use distributed_executor_backend="mp".
There was a problem hiding this comment.
why do we need to error if users explicitly specify the mp backend?
There was a problem hiding this comment.
The MP backend is not supported for TPUs at the moment. Without this line, the user will get the error:
"/vllm/engine/llm_engine.py", line 505, in _get_executor_cls
assert distributed_executor_backend is None
There was a problem hiding this comment.
I'm confused by Actually, the MP backend would also work for TPUs
so MP backend for TPU is actually not implemented yet.
There was a problem hiding this comment.
Yes. We don't have a executor for tpu + mp.
There was a problem hiding this comment.
cc @njhill if you have any misunderstanding. it is because we currently only have ray backend supported in tpu.
Signed-off-by: Alvant <[email protected]>
Signed-off-by: LeiWang1999 <[email protected]>
No description provided.