-
-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC]: Add runtime weight update API #5723
Comments
@youkaichao the transfer is preferred to be via NCCL |
More importantly, we need to support establishing an NCCL group between DeepSpeed and vLLM engines. |
Update: See: |
Motivation.
In online RL training, vLLM can significantly accelerate the rollout stage. To achieve this, we need weight sync from main training process to vLLM worker process, and then call the existing API in vLLM to update the weights by
model_runner.model.load_weights
An example of such implementation can be found in OpenRLHF, https://github.com/OpenLLMAI/OpenRLHF/blob/main/openrlhf/trainer/ray/vllm_worker_wrap.py
However, user has to monkey patch vLLM worker to introduce such behavior. It would be great if vLLM naturally supports weight sync at runtime.
Proposed Change.
def update_weight(self, name, dtype, shape)
then in master process, user can achieve weight sync via the following (modified from OpenRLHF):
Feedback Period.
No response
CC List.
No response
Any Other Things.
No response
The text was updated successfully, but these errors were encountered: