[Model Runner V2] Add KV Connector support#32742
[Model Runner V2] Add KV Connector support#32742WoosukKwon merged 7 commits intovllm-project:mainfrom
Conversation
njhill
force-pushed
the
mrv2-kvconnectors
branch
from
64bce72 to
f687265
Compare
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for KV Connector, which facilitates Key-Value cache transfers. The changes involve modifying the init_kv_cache function to return the initialized KV caches, updating the ModelRunner to register these caches with the KV Connector, and refactoring preemption handling into a dedicated method within a new mixin. The execute_model_state and ModelRunnerOutput are also updated to propagate KV Connector specific outputs. Overall, the changes are well-structured and integrate the new functionality effectively.
|
This pull request has merge conflicts that must be resolved before it can be |
njhill
force-pushed
the
mrv2-kvconnectors
branch
from
f00d986 to
96d00ed
Compare
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
force-pushed
the
mrv2-kvconnectors
branch
2 times, most recently
from
3f7c1cb to
d398473
Compare
Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
force-pushed
the
mrv2-kvconnectors
branch
from
d398473 to
9196aaf
Compare
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
Comment @cursor review or bugbot run to trigger another review on this PR
Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: 陈建华 <1647430658@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
3 participants
Tested with cpu offloading and NIXL P/D.