[KV Connector] Update lmcache kv_offloading_backend to use LMCacheMPConnector - #42865
Conversation
…onnector Signed-off-by: baoloongmao <baoloongmao@tencent.com>
There was a problem hiding this comment.
Code Review
This pull request updates the lmcache backend configuration in vllm/config/vllm.py by replacing LMCacheConnectorV1 with LMCacheMPConnector. This change transitions the system to use multi-process mode, where KV storage capacity is managed by a standalone LMCache server process. Consequently, the calculation for num_kv_ranks and the explicit configuration of local CPU size have been removed. I have no feedback to provide as there were no review comments.
|
@ApostaC This is the PR updated the |
ApostaC
left a comment
There was a problem hiding this comment.
This one looks good to me! @maobaolong, please make sure that the LMCache side has enough information to avoid the user's confusion.
@ApostaC Thanks for your review, will create a PR on lmcache side to give a warn if the lmcache mp server cannot reach. |
|
LMCache/LMCache#3365 @ApostaC The lmcache-side PR is here, please help to take a look when you have time, thanks! |
|
@maobaolong UT seems to be failing: |
Head branch was pushed to by a user without write access
6b9ca3c to
f9cd161
Compare
cpu_test image doesn't have lmcache installed, so loading the real LMCacheMPConnector class crashes the test at VllmConfig() construction. Swap the lazy loader for a placeholder via monkeypatch — the test only checks the connector name string and the extra-config dict, never builds a connector, so a bare class is enough. Signed-off-by: baoloongmao <baoloongmao@tencent.com>
f9cd161 to
7170a0a
Compare
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: JisoLya <523420504@qq.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: Waqar Ahmed <waqar.ahmed@amd.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: divineearthly <divineearthly@gmail.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com>
…onnector (vllm-project#42865) Signed-off-by: baoloongmao <baoloongmao@tencent.com>

Purpose
Switch the default kv_offloading_backend=lmcache path to LMCache's multi-process (MP) mode so that vLLM talks to a standalone LMCache server via LMCacheMPConnector instead of the legacy in-process LMCacheConnectorV1.
Use LMCacheMPConnector as the connector for --kv-offloading-backend lmcache.
Drop the now-irrelevant lmcache.local_cpu / lmcache.max_local_cpu_size extra config — KV capacity is owned by the standalone LMCache server in MP mode.
Rely on LMCacheMPConnector defaults (tcp://localhost:5555, mq_timeout=300s, heartbeat_interval=10s); users can still override via kv_connector_extra_config (lmcache.mp.host / lmcache.mp.port / ...).
Removed the unused num_kv_ranks local along the way.
Test Plan
Test Result
vLLM and lmcache multi-process server can start successfully, and double send a repeat request to the vllm then check the log from lmcache server to verify that chunk stored and retrieved.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.