Skip to content

[KV Connector] Update lmcache kv_offloading_backend to use LMCacheMPConnector - #42865

Merged
ApostaC merged 9 commits into
vllm-project:mainfrom
maobaolong:update_lmcache_shortcut
Jun 4, 2026
Merged

[KV Connector] Update lmcache kv_offloading_backend to use LMCacheMPConnector#42865
ApostaC merged 9 commits into
vllm-project:mainfrom
maobaolong:update_lmcache_shortcut

Conversation

@maobaolong

@maobaolong maobaolong commented May 17, 2026

Copy link
Copy Markdown
Contributor

Purpose

Switch the default kv_offloading_backend=lmcache path to LMCache's multi-process (MP) mode so that vLLM talks to a standalone LMCache server via LMCacheMPConnector instead of the legacy in-process LMCacheConnectorV1.

Use LMCacheMPConnector as the connector for --kv-offloading-backend lmcache.

Drop the now-irrelevant lmcache.local_cpu / lmcache.max_local_cpu_size extra config — KV capacity is owned by the standalone LMCache server in MP mode.

Rely on LMCacheMPConnector defaults (tcp://localhost:5555, mq_timeout=300s, heartbeat_interval=10s); users can still override via kv_connector_extra_config (lmcache.mp.host / lmcache.mp.port / ...).
Removed the unused num_kv_ranks local along the way.

Test Plan

  1. Start an LMCache server listening on tcp://localhost:5555.
  2. Launch vLLM with the new default path:
   vllm serve <model> \
       --kv-offloading-backend lmcache \
       --kv-offloading-size 20

Test Result

vLLM and lmcache multi-process server can start successfully, and double send a repeat request to the vllm then check the log from lmcache server to verify that chunk stored and retrieved.


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

…onnector

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the lmcache backend configuration in vllm/config/vllm.py by replacing LMCacheConnectorV1 with LMCacheMPConnector. This change transitions the system to use multi-process mode, where KV storage capacity is managed by a standalone LMCache server process. Consequently, the calculation for num_kv_ranks and the explicit configuration of local CPU size have been removed. I have no feedback to provide as there were no review comments.

@maobaolong

Copy link
Copy Markdown
Contributor Author

@ApostaC This is the PR updated the kv_offloading_backend to use LMCacheMPConnector by default. PTAL

@ApostaC ApostaC left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one looks good to me! @maobaolong, please make sure that the LMCache side has enough information to avoid the user's confusion.

@ApostaC
ApostaC enabled auto-merge (squash) May 21, 2026 04:19
@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label May 21, 2026
@maobaolong

Copy link
Copy Markdown
Contributor Author

This one looks good to me! @maobaolong, please make sure that the LMCache side has enough information to avoid the user's confusion.

@ApostaC Thanks for your review, will create a PR on lmcache side to give a warn if the lmcache mp server cannot reach.

@maobaolong

Copy link
Copy Markdown
Contributor Author

LMCache/LMCache#3365 @ApostaC The lmcache-side PR is here, please help to take a look when you have time, thanks!

@ApostaC

ApostaC commented May 27, 2026

Copy link
Copy Markdown
Collaborator

@maobaolong UT seems to be failing:
image

auto-merge was automatically disabled May 30, 2026 11:07

Head branch was pushed to by a user without write access

@mergify mergify Bot added the v1 label May 30, 2026
@maobaolong
maobaolong force-pushed the update_lmcache_shortcut branch from 6b9ca3c to f9cd161 Compare May 31, 2026 10:25
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
cpu_test image doesn't have lmcache installed, so loading the real
LMCacheMPConnector class crashes the test at VllmConfig() construction.
Swap the lazy loader for a placeholder via monkeypatch — the test only
checks the connector name string and the extra-config dict, never builds
a connector, so a bare class is enough.

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
@maobaolong
maobaolong force-pushed the update_lmcache_shortcut branch from f9cd161 to 7170a0a Compare June 1, 2026 01:27
@ApostaC
ApostaC merged commit b58e082 into vllm-project:main Jun 4, 2026
54 checks passed
mvanhorn pushed a commit to mvanhorn/vllm that referenced this pull request Jun 4, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
JisoLya pushed a commit to JisoLya/vllm that referenced this pull request Jun 5, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Signed-off-by: JisoLya <523420504@qq.com>
knight0528 pushed a commit to knight0528/vllm that referenced this pull request Jun 8, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
waqahmed-amd-fi pushed a commit to waqahmed-amd-fi/vllm that referenced this pull request Jun 10, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Signed-off-by: Waqar Ahmed <waqar.ahmed@amd.com>
Saddss pushed a commit to Saddss/vllm that referenced this pull request Jun 14, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
divineearthly pushed a commit to divineearthly/vllm that referenced this pull request Jun 19, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Signed-off-by: divineearthly <divineearthly@gmail.com>
nkzhenhua pushed a commit to nkzhenhua/vllm that referenced this pull request Jun 24, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
ohsono pushed a commit to ohsono/vllm that referenced this pull request Jul 3, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
zhangcong2019 pushed a commit to zhangcong2019/vllm that referenced this pull request Jul 18, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026
…onnector (vllm-project#42865)

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kv-connector ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants