Skip to content

fix(vllm): align KVBM decode KV cache layout #12821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jthomson04 merged 2 commits into from
Aug 10, 2026
Merged

fix(vllm): align KVBM decode KV cache layout #12821

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fde8962
fix(vllm): align KVBM decode KV cache layout
jthomson04 Aug 7, 2026
7657bc1
fix(vllm): align all PdConnector worker layouts
jthomson04 Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions examples/backends/vllm/deploy/disagg_kvbm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ spec:
- Qwen/Qwen3-8B
- --disaggregation-mode
- decode
- --disable-hybrid-kv-cache-manager
- --kv-transfer-config
- '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
- --max-model-len
Expand Down Expand Up @@ -53,6 +54,7 @@ spec:
- Qwen/Qwen3-8B
- --disaggregation-mode
- prefill
- --disable-hybrid-kv-cache-manager
- --max-model-len
- "32000"
- --enforce-eager
Expand Down
2 changes: 2 additions & 0 deletions examples/backends/vllm/deploy/disagg_kvbm_2p2d.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ spec:
- Qwen/Qwen3-8B
- --disaggregation-mode
- decode
- --disable-hybrid-kv-cache-manager
- --kv-transfer-config
- '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
- --max-model-len
Expand Down Expand Up @@ -53,6 +54,7 @@ spec:
- Qwen/Qwen3-8B
- --disaggregation-mode
- prefill
- --disable-hybrid-kv-cache-manager
- --max-model-len
- "32000"
- --enforce-eager
Expand Down
2 changes: 2 additions & 0 deletions examples/backends/vllm/deploy/disagg_kvbm_tp2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ spec:
- Qwen/Qwen3-8B
- --disaggregation-mode
- decode
- --disable-hybrid-kv-cache-manager
- --kv-transfer-config
- '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
- --gpu-memory-utilization
Expand Down Expand Up @@ -59,6 +60,7 @@ spec:
- Qwen/Qwen3-8B
- --disaggregation-mode
- prefill
- --disable-hybrid-kv-cache-manager
- --gpu-memory-utilization
- "0.23"
- --max-model-len
Expand Down
3 changes: 2 additions & 1 deletion examples/backends/vllm/launch/disagg_flexkv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ print_launch_banner "Launching Disaggregated Serving + FlexKV (2 GPUs)" "$MODEL"
python -m dynamo.frontend &

# Run decode worker without FlexKV
CUDA_VISIBLE_DEVICES=0 python -m dynamo.vllm --model $MODEL --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' &
CUDA_VISIBLE_DEVICES=0 python -m dynamo.vllm --model $MODEL --disaggregation-mode decode --disable-hybrid-kv-cache-manager --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' &

# Run prefill worker with FlexKV
DYN_VLLM_KV_EVENT_PORT=20081 \
Expand All @@ -28,6 +28,7 @@ CUDA_VISIBLE_DEVICES=1 \
python -m dynamo.vllm \
--model $MODEL \
--disaggregation-mode prefill \
--disable-hybrid-kv-cache-manager \
--kv-transfer-config '{"kv_connector":"PdConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"FlexKVConnectorV1","kv_role":"kv_both"},{"kv_connector":"NixlConnector","kv_role":"kv_both"}]},"kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20081","enable_kv_cache_events":true}' &

Expand Down
3 changes: 2 additions & 1 deletion examples/backends/vllm/launch/disagg_kvbm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ python -m dynamo.frontend &

# run decode worker on GPU 0, without enabling KVBM
# NOTE: remove --enforce-eager for production use
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm --model "$MODEL" --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' --enforce-eager &
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm --model "$MODEL" --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' --enforce-eager --disaggregation-mode decode --disable-hybrid-kv-cache-manager &
Comment thread
jthomson04 marked this conversation as resolved.
Resolved
Comment thread
jthomson04 marked this conversation as resolved.
Resolved

# run prefill worker on GPU 1 with KVBM enabled using 20GB of CPU cache
# NOTE: remove --enforce-eager for production use
Expand All @@ -28,6 +28,7 @@ CUDA_VISIBLE_DEVICES=1 \
python3 -m dynamo.vllm \
--model "$MODEL" \
--disaggregation-mode prefill \
--disable-hybrid-kv-cache-manager \
--kv-transfer-config '{"kv_connector":"PdConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"DynamoConnector","kv_connector_module_path":"kvbm.vllm_integration.connector","kv_role":"kv_both"},{"kv_connector":"NixlConnector","kv_role":"kv_both"}]},"kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
--enforce-eager \
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20081","enable_kv_cache_events":true}' &
Expand Down
6 changes: 4 additions & 2 deletions examples/backends/vllm/launch/disagg_kvbm_2p2d.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ python -m dynamo.frontend --router-mode kv &

# run decode workers on GPU 0 and 1, without enabling KVBM
# NOTE: remove --enforce-eager for production use
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm --model "$MODEL" --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' --enforce-eager --disaggregation-mode decode &
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm --model "$MODEL" --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' --enforce-eager --disaggregation-mode decode --disable-hybrid-kv-cache-manager &
VLLM_NIXL_SIDE_CHANNEL_PORT=20097 \
CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm --model "$MODEL" --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' --enforce-eager --disaggregation-mode decode &
CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm --model "$MODEL" --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' --enforce-eager --disaggregation-mode decode --disable-hybrid-kv-cache-manager &

# run prefill workers on GPU 2 and 3 with KVBM enabled using 20GB of CPU cache
# NOTE: use different barrier id prefixes for each prefill worker to avoid conflicts
Expand All @@ -31,6 +31,7 @@ CUDA_VISIBLE_DEVICES=2 \
python3 -m dynamo.vllm \
--model "$MODEL" \
--disaggregation-mode prefill \
--disable-hybrid-kv-cache-manager \
--kv-transfer-config '{"kv_connector":"PdConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"DynamoConnector","kv_connector_module_path":"kvbm.vllm_integration.connector","kv_role":"kv_both"},{"kv_connector":"NixlConnector","kv_role":"kv_both"}]},"kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
--enforce-eager \
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20082","enable_kv_cache_events":true}' &
Expand All @@ -43,6 +44,7 @@ CUDA_VISIBLE_DEVICES=3 \
python3 -m dynamo.vllm \
--model "$MODEL" \
--disaggregation-mode prefill \
--disable-hybrid-kv-cache-manager \
--kv-transfer-config '{"kv_connector":"PdConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"DynamoConnector","kv_connector_module_path":"kvbm.vllm_integration.connector","kv_role":"kv_both"},{"kv_connector":"NixlConnector","kv_role":"kv_both"}]},"kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
--enforce-eager \
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20083","enable_kv_cache_events":true}' &
Expand Down
4 changes: 4 additions & 0 deletions examples/backends/vllm/launch/disagg_kvbm_router.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,15 @@ CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm \
--model $MODEL \
--enforce-eager \
--disaggregation-mode decode \
--disable-hybrid-kv-cache-manager \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' &

VLLM_NIXL_SIDE_CHANNEL_PORT=20096 \
CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm \
--model $MODEL \
--enforce-eager \
--disaggregation-mode decode \
--disable-hybrid-kv-cache-manager \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' &

# two prefill workers with KVBM enabled
Expand All @@ -46,6 +48,7 @@ CUDA_VISIBLE_DEVICES=2 DYN_KVBM_CPU_CACHE_GB=20 \
--model $MODEL \
--enforce-eager \
--disaggregation-mode prefill \
--disable-hybrid-kv-cache-manager \
--kv-transfer-config '{"kv_connector":"PdConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"DynamoConnector","kv_connector_module_path":"kvbm.vllm_integration.connector","kv_role":"kv_both"},{"kv_connector":"NixlConnector","kv_role":"kv_both"}]},"kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20081"}' &

Expand All @@ -57,6 +60,7 @@ CUDA_VISIBLE_DEVICES=3 DYN_KVBM_CPU_CACHE_GB=20 \
--model $MODEL \
--enforce-eager \
--disaggregation-mode prefill \
--disable-hybrid-kv-cache-manager \
--kv-transfer-config '{"kv_connector":"PdConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"DynamoConnector","kv_connector_module_path":"kvbm.vllm_integration.connector","kv_role":"kv_both"},{"kv_connector":"NixlConnector","kv_role":"kv_both"}]},"kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20082"}' &

Expand Down
3 changes: 2 additions & 1 deletion examples/backends/vllm/launch/disagg_lmcache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ print_launch_banner "Launching Disaggregated Serving + LMCache (2 GPUs)" "$MODEL
python -m dynamo.frontend --router-mode kv &

# run decode worker on GPU 0, without enabling LMCache
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm --model "$MODEL" &
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm --model "$MODEL" --disaggregation-mode decode --disable-hybrid-kv-cache-manager --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' &

# wait for decode worker to initialize
sleep 20
Expand All @@ -28,6 +28,7 @@ CUDA_VISIBLE_DEVICES=1 \
python3 -m dynamo.vllm \
--model "$MODEL" \
--disaggregation-mode prefill \
--disable-hybrid-kv-cache-manager \
--kv-transfer-config '{"kv_connector":"PdConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"},{"kv_connector":"NixlConnector","kv_role":"kv_both"}]},"kv_connector_module_path":"kvbm.vllm_integration.connector"}' \
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20081","enable_kv_cache_events":true}' &

Expand Down
5 changes: 5 additions & 0 deletions tests/kvbm_integration/test_determinism_disagg.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,9 @@ def _set_up_vllm_config(self, gpu_cache_blocks):
"dynamo.vllm",
"--model",
os.environ.get("KVBM_MODEL_ID", "deepseek-ai/DeepSeek-R1-Distill-Llama-8B"),
"--disaggregation-mode",
"decode",
"--disable-hybrid-kv-cache-manager",
Comment thread
jthomson04 marked this conversation as resolved.
Resolved
"--block-size",
"16",
"--max-model-len",
Expand All @@ -166,6 +169,7 @@ def _set_up_vllm_config(self, gpu_cache_blocks):
os.environ.get("KVBM_MODEL_ID", "deepseek-ai/DeepSeek-R1-Distill-Llama-8B"),
"--disaggregation-mode",
"prefill",
"--disable-hybrid-kv-cache-manager",
"--block-size",
"16",
"--max-model-len",
Expand Down Expand Up @@ -300,6 +304,7 @@ def start_server(self, timeout: int = 300) -> bool:

prefiller_env = self.env.copy()
prefiller_env["CUDA_VISIBLE_DEVICES"] = "1"
prefiller_env["VLLM_NIXL_SIDE_CHANNEL_PORT"] = "20097"

# Launch frontend first
self.process_frontend = subprocess.Popen(
Expand Down
Loading