Skip to content

docs: add KV Cache Management section with KVCR pages for vLLM and SGLang #17544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
callanjfox wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

docs: add KV Cache Management section with KVCR pages for vLLM and SGLang #17544

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/fern/index.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -603,6 +603,34 @@ navigation:
contents:
- page: Mocker Engine Architecture
path: pages/developer-guide/knowledge-base/modular-components/backends/mocker/mocker-engine-architecture.md
- section: KV Cache Management
slug: kv-offload
collapsed: true
contents:
- page: KV Cache Management Overview
path: pages/developer-guide/knowledge-base/kv-offload/overview.md
- page: KV Cache Runner (KVCR)
path: pages/developer-guide/knowledge-base/kv-offload/kvcr.md
- page: How KV Hints Work
path: pages/developer-guide/knowledge-base/kv-offload/kv-hints.md
- page: Configure KVCR for vLLM
path: pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-vllm.md
- page: Run KVCR on Kubernetes with vLLM
path: pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-vllm.md
- page: Run KVCR Guard on Kubernetes with vLLM
path: pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-vllm.md
- page: Verify and Monitor KVCR with vLLM
path: pages/developer-guide/knowledge-base/kv-offload/kvcr-testing-monitoring-vllm.md
- page: Configure KVCR for SGLang
path: pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang.md
- page: Configure KVCR Linker Mode for SGLang
path: pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang-linker.md
- page: Run KVCR on Kubernetes with SGLang
path: pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-sglang.md
- page: Run KVCR Guard on Kubernetes with SGLang
path: pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-sglang.md
- page: Tune Routing for KVCR
path: pages/developer-guide/knowledge-base/kv-offload/kvcr-routing-tuning.md
- section: Developing and Extending Dynamo
slug: advanced-customizations
icon: sliders
Expand Down
144 changes: 144 additions & 0 deletions ...pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang-linker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: Configure KVCR Linker Mode for SGLang
subtitle: Load KV from other workers straight into GPU memory, with optional Guard, on one GPU server
---

> [!WARNING]
> Preview: needs Dynamo [#16827](https://github.com/ai-dynamo/dynamo/pull/16827) and SGLang [#45650](https://github.com/sgl-project/sglang/pull/45650) with the KVCR changes it is stacked on ([#45646](https://github.com/sgl-project/sglang/pull/45646) to [#45649](https://github.com/sgl-project/sglang/pull/45649)), none merged. One host only.

In Linker mode, KVCR loads KV pages directly between GPU memory and its own host-memory pool, layer by layer. Unlike HiCache mode ([Configure KVCR for SGLang](configure-kvcr-sglang.md)), there is no HiCache host tier in between.

## Before You Begin

- A runtime with these builds; no release image contains them:
- Dynamo with [#16827](https://github.com/ai-dynamo/dynamo/pull/16827)
- SGLang with the KVCR Linker ([#45650](https://github.com/sgl-project/sglang/pull/45650) and the PRs it is stacked on)
- KVCR `v0.2.0` and NIXL `1.5.0`
- Two GPUs, one per worker.
- On one host without etcd or NATS, set `DYN_DISCOVERY_BACKEND=file DYN_FILE_KV=/tmp/dynamo-kvcr DYN_REQUEST_PLANE=tcp DYN_RESPONSE_PLANE=tcp DYN_EVENT_PLANE=zmq` in every terminal; this list matches the tested setup. Without RDMA, also set `UCX_TLS=sm,self,cuda_copy,cuda_ipc,tcp`.

Run each command in its own terminal.

## Start the Frontend

```bash
python3 -m dynamo.frontend --router-mode kv --http-port 8000
```

## Start Two Workers

Worker 0:

```bash
CUDA_VISIBLE_DEVICES=0 DYN_SYSTEM_PORT=9200 python3 -m dynamo.sglang \
--model-path Qwen/Qwen3-0.6B --page-size 64 --mem-fraction-static 0.6 \
--kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:23000"}' \
--enable-unified-cache-external-linker --unified-cache-external-linker-backend kvcr \
--hicache-storage-backend-extra-config '{"advertise_host": "127.0.0.1", "control_port": 26800, "nixl_port": 20500, "kvcr_service_socket_path": null}'
```

Worker 1:

```bash
CUDA_VISIBLE_DEVICES=1 DYN_SYSTEM_PORT=9201 python3 -m dynamo.sglang \
--model-path Qwen/Qwen3-0.6B --page-size 64 --mem-fraction-static 0.6 \
--kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:23001"}' \
--enable-unified-cache-external-linker --unified-cache-external-linker-backend kvcr \
--hicache-storage-backend-extra-config '{"advertise_host": "127.0.0.1", "control_port": 26801, "nixl_port": 20501, "kvcr_service_socket_path": null}'
```

`"kvcr_service_socket_path": null` gives each worker its own in-process KVCR pool. Without it, the worker looks for a Guard at `/run/kvcr/memory.sock`, logs `KVCR-Service is unreachable`, and falls back to the in-process pool.

## Check It Works

1. Get the worker IDs. Each response names the worker that served it; repeat until you have seen both:

```bash
curl -s localhost:8000/v1/completions -H 'content-type: application/json' \
-d '{"model":"Qwen/Qwen3-0.6B","prompt":"hello","max_tokens":1,"nvext":{"extra_fields":["worker_id"]}}'
```

Use the `decode_worker_id` values as `<worker-a-id>` and `<worker-b-id>`.

2. Create `request.json` with a prompt of at least 1,000 tokens that you have not sent before:

```json
{"model": "Qwen/Qwen3-0.6B", "prompt": "<at least 1,000 tokens of new text>", "max_tokens": 8}
```

3. Send it to worker A, wait 15 seconds, then send it to worker B:

```bash
curl -s localhost:8000/v1/completions -H 'content-type: application/json' \
-H 'x-dynamo-worker-instance-id: <worker-a-id>' -d @request.json

curl -s localhost:8000/v1/completions -H 'content-type: application/json' \
-H 'x-dynamo-worker-instance-id: <worker-b-id>' -d @request.json
```

4. In the second response, `cached_tokens` is greater than 0 and a multiple of 64. Worker B logs `Prefill batch, #new-token: <a few>, #cached-token: <n>`: it computed only the last partial page and loaded the rest from worker A.

The first request after a fresh start can take a minute while SGLang builds a kernel extension. For transfer counters, add `"enable_telemetry": true` to the extra config; each worker then logs `KVCR telemetry` every 5 seconds, with `kvcr_transfer_blocks[source_write]` on worker A and the same `kvcr_transfer_blocks[remote_deliver]` count on worker B. Linker mode counts one block per page per layer.

## Add Guard

With Guard, a `kvcr_service` process owns the KVCR pools. If a worker's engine crashes, the restarted worker reattaches to its pool, and the KV state agent keeps the router sending the other worker to that pool while the crashed worker is down. See [The KV State Agent](kvcr.md#the-kv-state-agent).

1. Start one Guard service with one Guard per rank (here, one per worker):

```bash
mkdir -p /run/kvcr /dev/shm/kvcr
python3 -m kvcr.kvcr_service --socket-path /run/kvcr/memory.sock --pool-dir /dev/shm/kvcr \
--guard-count 2 --pool-sizes-gb 4 --compatibility-digest qwen3-0.6b-linker-v1 --health-port 9092
```

`curl localhost:9092/ready` returns `200` once the pools are ready.

2. Start the state agent. `--max-slots` is the number of cache owners it tracks, one per rank:

```bash
DYN_SYSTEM_PORT=9091 python3 -m dynamo.kv_state_agent --max-slots 2
```

3. Start each worker with the Guard settings and a cache-owner ID. Replace `<host-ip>` with this host's IP address (`hostname -i` shows it). With the state agent, `advertise_host` must not be a loopback address such as `127.0.0.1`, because the state agent connects to it. Worker 0:

```bash
CUDA_VISIBLE_DEVICES=0 DYN_SYSTEM_PORT=9200 python3 -m dynamo.sglang \
--model-path Qwen/Qwen3-0.6B --page-size 64 --mem-fraction-static 0.6 \
--kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:23000"}' \
--enable-unified-cache-external-linker --unified-cache-external-linker-backend kvcr \
--hicache-storage-backend-extra-config '{"advertise_host": "<host-ip>", "control_port": 26800, "nixl_port": 20500, "kvcr_service_socket_path": "/run/kvcr/memory.sock", "guard_index": 0, "compatibility_digest": "qwen3-0.6b-linker-v1"}' \
--kvcr-state-agent-owner 01010101010101010101010101010101:02020202020202020202020202020202/0303030303030303/00000000000000000000000000000000
```

For worker 1, use `CUDA_VISIBLE_DEVICES=1`, `DYN_SYSTEM_PORT=9201`, events port `23001`, `"control_port": 26801`, `"nixl_port": 20501`, `"guard_index": 1`, and a cache-owner ID ending in `...0001`.

4. To check recovery, send `request.json` to worker 0, wait 15 seconds, and stop worker 0's engine (`nvidia-smi` shows its PID; `kill -9 <pid>`). The Guard service logs `guard_promoted ... recovered_blocks=<n>`. Send `request.json` to worker 1: `cached_tokens` is positive, even after the router has removed worker 0. Restart worker 0 with the same command; it reattaches and serves the prompt from its pool.

## Settings

Fields of `--hicache-storage-backend-extra-config` in Linker mode. Unknown names stop the worker at startup. HiCache mode uses different names for the same settings (for example `control_advertise_host`, `nixl_listen_port`, `local_dram_bytes`), so do not copy fields between the two pages.

| Field | Required | Default | Meaning |
|---|---|---|---|
| `advertise_host` | For router hints | Local IP | Address peers use to reach this worker. Dynamo advertises a worker as a source only when this and `control_port` are set. |
| `control_port`, `nixl_port` | `control_port` for router hints; both unique per worker on a host | `19500`, `20500` | Base ports; the rank is added. The two must differ. |
| `kvcr_service_socket_path` | No | `/run/kvcr/memory.sock` | Guard socket. `null` uses an in-process pool. |
| `guard_index` | With Guard | `0` | Which of the service's Guards this worker uses; the rank is added. |
| `compatibility_digest` | With Guard | Derived from the model layout | Must match the service's `--compatibility-digest`. |
| `dram_size_gb` | No | `1` | In-process pool per rank. With Guard, `--pool-sizes-gb` sizes the pool instead. |
| `operation_timeout_ms`, `abandon_timeout_ms` | No | `1000`, `5000` | Transfer deadlines; the second must be at least twice the first. |
| `enable_telemetry` | No | `false` | Logs KVCR counters every 5 seconds. |

Dynamo worker flag: `--kvcr-state-agent-owner <cache-owner-id>` attaches the worker to the state agent. It requires Linker mode and ZMQ KV events with an empty topic.

Linker mode has no HiCache host tier, so give the host memory you budget for KV cache to `dram_size_gb` or the Guard pool.

## Related Topics

- [KV Cache Runner (KVCR)](kvcr.md)
- [How KV Hints Work](kv-hints.md)
- [Configure KVCR for SGLang](configure-kvcr-sglang.md)
- [Tune Routing for KVCR](kvcr-routing-tuning.md)
127 changes: 127 additions & 0 deletions docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: Configure KVCR for SGLang
subtitle: Run two SGLang workers on one GPU server that share KV cache through KVCR
---

> [!WARNING]
> Preview. Needs SGLang [#45647](https://github.com/sgl-project/sglang/pull/45647) (stacked on [#45646](https://github.com/sgl-project/sglang/pull/45646)) and the image recipe in KVCR [#114](https://github.com/ai-dynamo/kvcr/pull/114), both unmerged, and Dynamo with [#16117](https://github.com/ai-dynamo/dynamo/pull/16117). HiCache mode on one node only.

KVCR runs inside SGLang as a HiCache storage backend. SGLang stages KV in HiCache host memory; KVCR keeps it in its own KVCR pool and sends it to another worker when the Dynamo KV router attaches a source hint. To load KV straight into GPU memory instead, or to use Guard, see [Configure KVCR Linker Mode for SGLang](configure-kvcr-sglang-linker.md).

## Before You Begin

- The runtime image built from `Dockerfile.quick-start-sglang` in [KVCR #114](https://github.com/ai-dynamo/kvcr/pull/114) (pins SGLang `d365ebd`, Dynamo `7369d08`, KVCR `7186e7c`). No release image contains it. The image sets file-based discovery and TCP/ZMQ planes, so no etcd or NATS is needed on one host; this matches the tested setup.
- Two GPUs, one per worker.

## Start the Frontend

```bash
python3 -m dynamo.frontend --http-port 8000 --namespace kvcr --router-mode kv
```

## Start Two Workers

Run each worker in its own terminal. They differ only in GPU, ports, and the metrics port. `--max-total-tokens` keeps the example's GPU KV pool small, so the host tier sized from it stays small too.

Worker 0:

```bash
CUDA_VISIBLE_DEVICES=0 DYN_SYSTEM_PORT=24001 python3 -m dynamo.sglang \
--endpoint dyn://kvcr.backend.generate \
--model-path Qwen/Qwen3-0.6B --tp-size 1 --page-size 64 --max-total-tokens 65536 \
--enable-hierarchical-cache --hicache-ratio 1.5 \
--hicache-mem-layout page_first --hicache-write-policy write_through \
--hicache-storage-backend kvcr --hicache-storage-prefetch-policy wait_complete \
--hicache-storage-backend-extra-config '{"control_advertise_host": "127.0.0.1", "control_port": 24100, "nixl_listen_port": 24200, "local_dram_bytes": 8589934592}' \
--kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:24020"}'
```

Worker 1:

```bash
CUDA_VISIBLE_DEVICES=1 DYN_SYSTEM_PORT=24002 python3 -m dynamo.sglang \
--endpoint dyn://kvcr.backend.generate \
--model-path Qwen/Qwen3-0.6B --tp-size 1 --page-size 64 --max-total-tokens 65536 \
--enable-hierarchical-cache --hicache-ratio 1.5 \
--hicache-mem-layout page_first --hicache-write-policy write_through \
--hicache-storage-backend kvcr --hicache-storage-prefetch-policy wait_complete \
--hicache-storage-backend-extra-config '{"control_advertise_host": "127.0.0.1", "control_port": 24132, "nixl_listen_port": 24232, "local_dram_bytes": 8589934592}' \
--kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:24021"}'
```

Required flags: `--enable-hierarchical-cache`, `--hicache-storage-backend kvcr`, `--hicache-mem-layout page_first` (or `page_first_direct`), and `--kv-events-config`, plus `--router-mode kv` on the frontend. For more than one host, set `control_advertise_host` to an address the other workers can reach.

## Check It Works

1. Get the worker IDs. Each response names the worker that served it; repeat until you have seen both:

```bash
curl -s localhost:8000/v1/completions -H 'content-type: application/json' \
-d '{"model":"Qwen/Qwen3-0.6B","prompt":"hello","max_tokens":1,"nvext":{"extra_fields":["worker_id"]}}'
```

Use the `decode_worker_id` values as `<worker-0-id>` and `<worker-1-id>`.

2. Create `request.json` with a prompt of at least 1,000 tokens that you have not sent before:

```json
{"model": "Qwen/Qwen3-0.6B", "prompt": "<at least 1,000 tokens of new text>", "max_tokens": 8}
```

3. Send it to worker 0, wait 15 seconds, then send it to worker 1:

```bash
curl -s localhost:8000/v1/completions -H 'content-type: application/json' \
-H 'x-dynamo-worker-instance-id: <worker-0-id>' -d @request.json

curl -s localhost:8000/v1/completions -H 'content-type: application/json' \
-H 'x-dynamo-worker-instance-id: <worker-1-id>' -d @request.json
```

4. In the second response, `cached_tokens` is greater than 0 and a multiple of 64; the last partial page is recomputed. Worker 1 logs `HiCache prefetch success ... matched=0 loaded=<n>`: the `loaded` tokens came from worker 0 through KVCR. SGLang skips the KVCR lookup for prefixes shorter than its HiCache prefetch threshold (256 tokens by default), so use a long prompt.

For per-transfer detail, add `"diagnostic_telemetry": true` to the extra config. Worker 1 then logs `KVCR L3 operation` with `"direction":"deliver"` and `"failed_keys":[]`.

## Size the Tiers

Host memory for cached KV is split between two tiers:

| Tier | Set by | Shared with other workers |
|---|---|---|
| HiCache host tier (engine-managed) | `--hicache-ratio` (a multiple of the GPU KV pool) or `--hicache-size` (GB) | No. Peers fetch only from the KVCR pool; the host tier stages blocks between the GPU and KVCR. |
| KVCR pool | `local_dram_bytes`, per rank | Yes, under KVCR's policy |

Give most of the budget to `local_dram_bytes`, but keep the HiCache host tier larger than the GPU KV pool in tokens. Blocks fetched through KVCR are staged in the host tier before they reach the GPU, and with `write_through` the host tier also holds a copy of every block on the GPU. If the host tier is not larger than the GPU pool, SGLang drops KVCR hits and recomputes the prompt. A `--hicache-ratio` of `1.5` leaves half a GPU pool of room for hits; the startup log reports both pool sizes in tokens.

### Write Policy

`--hicache-write-policy` decides when a block is copied from the GPU to the host tier, and from there to KVCR:

| Policy | Blocks reach KVCR | Use with KVCR |
|---|---|---|
| `write_through` | As soon as they are cached | Recommended: peers can fetch a prefix right after the first request computes it |
| `write_through_selective` | After a block has been hit several times | Copies less, but a prefix is shareable only once it is hot |
| `write_back` | Only when the GPU evicts them | Avoid: peers cannot fetch a prefix while it is still on the source GPU |

## KVCR Settings

Fields of `--hicache-storage-backend-extra-config` in HiCache mode. Linker mode uses different names for the same settings (for example `advertise_host`, `nixl_port`, `dram_size_gb`), so do not copy fields between the two pages.

| Field | Required | Default | Meaning |
|---|---|---|---|
| `control_advertise_host` | Yes | - | Address peers use to reach this worker. Without it the worker is not advertised as a source. Not `0.0.0.0`. |
| `control_port` | No; unique per worker on a host | `32100` | Base port. Each rank uses `control_port + dp_rank * tp * cp * pp + rank offset`, so leave a gap between workers. |
| `nixl_listen_port` | No; unique per worker on a host | `control_port + 1024` | Same offsets. Must not overlap control ports. |
| `local_dram_bytes` | No | 8 GiB | KVCR pool per rank, in bytes. Separate from the HiCache host tier. |
| `enable_remote_hint` | No | `true` | Advertise this worker as a hint source. |
| `operation_timeout_ms`, `abandon_timeout_ms` | No | `30000`, `60000` | Transfer deadlines. |
| `diagnostic_telemetry` | No | `false` | Logs per-operation transfer records. |

## Related Topics

- [KV Cache Runner (KVCR)](kvcr.md)
- [Configure KVCR Linker Mode for SGLang](configure-kvcr-sglang-linker.md)
- [Tune Routing for KVCR](kvcr-routing-tuning.md)
- [HiCache](../modular-components/backends/sglang/hicache.md)
Loading
Loading