diff --git a/docs/fern/index.yml b/docs/fern/index.yml index 113c6cc803bf..f78fcc9d4b74 100644 --- a/docs/fern/index.yml +++ b/docs/fern/index.yml @@ -603,6 +603,34 @@ navigation: contents: - page: Mocker Engine Architecture path: pages/developer-guide/knowledge-base/modular-components/backends/mocker/mocker-engine-architecture.md + - section: KV Cache Management + slug: kv-offload + collapsed: true + contents: + - page: KV Cache Management Overview + path: pages/developer-guide/knowledge-base/kv-offload/overview.md + - page: KV Cache Runner (KVCR) + path: pages/developer-guide/knowledge-base/kv-offload/kvcr.md + - page: How KV Hints Work + path: pages/developer-guide/knowledge-base/kv-offload/kv-hints.md + - page: Configure KVCR for vLLM + path: pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-vllm.md + - page: Run KVCR on Kubernetes with vLLM + path: pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-vllm.md + - page: Run KVCR Guard on Kubernetes with vLLM + path: pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-vllm.md + - page: Verify and Monitor KVCR with vLLM + path: pages/developer-guide/knowledge-base/kv-offload/kvcr-testing-monitoring-vllm.md + - page: Configure KVCR for SGLang + path: pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang.md + - page: Configure KVCR Linker Mode for SGLang + path: pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang-linker.md + - page: Run KVCR on Kubernetes with SGLang + path: pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-sglang.md + - page: Run KVCR Guard on Kubernetes with SGLang + path: pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-sglang.md + - page: Tune Routing for KVCR + path: pages/developer-guide/knowledge-base/kv-offload/kvcr-routing-tuning.md - section: Developing and Extending Dynamo slug: advanced-customizations icon: sliders diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang-linker.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang-linker.md new file mode 100644 index 000000000000..28d9c2c85b7a --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang-linker.md @@ -0,0 +1,144 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: Configure KVCR Linker Mode for SGLang +subtitle: Load KV from other workers straight into GPU memory, with optional Guard, on one GPU server +--- + +> [!WARNING] +> Preview: needs Dynamo [#16827](https://github.com/ai-dynamo/dynamo/pull/16827) and SGLang [#45650](https://github.com/sgl-project/sglang/pull/45650) with the KVCR changes it is stacked on ([#45646](https://github.com/sgl-project/sglang/pull/45646) to [#45649](https://github.com/sgl-project/sglang/pull/45649)), none merged. One host only. + +In Linker mode, KVCR loads KV pages directly between GPU memory and its own host-memory pool, layer by layer. Unlike HiCache mode ([Configure KVCR for SGLang](configure-kvcr-sglang.md)), there is no HiCache host tier in between. + +## Before You Begin + +- A runtime with these builds; no release image contains them: + - Dynamo with [#16827](https://github.com/ai-dynamo/dynamo/pull/16827) + - SGLang with the KVCR Linker ([#45650](https://github.com/sgl-project/sglang/pull/45650) and the PRs it is stacked on) + - KVCR `v0.2.0` and NIXL `1.5.0` +- Two GPUs, one per worker. +- On one host without etcd or NATS, set `DYN_DISCOVERY_BACKEND=file DYN_FILE_KV=/tmp/dynamo-kvcr DYN_REQUEST_PLANE=tcp DYN_RESPONSE_PLANE=tcp DYN_EVENT_PLANE=zmq` in every terminal; this list matches the tested setup. Without RDMA, also set `UCX_TLS=sm,self,cuda_copy,cuda_ipc,tcp`. + +Run each command in its own terminal. + +## Start the Frontend + +```bash +python3 -m dynamo.frontend --router-mode kv --http-port 8000 +``` + +## Start Two Workers + +Worker 0: + +```bash +CUDA_VISIBLE_DEVICES=0 DYN_SYSTEM_PORT=9200 python3 -m dynamo.sglang \ + --model-path Qwen/Qwen3-0.6B --page-size 64 --mem-fraction-static 0.6 \ + --kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:23000"}' \ + --enable-unified-cache-external-linker --unified-cache-external-linker-backend kvcr \ + --hicache-storage-backend-extra-config '{"advertise_host": "127.0.0.1", "control_port": 26800, "nixl_port": 20500, "kvcr_service_socket_path": null}' +``` + +Worker 1: + +```bash +CUDA_VISIBLE_DEVICES=1 DYN_SYSTEM_PORT=9201 python3 -m dynamo.sglang \ + --model-path Qwen/Qwen3-0.6B --page-size 64 --mem-fraction-static 0.6 \ + --kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:23001"}' \ + --enable-unified-cache-external-linker --unified-cache-external-linker-backend kvcr \ + --hicache-storage-backend-extra-config '{"advertise_host": "127.0.0.1", "control_port": 26801, "nixl_port": 20501, "kvcr_service_socket_path": null}' +``` + +`"kvcr_service_socket_path": null` gives each worker its own in-process KVCR pool. Without it, the worker looks for a Guard at `/run/kvcr/memory.sock`, logs `KVCR-Service is unreachable`, and falls back to the in-process pool. + +## Check It Works + +1. Get the worker IDs. Each response names the worker that served it; repeat until you have seen both: + + ```bash + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -d '{"model":"Qwen/Qwen3-0.6B","prompt":"hello","max_tokens":1,"nvext":{"extra_fields":["worker_id"]}}' + ``` + + Use the `decode_worker_id` values as `` and ``. + +2. Create `request.json` with a prompt of at least 1,000 tokens that you have not sent before: + + ```json + {"model": "Qwen/Qwen3-0.6B", "prompt": "", "max_tokens": 8} + ``` + +3. Send it to worker A, wait 15 seconds, then send it to worker B: + + ```bash + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -H 'x-dynamo-worker-instance-id: ' -d @request.json + + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -H 'x-dynamo-worker-instance-id: ' -d @request.json + ``` + +4. In the second response, `cached_tokens` is greater than 0 and a multiple of 64. Worker B logs `Prefill batch, #new-token: , #cached-token: `: it computed only the last partial page and loaded the rest from worker A. + +The first request after a fresh start can take a minute while SGLang builds a kernel extension. For transfer counters, add `"enable_telemetry": true` to the extra config; each worker then logs `KVCR telemetry` every 5 seconds, with `kvcr_transfer_blocks[source_write]` on worker A and the same `kvcr_transfer_blocks[remote_deliver]` count on worker B. Linker mode counts one block per page per layer. + +## Add Guard + +With Guard, a `kvcr_service` process owns the KVCR pools. If a worker's engine crashes, the restarted worker reattaches to its pool, and the KV state agent keeps the router sending the other worker to that pool while the crashed worker is down. See [The KV State Agent](kvcr.md#the-kv-state-agent). + +1. Start one Guard service with one Guard per rank (here, one per worker): + + ```bash + mkdir -p /run/kvcr /dev/shm/kvcr + python3 -m kvcr.kvcr_service --socket-path /run/kvcr/memory.sock --pool-dir /dev/shm/kvcr \ + --guard-count 2 --pool-sizes-gb 4 --compatibility-digest qwen3-0.6b-linker-v1 --health-port 9092 + ``` + + `curl localhost:9092/ready` returns `200` once the pools are ready. + +2. Start the state agent. `--max-slots` is the number of cache owners it tracks, one per rank: + + ```bash + DYN_SYSTEM_PORT=9091 python3 -m dynamo.kv_state_agent --max-slots 2 + ``` + +3. Start each worker with the Guard settings and a cache-owner ID. Replace `` with this host's IP address (`hostname -i` shows it). With the state agent, `advertise_host` must not be a loopback address such as `127.0.0.1`, because the state agent connects to it. Worker 0: + + ```bash + CUDA_VISIBLE_DEVICES=0 DYN_SYSTEM_PORT=9200 python3 -m dynamo.sglang \ + --model-path Qwen/Qwen3-0.6B --page-size 64 --mem-fraction-static 0.6 \ + --kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:23000"}' \ + --enable-unified-cache-external-linker --unified-cache-external-linker-backend kvcr \ + --hicache-storage-backend-extra-config '{"advertise_host": "", "control_port": 26800, "nixl_port": 20500, "kvcr_service_socket_path": "/run/kvcr/memory.sock", "guard_index": 0, "compatibility_digest": "qwen3-0.6b-linker-v1"}' \ + --kvcr-state-agent-owner 01010101010101010101010101010101:02020202020202020202020202020202/0303030303030303/00000000000000000000000000000000 + ``` + + For worker 1, use `CUDA_VISIBLE_DEVICES=1`, `DYN_SYSTEM_PORT=9201`, events port `23001`, `"control_port": 26801`, `"nixl_port": 20501`, `"guard_index": 1`, and a cache-owner ID ending in `...0001`. + +4. To check recovery, send `request.json` to worker 0, wait 15 seconds, and stop worker 0's engine (`nvidia-smi` shows its PID; `kill -9 `). The Guard service logs `guard_promoted ... recovered_blocks=`. Send `request.json` to worker 1: `cached_tokens` is positive, even after the router has removed worker 0. Restart worker 0 with the same command; it reattaches and serves the prompt from its pool. + +## Settings + +Fields of `--hicache-storage-backend-extra-config` in Linker mode. Unknown names stop the worker at startup. HiCache mode uses different names for the same settings (for example `control_advertise_host`, `nixl_listen_port`, `local_dram_bytes`), so do not copy fields between the two pages. + +| Field | Required | Default | Meaning | +|---|---|---|---| +| `advertise_host` | For router hints | Local IP | Address peers use to reach this worker. Dynamo advertises a worker as a source only when this and `control_port` are set. | +| `control_port`, `nixl_port` | `control_port` for router hints; both unique per worker on a host | `19500`, `20500` | Base ports; the rank is added. The two must differ. | +| `kvcr_service_socket_path` | No | `/run/kvcr/memory.sock` | Guard socket. `null` uses an in-process pool. | +| `guard_index` | With Guard | `0` | Which of the service's Guards this worker uses; the rank is added. | +| `compatibility_digest` | With Guard | Derived from the model layout | Must match the service's `--compatibility-digest`. | +| `dram_size_gb` | No | `1` | In-process pool per rank. With Guard, `--pool-sizes-gb` sizes the pool instead. | +| `operation_timeout_ms`, `abandon_timeout_ms` | No | `1000`, `5000` | Transfer deadlines; the second must be at least twice the first. | +| `enable_telemetry` | No | `false` | Logs KVCR counters every 5 seconds. | + +Dynamo worker flag: `--kvcr-state-agent-owner ` attaches the worker to the state agent. It requires Linker mode and ZMQ KV events with an empty topic. + +Linker mode has no HiCache host tier, so give the host memory you budget for KV cache to `dram_size_gb` or the Guard pool. + +## Related Topics + +- [KV Cache Runner (KVCR)](kvcr.md) +- [How KV Hints Work](kv-hints.md) +- [Configure KVCR for SGLang](configure-kvcr-sglang.md) +- [Tune Routing for KVCR](kvcr-routing-tuning.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang.md new file mode 100644 index 000000000000..89237eb474f7 --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-sglang.md @@ -0,0 +1,127 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: Configure KVCR for SGLang +subtitle: Run two SGLang workers on one GPU server that share KV cache through KVCR +--- + +> [!WARNING] +> Preview. Needs SGLang [#45647](https://github.com/sgl-project/sglang/pull/45647) (stacked on [#45646](https://github.com/sgl-project/sglang/pull/45646)) and the image recipe in KVCR [#114](https://github.com/ai-dynamo/kvcr/pull/114), both unmerged, and Dynamo with [#16117](https://github.com/ai-dynamo/dynamo/pull/16117). HiCache mode on one node only. + +KVCR runs inside SGLang as a HiCache storage backend. SGLang stages KV in HiCache host memory; KVCR keeps it in its own KVCR pool and sends it to another worker when the Dynamo KV router attaches a source hint. To load KV straight into GPU memory instead, or to use Guard, see [Configure KVCR Linker Mode for SGLang](configure-kvcr-sglang-linker.md). + +## Before You Begin + +- The runtime image built from `Dockerfile.quick-start-sglang` in [KVCR #114](https://github.com/ai-dynamo/kvcr/pull/114) (pins SGLang `d365ebd`, Dynamo `7369d08`, KVCR `7186e7c`). No release image contains it. The image sets file-based discovery and TCP/ZMQ planes, so no etcd or NATS is needed on one host; this matches the tested setup. +- Two GPUs, one per worker. + +## Start the Frontend + +```bash +python3 -m dynamo.frontend --http-port 8000 --namespace kvcr --router-mode kv +``` + +## Start Two Workers + +Run each worker in its own terminal. They differ only in GPU, ports, and the metrics port. `--max-total-tokens` keeps the example's GPU KV pool small, so the host tier sized from it stays small too. + +Worker 0: + +```bash +CUDA_VISIBLE_DEVICES=0 DYN_SYSTEM_PORT=24001 python3 -m dynamo.sglang \ + --endpoint dyn://kvcr.backend.generate \ + --model-path Qwen/Qwen3-0.6B --tp-size 1 --page-size 64 --max-total-tokens 65536 \ + --enable-hierarchical-cache --hicache-ratio 1.5 \ + --hicache-mem-layout page_first --hicache-write-policy write_through \ + --hicache-storage-backend kvcr --hicache-storage-prefetch-policy wait_complete \ + --hicache-storage-backend-extra-config '{"control_advertise_host": "127.0.0.1", "control_port": 24100, "nixl_listen_port": 24200, "local_dram_bytes": 8589934592}' \ + --kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:24020"}' +``` + +Worker 1: + +```bash +CUDA_VISIBLE_DEVICES=1 DYN_SYSTEM_PORT=24002 python3 -m dynamo.sglang \ + --endpoint dyn://kvcr.backend.generate \ + --model-path Qwen/Qwen3-0.6B --tp-size 1 --page-size 64 --max-total-tokens 65536 \ + --enable-hierarchical-cache --hicache-ratio 1.5 \ + --hicache-mem-layout page_first --hicache-write-policy write_through \ + --hicache-storage-backend kvcr --hicache-storage-prefetch-policy wait_complete \ + --hicache-storage-backend-extra-config '{"control_advertise_host": "127.0.0.1", "control_port": 24132, "nixl_listen_port": 24232, "local_dram_bytes": 8589934592}' \ + --kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:24021"}' +``` + +Required flags: `--enable-hierarchical-cache`, `--hicache-storage-backend kvcr`, `--hicache-mem-layout page_first` (or `page_first_direct`), and `--kv-events-config`, plus `--router-mode kv` on the frontend. For more than one host, set `control_advertise_host` to an address the other workers can reach. + +## Check It Works + +1. Get the worker IDs. Each response names the worker that served it; repeat until you have seen both: + + ```bash + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -d '{"model":"Qwen/Qwen3-0.6B","prompt":"hello","max_tokens":1,"nvext":{"extra_fields":["worker_id"]}}' + ``` + + Use the `decode_worker_id` values as `` and ``. + +2. Create `request.json` with a prompt of at least 1,000 tokens that you have not sent before: + + ```json + {"model": "Qwen/Qwen3-0.6B", "prompt": "", "max_tokens": 8} + ``` + +3. Send it to worker 0, wait 15 seconds, then send it to worker 1: + + ```bash + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -H 'x-dynamo-worker-instance-id: ' -d @request.json + + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -H 'x-dynamo-worker-instance-id: ' -d @request.json + ``` + +4. In the second response, `cached_tokens` is greater than 0 and a multiple of 64; the last partial page is recomputed. Worker 1 logs `HiCache prefetch success ... matched=0 loaded=`: the `loaded` tokens came from worker 0 through KVCR. SGLang skips the KVCR lookup for prefixes shorter than its HiCache prefetch threshold (256 tokens by default), so use a long prompt. + +For per-transfer detail, add `"diagnostic_telemetry": true` to the extra config. Worker 1 then logs `KVCR L3 operation` with `"direction":"deliver"` and `"failed_keys":[]`. + +## Size the Tiers + +Host memory for cached KV is split between two tiers: + +| Tier | Set by | Shared with other workers | +|---|---|---| +| HiCache host tier (engine-managed) | `--hicache-ratio` (a multiple of the GPU KV pool) or `--hicache-size` (GB) | No. Peers fetch only from the KVCR pool; the host tier stages blocks between the GPU and KVCR. | +| KVCR pool | `local_dram_bytes`, per rank | Yes, under KVCR's policy | + +Give most of the budget to `local_dram_bytes`, but keep the HiCache host tier larger than the GPU KV pool in tokens. Blocks fetched through KVCR are staged in the host tier before they reach the GPU, and with `write_through` the host tier also holds a copy of every block on the GPU. If the host tier is not larger than the GPU pool, SGLang drops KVCR hits and recomputes the prompt. A `--hicache-ratio` of `1.5` leaves half a GPU pool of room for hits; the startup log reports both pool sizes in tokens. + +### Write Policy + +`--hicache-write-policy` decides when a block is copied from the GPU to the host tier, and from there to KVCR: + +| Policy | Blocks reach KVCR | Use with KVCR | +|---|---|---| +| `write_through` | As soon as they are cached | Recommended: peers can fetch a prefix right after the first request computes it | +| `write_through_selective` | After a block has been hit several times | Copies less, but a prefix is shareable only once it is hot | +| `write_back` | Only when the GPU evicts them | Avoid: peers cannot fetch a prefix while it is still on the source GPU | + +## KVCR Settings + +Fields of `--hicache-storage-backend-extra-config` in HiCache mode. Linker mode uses different names for the same settings (for example `advertise_host`, `nixl_port`, `dram_size_gb`), so do not copy fields between the two pages. + +| Field | Required | Default | Meaning | +|---|---|---|---| +| `control_advertise_host` | Yes | - | Address peers use to reach this worker. Without it the worker is not advertised as a source. Not `0.0.0.0`. | +| `control_port` | No; unique per worker on a host | `32100` | Base port. Each rank uses `control_port + dp_rank * tp * cp * pp + rank offset`, so leave a gap between workers. | +| `nixl_listen_port` | No; unique per worker on a host | `control_port + 1024` | Same offsets. Must not overlap control ports. | +| `local_dram_bytes` | No | 8 GiB | KVCR pool per rank, in bytes. Separate from the HiCache host tier. | +| `enable_remote_hint` | No | `true` | Advertise this worker as a hint source. | +| `operation_timeout_ms`, `abandon_timeout_ms` | No | `30000`, `60000` | Transfer deadlines. | +| `diagnostic_telemetry` | No | `false` | Logs per-operation transfer records. | + +## Related Topics + +- [KV Cache Runner (KVCR)](kvcr.md) +- [Configure KVCR Linker Mode for SGLang](configure-kvcr-sglang-linker.md) +- [Tune Routing for KVCR](kvcr-routing-tuning.md) +- [HiCache](../modular-components/backends/sglang/hicache.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-vllm.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-vllm.md new file mode 100644 index 000000000000..8258b2f56ea0 --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/configure-kvcr-vllm.md @@ -0,0 +1,197 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: Configure KVCR for vLLM +subtitle: Run the Dynamo KV router and a KVCR-enabled vLLM worker on one GPU server +--- + +These commands run the Dynamo frontend and a vLLM worker with a KVCR tier on one host with two GPUs. Pick a variant: + +- **KVCR pool** (recommended): KVCR keeps its own pool of shareable blocks, sized separately from vLLM's CPU cache. +- **Guard**: the pool lives in a separate Guard service and survives a worker crash. +- **Peer-only** (testing only): workers fetch from each other's vLLM CPU cache. + +## Before You Begin + +- A Linux host with two NVIDIA GPUs. +- A runtime with matching Dynamo, vLLM, KVCR v0.2.0, and NIXL builds. Dynamo release images do not include the KVCR package yet; build one with the [KVCR quick start](https://github.com/ai-dynamo/kvcr/blob/main/docs/quick-start.md) (`Dockerfile.quick-start`) and run the commands below inside it. +- Enough `/dev/shm` for vLLM's CPU cache (`cpu_bytes_to_use` for every rank) and any Guard pools. The quick start gives the container 8 GB, which covers the KVCR pool example; for the Guard example, start the container with about 35 GB of `/dev/shm` per worker (`docker run --shm-size`). +- These environment variables, for one host without etcd or NATS. The quick-start image already sets them: + + | Variable | Value | Purpose | + |---|---|---| + | `DYN_DISCOVERY_BACKEND`, `DYN_FILE_KV` | `file`, `/tmp/dynamo-kvcr` | The frontend and workers find each other through files in this directory instead of etcd. | + | `DYN_REQUEST_PLANE` | `tcp` | Sends requests to workers over TCP instead of NATS. | + | `DYN_EVENT_PLANE` | `zmq` | Sends KV events to the router over ZeroMQ instead of NATS. | + | `PYTHONHASHSEED` | `0` | Makes block hashes identical across processes, so the router matches blocks from different workers. | + | `VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES` | `1` | Publishes block hashes as integers, the form the router indexes. | + +Each command below runs in the foreground; use a separate terminal for each. A worker is ready when `curl -s localhost:8000/v1/models` lists the model. + +## Start the Frontend + +KVCR fetches are planned by the KV router, so the frontend must run with `--router-mode kv`: + +```bash +python3 -m dynamo.frontend --router-mode kv --http-port 8000 +``` + +## Start a Worker + +This worker has two data-parallel ranks, and they act as peers. Each rank has a small vLLM CPU cache (`cpu_bytes_to_use`) and a KVCR pool (`secondary_g2_slots`), and fetches blocks from the other rank's pool. + +```bash +export MODEL=Qwen/Qwen3-0.6B + +export KV_TRANSFER_CONFIG='{ + "kv_connector": "OffloadingConnector", + "kv_role": "kv_both", + "kv_connector_extra_config": { + "spec_name": "TieringOffloadingSpec", + "cpu_bytes_to_use": 2000000000, + "self_describing_kv_events": true, + "secondary_tiers": [{ + "type": "kvcr", + "router_capabilities": ["router_hint"], + "control_ports": [17771, 17772], + "control_advertise_host": "127.0.0.1", + "secondary_g2_slots": 1024, + "enable_telemetry": true + }] + } +}' + +CUDA_VISIBLE_DEVICES=0,1 DYN_SYSTEM_PORT=8081 python3 -m dynamo.vllm \ + --model "$MODEL" \ + --data-parallel-size 2 \ + --block-size 64 \ + --enable-prefix-caching \ + --kv-transfer-config "$KV_TRANSFER_CONFIG" \ + --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20081","enable_kv_cache_events":true}' +``` + +`--kv-events-config` publishes the worker's cache contents to the router; without it the router never sends hints. `DYN_SYSTEM_PORT` exposes the worker's `/metrics`. To confirm a transfer, see [Verify and Monitor KVCR with vLLM](kvcr-testing-monitoring-vllm.md). + +For a quick functional test you can remove `secondary_g2_slots` (peer-only): ranks then fetch from each other's vLLM CPU cache. Keep the pool for real use: it is faster, because KVCR registers it once at startup and transfers do not wait for vLLM to pin memory. + +## Size the Tiers + +`secondary_g2_slots` is a number of KV blocks, not bytes. The pool is ordinary host memory in the worker process, not `/dev/shm`. Pool size per rank = slots x bytes per block. Bytes per block for a dense model = 2 x layers x KV heads x head dim x dtype bytes x block size. For Qwen3-0.6B at block size 64 that is 7.34 MB, so 1024 slots is 7.5 GB per rank, 15 GB for this two-rank worker. After one transfer you can also read it from the worker metrics: `vllm:kvcr_transfer_bytes_total / vllm:kvcr_transfer_blocks_total`. + +Host memory for cached KV is split between two tiers: + +| Tier | Set by | Memory | Shared with other workers | +|---|---|---|---| +| vLLM CPU cache (engine-managed) | `cpu_bytes_to_use`, per rank | `/dev/shm` | Only until vLLM evicts the block | +| KVCR pool | `secondary_g2_slots`, or `--pool-sizes-gb` with Guard | Host memory (Guard: `/dev/shm`) | Yes, under KVCR's policy; survives a crash with Guard | + +Keep the vLLM CPU cache small and give the rest of the budget to KVCR. vLLM's cache still stages every block that moves between the GPU and KVCR, and receives blocks fetched from peers, so it cannot be zero. For example, with 200 GiB of host memory per worker for Qwen3.8-27B-FP8, use 25 GiB for `cpu_bytes_to_use` and 175 GiB for KVCR: `secondary_g2_slots: 3584` (50 MiB blocks), or `--pool-sizes-gb 175` with Guard. + +## Add Guard + +The Guard service (`kvcr_service`) owns the pool. After a worker crash, the restarted worker reattaches to its blocks, and other workers can fetch them in the meantime. The [KV state agent](kvcr.md#the-kv-state-agent) (`dynamo.kv_state_agent`) keeps the router pointing at a crashed worker's blocks. + +Run one Guard service per worker process: several worker processes cannot share one Guard service yet. Each data-parallel rank uses the Guard with its rank's index, so `--guard-count` must be at least the worker's data-parallel size. + +1. Start the Guard service: + + ```bash + mkdir -p /run/kvcr /dev/shm/kvcr + python3 -m kvcr.kvcr_service \ + --socket-path /run/kvcr/memory.sock \ + --pool-dir /dev/shm/kvcr \ + --guard-count 1 \ + --pool-sizes-gb 32 \ + --compatibility-digest qwen3-0.6b-v1 \ + --health-port 9092 + ``` + + `curl localhost:9092/ready` returns `200` once the pool is ready. Registering a large pool can take tens of seconds; if the service times out at startup, raise `--startup-timeout-seconds` (default `60`). The pool lives in `/dev/shm`, and each Guard adds a 100 MiB journal to it. `--pool-sizes-gb` is the pool size in GB per Guard; size it as in [Size the Tiers](#size-the-tiers). + +2. Start one state agent. All workers in the Dynamo namespace share it, and `--max-slots` is the number of cache owners it tracks, one per rank across those workers: + + ```bash + DYN_SYSTEM_PORT=9091 python3 -m dynamo.kv_state_agent --max-slots 2 + ``` + +3. Start the worker with a Guard attachment and a `dynamo_state_agent` block: + + - `raw_advertise_host` is the address the state agent uses to read this worker's KV events. Replace `` with this host's IP address (`hostname -i` shows it). Dynamo rejects loopback addresses here, even on one host; `control_advertise_host` can stay `127.0.0.1`. + - `cache_owner_ids` names this worker's cache so the router can find it after a crash. It must be unique per worker and stay the same across restarts: copy the example value and change only the last segment for each worker. + + ```bash + export KV_TRANSFER_CONFIG='{ + "kv_connector": "OffloadingConnector", + "kv_role": "kv_both", + "kv_connector_extra_config": { + "spec_name": "TieringOffloadingSpec", + "cpu_bytes_to_use": 2147483648, + "self_describing_kv_events": true, + "dynamo_state_agent": { + "raw_advertise_host": "", + "cache_owner_ids": {"0": "01010101010101010101010101010101:02020202020202020202020202020202/0303030303030303/00000000000000000000000000000000"} + }, + "secondary_tiers": [{ + "type": "kvcr", + "kvcr_service_socket_path": "/run/kvcr/memory.sock", + "compatibility_digest": "qwen3-0.6b-v1", + "router_capabilities": ["router_hint"], + "control_ports": [23280], + "control_advertise_host": "127.0.0.1", + "enable_telemetry": true + }] + } + }' + + CUDA_VISIBLE_DEVICES=0 DYN_SYSTEM_PORT=8081 python3 -m dynamo.vllm \ + --model Qwen/Qwen3-0.6B \ + --block-size 64 \ + --enable-prefix-caching \ + --kv-transfer-config "$KV_TRANSFER_CONFIG" \ + --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20081","enable_kv_cache_events":true}' + ``` + + For a second worker on the same host, repeat steps 1 and 3 with its own values: + + | Setting | Worker A | Worker B | + |---|---|---| + | `--socket-path`, `kvcr_service_socket_path` | `/run/kvcr/memory.sock` | `/run/kvcrB/memory.sock` | + | `--pool-dir` | `/dev/shm/kvcr` | `/dev/shm/kvcrB` | + | `--health-port` | `9092` | `9093` | + | `CUDA_VISIBLE_DEVICES` | `0` | `1` | + | `control_ports` | `[23280]` | `[23281]` | + | KV-events endpoint | `tcp://*:20081` | `tcp://*:20082` | + | `DYN_SYSTEM_PORT` | `8081` | `8082` | + | `cache_owner_ids` last segment | `...0000` | `...0001` | + +## Configuration Reference + +Fields of the KVCR entry in `secondary_tiers`: + +| Field | Required | Default | Meaning | +|---|---|---|---| +| `router_capabilities` | Yes | - | Must include `router_hint`; the worker does not start without it. | +| `control_ports` | Yes | - | One port per local data-parallel rank, in rank order. | +| `control_advertise_host` | On more than one host | Hostname | Address peers use to reach this worker. Loopback works only on one host. | +| `control_host` | No | `0.0.0.0` | Bind address for the control port. | +| `secondary_g2_slots` | No | `0` | KVCR pool size, in KV blocks. Ignored when Guard is configured. | +| `kvcr_service_socket_path`, `compatibility_digest` | With Guard, together | - | Attach to a Guard service. The digest must match the service's and change when the model, dtype, or block layout changes. | +| `operation_timeout_ms` | No | `1000` | Deadline for one KVCR operation. | +| `abandon_timeout_ms` | No | `5000` | At least twice `operation_timeout_ms`. | +| `enable_telemetry` | No | `false` | Exposes `vllm:kvcr_*` metrics on the worker's `DYN_SYSTEM_PORT`. | + +Fields of `kv_connector_extra_config` that KVCR depends on: + +| Field | Required | Default | Meaning | +|---|---|---|---| +| `cpu_bytes_to_use` | Yes | - | Size of vLLM's own CPU cache per rank, in `/dev/shm`. Not a KVCR pool. | +| `self_describing_kv_events` | With a pool, Guard, or disk tier | `false` | Set to `true` in every configuration; the router needs it to index KVCR's tiers. The worker does not start without it when a pool, Guard, or disk tier is configured. | +| `dynamo_state_agent.raw_advertise_host` | With the state agent | - | Non-loopback address where the state agent reads this worker's KV events. | +| `dynamo_state_agent.cache_owner_ids` | With the state agent | - | Stable, unique ID of this worker's cache, per local rank. | + +## Related Topics + +- [Verify and Monitor KVCR with vLLM](kvcr-testing-monitoring-vllm.md) +- [Run KVCR on Kubernetes with vLLM](kvcr-kubernetes-vllm.md) +- [Run KVCR Guard on Kubernetes with vLLM](kvcr-guard-kubernetes-vllm.md) +- [KV Cache Runner (KVCR)](kvcr.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kv-hints.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kv-hints.md new file mode 100644 index 000000000000..2b1858039c09 --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kv-hints.md @@ -0,0 +1,70 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: How KV Hints Work +subtitle: The per-request signal that connects router decisions to KVCR fetches +--- + +A **KV hint** tells KVCR where to fetch reusable blocks for one request. The router picks a source from its cluster-wide cache view and attaches the source's identity and control endpoint to the request. The hint carries no KV data: KVCR transfers the blocks over NIXL and decides locally whether to keep them. + +## Request Flow + +```mermaid +sequenceDiagram + participant R as KV Router + participant W as Target Worker (KVCR) + participant P as Source Worker (KVCR) + + R->>R: Choose the target worker from cache state and load + R->>R: Find an eligible source within the target's topology rules + alt Source holds a longer cached prefix + R->>W: Request + kv.fetch hint (source identity and control endpoint) + W->>P: Ask for the hinted blocks + P-->>W: Write the blocks over NIXL + W->>W: Use the blocks, apply local cache policy + else No eligible source + R->>W: Request without a hint + W->>W: Compute the missing prefix + end +``` + +## Generation and Attachment + +The router generates a hint for a request only when all of these hold: + +- The frontend runs `--router-mode kv`. Round-robin routing never generates hints. +- The target worker advertises the `router_hint` capability and a control endpoint. +- A source of the same worker type (aggregated, prefill, or decode) holds a longer cached prefix than the target. +- The source passes the target's topology rules (below). + +The hint is a `kv.fetch` action in a versioned hint envelope, built from the same block index that drives [KV-Aware Routing](../concepts/system-architecture/kv-aware-routing.md). Each hint belongs to one request, is not reused, and names one source: KVCR does not fetch one prefix from several peers. + +## Topology Filtering + +The **target** worker's `kv_transfer_domain` and `kv_transfer_enforcement` decide which sources are eligible. This is the reverse of prefill-to-decode selection: the receiving worker's settings govern the fetch. + +| Target setting | Eligible sources | +|---|---| +| Domain set, enforcement `required` or unset | Only sources in the target's domain | +| Domain set, enforcement `preferred` | Sources in the target's domain; others only when none in the domain qualifies | +| No domain | Any source | + +The `preferredWeight` setting of [Topology-Aware KV Transfer](../modular-components/router/topology-aware-kv-transfer.md) does not affect which source a hint names. If no eligible source holds a longer prefix than the target, the router sends no hint and the worker computes the prefix. + +A KVCR cache owner, such as a Guard pool, that publishes no topology of its own is treated as part of its attached worker's domain. + +See [Tune Routing for KVCR](kvcr-routing-tuning.md#keep-fetches-on-fast-links) to set these, and [Topology-Aware KV Transfer](../modular-components/router/topology-aware-kv-transfer.md) for the domain model. + +## Fetching and Local Cache Policy + +KVCR on the target fetches the hinted blocks directly from the source over NIXL; the router plans the fetch but does not take part in it. If only part of the prefix is still available at the source, the target uses what arrived and computes the rest. + +A fetched block is not guaranteed to stay cached. KVCR's local policy still decides admission, retention, and eviction. + +## Related Topics + +- [KV Cache Runner (KVCR)](kvcr.md) +- [KV Cache Management Overview](overview.md) +- [Tune Routing for KVCR](kvcr-routing-tuning.md) +- [Topology-Aware KV Transfer](../modular-components/router/topology-aware-kv-transfer.md) +- [KV-Aware Routing](../concepts/system-architecture/kv-aware-routing.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-sglang.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-sglang.md new file mode 100644 index 000000000000..7411c135d993 --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-sglang.md @@ -0,0 +1,90 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: Run KVCR Guard on Kubernetes with SGLang +subtitle: Deploy SGLang Linker-mode workers whose KVCR cache survives an engine crash +--- + +> [!WARNING] +> Preview: needs Dynamo [#16827](https://github.com/ai-dynamo/dynamo/pull/16827) and SGLang [#45650](https://github.com/sgl-project/sglang/pull/45650) with the KVCR changes it is stacked on ([#45646](https://github.com/sgl-project/sglang/pull/45646) to [#45649](https://github.com/sgl-project/sglang/pull/45649)), none merged, and KVCR v0.2.0. +> +> Not yet run end to end on Kubernetes. Validated: the manifest renders with `envsubst` and passes `kubectl apply --dry-run=server` against the DynamoGraphDeployment CRD; the same worker flags passed on one server in [Configure KVCR Linker Mode for SGLang](configure-kvcr-sglang-linker.md), including crash recovery. Whether `main` restarts on its own after the engine is killed is not yet verified on Kubernetes. + +The [`sglang-agg-linker-guard.yaml`](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/sglang/deploy/kvcr/sglang-agg-linker-guard.yaml) example deploys a frontend and two SGLang workers in Linker mode on two nodes. Guard needs Linker mode; HiCache mode has no Guard support. Each worker Pod has two containers: + +- `main` runs `dynamo.sglang` with the KVCR Linker. `--kvcr-state-agent-owner` registers the worker's cache with the [KV state agent](kvcr.md#the-kv-state-agent) under a stable ID taken from the Pod's Grove index. +- `kvcr-services` runs the Guard service, which owns the KVCR pool. On the first replica only, it also runs the KV state agent, which keeps the router sending peers to a crashed worker's Guard; one state agent serves the whole DynamoGraphDeployment. + +SGLang workers on one host can share a Guard service through `guard_index`, but on Kubernetes keep one sidecar per Pod, as in this example. + +When the engine in `main` crashes, `main` restarts and reattaches to its pool; while it is down, the other worker can fetch the crashed worker's blocks from its Guard. Deleting the Pod loses the cache. + +This example uses Kubernetes discovery; it does not need etcd. + +## Before You Begin + +- Two GPU nodes, each with one free GPU and two shares of an RDMA resource (`rdma/shared_ib` by default; one for each container). +- The Dynamo operator with the Grove workload provider, and `kubectl` set to the target cluster context. +- An `hf-token-secret` in the target namespace. +- A runtime image with the SGLang Linker, Dynamo with #16827, KVCR v0.2.0, and NIXL; see [Configure KVCR Linker Mode for SGLang](configure-kvcr-sglang-linker.md). + +## Deploy + +1. From `examples/backends/sglang/deploy/kvcr` in the Dynamo repository, set the variables and apply the manifest. `envsubst` fills in only the listed variables; the startup scripts use other `$` variables that must stay as they are. + + ```bash + export NAMESPACE= + export DYNAMO_SGLANG_IMAGE=@sha256: + export DYNAMO_VERSION= + export DYNAMO_KVCR_COMPATIBILITY_DIGEST=qwen3-0.6b-linker-v1 + export DYNAMO_UCX_NET_DEVICES=:1 + export DYNAMO_RDMA_RESOURCE=rdma/shared_ib + + envsubst '${DYNAMO_SGLANG_IMAGE} ${DYNAMO_VERSION} ${DYNAMO_RDMA_RESOURCE} ${DYNAMO_UCX_NET_DEVICES} ${DYNAMO_KVCR_COMPATIBILITY_DIGEST}' \ + < sglang-agg-linker-guard.yaml | kubectl apply -n "$NAMESPACE" -f - + ``` + + | Variable | Value | + |---|---| + | `DYNAMO_VERSION` | The Dynamo version in the image, for example `1.7.0`. The operator rejects an image without a Dynamo version tag unless `runtimeVersionOverride` is set. | + | `DYNAMO_KVCR_COMPATIBILITY_DIGEST` | Any string shared by `main` and `kvcr-services`. Change it when the model, dtype, or block layout changes. | + | `DYNAMO_UCX_NET_DEVICES` | The example requires it. NIXL picks network devices on its own, but on nodes with several RDMA NICs, workers that see every NIC can stall at startup, so pin the NIC attached directly to the worker's GPU (marked `PIX` for that GPU in `nvidia-smi topo -m`), for example `mlx5_5:1`. On a multi-GPU node the right NIC depends on which GPU the Pod gets. | + | `DYNAMO_RDMA_RESOURCE` | The RDMA extended resource your cluster exposes. | + +2. Wait until both worker Pods show `2/2 Running`: + + ```bash + kubectl get pods -n "$NAMESPACE" -w + ``` + +3. Port-forward port 8000 of the frontend Pod and confirm a cross-worker transfer as in [Run KVCR on Kubernetes with SGLang](kvcr-kubernetes-sglang.md#check-it-works). In Linker mode the second response shows `cached_tokens` greater than 0; the worker log does not show the HiCache prefetch line. + +## Verify Recovery + +Pin each request to a worker as in [Check It Works](kvcr-kubernetes-sglang.md#check-it-works). + +1. Send `request.json` to worker A and wait 15 seconds. + +2. Find worker A's engine process and stop it: + + ```bash + kubectl exec -n "$NAMESPACE" -c main -- nvidia-smi --query-compute-apps=pid --format=csv,noheader + kubectl exec -n "$NAMESPACE" -c main -- kill -9 + ``` + + `kvcr-services` logs `guard_promoted ... recovered_blocks=`. + +3. Wait 75 seconds, so that the router has removed worker A, then send `request.json` to worker B. `cached_tokens` is positive: B fetched the blocks from worker A's Guard. + +4. After worker A is Ready again, send `request.json` to it. `cached_tokens` is positive, served from its own pool. + +## Limits + +- If `kvcr-services` restarts, recovery no longer works for that Pod. Replace the whole Pod. +- Without the KV state agent, peers stop finding a crashed worker's blocks once the router removes that worker. + +## Related Topics + +- [Configure KVCR Linker Mode for SGLang](configure-kvcr-sglang-linker.md) +- [Run KVCR on Kubernetes with SGLang](kvcr-kubernetes-sglang.md) +- [KV Cache Runner (KVCR)](kvcr.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-vllm.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-vllm.md new file mode 100644 index 000000000000..4d66ac0a2afd --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-guard-kubernetes-vllm.md @@ -0,0 +1,86 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: Run KVCR Guard on Kubernetes with vLLM +subtitle: Deploy vLLM workers whose KVCR cache survives an engine crash +--- + +The [`agg-memory-service.yaml`](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/vllm/deploy/kvcr/agg-memory-service.yaml) example deploys a frontend and two vLLM workers on two nodes. Each worker Pod has two containers: + +- `main` runs `dynamo.vllm`. +- `kvcr-services` runs the Guard service, which owns the KVCR pool. On the first replica only, it also runs the [KV state agent](kvcr.md#the-kv-state-agent), which keeps the router routing to a crashed worker's cache; one state agent serves the whole DynamoGraphDeployment. + +When the engine in `main` crashes, `main` restarts and reattaches to its pool; while it is down, the other worker can fetch the crashed worker's blocks from its Guard. Deleting the Pod loses the cache. + +This example uses Kubernetes discovery; it does not need etcd. + +## Before You Begin + +- Two GPU nodes, each with one free GPU and two shares of an RDMA resource (`rdma/shared_ib` by default; one for each container). +- The Dynamo operator with the Grove workload provider, and `kubectl` set to the target cluster context. +- An `hf-token-secret` in the target namespace. +- A runtime image with matching Dynamo, vLLM, KVCR v0.2.0, and NIXL builds (see [Configure KVCR for vLLM](configure-kvcr-vllm.md#before-you-begin)). If its tag is not a Dynamo release version, set `runtimeVersionOverride` on every component. + +## Deploy + +1. From `examples/backends/vllm/deploy/kvcr` in the Dynamo repository, set the variables and apply `agg-memory-service.yaml`. `envsubst` fills in only the listed variables; the manifest's startup scripts use other `$` variables that must stay as they are. + + ```bash + export NAMESPACE= + export DYNAMO_VLLM_IMAGE=@sha256: + export DYNAMO_KVCR_COMPATIBILITY_DIGEST=qwen3-0.6b-v1 + export DYNAMO_UCX_NET_DEVICES=:1 + export DYNAMO_RDMA_RESOURCE=rdma/shared_ib + + envsubst '${DYNAMO_VLLM_IMAGE} ${DYNAMO_RDMA_RESOURCE} ${DYNAMO_UCX_NET_DEVICES} ${DYNAMO_KVCR_COMPATIBILITY_DIGEST}' \ + < agg-memory-service.yaml | kubectl apply -n "$NAMESPACE" -f - + ``` + + | Variable | Value | + |---|---| + | `DYNAMO_KVCR_COMPATIBILITY_DIGEST` | Any string shared by `main` and `kvcr-services`. Change it when the model, dtype, or block layout changes. | + | `DYNAMO_UCX_NET_DEVICES` | The example requires it. NIXL picks network devices on its own, but on nodes with several RDMA NICs, workers that see every NIC can stall at startup, so pin the NIC attached directly to the worker's GPU (marked `PIX` for that GPU in `nvidia-smi topo -m`), for example `mlx5_5:1`. On a multi-GPU node the right NIC depends on which GPU the Pod gets. | + | `DYNAMO_RDMA_RESOURCE` | The RDMA extended resource your cluster exposes. | + +2. Wait until both worker Pods show `2/2 Running`: + + ```bash + kubectl get pods -n "$NAMESPACE" -w + ``` + +3. Port-forward port 8000 of the frontend Pod, then confirm a cross-worker transfer with [Verify and Monitor KVCR with vLLM](kvcr-testing-monitoring-vllm.md), using the two worker IDs. Each worker serves its metrics on port 9090 of its Pod; reach a worker's metrics with `kubectl port-forward -n "$NAMESPACE" pod/ 9090` and read them at `localhost:9090/metrics`. + +## Verify Recovery + +Pin each request to a worker as in [Verify a Transfer](kvcr-testing-monitoring-vllm.md#verify-a-transfer). + +1. Send a prompt to worker A, then crash its engine: + + ```bash + kubectl exec -n "$NAMESPACE" -c main -- pkill -9 -f '[V]LLM::EngineCore' + ``` + +2. Check that only `main` restarted: + + ```bash + kubectl get pod -n "$NAMESPACE" \ + -o jsonpath='{range .status.containerStatuses[*]}{.name}={.restartCount}{"\n"}{end}' + ``` + + `kvcr-services` logs `guard_promoted ... recovered_blocks=`. + +3. Send the same prompt to worker B. `cached_tokens` is positive, and B's `vllm:kvcr_transfer_blocks_total{operation="remote_deliver"}` increases. + +4. When worker A's Pod is Ready again, send the prompt to A. `cached_tokens` is positive, served from its pool (`operation="local_deliver"`). + +## Limits + +- Each worker Pod has its own Guard service; several workers cannot share one Guard service yet. +- If `kvcr-services` restarts, recovery no longer works for that Pod. Replace the whole Pod. +- Each crash leaves the dead engine's CPU-offload file (`cpu_bytes_to_use`, 2 GiB in the example) in `main`'s `/dev/shm` until vLLM [#53073](https://github.com/vllm-project/vllm/pull/53073) is in your vLLM build. Size `/dev/shm` for it, or remove `/dev/shm/vllm_offload_*` before starting vLLM. + +## Related Topics + +- [Configure KVCR for vLLM](configure-kvcr-vllm.md) +- [Verify and Monitor KVCR with vLLM](kvcr-testing-monitoring-vllm.md) +- [Run KVCR on Kubernetes with vLLM](kvcr-kubernetes-vllm.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-sglang.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-sglang.md new file mode 100644 index 000000000000..7c35897d83df --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-sglang.md @@ -0,0 +1,92 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: Run KVCR on Kubernetes with SGLang +subtitle: Deploy SGLang workers that share KV cache through KVCR in HiCache mode +--- + +> [!WARNING] +> Preview. Needs SGLang [#45647](https://github.com/sgl-project/sglang/pull/45647) (stacked on [#45646](https://github.com/sgl-project/sglang/pull/45646)) and the image recipe in KVCR [#114](https://github.com/ai-dynamo/kvcr/pull/114), both unmerged, and Dynamo with [#16117](https://github.com/ai-dynamo/dynamo/pull/16117). +> +> Run end to end on two H200 nodes on 2026-10-29: the second worker reused 2,112 of 2,161 prompt tokens from the first through KVCR. The worker's liveness probe allows 10 minutes, because SGLang builds a kernel on its first request and the default probe restarts the worker before it finishes. + +The [`sglang-agg-hicache.yaml`](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/sglang/deploy/kvcr/sglang-agg-hicache.yaml) example deploys a frontend with the KV router and two SGLang workers on two nodes. Each worker runs KVCR as its HiCache storage backend with its own KVCR pool (`local_dram_bytes`, 8 GiB in the example). The cache is lost when a worker restarts; for crash recovery, see [Run KVCR Guard on Kubernetes with SGLang](kvcr-guard-kubernetes-sglang.md). + +This example uses Kubernetes discovery; it does not need etcd. + +## Before You Begin + +- Two GPU nodes, each with one free GPU and one share of an RDMA resource (`rdma/shared_ib` by default). +- The Dynamo operator with the Grove workload provider, and `kubectl` set to the target cluster context. +- An `hf-token-secret` in the target namespace. +- A runtime image built from `Dockerfile.quick-start-sglang` in [KVCR #114](https://github.com/ai-dynamo/kvcr/pull/114); see [Configure KVCR for SGLang](configure-kvcr-sglang.md#before-you-begin). + +## Deploy + +1. From `examples/backends/sglang/deploy/kvcr` in the Dynamo repository, set the variables and apply the manifest. `envsubst` fills in only the listed variables; the worker command uses `$POD_IP`, which must stay as it is. + + ```bash + export NAMESPACE= + export DYNAMO_SGLANG_IMAGE=@sha256: + export DYNAMO_VERSION= + export DYNAMO_UCX_NET_DEVICES=:1 + export DYNAMO_RDMA_RESOURCE=rdma/shared_ib + + envsubst '${DYNAMO_SGLANG_IMAGE} ${DYNAMO_VERSION} ${DYNAMO_RDMA_RESOURCE} ${DYNAMO_UCX_NET_DEVICES}' \ + < sglang-agg-hicache.yaml | kubectl apply -n "$NAMESPACE" -f - + ``` + + | Variable | Value | + |---|---| + | `DYNAMO_VERSION` | The Dynamo version in the image, for example `1.7.0`. The operator reads it from `runtimeVersionOverride`, because the image tag is not a Dynamo version. | + | `DYNAMO_UCX_NET_DEVICES` | The example requires it. NIXL picks network devices on its own, but on nodes with several RDMA NICs, workers that see every NIC can stall at startup, so pin the NIC attached directly to the worker's GPU (marked `PIX` for that GPU in `nvidia-smi topo -m`), for example `mlx5_5:1`. On a multi-GPU node the right NIC depends on which GPU the Pod gets. | + | `DYNAMO_RDMA_RESOURCE` | The RDMA extended resource your cluster exposes. | + +2. Wait until the frontend and both worker Pods are `1/1 Running`: + + ```bash + kubectl get pods -n "$NAMESPACE" -w + ``` + +## Check It Works + +1. Port-forward port 8000 of the frontend Pod. + +2. Get the two worker IDs. Send any short request a few times and read `"decode_worker_id"` in each response: + + ```bash + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -d '{"model":"Qwen/Qwen3-0.6B","prompt":"hello","max_tokens":1,"nvext":{"extra_fields":["worker_id"]}}' + ``` + +3. Create `request.json` with a prompt of at least 1,000 tokens that you have not sent before: + + ```json + {"model": "Qwen/Qwen3-0.6B", "prompt": "", "max_tokens": 8} + ``` + +4. Send it to worker 0, wait 15 seconds, then send it to worker 1: + + ```bash + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -H 'x-dynamo-worker-instance-id: ' -d @request.json + + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -H 'x-dynamo-worker-instance-id: ' -d @request.json + ``` + +5. In the second response, `cached_tokens` is greater than 0 and a multiple of 64. The second worker's log shows the transfer: + + ```bash + kubectl logs -n "$NAMESPACE" | grep 'HiCache prefetch success' + ``` + + `loaded=` with `matched=0` means all `` tokens came from the other worker through KVCR. + +To change the pool size, edit `local_dram_bytes` in the worker's `--hicache-storage-backend-extra-config`; see [Size the Tiers](configure-kvcr-sglang.md#size-the-tiers). + +## Related Topics + +- [Configure KVCR for SGLang](configure-kvcr-sglang.md) +- [Run KVCR Guard on Kubernetes with SGLang](kvcr-guard-kubernetes-sglang.md) +- [KV Cache Runner (KVCR)](kvcr.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-vllm.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-vllm.md new file mode 100644 index 000000000000..e6bf5c780ba7 --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-kubernetes-vllm.md @@ -0,0 +1,55 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: Run KVCR on Kubernetes with vLLM +subtitle: Deploy vLLM workers with an in-process KVCR pool +--- + +The [`agg.yaml`](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/vllm/deploy/kvcr/agg.yaml) example deploys a frontend with the KV router and two vLLM workers on two nodes. Each worker runs KVCR in-process with its own KVCR pool (`secondary_g2_slots: 1024`). The cache is lost when a worker restarts; for crash recovery, see [Run KVCR Guard on Kubernetes with vLLM](kvcr-guard-kubernetes-vllm.md). + +This example uses etcd discovery. + +## Before You Begin + +- Two GPU nodes, each with one free GPU and one share of an RDMA resource (`rdma/shared_ib` by default). +- The Dynamo operator with the Grove workload provider, and `kubectl` set to the target cluster context. +- etcd. If the operator has no etcd configured, run one in the namespace and set `ETCD_ENDPOINTS` (for example `http://etcd:2379`) in the `env` of every container in the manifest; otherwise pods log `etcd not reachable`. +- An `hf-token-secret` in the target namespace. +- A runtime image with matching Dynamo, vLLM, KVCR v0.2.0, and NIXL builds (see [Configure KVCR for vLLM](configure-kvcr-vllm.md#before-you-begin)). If its tag is not a Dynamo release version, set `runtimeVersionOverride` on every component. + +## Deploy + +1. From `examples/backends/vllm/deploy/kvcr` in the Dynamo repository, set the variables and apply `agg.yaml`. `envsubst` fills in only the listed variables; the manifest's startup scripts use other `$` variables that must stay as they are. + + ```bash + export NAMESPACE= + export DYNAMO_VLLM_IMAGE=@sha256: + export DYNAMO_UCX_NET_DEVICES=:1 + export DYNAMO_RDMA_RESOURCE=rdma/shared_ib + + envsubst '${DYNAMO_VLLM_IMAGE} ${DYNAMO_RDMA_RESOURCE} ${DYNAMO_UCX_NET_DEVICES}' \ + < agg.yaml | kubectl apply -n "$NAMESPACE" -f - + ``` + + | Variable | Value | + |---|---| + | `DYNAMO_UCX_NET_DEVICES` | The example requires it. NIXL picks network devices on its own, but on nodes with several RDMA NICs, workers that see every NIC can stall at startup, so pin the NIC attached directly to the worker's GPU (marked `PIX` for that GPU in `nvidia-smi topo -m`), for example `mlx5_5:1`. On a multi-GPU node the right NIC depends on which GPU the Pod gets. | + | `DYNAMO_RDMA_RESOURCE` | The RDMA extended resource your cluster exposes. | + +2. Wait until the frontend and both worker Pods are `1/1 Running`: + + ```bash + kubectl get pods -n "$NAMESPACE" -w + ``` + +3. Port-forward port 8000 of the frontend Pod, then confirm a cross-worker transfer with [Verify and Monitor KVCR with vLLM](kvcr-testing-monitoring-vllm.md), using the two worker IDs. Each worker serves its metrics on port 9090 of its Pod; reach a worker's metrics with `kubectl port-forward -n "$NAMESPACE" pod/ 9090` and read them at `localhost:9090/metrics`. + + The example sets no `--block-size`, so blocks are 16 tokens and `cached_tokens` is a multiple of 16. + +To change the pool size, edit `secondary_g2_slots` in the worker's `kv-transfer-config`; see [Size the Tiers](configure-kvcr-vllm.md#size-the-tiers). + +## Related Topics + +- [Configure KVCR for vLLM](configure-kvcr-vllm.md) +- [Run KVCR Guard on Kubernetes with vLLM](kvcr-guard-kubernetes-vllm.md) +- [Verify and Monitor KVCR with vLLM](kvcr-testing-monitoring-vllm.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-routing-tuning.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-routing-tuning.md new file mode 100644 index 000000000000..7787c46ef668 --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-routing-tuning.md @@ -0,0 +1,87 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: Tune Routing for KVCR +subtitle: Recommended router settings for KVCR, tuning the cache weights, and keeping fetches on fast links +--- + +The KV router scores each worker by the prefill work it would have to do. Cached KV lowers that cost. Three settings control how much credit KV outside the GPU gets: + +| Setting | Default | Applies to | +|---|---|---| +| `host_cache_hit_weight` | `0.75` | Prefix blocks the worker holds in host memory | +| `disk_cache_hit_weight` | `0.25` | Prefix blocks the worker holds on disk | +| `shared_cache_multiplier` | `0` (`0.5` with `--shared-cache-type hicache`) | Prefix blocks the worker could fetch from a peer through KVCR, and shared-cache hits | + +`0` ignores those blocks; `1` weights them the same as GPU hits. + +> [!IMPORTANT] +> With the default `shared_cache_multiplier` of `0`, the router does not prefer an idle worker that could fetch a prefix over a busy worker that holds it. KVCR still fetches when the router picks a worker for other reasons. Credit for KVCR-fetchable blocks requires Dynamo with [#16701](https://github.com/ai-dynamo/dynamo/pull/16701); on Dynamo v1.6, `shared_cache_multiplier` is accepted but credits only HiCache shared-cache hits. + +## Recommended Routing + +Combine three settings so sessions stay on their worker until it is busy, then move to a worker that can fetch their prefix: + +- **Soft session affinity.** With the default policy, a session goes to the worker it is bound to regardless of load until the worker crosses the busy threshold, so most turns reuse that worker's GPU cache. Unlike hard affinity, a soft binding is dropped for a request when the worker is overloaded, and the request goes through normal KV-aware selection. With hard affinity, an overloaded bound worker fails the request instead of moving it. +- **A busy threshold** (`--active-decode-blocks-threshold`). This is what marks a worker overloaded. Without one, no worker is ever overloaded and soft affinity never moves a session. +- **Fetchable-block credit** (`shared_cache_multiplier`). When a session moves, this lets the router prefer a worker that can fetch the prefix through KVCR, instead of treating the prefix as lost. Use it with a raised `prefill_load_scale`, so the router weighs prefill work strongly enough not to move sessions between workers for small gains. + +Fetchable-block credit needs Dynamo with [#16701](https://github.com/ai-dynamo/dynamo/pull/16701) (not in v1.6.x). The values below come from simulations in that PR: a 24-hour agentic trace on 8 workers with KVCR fetches restricted to the same network domain ([`required`](#keep-fetches-on-fast-links)). There, a multiplier of `0.75` with a `prefill_load_scale` of 16 to 64 beat no credit. If fetches may cross network domains (`preferred`), leave `shared_cache_multiplier` at `0`: the router credits a cross-domain source at the full multiplier, and that did worse than no credit. + +1. Create `policy.yaml`: + + ```yaml + router: + host_cache_hit_weight: 0.75 + disk_cache_hit_weight: 0.25 + worker_selection: + aggregated: tuned + instances: + - name: tuned + type: dynamo-default-cost-fn + parameters: + shared_cache_multiplier: 0.75 + prefill_load_scale: 16 + ``` + + `worker_selection` names the policy for aggregated workers; `dynamo-default-cost-fn` is the default policy, so this keeps default routing and sets only these two parameters. For disaggregated serving, set `prefill` and `decode` instead of `aggregated`. + +2. Start the frontend with the policy, session affinity, and a busy threshold: + + ```bash + python3 -m dynamo.frontend --router-mode kv \ + --router-policy-config policy.yaml \ + --router-session-affinity-ttl-secs 86400 \ + --router-session-affinity-mode soft \ + --active-decode-blocks-threshold 0.9 + ``` + + `--router-session-affinity-ttl-secs` turns affinity on and sets how long an idle session keeps its worker. Agent sessions can sit idle for a long time between turns, so use a long TTL; in the same simulations, 1 hour did worse than 24 hours. `--active-decode-blocks-threshold 0.9` marks a worker overloaded above 90% KV cache use; tune it for your load, at runtime if needed through the frontend's `/busy_threshold` endpoint. + +3. Send `X-Dynamo-Session-ID: ` on every request of a session. Requests without it are routed normally. + +The policy file is read at startup; restart the frontend after changing it. A frontend started with `shared_cache_multiplier` at `0` does not track fetchable blocks, and per-request overrides cannot turn it on. On Kubernetes, mount the file from a `ConfigMap` on the frontend; see [Swap the Worker-Selection Policy](../../../kubernetes/kv-aware-routing/dynamo-frontend.md#swap-the-worker-selection-policy). For details, see [Session Affinity](../modular-components/router/configuration-and-tuning.md#session-affinity) and [Busy Thresholds](../modular-components/router/worker-filtering.md#busy-thresholds). + +## Tune the Weights + +For the host and disk weights alone, the frontend flags `--router-host-cache-hit-weight` and `--router-disk-cache-hit-weight` also work. Set `shared_cache_multiplier` only in the policy file; its `--shared-cache-multiplier` flag is deprecated. + +Lower `host_cache_hit_weight` if host-memory hits are slower than recomputing on an idle worker. Raise `disk_cache_hit_weight` toward the host weight for fast local NVMe. Lower `shared_cache_multiplier` if fetches across your network are slow. Set all three to `0` for a baseline that routes on GPU cache and load only. + +## Keep Fetches on Fast Links + +A KVCR fetch over a slow link can be slower than recomputing. Restrict fetch sources to the target worker's network domain, such as a host or an RDMA rack: + +```bash +DYN_TOPOLOGY_ENABLED=true +DYN_KV_TRANSFER_DOMAIN=rack # topology key; its value is read from /etc/dynamo/topology/rack +DYN_KV_TRANSFER_ENFORCEMENT=required # or preferred +``` + +Set these on each worker. Use `required` for KVCR: it allows only sources in the same domain. `preferred` also uses another domain when no same-domain source has a longer prefix. On Kubernetes, the operator sets these from the DGD's `KvTransferPolicy`; see [Topology-Aware KV Transfer](../modular-components/router/topology-aware-kv-transfer.md). + +## Related Topics + +- [KV Cache Runner (KVCR)](kvcr.md) +- [Configure KVCR for vLLM](configure-kvcr-vllm.md) +- [Configuration and Tuning](../modular-components/router/configuration-and-tuning.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-testing-monitoring-vllm.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-testing-monitoring-vllm.md new file mode 100644 index 000000000000..478e47f52fee --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr-testing-monitoring-vllm.md @@ -0,0 +1,92 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: Verify and Monitor KVCR with vLLM +subtitle: Confirm a cross-worker KV transfer and watch KVCR in production +--- + +## Verify a Transfer + +Send a long prompt to one worker, then the same prompt to another worker, and check that the second request reuses the first one's KV cache. The routing headers pin the worker for this test only; in normal use, omit them and let the router choose. + +The steps below use one worker with two data-parallel ranks, as in [Configure KVCR for vLLM](configure-kvcr-vllm.md). With two separate workers, repeat step 1 until both worker IDs appear, send each request with its worker's ID and without the `x-dynamo-dp-rank` header, and in step 6 read both workers' metrics: ports `8081` and `8082` in the Guard example, or port `9090` of each worker Pod through `kubectl port-forward` on Kubernetes. The source worker shows `source_write` and the target worker shows `remote_deliver`. + +1. Get the worker ID: + + ```bash + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -d '{"model":"Qwen/Qwen3-0.6B","prompt":"hello","max_tokens":1,"nvext":{"extra_fields":["worker_id"]}}' + ``` + + The response contains `"decode_worker_id": `. Use that number as `` below. + +2. Create `request.json` with a prompt of at least 1,000 tokens that you have not sent before, for example a few pages of any document: + + ```json + {"model": "Qwen/Qwen3-0.6B", "prompt": "", "max_tokens": 8} + ``` + +3. Send it to rank 0: + + ```bash + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -H 'x-dynamo-worker-instance-id: ' -H 'x-dynamo-dp-rank: 0' -d @request.json + ``` + + The response shows `"cached_tokens": 0`. + +4. Wait 15 seconds. Offload, KV events, and router indexing are asynchronous. + +5. Send the same request to rank 1: + + ```bash + curl -s localhost:8000/v1/completions -H 'content-type: application/json' \ + -H 'x-dynamo-worker-instance-id: ' -H 'x-dynamo-dp-rank: 1' -d @request.json + ``` + + `cached_tokens` in `usage.prompt_tokens_details` must be positive and a multiple of the block size. If it is `0`, see [Troubleshooting](#troubleshooting). + +6. Confirm the transfer on the worker's metrics port (`DYN_SYSTEM_PORT`, `8081` in the configure examples): + + ```bash + curl -s localhost:8081/metrics | grep '^vllm:kvcr_transfer_blocks_total' + ``` + + Rank 0 (`engine="0"`) shows `operation="source_write"` and rank 1 (`engine="1"`) shows `operation="remote_deliver"` with the same count. With a KVCR pool, the source also shows `local_fill`. + +To run the check again, change the prompt in `request.json`; a prompt that is already cached on both ranks does not transfer. + +## Monitor + +Scrape each worker's `/metrics` (requires `DYN_SYSTEM_PORT` and `"enable_telemetry": true`): + +| Metric | Use | +|---|---| +| `vllm:kvcr_transfer_blocks_total{operation}` | Blocks moved: `source_write`, `remote_deliver`, `local_fill`, `local_deliver`. | +| `vllm:kvcr_transfer_bytes_total{operation}` | Bytes moved. | +| `vllm:kvcr_duration_seconds{scope,result}` | Latency and outcome per stage. | +| `vllm:kvcr_state{resource}` | Current state, such as `block_records` and `in_flight_ops`. | + +Alert on failed or partial deliveries: + +```text +increase(vllm:kvcr_duration_seconds_count{scope=~"remote_deliver|remote_fetch",result=~"partial|failed"}[5m]) > 0 +``` + +## Troubleshooting + +| Symptom | Check | +|---|---| +| Worker does not start | `router_capabilities` includes `router_hint`; `self_describing_kv_events` is `true`; every control and KV-events port is unique; `/dev/shm` covers `cpu_bytes_to_use` for every rank. | +| `cached_tokens` stays `0` | The frontend runs `--router-mode kv`; the frontend's `dynamo_component_kv_cache_events_applied` is increasing; the worker log has no `KV routing remains disabled`. Then retry with a new prompt in `request.json`. | +| Worker hangs at startup, or transfers fail across hosts | Set `UCX_NET_DEVICES` to the one NIC attached to the worker's GPU (`PIX` in `nvidia-smi topo -m`); with every NIC visible, workers can stall while exchanging transfer metadata. | +| `remote_deliver` partial or failed counts rise under load | Raise `operation_timeout_ms` (default `1000`) and keep `abandon_timeout_ms` at least twice it. | +| No transfer across hosts | `control_advertise_host` is reachable from the other worker, not `127.0.0.1`. | +| Peers cannot read a crashed worker's Guard | The state agent is running and the worker sets `dynamo_state_agent`. | +| Log shows `kv_offload_tiering_cascade_job_failures` | Expected in peer-only setups, once per new prefix. It does not affect transfers. | + +## Related Topics + +- [Configure KVCR for vLLM](configure-kvcr-vllm.md) +- [Run KVCR on Kubernetes with vLLM](kvcr-kubernetes-vllm.md) +- [Run KVCR Guard on Kubernetes with vLLM](kvcr-guard-kubernetes-vllm.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr.md new file mode 100644 index 000000000000..dad78963d3d7 --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/kvcr.md @@ -0,0 +1,161 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: KV Cache Runner (KVCR) +subtitle: Reuse KV cache across workers and nodes, directed by the KV router +--- + +**KV Cache Runner (KVCR)** is Dynamo's distributed KV cache manager. It lets a worker reuse KV blocks that another worker already computed, on the same node or another one, instead of recomputing the prefix. + +KVCR runs inside each worker: in vLLM as a secondary tier of `OffloadingConnector`, alongside the worker's own CPU cache; in SGLang as a HiCache storage backend or through the Linker. + +## What It Is and Why It Is Split This Way + +Reusing cache across workers needs a cluster-wide view to decide, and fast local execution to move the data. KVCR keeps those apart: + +- **The KV router decides.** It already knows which worker holds which prefix, how loaded each worker is, and how the network is laid out. When it sends a request to a worker that lacks part of the prefix, it attaches a **KV hint** naming a peer that has it. +- **KVCR moves and manages.** The receiving worker's KVCR fetches the blocks from that peer over NIXL, then applies its own admission and eviction policy. A fetched block is not guaranteed to stay cached; that is a local decision. + +Splitting the two lets the router decide early, when it schedules the request and before any data moves. Because the decision does not wait on a transfer, the router has room to weigh every worker, every tier, and every possible source, and makes a better scheduling decision than a worker that sees only its own cache. KVCR keeps no cluster-wide inventory of its own, and the router leaves local cache policy to KVCR. See [How KV Hints Work](kv-hints.md) for when hints are sent and how sources are chosen. + +By default the router credits only blocks a worker already holds; it does not choose a worker because that worker could fetch a prefix. To make fetchable blocks count, set `shared_cache_multiplier`; see [Tune Routing for KVCR](kvcr-routing-tuning.md). + +## Events and Hints + +KVCR uses KV events like the other tiers, and adds hints on top: + +- **Events tell the router where blocks are.** Each worker publishes KV events for its GPU cache and vLLM's CPU tier, and KVCR publishes events for its own pool. This is why `self_describing_kv_events` must be on: the events carry the block metadata the router needs to index KVCR's tiers. +- **Hints tell a worker where to fetch.** When the router sends a request to a worker that lacks part of a prefix another worker holds, it attaches a hint naming that worker. The target's KVCR fetches the blocks; the router never moves data. + +Events alone let the router credit a worker for blocks it already holds. Hints let a worker use blocks that another worker holds. + +## In-Process vs. Guard Sidecar + +KVCR always runs in the worker process for serving, so cache access needs no extra hop. **Guard** owns the KVCR memory pool outside the worker, and the worker maps the pool directly. A Guard service (`kvcr_service`) is a separate process that hosts one Guard per rank. Because the pool lives outside the worker, it survives an engine crash: the restarted worker reattaches to its blocks, and peers can fetch them while it is down. Guard adds to the in-process KVCR; it does not replace it. For peers to keep using a crashed worker's Guard, also run the [KV state agent](#the-kv-state-agent). + +### Guard Limits + +- One Guard per rank: several ranks cannot share one Guard yet. A Guard service hosts as many Guards as `--guard-count`, one per rank. +- vLLM picks the Guard whose index matches the local data-parallel rank, so separate vLLM worker processes need separate Guard services. SGLang ranks choose their Guard with `guard_index`, so several SGLang workers can share one service. +- If the Guard service itself restarts, recovery no longer works for its ranks; restart their workers with it. + +## The KV State Agent + +The router normally ties cached blocks to the worker that holds them: when a worker deregisters, for example after a crash, the router drops its blocks from the index. With Guard, those blocks are still in memory, so dropping them throws away reusable cache. + +The **KV state agent** (`dynamo.kv_state_agent`) is a separate Dynamo process that fixes this by indexing caches by **cache owner** instead of by worker process: + +- Each rank declares a cache owner ID and the address of its raw KV event stream: for vLLM, `dynamo_state_agent.cache_owner_ids` and `raw_advertise_host`; for SGLang Linker mode, `--kvcr-state-agent-owner` and `advertise_host`. The address must not be a loopback address, because the state agent connects to it. +- The state agent reads that event stream and keeps the owner's blocks registered with the router, together with the owner's control endpoint. +- When the worker crashes, the owner stays registered. The router keeps sending peers hints that point at the owner, and the Guard serves them. +- When the worker restarts with the same cache owner ID, it takes the owner back. + +Run one state agent per Dynamo namespace, which on Kubernetes means one per DynamoGraphDeployment: workers find the state agent through their namespace, so every worker in it shares the same one. `--max-slots` is the number of cache owners it tracks, one per rank across all those workers. Cache owner IDs must be unique per rank and stay the same across restarts. Without the state agent, a restarted worker still reattaches to its own Guard pool, but the router stops pointing peers at a crashed worker's Guard as soon as that worker deregisters. + +## Memory Options + +Use a KVCR pool, and add Guard when cached blocks must survive an engine crash. Peer-only is for functional testing: + +| Option | Where cached blocks live | Survives an engine crash | Use | +|---|---|---|---| +| KVCR pool | A KVCR-owned host-memory pool in the worker process, sized separately from the engine's CPU cache | No | Recommended | +| Guard | A KVCR pool owned by the Guard process | Yes | When the cache must survive a crash; also needs the [KV state agent](#the-kv-state-agent) | +| Peer-only | Each worker's engine-managed CPU cache | No | Testing only | + +A KVCR pool is faster than peer-only: KVCR registers the pool once at startup, so transfers do not wait for the engine to pin its CPU cache memory. With vLLM and SGLang HiCache, the engine keeps a small CPU cache that stages blocks between the GPU and KVCR. Some modes do not need an engine-managed CPU cache at all: in SGLang Linker mode, KVCR manages all of the host memory and delivers blocks straight into the GPU. + +### KVCR Pool + +Each worker adds a KVCR pool next to a small engine CPU cache. Blocks in the pool are kept by KVCR's policy and served to any peer. + +```mermaid +flowchart LR + R[KV router] -. hint .-> B + subgraph A[Worker A] + AG[GPU KV] --> AC[Engine CPU cache] --> AP[(KVCR pool)] + end + subgraph B[Worker B] + BC[Engine CPU cache] --> BG[GPU KV] + end + AP -- NIXL --> BC +``` + +### Guard + +The pool moves into a separate Guard process, and the KV state agent keeps the pool indexed under its cache owner. If worker A crashes, the router can still send B to A's Guard, and A reattaches to the pool when it restarts. + +```mermaid +flowchart LR + R[KV router] -. hint .-> B + S[KV state agent] -- cache owner index --> R + subgraph A[Worker A] + AG[Engine] + GA[(Guard process: KVCR pool)] + AG -- maps --> GA + end + subgraph B[Worker B] + BC[Engine CPU cache] --> BG[GPU KV] + end + GA -- NIXL --> BC + AG -. KV events .-> S +``` + +### Peer-Only + +Each worker keeps offloaded KV only in the engine's own CPU cache, under the engine's eviction policy. When the router sends B a hint, B's KVCR copies the blocks from A's CPU cache into B's CPU cache over NIXL, and the engine loads them into the GPU. + +```mermaid +flowchart LR + R[KV router] -. hint .-> B + subgraph A[Worker A] + AG[GPU KV] --> AC[Engine CPU cache] + end + subgraph B[Worker B] + BC[Engine CPU cache] --> BG[GPU KV] + end + AC -- NIXL --> BC +``` + +### SGLang: HiCache and Linker Modes + +In HiCache mode, KVCR is SGLang's HiCache storage backend, so KV passes through SGLang's HiCache host tier. In Linker mode, KVCR delivers KV directly into the GPU. + +```mermaid +flowchart LR + subgraph H[HiCache mode] + HP[(KVCR pool)] --> HH[HiCache host tier] --> HG[GPU KV] + end + subgraph L[Linker mode] + LP[(KVCR pool)] --> LG[GPU KV] + end +``` + +Local disk (`g3`) extends capacity beyond host memory and is experimental; see the [KVCR developer guide](https://github.com/ai-dynamo/kvcr/blob/main/docs/dev-guide.md). Remote disk is planned. + +## Support + +| Framework | Status | +|---|---| +| vLLM | Supported with KVCR v0.2.0 and a vLLM build with the KVCR adapter. | +| SGLang | Preview: HiCache mode on one node, needs unmerged SGLang changes; see [Configure KVCR for SGLang](configure-kvcr-sglang.md). Linker mode: see [Configure KVCR Linker Mode for SGLang](configure-kvcr-sglang-linker.md). Preview: needs the SGLang KVCR series ([#45646](https://github.com/sgl-project/sglang/pull/45646) to [#45650](https://github.com/sgl-project/sglang/pull/45650)) and the Dynamo change in [#16827](https://github.com/ai-dynamo/dynamo/pull/16827), neither merged. | +| TensorRT-LLM | Planned. | + +See the [KVCR framework support list](https://github.com/ai-dynamo/kvcr#framework-support) for the work in progress. + +## Next Steps + +| | vLLM | SGLang (preview) | +|---|---|---| +| Configure on one server | [Configure KVCR for vLLM](configure-kvcr-vllm.md) | [Configure KVCR for SGLang](configure-kvcr-sglang.md) | +| Linker mode, with Guard | Not applicable | [Configure KVCR Linker Mode for SGLang](configure-kvcr-sglang-linker.md) | +| Kubernetes | [Run KVCR on Kubernetes with vLLM](kvcr-kubernetes-vllm.md) | [Run KVCR on Kubernetes with SGLang](kvcr-kubernetes-sglang.md) | +| Kubernetes with Guard | [Run KVCR Guard on Kubernetes with vLLM](kvcr-guard-kubernetes-vllm.md) | [Run KVCR Guard on Kubernetes with SGLang](kvcr-guard-kubernetes-sglang.md) | +| Verify and monitor | [Verify and Monitor KVCR with vLLM](kvcr-testing-monitoring-vllm.md) | [Check It Works](configure-kvcr-sglang.md#check-it-works) | + +Both engines: [How KV Hints Work](kv-hints.md), [Tune Routing for KVCR](kvcr-routing-tuning.md). + +## Related Topics + +- [KV Cache Management Overview](overview.md) +- [How KV Hints Work](kv-hints.md) +- [KVCR Design Overview](https://github.com/ai-dynamo/kvcr/blob/main/docs/design_overview.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/kv-offload/overview.md b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/overview.md new file mode 100644 index 000000000000..9ceae6ceb930 --- /dev/null +++ b/docs/fern/pages/developer-guide/knowledge-base/kv-offload/overview.md @@ -0,0 +1,94 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: KV Cache Management Overview +subtitle: Store reusable KV cache efficiently across tiers and route requests to it +--- + +Recomputing the same prompt prefix across requests is expensive, especially in agentic loops that resend a large context. Reuse depends on two things: keeping the right blocks after they leave GPU memory, and sending each request to a worker that holds them or can fetch them. + +Dynamo tracks where cached blocks are, so it can make both decisions. This page explains how routing and cache management work together, and which cache manager to use. + +## Cache Tiers + +Instead of discarding KV cache when GPU memory fills up, a worker can keep it in lower tiers: host memory, local disk, or storage shared with other workers. Blocks do not have to move down one tier at a time. Depending on the cache manager, blocks can be written through to every tier as they are stored, or go from GPU memory straight to disk. + +```mermaid +flowchart LR + G["GPU memory"] --> H["Host memory"] + G --> D["Disk"] + H --> D + H -.->|peer or shared access| S["Other workers"] + R["KV router"] -.->|indexes every tier| G & H & D +``` + +The tiers work together as one cache. GPU memory serves the request. Host memory holds far more blocks and returns them quickly. Disk and shared storage hold the most but are the slowest to read back. A later request that reuses a prefix recalls its blocks from the fastest tier that has them instead of recomputing prefill. What matters is that every tier is addressable from one place: the router knows where each block is, so it can weigh all of them when it picks a worker. + +### Write-Through and Write-Back + +The write policy decides when a block is copied to the tier below. It changes how soon a block can be reused elsewhere and how much copying the worker does: + +| Policy | When a block is copied down | Benefit | Cost | +|---|---|---|---| +| Write-through | As soon as it is stored in the higher tier | The lower tier always has a copy, so other workers can fetch it sooner and the higher tier can drop it without waiting | Copies every block, including blocks that are never reused | +| Write-back | Only when the higher tier approaches its eviction watermark | Copies only blocks that are about to be evicted | Until then the block exists only in the higher tier, and eviction waits for the copy | + +The cache manager sets the policy. For example, SGLang HiCache sets it with `--hicache-write-policy`. + +## Internal vs. External Cache Managers + +A worker's offloaded KV cache is managed either by the engine itself or by an external manager. Dynamo works with whatever offloading the engine supports, and adds external managers on top: + +- **Internal (engine-native).** The engine manages its own lower tiers, for example vLLM's `OffloadingConnector` or SGLang's HiCache. Nothing extra to deploy, but the cache serves only that worker unless another mechanism shares it. +- **External.** A separate component manages the cache and can make blocks available to other engine instances, which gives the router more freedom to balance cache reuse against load. "External" describes ownership, not deployment: the manager can run inside the engine process, beside it, or as a shared service. + +You can use one or both: for example, vLLM's own CPU tier with KVCR as a secondary tier behind it. + +| Manager | Type | Approach | Setup | +|---|---|---|---| +| vLLM `OffloadingConnector` | Internal | Per-worker host and disk tiers | [Native KV Offloading](../modular-components/backends/vllm/native-kv-offloading.md) | +| SGLang HiCache | Internal | Host tier, optional external storage such as Mooncake | [HiCache](../modular-components/backends/sglang/hicache.md) | +| Mooncake | External | Storage pool shared by workers, as a HiCache storage backend | [Tier-Aware Shared KV Cache Routing](../modular-components/backends/sglang/hicache.md#tier-aware-shared-kv-cache-routing) | +| LMCache | External | Local and remote storage options | [KV Cache Offloading](../modular-components/backends/vllm/kv-cache-offloading.md#lmcache) | +| FlexKV | External | Linked into the engine process | [KV Cache Offloading](../modular-components/backends/vllm/kv-cache-offloading.md#flexkv) | +| **KVCR** | External | Worker-to-worker transfers directed by the KV router | [KV Cache Runner (KVCR)](kvcr.md) | + +> [!NOTE] +> KVBM is deprecated. Migrate to the engine's native offloading or KVCR; see [Deprecations](../../../reference/general/releases/deprecations.mdx). + +Use engine-native offloading when each worker reuses its own prefixes. Add an external cache manager, such as KVCR for vLLM and SGLang, when you need more: workers fetching each other's blocks peer to peer, a shared pool of host memory, or shared or remote storage. + +## How Routing Affects Cache Management + +The KV router combines cache availability, worker load, and network topology to decide where to serve a request. Workers publish KV events as blocks are stored and evicted, and the router indexes them, so it knows which worker holds which prefix and in which tier. When a remote fetch is worthwhile, it can also name a peer that supplies the cached blocks, and the serving worker's cache manager fetches them. + +[Routing Concepts](../modular-components/router/routing-concepts.md) explains the cost model, [Configuration and Tuning](../modular-components/router/configuration-and-tuning.md) covers router settings, and [Topology-Aware KV Transfer](../modular-components/router/topology-aware-kv-transfer.md) covers network-aware peer selection. + +## Dynamo's Approach to Distributed Cache Management + +Sharing cached blocks across workers takes a decision and data movement. The decision is which worker serves the request, whether fetching is worth it, and where the blocks come from. The data movement is transferring the blocks to that worker and keeping them there. Dynamo splits the two: + +- **The router coordinates.** It already holds the cluster-wide view of cache, load, and topology, so it chooses the serving worker and the source for any remote fetch. +- **The cache manager moves and keeps blocks.** It transfers blocks and applies local admission and eviction policy. + +KVCR is built to take advantage of this split: it moves blocks where the router directs, instead of keeping a second cluster-wide inventory. See [KV Cache Runner (KVCR)](kvcr.md). + +## What the Router Sees per Engine + +Each engine reports offloaded blocks differently, which decides what the router can credit: + +| Engine | Version | GPU | Host memory | Disk | Shared or peer KV | +|---|---|---|---|---|---| +| vLLM | vLLM v0.24.0+, Dynamo 1.3.0+ | Per tier | Per tier (`OffloadingConnector` events) | Per tier (experimental) | Through [KV Cache Runner (KVCR)](kvcr.md) | +| SGLang | SGLang 0.5.11+ (0.5.13+ with Mooncake), Dynamo 1.2+ | Per tier | Per tier (HiCache events) | - | HiCache + Mooncake (`--shared-cache-type hicache`) | +| TensorRT-LLM | Dynamo 1.3.0+ | Cached or not, no tier (`--publish-kv-events`) | Same as GPU | - | - | + +Per-tier engines get separate credit for host and disk hits; see [Tune Routing for KVCR](kvcr-routing-tuning.md). For TensorRT-LLM the router sees that a block is cached, not where, so per-tier weights do not apply. To confirm the router receives KV events, check `dynamo_component_kv_cache_events_applied` on the frontend's `/metrics`. + +## Related Topics + +- [KV Cache Runner (KVCR)](kvcr.md) +- [How KV Hints Work](kv-hints.md) +- [Configure KVCR for vLLM](configure-kvcr-vllm.md) +- [KV-Aware Routing](../concepts/system-architecture/kv-aware-routing.md) +- [Topology-Aware KV Transfer](../modular-components/router/topology-aware-kv-transfer.md) diff --git a/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md b/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md index dbce0e139bdd..b8e44da8bad3 100644 --- a/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md +++ b/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md @@ -152,7 +152,7 @@ worker_selection: router_temperature: 0.0 ``` -Explicit parameters take precedence over router flags and environment variables. Omitted parameters inherit their existing values or defaults. The default policy uses `shared_cache_multiplier: 0.5` when shared cache is enabled; an explicit `0` disables shared-cache credit. The same multiplier credits the blocks a worker could fetch from a peer through a [KV fetch hint](topology-aware-kv-transfer.md#kv-fetch-hints), such as from KVCR. Without shared cache it defaults to `0`, so set it to enable that credit. Select the instance for each stage you want to tune; stages omitted from `worker_selection` keep the default selector. +Explicit parameters take precedence over router flags and environment variables. Omitted parameters inherit their existing values or defaults. The default policy uses `shared_cache_multiplier: 0.5` when shared cache is enabled; an explicit `0` disables shared-cache credit. The same multiplier credits the blocks a worker could fetch from a peer through a [KV fetch hint](topology-aware-kv-transfer.md#kv-fetch-hints), such as from KVCR. Without shared cache it defaults to `0`, so set it to enable that credit; for recommended KVCR values, see [Tune Routing for KVCR](../../kv-offload/kvcr-routing-tuning.md). Select the instance for each stage you want to tune; stages omitted from `worker_selection` keep the default selector. > [!WARNING] > The flags below and their environment variables are deprecated for removal in v1.7. They still work and emit a warning. Move their values into the policy’s `parameters` mapping. diff --git a/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/offloading-support-matrix.md b/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/offloading-support-matrix.md index 0345ca7b1f83..e950f409650e 100644 --- a/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/offloading-support-matrix.md +++ b/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/offloading-support-matrix.md @@ -127,7 +127,7 @@ logit = prefill_load_scale * adjusted_prefill_blocks // default 1.0 Before applying the device term, the router computes `effective_device_credit = overlap_score_credit / (1 + overlap_score_credit_decay * normalized_excess_prefill)`. -The host, disk, shared-cache, and fetchable-block credits do not decay. `fetchable_blocks` are the blocks a worker could fetch from a peer through a KV fetch hint, such as from KVCR, beyond everything it holds. +The host, disk, shared-cache, and fetchable-block credits do not decay. `fetchable_blocks` are the blocks a worker could fetch from a peer through a KV fetch hint, such as from [KVCR](../../kv-offload/kvcr.md), beyond everything it holds. The router picks the worker with the **minimum** logit. @@ -149,4 +149,5 @@ See [Configuration and Tuning](configuration-and-tuning.md) for the cache-hit we - [Native KV Offloading](../backends/vllm/native-kv-offloading.md) — vLLM detail page - [Using HiCache](../backends/sglang/hicache.md) — SGLang detail page - [KV Cache Offloading](../backends/vllm/kv-cache-offloading.md) — vLLM offloading backend catalog (native, KVBM, LMCache, FlexKV) +- [KV Cache Management Overview](../../kv-offload/overview.md) - offload tiers, cache managers, and KVCR - [Feature Matrix](../../../../reference/general/compatibility.mdx) — per-backend feature support across Dynamo diff --git a/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/topology-aware-kv-transfer.md b/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/topology-aware-kv-transfer.md index c68b26404fb7..a53a93a61bb2 100644 --- a/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/topology-aware-kv-transfer.md +++ b/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/topology-aware-kv-transfer.md @@ -150,7 +150,7 @@ User-provided constraints still apply. A decode worker must satisfy all required ## KV Fetch Hints -When workers advertise router-hint support, as vLLM workers with KVCR do, the router can attach a `kv.fetch` hint to the selected worker. The hint names a peer that holds a longer prefix of the request, so the worker can fetch that KV instead of computing it. A fetch over a slow link, such as TCP between hosts without RDMA, is usually slower than recomputing, so with topology enabled the fetch source must be in the target worker's island: its value for the target's `kv_transfer_domain` must equal the target's value. +When workers advertise router-hint support, as vLLM workers with [KVCR](../../kv-offload/kvcr.md) do, the router can attach a `kv.fetch` hint to the selected worker. The hint names a peer that holds a longer prefix of the request, so the worker can fetch that KV instead of computing it. A fetch over a slow link, such as TCP between hosts without RDMA, is usually slower than recomputing, so with topology enabled the fetch source must be in the target worker's island: its value for the target's `kv_transfer_domain` must equal the target's value. - The target worker's `kv_transfer_domain` and `kv_transfer_enforcement` apply. This is the reverse of prefill-to-decode routing, where the prefill worker's policy applies. - A source with no value for that domain is in no island. diff --git a/examples/backends/sglang/deploy/kvcr/sglang-agg-hicache.yaml b/examples/backends/sglang/deploy/kvcr/sglang-agg-hicache.yaml new file mode 100644 index 000000000000..580549a505ac --- /dev/null +++ b/examples/backends/sglang/deploy/kvcr/sglang-agg-hicache.yaml @@ -0,0 +1,97 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: nvidia.com/v1beta1 +kind: DynamoGraphDeployment +metadata: + annotations: + nvidia.com/dynamo-discovery-backend: kubernetes + name: sglang-agg-kvcr +spec: + components: + - name: Frontend + podTemplate: + spec: + containers: + - env: + - name: DYN_ROUTER_MODE + value: kv + image: ${DYNAMO_SGLANG_IMAGE} + name: main + replicas: 1 + type: frontend + runtimeVersionOverride: ${DYNAMO_VERSION} + - name: worker + podTemplate: + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: nvidia.com/dynamo-component + operator: In + values: + - worker + - key: nvidia.com/dynamo-graph-deployment-name + operator: In + values: + - sglang-agg-kvcr + topologyKey: kubernetes.io/hostname + containers: + - args: + - | + set -euo pipefail + exec python3 -m dynamo.sglang \ + --model-path Qwen/Qwen3-0.6B --tp-size 1 --page-size 64 --max-total-tokens 65536 \ + --enable-hierarchical-cache --hicache-ratio 1.5 \ + --hicache-mem-layout page_first --hicache-write-policy write_through \ + --hicache-storage-backend kvcr --hicache-storage-prefetch-policy wait_complete \ + --hicache-storage-backend-extra-config "{\"control_advertise_host\": \"$POD_IP\", \"control_port\": 24100, \"nixl_listen_port\": 24200, \"local_dram_bytes\": 8589934592}" \ + --kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:24020"}' + command: + - /bin/bash + - -c + env: + - name: DYN_SYSTEM_PORT + value: "9090" + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: PYTHONHASHSEED + value: "0" + - name: UCX_TLS + value: rc_x,cuda + - name: UCX_NET_DEVICES + value: ${DYNAMO_UCX_NET_DEVICES} + envFrom: + - secretRef: + name: hf-token-secret + image: ${DYNAMO_SGLANG_IMAGE} + livenessProbe: + httpGet: + path: /live + port: system + periodSeconds: 10 + timeoutSeconds: 4 + failureThreshold: 60 + name: main + resources: + limits: + ${DYNAMO_RDMA_RESOURCE}: "1" + nvidia.com/gpu: "1" + requests: + ${DYNAMO_RDMA_RESOURCE}: "1" + nvidia.com/gpu: "1" + securityContext: + capabilities: + add: + - IPC_LOCK + tolerations: + - effect: NoSchedule + key: nvidia.com/gpu + operator: Exists + replicas: 2 + type: worker + runtimeVersionOverride: ${DYNAMO_VERSION} diff --git a/examples/backends/sglang/deploy/kvcr/sglang-agg-linker-guard.yaml b/examples/backends/sglang/deploy/kvcr/sglang-agg-linker-guard.yaml new file mode 100644 index 000000000000..b9fb6d003e57 --- /dev/null +++ b/examples/backends/sglang/deploy/kvcr/sglang-agg-linker-guard.yaml @@ -0,0 +1,278 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: nvidia.com/v1beta1 +kind: DynamoGraphDeployment +metadata: + annotations: + nvidia.com/dynamo-discovery-backend: kubernetes + nvidia.com/dynamo-kube-discovery-mode: container + name: sglang-agg-kvcr-guard +spec: + components: + - name: Frontend + podTemplate: + spec: + containers: + - env: + - name: DYN_ROUTER_MODE + value: kv + image: ${DYNAMO_SGLANG_IMAGE} + name: main + replicas: 1 + type: frontend + runtimeVersionOverride: ${DYNAMO_VERSION} + - name: worker + podTemplate: + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: nvidia.com/dynamo-component + operator: In + values: + - worker + - key: nvidia.com/dynamo-graph-deployment-name + operator: In + values: + - sglang-agg-kvcr-guard + topologyKey: kubernetes.io/hostname + containers: + - args: + - | + set -euo pipefail + for _ in $(seq 1 60); do + if python3 -c "import socket; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close()" >/dev/null 2>&1; then + break + fi + sleep 1 + done + if ! python3 -c "import socket; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close()"; then + echo "KVCR memory service did not become ready" >&2 + exit 1 + fi + + cache_owner=$( + python3 - <<'PY' + import os + + raw_slot = os.environ["KVCR_CACHE_SLOT"] + raw_count = os.environ["KVCR_CACHE_SLOT_COUNT"] + if not raw_slot.isdecimal() or not raw_count.isdecimal(): + raise SystemExit("KVCR cache slot and count must be decimal integers") + slot = int(raw_slot) + if int(raw_count) <= 0 or slot >= int(raw_count): + raise SystemExit(f"KVCR cache slot {slot} is outside [0, {raw_count})") + print( + "01010101010101010101010101010101:" + "02020202020202020202020202020202/" + f"0303030303030303/{slot:032x}" + ) + PY + ) + exec python3 -m dynamo.sglang \ + --model-path Qwen/Qwen3-0.6B --page-size 64 --mem-fraction-static 0.6 \ + --kv-events-config '{"publisher": "zmq", "topic": "", "endpoint": "tcp://*:23000"}' \ + --enable-unified-cache-external-linker --unified-cache-external-linker-backend kvcr \ + --hicache-storage-backend-extra-config "{\"advertise_host\": \"$POD_IP\", \"control_port\": 26800, \"nixl_port\": 20500, \"kvcr_service_socket_path\": \"/run/kvcr/memory.sock\", \"guard_index\": 0, \"compatibility_digest\": \"${DYNAMO_KVCR_COMPATIBILITY_DIGEST}\"}" \ + --kvcr-state-agent-owner "$cache_owner" + command: + - /bin/bash + - -c + env: + - name: CONTAINER_NAME + value: main + - name: DYN_KUBE_DISCOVERY_MODE + value: container + - name: DYN_SYSTEM_PORT + value: '9090' + - name: KVCR_CACHE_SLOT + valueFrom: + fieldRef: + fieldPath: metadata.labels['grove.io/podclique-pod-index'] + - name: KVCR_CACHE_SLOT_COUNT + value: '2' + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: POD_INDEX + valueFrom: + fieldRef: + fieldPath: metadata.labels['grove.io/podclique-pod-index'] + - name: PYTHONHASHSEED + value: '0' + - name: UCX_TLS + value: rc_x,cuda + - name: UCX_NET_DEVICES + value: ${DYNAMO_UCX_NET_DEVICES} + envFrom: + - secretRef: + name: hf-token-secret + image: ${DYNAMO_SGLANG_IMAGE} + livenessProbe: + httpGet: + path: /live + port: system + periodSeconds: 10 + timeoutSeconds: 4 + failureThreshold: 60 + name: main + resources: + limits: + ${DYNAMO_RDMA_RESOURCE}: '1' + nvidia.com/gpu: '1' + requests: + ${DYNAMO_RDMA_RESOURCE}: '1' + ephemeral-storage: 2Gi + nvidia.com/gpu: '1' + securityContext: + capabilities: + add: + - IPC_LOCK + volumeMounts: + - mountPath: /run/kvcr + name: kvcr-socket + - mountPath: /dev/shm/kvcr + name: kvcr-memory + - args: + - | + set -euo pipefail + state_namespace=$DYN_NAMESPACE + if [ -n "${DYN_NAMESPACE_WORKER_SUFFIX:-}" ]; then + state_namespace="${state_namespace}-${DYN_NAMESPACE_WORKER_SUFFIX}" + fi + + kvcr_pid= + state_agent_pid= + cleanup() { + [ -z "$kvcr_pid" ] || kill -TERM "$kvcr_pid" 2>/dev/null || true + [ -z "$state_agent_pid" ] || kill -TERM "$state_agent_pid" 2>/dev/null || true + [ -z "$kvcr_pid" ] || wait "$kvcr_pid" 2>/dev/null || true + [ -z "$state_agent_pid" ] || wait "$state_agent_pid" 2>/dev/null || true + } + trap cleanup EXIT INT TERM + + python3 -m kvcr.kvcr_service \ + --socket-path /run/kvcr/memory.sock \ + --pool-dir /dev/shm/kvcr \ + --guard-count 1 \ + --pool-sizes-gb 2 \ + --compatibility-digest "${DYNAMO_KVCR_COMPATIBILITY_DIGEST}" & + kvcr_pid=$! + if [ "$POD_INDEX" = "0" ]; then + env DYN_NAMESPACE="$state_namespace" \ + DYN_HEALTH_CHECK_ENABLED=false \ + python3 -m dynamo.kv_state_agent \ + --max-slots "$KVCR_CACHE_SLOT_COUNT" & + state_agent_pid=$! + wait -n "$kvcr_pid" "$state_agent_pid" + else + wait "$kvcr_pid" + fi + command: + - /bin/bash + - -c + env: + - name: CONTAINER_NAME + value: kvcr-services + - name: DYN_COMPONENT + value: kv_state_agent + - name: DYN_DISCOVERY_BACKEND + value: kubernetes + - name: DYN_KUBE_DISCOVERY_MODE + value: container + - name: DYN_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.labels['nvidia.com/dynamo-namespace'] + - name: DYN_NAMESPACE_WORKER_SUFFIX + valueFrom: + fieldRef: + fieldPath: metadata.labels['nvidia.com/dynamo-worker-hash'] + - name: DYN_SYSTEM_PORT + value: '9091' + - name: KVCR_CACHE_SLOT_COUNT + value: '2' + - name: POD_INDEX + valueFrom: + fieldRef: + fieldPath: metadata.labels['grove.io/podclique-pod-index'] + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + - name: UCX_TLS + value: rc_x,cuda + - name: UCX_NET_DEVICES + value: ${DYNAMO_UCX_NET_DEVICES} + image: ${DYNAMO_SGLANG_IMAGE} + livenessProbe: + exec: + command: + - python3 + - -c + - import os,socket,urllib.request; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close(); os.environ['POD_INDEX'] != '0' or urllib.request.urlopen('http://127.0.0.1:9091/live', timeout=1).read() + failureThreshold: 3 + periodSeconds: 5 + timeoutSeconds: 4 + name: kvcr-services + readinessProbe: + exec: + command: + - python3 + - -c + - import os,socket,urllib.request; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close(); os.environ['POD_INDEX'] != '0' or urllib.request.urlopen('http://127.0.0.1:9091/live', timeout=1).read() + failureThreshold: 3 + periodSeconds: 5 + timeoutSeconds: 4 + resources: + limits: + ${DYNAMO_RDMA_RESOURCE}: '1' + memory: 4Gi + requests: + ${DYNAMO_RDMA_RESOURCE}: '1' + cpu: '1' + memory: 3Gi + securityContext: + capabilities: + add: + - IPC_LOCK + startupProbe: + exec: + command: + - python3 + - -c + - import os,socket,urllib.request; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close(); os.environ['POD_INDEX'] != '0' or urllib.request.urlopen('http://127.0.0.1:9091/live', timeout=1).read() + failureThreshold: 60 + periodSeconds: 2 + timeoutSeconds: 4 + volumeMounts: + - mountPath: /run/kvcr + name: kvcr-socket + - mountPath: /dev/shm/kvcr + name: kvcr-memory + tolerations: + - effect: NoSchedule + key: nvidia.com/gpu + operator: Exists + volumes: + - emptyDir: {} + name: kvcr-socket + - emptyDir: + medium: Memory + sizeLimit: 3Gi + name: kvcr-memory + replicas: 2 + type: worker + runtimeVersionOverride: ${DYNAMO_VERSION}