Skip to content

feat(kvcr): supervise startup and configure memory-service resources #17290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aranadive merged 1 commit into from
Nov 3, 2026
Merged

feat(kvcr): supervise startup and configure memory-service resources #17290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
173 changes: 152 additions & 21 deletions examples/backends/vllm/deploy/kvcr/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,30 +23,29 @@ KVCR-capable Dynamo vLLM runtime image for every process.
and UCX builds.
- Linux 6.5 or a kernel with equivalent `SO_PEERPIDFD` support for the
memory-service variant.
- `envsubst` and `kubectl` on the deployment host.
- `envsubst`, `kubectl`, and Python 3 on the deployment host.

The manifests default to the `rdma/shared_ib` extended resource. The
process-local variant requests one share per GPU node; the memory-service
variant requests two shares per node because both `main` and `kvcr-services`
use RDMA. Set
`DYNAMO_RDMA_RESOURCE=rdma/ib` or another cluster-specific resource name when
required. `UCX_TLS=rc_x,cuda` prevents TCP fallback. Set
`DYNAMO_UCX_NET_DEVICES` to the GPU-local HCA and port exposed by the selected
RDMA resource, such as `mlx5_0:1`; do not copy that example device name without
checking the target nodes.
required. `UCX_TLS=rc_x,cuda` prevents TCP payload fallback. Set
`DYNAMO_UCX_NET_DEVICES=all` for per-container single-HCA allocations, or pin
an HCA and port exposed in every engine and sidecar, such as `mlx5_0:1`.

The image tag must contain a Dynamo semantic release version. Pin the same
image by digest for every component, for example
`nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0@sha256:REPLACE_ME`. Until KVCR is
available in a Dynamo release image, build that image from matching Dynamo,
KVCR, and vLLM revisions. Use [KVCR tag
`v0.1.0`](https://github.com/ai-dynamo/kvcr/tree/v0.1.0). Until
[vLLM PR 53624](https://github.com/vllm-project/vllm/pull/53624) is merged, pin
vLLM to the PR's immutable head SHA; at the time of this update it is
[`7015687c`](https://github.com/vllm-project/vllm/commit/7015687c390442fae384b78fc62e6621b3bf6e56).
After the PR merges, pin the commit that lands on vLLM `main`. Record the exact
Dynamo, KVCR, and vLLM commit IDs with the image digest used for every qualified
deployment.
KVCR, and vLLM revisions. This example targets KVCR **0.2.0** at
[`7ce17c358b867bd89f40a4a42bee9c76839506d0`](https://github.com/ai-dynamo/kvcr/commit/7ce17c358b867bd89f40a4a42bee9c76839506d0)
and vLLM's 0.2.0 adapter at
[`fed7de909f894fbee0c45130376aed2e2c9bf0d3`](https://github.com/vllm-project/vllm/commit/fed7de909f894fbee0c45130376aed2e2c9bf0d3).
KVCR requires NIXL 1.5.0 and the runtime's UCX, CUDA, and `libatomic` libraries.
Record the Dynamo, vLLM, and KVCR revisions and image digest for each deployment;
these source pins alone do not establish that an existing image contains a
compatible stack.

`DYNAMO_KVCR_COMPATIBILITY_DIGEST` is an opaque layout version shared by the
engine and memory service. Change it whenever model, dtype, block layout, or
Expand All @@ -67,7 +66,26 @@ KVCR_MEMORY_SERVICE_ENABLED=false ./deploy.sh
KVCR_MEMORY_SERVICE_ENABLED=true ./deploy.sh
```

Run `./deploy.sh --render-only` to inspect the selected DGD. Both manifests
To test KVCR changes without rebuilding an otherwise compatible runtime image,
build a pure-Python wheel from the pinned KVCR tree and export
`KVCR_WHEEL=/absolute/path/to/kvcr-0.2.0-py3-none-any.whl` before deployment.
The script mounts it through `PYTHONPATH` in both containers; it does not install
packages, update vLLM, or change image environment settings. Rebuild the image
for native dependency or vLLM changes. Without `KVCR_WHEEL`, the installed KVCR
must be compatible with this example. A wheel overlay does not upgrade an old
vLLM adapter to the 0.2.0 API. For pinned registration, ensure
`UCX_REG_NONBLOCK_MEM_TYPES` and `UCX_IB_ODP_PREFETCH` are unset in both containers.

`deploy.sh` applies an immutable, content-addressed ConfigMap containing
`services.sh`, Dynamo's shared `launch_utils.sh`, and the optional wheel before
applying the DGD. The bundle is limited to 1 MiB. It is retained after deployment
deletion because other deployments may share it; delete an old bundle only after
no Pods or deployment templates reference it. Do not replace a live bundle or
roll an incompatible KVCR wheel under engines that still map the old pools.

Run `./deploy.sh --render-only` to inspect the selected DGD. For the memory-service
variant, `./deploy.sh --render-configmap` separately renders its prerequisite
ConfigMap; apply that first when deploying rendered files manually. Both manifests
use required Pod anti-affinity, so the second worker remains Pending unless a
second eligible node is available. After apply, the script also verifies that
the operator materialized Grove as the DGD's workload provider. Each engine's
Expand All @@ -92,11 +110,61 @@ supported, restart both workers to restore state tracking after worker 0
restarts.

In `agg-memory-service.yaml`, `dynamo.vllm` runs in `main`. A regular
`kvcr-services` sidecar runs the KVCR memory service and state agent. Both
containers mount `/run/kvcr` and the memory-backed `/dev/shm/kvcr` volume. A
vLLM container restart leaves the service, Guard, and pool running. The
sidecar's probes check the KVCR Unix socket. The state agent runs in worker 0's
sidecar, and that sidecar's probe also checks state-agent health. Kubernetes
`kvcr-services` sidecar runs the KVCR memory service (`kvcr.service.server`)
and, on worker 0 only, the state agent. Both containers mount `/run/kvcr` for
the Unix socket. Only the sidecar mounts the memory-backed `/dev/shm/kvcr`
volume. Guard passes the pool and control-listener file descriptors over the
socket using `SCM_RIGHTS`; the engine maps the same backing pages without
opening a pool path. No shared pool mount, `hostPath`, or permission-setting
init container is needed in the engine. Both containers must use the same UID
to access the service socket. A vLLM container restart leaves the service,
Guard, and pool running.

Both engines report residency to worker 0's state agent. Each Guard can serve
its preserved pool after its engine exits. The FD handoff
does not let a replacement service adopt a live engine's pool; the coupled
failure policy below still applies.

`services.sh` starts KVCR and, on worker 0, the state agent. It uses Dynamo's
`wait_any_exit` and process-group cleanup helpers: any child exit ends the
wrapper and terminates/reaps its companion; SIGTERM does the same. Kubernetes'
Pod termination grace period (default: 30 seconds) is the final bound for stuck
children, not a timeout implemented by the shell helper. For a planned shutdown,
drain and stop engines before their service; the wrapper does not impose
cross-container shutdown ordering.

Memory-service settings accept positive decimal integers without leading zeros:

| Variable | Default | Controls |
|---|---:|---|
| `KVCR_POOL_SIZE_GIB` | 2 | Guard pool capacity per worker, in GiB |
| `KVCR_READY_TIMEOUT_SECONDS` | 120 | HTTP `/ready` wait before launching vLLM |
| `KVCR_GUARD_STARTUP_TIMEOUT_SECONDS` | 30 | Guard preparation and activation, each with this budget |
| `KVCR_TERMINATION_GRACE_PERIOD_SECONDS` | 30 | Pod termination grace for drain and cleanup |

Timeouts are in seconds and independent. HTTP readiness is not inference readiness.
Engine-side KVCR initialization uses KVCR 0.2.0's default 60-second budget;
the pinned vLLM adapter does not expose an override. Increasing the Guard or
readiness timeout does not extend that budget. Pools whose engine-side memory
registration exceeds it require a vLLM adapter that exposes the startup timeout.

Shared memory is pool size + 1 GiB; the sidecar requests pool size + 1%
(rounded up to GiB), with a limit 1 GiB above that. Engine CPU offload stays 2 GiB.
For a 1 TiB pool, reserve 1035 GiB per sidecar (1036 GiB limit), for example:

```bash
KVCR_MEMORY_SERVICE_ENABLED=true KVCR_POOL_SIZE_GIB=1024 \
KVCR_READY_TIMEOUT_SECONDS=600 KVCR_GUARD_STARTUP_TIMEOUT_SECONDS=600 \
KVCR_TERMINATION_GRACE_PERIOD_SECONDS=180 \
./deploy.sh
```

The sidecar's HTTP readiness probe checks that the Unix accept
loop and Guard actor loops progress, plus worker 0's state-agent `/live` endpoint.
It does not require an attached engine or mutate a pool. No liveness/startup
restart probe is set on the sidecar: a failed probe must not restart it beneath
a live engine. Readiness does not itself fence engines or remove Dynamo routing
state. Kubernetes
container discovery gives `main` and the real `kvcr-services` sidecar separate
metadata writers. The explicit sidecar supplies its own downward-API Pod UID;
the operator injects that identity only into its generated `main` container.
Expand All @@ -106,11 +174,73 @@ deployment-fatal: a replacement service cannot adopt the original Guard and
pool while vLLM remains alive. An external deployment controller must replace
the entire affected worker Pod instead of restarting only the sidecar. If
worker 0's sidecar restarts, recreate the worker group or DGD because its state
agent's routing state was also lost.
agent's routing state was also lost. State-agent loss alone does not destroy KV
payloads, but current engines bind a discovered agent instance and do not
automatically reselect its replacement. Group recreation is the conservative
way to restore those bindings. This example requires an external controller to
coordinate Pod or worker-group replacement.

The state agent carries routing and residency information; it does not move
KV payloads. KVCR uses NIXL and UCX for the remote payload transfer.

## Guard HugePages
Comment thread
aranadive marked this conversation as resolved.
Resolved

Guard supports native HugeTLB and transparent huge pages (THP), with base-page
fallback. Check Guard logs to confirm the backing in use. For allocation and
fallback details, see the [KVCR memory-service guide](https://github.com/ai-dynamo/kvcr/blob/7ce17c358b867bd89f40a4a42bee9c76839506d0/docs/dev-guide.md).

### Native hugetlbfs

Have the cluster administrator reserve huge pages on eligible nodes. Add a
`HugePages-2Mi` emptyDir mounted **only in `kvcr-services`**, with matching
`hugepages-2Mi` requests and limits. Kubernetes creates the mount; no `hostPath`,
privileged mount command, or permission-setting init container is needed.
See [Kubernetes HugePages configuration](https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/).

Merge these additions into the worker's `podTemplate.spec` in
`agg-memory-service.yaml`; retain its existing containers, resources, and volumes:

```yaml
containers:
- name: kvcr-services
resources:
requests:
hugepages-2Mi: 3Gi
limits:
hugepages-2Mi: 3Gi
volumeMounts:
- name: kvcr-hugepages
mountPath: /hugepages
volumes:
- name: kvcr-hugepages
emptyDir:
medium: HugePages-2Mi
```

This illustrates a 2 GiB pool with headroom: budget for the additional 100 MiB
journal per Guard, page-size rounding, and recovery snapshots. Adjust the
reservation for larger pools. For 1 GiB pages, change both resource keys to
`hugepages-1Gi` and the volume medium to `HugePages-1Gi`.

The mount path is arbitrary; no mount-path CLI option is needed. For non-root
containers, use a suitable Pod `fsGroup` so Guard can write the volume, and
retain the same UID in Guard and vLLM for socket access. The engine needs neither
this mount nor a hugepage reservation for the pool received by FD.

Keep the existing `/dev/shm/kvcr` volume, `--pool-dir`, and ordinary memory
budget for runtime fallback. A Pod whose hugepage request cannot be scheduled
stays Pending; KVCR's fallback only applies after it starts. To skip native
HugeTLB, add `--disable-hugepages` to the service command in `services.sh`.
That flag does not disable THP.

### Transparent HugePages

Retain the default memory-backed emptyDir at `/dev/shm/kvcr`; no extra volume
or `hugepages-*` resource is needed. THP uses the ordinary memory budget and
depends on the host's kernel and tmpfs/shmem policy. Have the cluster
administrator configure that policy at node level. THP is best-effort and
can fall back to base pages.

## Verify Guard recovery

Use the memory-service variant and the opt-in live-cluster test. The test adds
Expand All @@ -134,4 +264,5 @@ the standard `test_output` directory). It verifies that the workers are on
separate hosts, one source engine restarts while its KVCR sidecar remains up,
the Guard serves the preserved cache to the other engine, the response matches,
the KVCR transfer metrics increase, and the selected active HCA carries the
transfer.
transfer. With `DYNAMO_UCX_NET_DEVICES=all`, each measured container must expose
exactly one active RDMA port; its counters are read independently.
103 changes: 30 additions & 73 deletions examples/backends/vllm/deploy/kvcr/agg-memory-service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

apiVersion: nvidia.com/v1beta1
Expand Down Expand Up @@ -42,16 +42,11 @@ spec:
- args:
- |
set -euo pipefail
for _ in $(seq 1 60); do
if python3 -c "import socket; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close()" >/dev/null 2>&1; then
break
fi
sleep 1
done
if ! python3 -c "import socket; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close()"; then
echo "KVCR memory service did not become ready" >&2
exit 1
source /opt/kvcr-startup/launch_utils.sh
if [ -f /opt/kvcr-startup/kvcr.whl ]; then
export PYTHONPATH="/opt/kvcr-startup/kvcr.whl${PYTHONPATH:+:$PYTHONPATH}"
fi
wait_for_ready http://127.0.0.1:9092/ready "${KVCR_READY_TIMEOUT_SECONDS}"

kv_transfer_config=$(
python3 - <<'PY'
Expand Down Expand Up @@ -152,50 +147,23 @@ spec:
add:
- IPC_LOCK
volumeMounts:
- mountPath: /opt/kvcr-startup
name: kvcr-startup
readOnly: true
- mountPath: /run/kvcr
name: kvcr-socket
- mountPath: /dev/shm/kvcr
name: kvcr-memory
workingDir: /workspace/examples/backends/vllm
- args:
- |
set -euo pipefail
state_namespace=$DYN_NAMESPACE
if [ -n "${DYN_NAMESPACE_WORKER_SUFFIX:-}" ]; then
state_namespace="${state_namespace}-${DYN_NAMESPACE_WORKER_SUFFIX}"
fi

kvcr_pid=
state_agent_pid=
cleanup() {
[ -z "$kvcr_pid" ] || kill -TERM "$kvcr_pid" 2>/dev/null || true
[ -z "$state_agent_pid" ] || kill -TERM "$state_agent_pid" 2>/dev/null || true
[ -z "$kvcr_pid" ] || wait "$kvcr_pid" 2>/dev/null || true
[ -z "$state_agent_pid" ] || wait "$state_agent_pid" 2>/dev/null || true
}
trap cleanup EXIT INT TERM

python3 -m kvcr.kvcr_service \
--socket-path /run/kvcr/memory.sock \
--pool-dir /dev/shm/kvcr \
--guard-count 1 \
--pool-sizes-gb 2 \
--compatibility-digest "${DYNAMO_KVCR_COMPATIBILITY_DIGEST}" &
kvcr_pid=$!
if [ "$POD_INDEX" = "0" ]; then
env DYN_NAMESPACE="$state_namespace" \
DYN_HEALTH_CHECK_ENABLED=false \
python3 -m dynamo.kv_state_agent \
--max-slots "$KVCR_CACHE_SLOT_COUNT" &
state_agent_pid=$!
wait -n "$kvcr_pid" "$state_agent_pid"
else
wait "$kvcr_pid"
fi
- /opt/kvcr-startup/services.sh
command:
- /bin/bash
- -c
env:
- name: DYNAMO_KVCR_COMPATIBILITY_DIGEST
value: "${DYNAMO_KVCR_COMPATIBILITY_DIGEST}"
- name: KVCR_POOL_SIZE_GIB
value: "${KVCR_POOL_SIZE_GIB}"
- name: KVCR_GUARD_STARTUP_TIMEOUT_SECONDS
value: "${KVCR_GUARD_STARTUP_TIMEOUT_SECONDS}"
- name: CONTAINER_NAME
value: kvcr-services
- name: DYN_COMPONENT
Expand Down Expand Up @@ -237,61 +205,50 @@ spec:
- name: UCX_NET_DEVICES
value: ${DYNAMO_UCX_NET_DEVICES}
image: ${DYNAMO_VLLM_IMAGE}
livenessProbe:
exec:
command:
- python3
- -c
- import os,socket,urllib.request; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close(); os.environ['POD_INDEX'] != '0' or urllib.request.urlopen('http://127.0.0.1:9091/live', timeout=1).read()
failureThreshold: 3
periodSeconds: 5
timeoutSeconds: 4
name: kvcr-services
# Readiness observes failures without triggering an unsafe sidecar-only
# restart. A deployment controller must coordinate Pod/group replacement.
readinessProbe:
exec:
command:
- python3
- -c
- import os,socket,urllib.request; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close(); os.environ['POD_INDEX'] != '0' or urllib.request.urlopen('http://127.0.0.1:9091/live', timeout=1).read()
httpGet:
path: /ready
port: 9092
failureThreshold: 3
periodSeconds: 5
timeoutSeconds: 4
resources:
limits:
${DYNAMO_RDMA_RESOURCE}: "1"
memory: 4Gi
memory: ${KVCR_MEMORY_LIMIT_GIB}Gi
requests:
${DYNAMO_RDMA_RESOURCE}: "1"
cpu: "1"
memory: 3Gi
memory: ${KVCR_MEMORY_REQUEST_GIB}Gi
securityContext:
capabilities:
add:
- IPC_LOCK
startupProbe:
exec:
command:
- python3
- -c
- import os,socket,urllib.request; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect('/run/kvcr/memory.sock'); s.close(); os.environ['POD_INDEX'] != '0' or urllib.request.urlopen('http://127.0.0.1:9091/live', timeout=1).read()
failureThreshold: 60
periodSeconds: 2
timeoutSeconds: 4
volumeMounts:
- mountPath: /opt/kvcr-startup
name: kvcr-startup
readOnly: true
- mountPath: /run/kvcr
name: kvcr-socket
- mountPath: /dev/shm/kvcr
name: kvcr-memory
terminationGracePeriodSeconds: ${KVCR_TERMINATION_GRACE_PERIOD_SECONDS}
tolerations:
- effect: NoSchedule
key: nvidia.com/gpu
operator: Exists
volumes:
- configMap:
name: ${KVCR_STARTUP_CONFIG_MAP}
name: kvcr-startup
- emptyDir: {}
name: kvcr-socket
- emptyDir:
medium: Memory
sizeLimit: 3Gi
sizeLimit: ${KVCR_SHM_SIZE_GIB}Gi
name: kvcr-memory
replicas: 2
type: worker
Loading
Loading