[Docs] Rewrite the preload (`ipc_cache`) guide and add Docker/Kubernetes usage by gaby · Pull Request #60006 · vllm-project/vllm

gaby · 2026-10-05T03:10:50Z

Overview

Rewrites docs/features/preload.md (the vllm preload / --load-format ipc_cache weight cache daemon) around the operator workflow and adds the missing Docker and Kubernetes guidance, plus pointers from the Docker deployment guide, the faster-startup list and the CLI reference. It also folds in the preload changes that landed on main while this PR was open: FlashInfer autotune preloading (#60085), warmup cache seeding between daemons (#63088) and the daemon's GPU memory handling (#62583).

Claims

Documents how to run the daemon and the engine in separate containers: a Docker run pair, a Docker Compose file gated on the daemon's health check, a scoped-namespace alternative (daemon started with --ipc=shareable, engine joining its IPC and PID namespaces), and the reason behind each requirement (--ipc=host, --pid=host, shared socket and autotune cache volume, same GPUs/UID/image tag, --entrypoint vllm).
Documents the Kubernetes pattern: GPU device plugins allocate a GPU to one container, so the daemon and a vllm serve restart loop run in one container with startup/liveness probes on the daemon's health port and readiness on the engine.
Documents previously undocumented behaviour: the /health readiness endpoint (--weight-cache-health-port/-host), the fallback: false wait, GPU memory accounting of daemon-held weights, the cold-start ordering pitfall, the daemon flags, and a troubleshooting table keyed on the actual error strings.
Keeps the guide in step with main: where the FlashInfer autotune cache must live in Docker, how to seed a new replica's autotune cache from a running one (start order, per-rank ports to publish, token), what the engine counts toward --gpu-memory-utilization, the full cache fingerprint, and that EPLB is not supported.
Corrects stale or wrong statements: pipeline parallelism was listed as rejected but the daemon plans PP ranks; copy mode was described as reusable per restart but the daemon releases its cache after the first handoff (the next restart falls back to disk only while fallback is enabled); the quantization support list now matches the supports_pre_processed_weights declarations (unquantized except MoE on the MoonEP all2all backend, fp8, ModelOpt NVFP4, MXFP4 except GPT-OSS).

Validation

Docs-only change (plus a one-word docstring fix in vllm/entrypoints/cli/preload.py), so no model evaluation applies. All checks below were re-run after the latest rebase onto main (94a29d3).

# markdownlint with the repo config: 0 errors
npx markdownlint-cli2@0.21.0 --config tools/pre_commit/.markdownlint.yaml \
    docs/features/preload.md docs/cli/README.md docs/deployment/docker.md docs/configuration/optimization.md

# ruff on the touched Python file: all checks passed, already formatted
uvx ruff@0.14.0 check vllm/entrypoints/cli/preload.py && uvx ruff@0.14.0 format --check vllm/entrypoints/cli/preload.py

# full docs build in strict mode: exit 0, no warnings
uv venv --python 3.12 .docs-venv && uv pip install --python .docs-venv/bin/python -r requirements/docs.txt
DISABLE_MKDOCS_2_WARNING=true .docs-venv/bin/mkdocs build --strict

The embedded Compose and Kubernetes YAML parse, every bash block passes bash -n, and every relative link and anchor resolves. Every statement about behaviour was checked against weight_cache/{daemon,ipc_loader,protocol,seed,warmup_cache}.py, cli/preload.py, warmup/{kernel_warmup,flashinfer_autotune_cache}.py, v1/worker/{gpu_worker,utils}.py and the supports_pre_processed_weights declarations of the quantization methods. As the contributing guide asks, three fresh agents limited to the checkout answered a user question each (seeding a new replica in Docker, the free-memory startup error, the two-container setup) from the guide, and the wording they found confusing was fixed.

Not validated: the Docker, Kubernetes and seeding examples were not executed on GPU hardware. The namespace requirements follow from the mechanism (legacy CUDA IPC handles are bound to the exporting process's PID namespace, see openucx/ucx#11978 and this NVIDIA forum thread; PyTorch's reference counters for shared CUDA tensors are shared-memory files, so /dev/shm must be shared). The seeding behaviour (listener start order, per-rank ports, fallbacks) is read from seed.py and warmup_cache.py, not observed. A two-container run on a GPU host would confirm both before merge.

Details

docs/features/preload.md: restructured into requirements, quick start with the expected log lines, autotune preloading and seeding between daemons, how it works, memory accounting, cache modes, health endpoint, Docker, Kubernetes, loader/daemon configuration tables, multi-node/DP, speculative decoding, security, lifecycle, troubleshooting. Upstream's own paragraphs on autotune preloading, seeding, multi-node/DP and security are kept as written.
docs/deployment/docker.md: new "Keep model weights in GPU memory across engine restarts" section with the two-container example, linking to the full guide.
docs/configuration/optimization.md: preload added as a fourth faster-startup mechanism.
docs/cli/README.md and the preload.py module docstring: supported-parallelism wording updated; the CLI summary now lists the health, socket-dir, master-port and seeding flags and the autotune pass.

Review history: the three Codex findings on the first revision (MoonEP exception, copy mode with fallback: false, --ipc=shareable for the scoped-namespace variant) were verified against the code and Docker's documentation and addressed; the threads are resolved. Maintainer comments on docs/features/preload.md are resolved: the overlap with #60085 was settled by integrating its text, and the text added here is not hard-wrapped.

Duplicate-work check (2026-10-25): no open upstream PR documents preload for Docker/Kubernetes; searching open PRs for preload and ipc_cache returns no docs PR other than this one; the matching PRs are fixes and features for the daemon itself. Open PRs that change facts this guide states and will need a one-line reconcile if they land first: #59648 (checkpoint fingerprint from sampled tensor bytes instead of safetensors headers), #63506 (MoE and linear kernel backends added to the fingerprint), and #63488 and #61841, which also edit docs/features/preload.md (engine memory budget after an IPC fallback; sharing FlashInfer autotune artifacts between daemons).

AI assistance (Claude Code) was used to draft this PR; the submitting human has to review every line and run the Docker example before upstreaming.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

mergify · 2026-10-05T03:11:26Z

Documentation preview: https://vllm--60006.org.readthedocs.build/en/60006/

mergify · 2026-10-10T16:24:52Z