[gaby]

Overview

Rewrites docs/features/preload.md (the vllm preload / --load-format ipc_cache weight cache daemon) around the operator workflow and adds the missing Docker and Kubernetes guidance, plus pointers from the Docker deployment guide, the faster-startup list and the CLI reference. It also folds in the preload changes that landed on main while this PR was open: FlashInfer autotune preloading (vllm-project#60085), warmup cache seeding between daemons (vllm-project#63088), the daemon's GPU memory handling (vllm-project#62583 and vllm-project#63451), the launcher shutdown when a rank exits (vllm-project#63692), the engine's warning when its daemon exits (vllm-project#62734) and the kernel backends in the cache fingerprint (vllm-project#63506).

Claims

• Documents how to run the daemon and the engine in separate containers: a Docker run pair, a Docker Compose file gated on the daemon's health check, a scoped-namespace alternative (daemon started with --ipc=shareable, engine joining its IPC and PID namespaces), and the reason behind each requirement (--ipc=host, --pid=host, shared socket and autotune cache volume, same GPUs/UID/image tag, --entrypoint vllm).
• Documents the Kubernetes pattern: GPU device plugins allocate a GPU to one container, so the daemon and a vllm serve restart loop run in one container with startup/liveness probes on the daemon's health port and readiness on the engine.
• Documents previously undocumented behaviour: the /health readiness endpoint (--weight-cache-health-port/-host), the fallback: false wait, GPU memory accounting of daemon-held weights, the cold-start ordering pitfall, the daemon flags, and a troubleshooting table keyed on the actual error strings.
• Keeps the guide in step with main: where the FlashInfer autotune cache must live in Docker, how to seed a new replica's autotune cache from a running one (start order, per-rank ports to publish, token), what the engine counts toward --gpu-memory-utilization, the full cache fingerprint, what happens when a rank or a daemon exits (exit statuses, the engine keeps serving), and that EPLB is not supported.
• Corrects stale or wrong statements: pipeline parallelism was listed as rejected but the daemon plans PP ranks; copy mode was described as reusable per restart but the daemon releases its cache after the first handoff (the next restart falls back to disk only while fallback is enabled); the quantization support list now matches the supports_pre_processed_weights declarations (unquantized except MoE on the MoonEP all2all backend, fp8, ModelOpt NVFP4, MXFP4 except GPT-OSS).

Validation

Docs-only change (plus a one-word docstring fix in vllm/entrypoints/cli/preload.py), so no model evaluation applies. All checks below were re-run after the latest rebase onto main (d2d550c).

# markdownlint with the repo config: 0 errors
npx markdownlint-cli2@0.21.0 --config tools/pre_commit/.markdownlint.yaml \
    docs/features/preload.md docs/cli/README.md docs/deployment/docker.md docs/configuration/optimization.md

# ruff on the touched Python file: all checks passed, already formatted
uvx ruff@0.14.0 check vllm/entrypoints/cli/preload.py && uvx ruff@0.14.0 format --check vllm/entrypoints/cli/preload.py

# full docs build in strict mode: exit 0, no warnings
uv venv --python 3.12 .docs-venv && uv pip install --python .docs-venv/bin/python -r requirements/docs.txt
DISABLE_MKDOCS_2_WARNING=true .docs-venv/bin/mkdocs build --strict

The embedded Compose and Kubernetes YAML parse, every bash block passes bash -n, and every relative link and anchor resolves. Every statement about behaviour was checked against weight_cache/{daemon,ipc_loader,protocol,seed,warmup_cache}.py, cli/preload.py, warmup/{kernel_warmup,flashinfer_autotune_cache}.py, v1/worker/{gpu_worker,utils}.py and the supports_pre_processed_weights declarations of the quantization methods. As the contributing guide asks, five fresh agents limited to the checkout answered a user question each from the guide (seeding a new replica in Docker, the free-memory startup error, the two-container setup, the daemon-exit warning, a launcher exit status), and the wording they found confusing was fixed.

Not validated: the Docker, Kubernetes and seeding examples were not executed on GPU hardware. The namespace requirements follow from the mechanism (legacy CUDA IPC handles are bound to the exporting process's PID namespace, see openucx/ucx#11978 and this NVIDIA forum thread; PyTorch's reference counters for shared CUDA tensors are shared-memory files, so /dev/shm must be shared). The seeding behaviour (listener start order, per-rank ports, fallbacks) is read from seed.py and warmup_cache.py, and the exit handling (launcher shutdown, the engine's warning) from cli/preload.py and ipc_loader.py; none of it was observed. A two-container run on a GPU host would confirm it before merge.

Details

• docs/features/preload.md: restructured into requirements, quick start with the expected log lines, autotune preloading and seeding between daemons, how it works, memory accounting, cache modes, health endpoint, Docker, Kubernetes, loader/daemon configuration tables, multi-node/DP, speculative decoding, security, lifecycle, troubleshooting. Upstream's own paragraphs on autotune preloading, seeding, multi-node/DP and security are kept as written.
• docs/deployment/docker.md: new "Keep model weights in GPU memory across engine restarts" section with the two-container example, linking to the full guide.
• docs/configuration/optimization.md: preload added as a fourth faster-startup mechanism.
• docs/cli/README.md and the preload.py module docstring: supported-parallelism wording updated; the CLI summary now lists the health, socket-dir, master-port and seeding flags and the autotune pass.

Review history: the three Codex findings on the first revision (MoonEP exception, copy mode with fallback: false, --ipc=shareable for the scoped-namespace variant) were verified against the code and Docker's documentation and addressed; the threads are resolved. Maintainer comments on docs/features/preload.md are resolved: the overlap with vllm-project#60085 was settled by integrating its text, and the text added here is not hard-wrapped.

Duplicate-work check (2026-10-30): no open upstream PR documents preload for Docker/Kubernetes; searching open PRs for preload and ipc_cache returns no docs PR other than this one. Open PRs that also edit docs/features/preload.md and will need a reconcile if they land first: vllm-project#64641 (weights mirror for daemon-to-daemon seeding), vllm-project#63488 (engine memory budget after an IPC fallback), and vllm-project#63672 and vllm-project#64182 (new quantization schemes for the weight cache, which widen the requirement stated here; vllm-project#63673, vllm-project#63754, vllm-project#63976, vllm-project#64119 and vllm-project#64738 are further PRs in the same series). vllm-project#59648 changes the checkpoint fingerprint to sampled tensor bytes.

AI assistance (Claude Code) was used to draft this PR; the submitting human has to review every line and run the Docker example before upstreaming.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MZTcfM8JsFUMWPHCsoLsuE

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-05T01:56:39.901210Z	270b1ab	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 270b1aba78

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".