Repository navigation
[Bugfix] Release cached cuBLAS workspaces after daemon warmup - #63451
Merged
Isotr0py merged 5 commits intoOct 25, 2026
Merged
Conversation
Release PyTorch BLAS workspace caches before empty_cache so warmup scratch does not remain resident in the weight cache daemon. Add a regression using the existing Qwen model and document the cleanup behavior. Co-authored-by: Codex Signed-off-by: lichongmin <chongminli.cn@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
Contributor
|
Documentation preview: https://vllm--63451.org.readthedocs.build/en/63451/ |
tryryry
changed the title
Member
|
/ci run |
|
✅ @tryryry, CI is now available for this PR.
|
Isotr0py
enabled auto-merge (squash)
October 24, 2026 06:50
|
✅ Triggered Buildkite CI #97024 for commit ℹ️ This PR is 5 commits behind upstream |
Member
|
/ci retry |
|
✅ Triggered Buildkite CI #97047 for commit |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Resolve the preload documentation conflict by retaining both the BLAS workspace cleanup description and D2D warmup cache seeding instructions. Co-authored-by: Codex Signed-off-by: lichongmin <chongminli.cn@gmail.com>
auto-merge was automatically disabled
October 25, 2026 14:16
Head branch was pushed to by a user without write access
Isotr0py
enabled auto-merge (squash)
October 25, 2026 14:33
Member
|
/ci retry |
|
✅ Triggered Buildkite CI #97138 for commit |
gaby
added a commit
to gaby/vllm
that referenced
this pull request
Oct 25, 2026
…s usage Restructure docs/features/preload.md around operating the weight cache daemon and add the missing deployment documentation. - Add Docker, Docker Compose and Kubernetes deployment: running the daemon and the engine as separate containers, the IPC and PID namespace requirements, the shared volume for the sockets and the FlashInfer autotune cache, and the probes. - Document the /health endpoint, the loader and daemon configuration tables, the start order, GPU memory accounting and troubleshooting. - Integrate the FlashInfer autotune preloading from vllm-project#60085. Readiness now includes tuning, and the engine finds the tuned configs through the autotune cache location. - Integrate the daemon-to-daemon warmup cache seeding from vllm-project#63088: the seeding flags, the start order, the Docker ports, and what a mirror logs when it falls back to local tuning. - Describe the daemon memory handling from vllm-project#62583 and vllm-project#63451: the engine counts only the exported weights, and the daemon releases its process groups and cached cuBLAS workspaces after loading and tuning. - Update the fingerprint description for the RoPE parameters, expert parallelism, KV cache dtype settings and dense data parallel keying, and document that EPLB is not supported. - Correct statements that did not match the code: pipeline parallelism is supported, copy mode releases the daemon's cache after the first handoff, and the quantization support list now follows the code. - Add pointers from the Docker deployment guide, the faster-startup list and the CLI reference. Co-authored-by: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZTcfM8JsFUMWPHCsoLsuE Signed-off-by: Juan Calderon-Perez <835733+gaby@users.noreply.github.com>
6 tasks done
Micador
pushed a commit
to Micador/vllm
that referenced
this pull request
Oct 27, 2026
…roject#63451) Signed-off-by: lichongmin <chongminli.cn@gmail.com> Co-authored-by: vllm-agent <claw@inferact.ai> Signed-off-by: Christopher Mackenzie <christopher.mackenzie.software@gmail.com>
Micador
pushed a commit
to Micador/vllm
that referenced
this pull request
Oct 27, 2026
…roject#63451) Signed-off-by: lichongmin <chongminli.cn@gmail.com> Co-authored-by: vllm-agent <claw@inferact.ai> Signed-off-by: Christopher Mackenzie <christopher.mackenzie.software@gmail.com>
Micador
pushed a commit
to Micador/vllm
that referenced
this pull request
Oct 27, 2026
…roject#63451) Signed-off-by: lichongmin <chongminli.cn@gmail.com> Co-authored-by: vllm-agent <claw@inferact.ai> Signed-off-by: Christopher Mackenzie <christopher.mackenzie.software@gmail.com>
gaby
added a commit
to gaby/vllm
that referenced
this pull request
Oct 30, 2026
…s usage Restructure docs/features/preload.md around operating the weight cache daemon and add the missing deployment documentation. - Add Docker, Docker Compose and Kubernetes deployment: running the daemon and the engine as separate containers, the IPC and PID namespace requirements, the shared volume for the sockets and the FlashInfer autotune cache, and the probes. - Document the /health endpoint, the loader and daemon configuration tables, the start order, GPU memory accounting and troubleshooting. - Integrate the FlashInfer autotune preloading from vllm-project#60085. Readiness now includes tuning, and the engine finds the tuned configs through the autotune cache location. - Integrate the daemon-to-daemon warmup cache seeding from vllm-project#63088: the seeding flags, the start order, the Docker ports, and what a mirror logs when it falls back to local tuning. - Describe the daemon memory handling from vllm-project#62583 and vllm-project#63451: the engine counts only the exported weights, and the daemon releases its process groups and cached cuBLAS workspaces after loading and tuning. - Describe how `vllm preload` stops every rank when one exits (vllm-project#63692), what a zero-copy engine does when its daemon exits (vllm-project#62734), and the kernel backends that vllm-project#63506 added to the fingerprint. - Update the fingerprint description for the RoPE parameters, expert parallelism, KV cache dtype settings and dense data parallel keying, and document that EPLB is not supported. - Correct statements that did not match the code: pipeline parallelism is supported, copy mode releases the daemon's cache after the first handoff, and the quantization support list now follows the code. - Add pointers from the Docker deployment guide, the faster-startup list and the CLI reference. Co-authored-by: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZTcfM8JsFUMWPHCsoLsuE Signed-off-by: Juan Calderon-Perez <835733+gaby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Release cached cuBLAS/cuBLASLt workspaces after weight-cache daemon warmup, before accepting engine connections.
Follow-up to #62583: communication teardown leaves PyTorch-owned BLAS workspaces resident even though the daemon no longer performs inference.
Claims
Validation
Memory comparison
nvidia/Qwen3-30B-A3B-NVFP4, B300, TP=4,gpu_memory_utilization=0.1,max_model_len=8192,max_num_batched_tokens=16384,max_num_seqs=1024.“Before” includes #62583. Memory figures are per GPU / daemon process.
Workspace allocations decreased by 32 MiB per daemon. NVML savings depend on allocator fragmentation; KV capacity and exported weights remained unchanged.
These development measurements used different B300 GPU sets and base revisions (
c569393ed2/404c2b510e, with unchanged daemon code). Externally disturbed baseline runs were excluded from the KV comparison. Results are individual observations, not repeated-run averages.Final implementation validation
Qwen/Qwen3.5-0.8B, B300, TP=4, compilation enabled, utilization=0.06, maximum length=8192, batched tokens=2048, maximum sequences=32.All five runs—cold/warm disk, cold IPC, warm IPC with the existing daemon, and warm IPC after daemon restart—completed successfully with identical 32-token outputs. Exported weight sizes remained unchanged. This is targeted correctness validation, not a broad accuracy evaluation; the disk/IPC KV difference is not attributed to this fix.
Tests
CUDA_VISIBLE_DEVICES=5 OMP_NUM_THREADS=1 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ .venv/bin/python -m pytest \ tests/model_executor/model_loader/test_weight_cache.py::test_daemon_releases_cublas_workspaces \ 'tests/model_executor/model_loader/test_weight_cache.py::test_ipc_cache_cold_start_and_warm_restart[qwen3.5]' \ -v -s2 passed. The development reproducer failed without the fix with
daemon retained BLAS workspaces. A separate two-stream probe released 16 MiB with GPU work pending at cleanup entry and preserved correct outputs.All checks passed.
Details
PyTorch caches BLAS workspaces independently of model tensors, so garbage collection and
empty_cache()cannot release them while those caches retain ownership.This change clears the caches after communication teardown and before the existing
empty_cache(). Cleanup is CUDA-only and usesgetattrto skip gracefully when the private PyTorch API is unavailable. No additional explicit device synchronization is introduced.Duplicate-work searches found no equivalent open fix. Related #52553 addresses workspace address stability during graph execution; this PR releases caches after daemon warmup has finished.
AI assistance was used for implementation, investigation, validation, and this description.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.