[Doc] Sleep mode: what stays on the GPU

[aoshen02]

Purpose

Adds a "What stays on the GPU" section to the sleep mode docs. It covers:

• the sleep-mode tags and what sleep does with each;
• what cannot be released without restarting the process, and how to reduce it;
• the NCCL_GRAPH_REGISTER default.

The table describes behavior after the related PRs land: #59156 (workspace), #59158 and #59162 (runtime), #59160 (graph pool and NCCL_GRAPH_REGISTER). Keep this PR in draft until they are merged.

Test Plan

Docs only. Numbers come from DeepSeek-V4-Flash runs on 4x GB300.

Test Result

Docs only. Below are the results with all four PRs applied together (#59156, #59158, #59160, #59162) on GB300. Every run is the same image with and without the PRs. GSM8K is 1319 questions, 5-shot, temperature 0.

Memory per GPU while asleep (level 1):

Model	Setup	Unpatched	All PRs	Saved
DeepSeek-V4-Flash	TP4 + EP, MTP, 1 node	14029 MiB	5161 MiB	8868 MiB (63%)
DeepSeek-V4-Flash	DP4 + EP4, auto MoE, fp8 KV	14193 MiB	4975 MiB	9218 MiB (65%)
DeepSeek-V4-Flash	DP4 + EP4, DeepGEMM MoE, fp8 KV	14528 MiB	4897 MiB	9631 MiB (66%)
DeepSeek-V4.1	TP8, 2 nodes (mp)	21035 MiB	5819 MiB	15216 MiB (72%)
GLM-5.3	TP8, 2 nodes (mp), MTP	9449 MiB	7405 MiB	2044 MiB (22%)

GSM8K accuracy (unpatched → all PRs):

State	V4-Flash	V4.1	GLM-5.3
Awake	0.948 → 0.946	0.892 → 0.904	0.914 → 0.917
Level-1 sleep, full wake	0.943 → 0.954	0.899 → 0.898	0.914 → 0.910
Level-1 sleep, selective wake	0.948 → 0.949	0.902 → 0.905	0.918 → 0.912
Level-2 sleep, reload_weights	0.017 → 0.021 *	0.000 → 0.000 *	0.911 → 0.919

* reload_weights after a level-2 sleep is already broken for DeepSeek-V4 models on unpatched vLLM. layerwise.py logs "Failed to load weights" for the V4 attention, rotary and n-gram modules. This is unrelated to these PRs.

• What remains (V4-Flash DP4, about 5.0 GiB per GPU): mostly driver-side state that no allocator tag can release. CUDA graph executables take 1944 MiB and other memory charged during capture takes 1142 MiB; the rest is the CUDA context (390 MiB), local memory (480 MiB), kernel modules (about 300 MiB), NCCL buffers (108 MiB) and cuBLAS (68 MiB). Releasing the graph executables would mean destroying and recapturing the graphs on wake.
• Serving performance while awake is unchanged (DeepSeek-V4-Flash TP4, vllm bench serve, random 1024/256 at concurrency 1–256 and 4096/512 at concurrency 32).
  • Throughput and TTFT/TPOT match unpatched within noise, both on a fresh server and after a sleep/wake cycle.
  • Throughput differs by up to ~45% between launches of the same code, because the FlashInfer MoE autotuner picks different tactics on each launch. Matched comparisons need 2–3 launches per arm.
  • On GB300 the allreduce is FlashInfer MNNVL, so [Feature] Release the CUDA graph pool on sleep #59160's custom-allreduce copy path is not exercised.
• RL-style soak, 100 iterations per model:
  • Qwen3-235B-A22B FP8 TP4+EP and DeepSeek-V4-Flash TP4.
  • Each iteration runs a rollout of 64–256 concurrent requests, then a level-1 sleep; on about a third to a half of the iterations, another process fills ~270 GiB/GPU of the freed VRAM during the sleep. Wakes alternate full and selective, and every 25th iteration is level 2 + reload_weights (Qwen).
  • Probes match iteration 0 bit-exactly. GPU memory, host RSS and latency show no drift. Qwen3 also has a one-time +14 MiB after the first level-2 reload. 16,345/16,345 requests succeeded.
• Sleep and wake latencies are unchanged, and there were no errors.
• Unit tests pass for all PRs combined (test_cumem.py, test_sleep_mode_backend.py, test_workspace.py: 37/37).

---

This PR contains AI-assisted text (Claude Code); the commit carries a Co-authored-by trailer.

🤖 Generated with Claude Code

[mergify]

Documentation preview: https://vllm--59165.org.readthedocs.build/en/59165/