[RL][Sleep] Retain frozen weights across level-2 sleep

[aoshen02]

Purpose

Level-2 sleep discards weights, including frozen parameters omitted by the
trainer. Preserve selected parameters so they remain usable after wake-up
without retransmission.

sleep_preserve_parameter_names (CLI: --sleep-preserve-parameter-names) matches runtime parameter names
with globs (default: empty; unmatched patterns raise). Selected GPU parameters
are backed up to pageable CPU memory and restored in place on weights wake-up;
CPU-resident parameters need no copy. Backups are released after restoration.
Level-1 behavior is unchanged. The option lives in ModelConfig alongside the
other sleep settings and does not require a weight-transfer engine.

Two private Worker methods keep selection and backup state within the sleep/wake lifecycle. XPU inherits the implementation; CPU workers do not support sleep. Selection
covers the target model only. The trainer must omit selected weights, and
reload/post-processing must preserve their values and storage. Budget host
memory for backups; this option does not filter updates.

Validation

Focused checks

• Empty selection, CPU-resident parameters, unmatched-pattern rejection,
  in-place restoration, and backup release passed.
• Commit hooks passed, including Ruff and mypy; git diff --check passed.

Real GPU sleep/wake

On an NVIDIA GB200 using the real CuMem sleep backend and a small Llama model:

• Three level-2 cycles retained lm_head.weight. All 16,384 bytes and its GPU
  data_ptr remained exact after every wake, while a non-retained parameter was
  changed each cycle.
• Level-1 sleep preserved all 180,864 parameter bytes and addresses.
• With the default empty selection, no parameter was backed up and a complete
  reload restored all 180,864 bytes.

Commands run in the pinned serving container:

python3 /opt/test_frozen_gpu.py configured
python3 /opt/test_frozen_gpu.py empty

Both returned PASS. After moving the option to ModelConfig, the same GPU
tests were rerun with weight_transfer_config=None.

The integration results below precede the configuration relocation; the GPU
sleep/wake regression above exercises the current implementation.

Default VERL update path

An eight-rank run exercised VERL's standard non-layerwise update path. The
trainer omitted four frozen weight/scale tensors per rank. Across all ranks,
32 CPU-resident tensors totaling 202,758,032,400 bytes retained their complete
SHA-256 digest, data_ptr, shape, dtype, and device through level-2 sleep and
two updates. All eight trainable shards changed after an optimizer step, and
the subsequent synchronization did not mutate trainer state.

Generation succeeded before and after the update. On the same 30-question
held-out math set, pre/post accuracy was 8/30; all eight non-truncated responses
were correct in both runs. This is integration evidence, not an isolated model
accuracy claim.

Cross-host TP8

A two-host NVIDIA GB200 test ran TP8 with four ranks per host and default
FULL_AND_PIECEWISE CUDA Graphs. Across two real level-2 cycles, the same 32
CPU-resident tensors retained complete SHA-256 digests and addresses both before
and after restoring the other weights. All three generations produced identical
token IDs. There was no OOM and no CUDA Graph invalidation or recapture during
reload.

XPU execution remains untested.

Duplicate checks

Reviewed #50079 and open sleep/retention PRs. This preserves selected parameters
across level-2 sleep; #55206 releases cached pinned memory and #49789 preserves
post-load runtime state.

AI assistance was used. Human review of the latest revision remains pending.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--57891.org.readthedocs.build/en/57891/

[aoshen02]

/ci run

[github-actions]

✅ Triggered Buildkite CI #90329 for commit 943a2410a9c4.