Kimi K3: --colocate-memory-peak-device — overlap the trainer/rollout handoff on the GPU

[yueming-yuan]

Stacked on the kimi-k3 branch (#1825). One squashed commit. The persistent CUDA-IPC LoRA transfer buffer was split out to #1923.

What

--colocate-memory-peak-device {cpu,gpu} (default cpu = existing behavior).

In colocated RL, each trainer<->rollout handoff briefly holds two host copies:
the engine's weight mirror (built when the engine releases its weights under
--enable-weights-cpu-backup) and the trainer's own TMS backup. On hosts where
the two together exceed the memory budget (GB300: 4x140 GB mirror + trainer
backup vs the job cgroup), gpu mode reorders both handoffs so the overlap
lives on the GPU instead and the host copies never coexist:

• rollout->train: drop the engine KV cache + CUDA graphs (pure GPU memory),
  wake the trainer while the engine weights are still resident, then release
  the weights — their mirror replaces the trainer's just-freed backup.
• train->rollout: the trainer returns its allocator caches and the no-backup
  LoRA grad-buffer region to the device, the engine resumes its weights, and
  only then does the trainer sleep into the space the mirror vacated.
• init: the engine weights stay resident through the trainer checkpoint load
  (their mirror cannot coexist with the load's footprint).

RolloutManager gains split offload_kv/offload_weights/onload_weights;
the trainer actor gains an idempotent sleep/wake_up pair plus
offload_grad_buffer; the K3 launcher hardcodes gpu.

Validation

16 nodes x 4 GPU (GB300), full Kimi-K3 LoRA RL (together with #1923): both
handoffs stable cycle over cycle, host anon+shmem peaking ~770 GB of the
898 GB cgroup with a single large host copy at any time; eval/aime
0.367 -> 0.467 by the second eval. This is the configuration inside the
released radixark/miles:kimi-k3 image, which the Reproduce section of
#1825 describes.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.