build(vllm): upgrade to 0.25.1 and make CSCS images reproducible

[Alvorecer721]

Summary

• upgrade the Apertus generation stack from vLLM 0.20.0 to 0.25.1, including the renderer, fused-MoE/refit, RayExecutorV2, and environment-propagation API moves
• make worker environments dependency-aware and fail closed on interrupted or stale builds, while avoiding redundant rebuilds across checkout aliases
• add a resumable CSCS Podman -> persistent registry -> Enroot/SquashFS release pipeline with bounded actor layers, pinned inputs, clean-tree source contexts, and baked-image probes
• preserve FP8 tensor storage across refits and backport vLLM's post-0.25.1 invalid-MNNVL-workspace fix for compiled TP engines on GH200
• document the resulting image/cache lifecycle, failure recovery, and CSCS launch constraints

Upstream provenance

• the vLLM 0.25.1 integration tracks NVIDIA-NeMo/RL#3280, specifically cf766d5f38eca80423df904a6cd68d09eb1f893c; the checkpoint-engine MoE refit hunks are intentionally omitted because that loader does not exist in this v0.7.0-era tree
• the FP8 in-place refit correction and its regression coverage are backported from that same NVIDIA commit
• the GH200 TP2 failure fix backports vllm-project/vllm#49043, 81962bb6995eaebd1e49998c2a91c9e01e24da27, which was merged after v0.25.1

Validation

• compiled vLLM TP2 generation/refit unit test on two GH200s (test_vllm_policy_generation_async[2-1]), including CUDA graph capture and TRT-LLM workspace fallback
• real Apertus-8B Async-GRPO on four GH200s: one TP2 vLLM engine + one TP2 Megatron policy, two complete training steps, repeated refit, exit 0 (Slurm allocation 3076785)
• observed final-step group throughput: 2,442.66 tokens/s end-to-end; 3,006.32 training tokens/s; 2,906.17 generation tokens/s
• focused regression suite: 24 passed (test_vllm_patches.py, test_venvs.py)
• Ruff check and format check on changed Python files
• branch is 0 commits behind current main; full diff passes git diff --check
• clean final image build and SquashFS export from code-bearing revision 7c68228e4f09 (Slurm job 3077164, 2:03:51)
• baked-image TP2 Async-GRPO certification on four GH200s: compiled CUDA graphs, invalid-MNNVL guard/fallback, successful refit, two complete training steps, exit 0 (Slurm job 3077837, 4:48)
• baked-image compiled TP4 generation on one quad-GH200 node: invalid MNNVL workspace rejected, TRT-LLM FlashInfer fallback initialized, CUDA graphs captured, 4 x 64 tokens generated, exit 0 (Slurm job 3078831, 3:27)

The single-node TP4 result is the maximum relevant tensor-parallel certification for Clariden's quad-GH200 nodes. Two-node TP8 remains optional, non-blocking scalability evidence; the current Apertus-8B Async-GRPO placement uses TP2 inference plus TP2 training on one node and does not require TP8.

CSCS image notes

The certified SquashFS is /iopsstor/scratch/cscs/xyixuan/ce-images/nemo-rl/nemo-rl-apertus-vllm-0.25.1-7c68228e4f09-38c6b702948c.sqsh (48,752,754,688 bytes), SHA-256 d50f39e45f6104d13e12b9323dbe28cc91b0f13e3a250d029ce6cc7e7646742a, OCI image ID 18c36e6a31df01fc0370f65f9446c373d741198abf821b35a89be8214c11e79e. The later PR commits add reusable certification probes and record their evidence; they do not change image contents.

The persistent OCI cache lives under /iopsstor/scratch/cscs/xyixuan/podman-cache/nemo-rl; the local registry process is allocation-scoped, while its blobs and exported SquashFS survive node changes. Release builds reject dirty source/submodules and encode the source commit, recursive submodule SHAs, base manifest, and build arguments in the artifact identity.

The node-spanning TP8 probe deliberately routes collectives through PyTorch NCCL over the CSCS AWS OFI plugin because vLLM 0.25.1's direct PyNccl and Hopper symmetric-memory paths are incompatible with that topology. On the tested single-node TP2 and TP4 topologies, vLLM rejects the unusable FlashInfer MNNVL multicast workspace and safely initializes the compiled TRT-LLM FlashInfer all-reduce backend.