async grpo native weight sync with vllm>=0.22.0

[AmineDiro]

What this implements

Wires AsyncGRPO's WeightTransferClient to vLLM's native RL weight-transfer API (the 4-phase protocol from the 2026-05-28 native RL APIs post
(https://vllm.ai/blog/2026-05-28-native-rl-apis)), replacing the previous 2-call flow.

Per weight sync, WeightTransferClient.send_weights() now drives the full lifecycle:

pause(mode="keep")
→ start_weight_update(is_checkpoint_format=True)   # workers run initialize_layerwise_reload
→ update_weights(update_info)                      # threaded; concurrent with the trainer NCCL broadcast
→ finish_weight_update()                           # workers run finalize_layerwise_reload
→ resume()

Reference implementations

• vLLM weight-transfer docs: https://docs.vllm.ai/en/latest/training/weight_transfer/
• vLLM RL examples (HTTP + NCCL): https://github.com/vllm-project/vllm/tree/main/examples/rl in particular rlhf_http_nccl.py (same 4-phase order an concurrent update_weights/broadcast) and rlhf_async_new_apis.py (pause(mode="keep")).

Validation

End-to-end on H100 (Qwen3-0.6B, GSM8K) at three scales, all completing with healthy GRPO metrics; server logs show paired start/finish bracketing every update_weights:

Scale	Layout	weight_sync_time
1 node	trainer×1 + vLLM TP=1	~0.18s
1 node	trainer FSDP2×4 + vLLM TP=4	~0.24s
2 nodes	trainer FSDP2×4 + vLLM DP=2×TP=4	~0.8s (cross-node)

Runtime note: vLLM ≥0.22 ships CUDA-13 wheels by default; on a CUDA-12.9 driver use the +cu129 vLLM release wheel + torch==2.11.0+cu128.

---

Note

Medium Risk
Changes the hot path that pushes trainer weights to the vLLM rollout server; mis-ordering or version skew would break training sync, though it follows vLLM’s documented RL API.

Overview
Updates Async GRPO weight sync to match vLLM ≥0.22.0’s native RL weight-transfer lifecycle instead of the older two-endpoint flow.

WeightTransferClient.send_weights() now calls start_weight_update (is_checkpoint_format: true) before weights, then the existing threaded update_weights + NCCL broadcast, then finish_weight_update. Checkpoint-format is no longer passed in the static weight_update_info built in AsyncGRPOTrainer. The minimum vLLM version for WeightTransferClient is raised from 0.17.1 to 0.22.0 (imports and error messages updated). Per-phase debug timing logs in send_weights are collapsed to a single total timing line.

^{Reviewed by Cursor Bugbot for commit e828e14. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e828e14. Configure here.}

[cursor]

Finish not called after sync errors

High Severity

send_weights now calls start_weight_update before NCCL and finish_weight_update only on the happy path. If trainer_send_weights, the threaded /update_weights call, or join raises or aborts, finish_weight_update is skipped while vLLM workers may still be in layerwise reload, breaking later rollouts or weight syncs until the server is restarted.

 

^{Reviewed by Cursor Bugbot for commit e828e14. Configure here.}

[AmineDiro]

mirrors upstream : https://github.com/vllm-project/vllm/tree/main/examples/rl

no example handles this failure mode

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[AmineDiro]

small question. @qgallouedec where should we bump the minimum vllm version for asyncgrpo specifically ?

[qgallouedec]

sorry just saw your question.

in the init, but this is already what you do, so I think it can be merged

[adithya-s-k]

@AmineDiro @qgallouedec can we get this merged as it will help out with #6018 as well
thank you