[qgallouedec]

trl vllm-serve ran a custom FastAPI app around LLM, with its own data-parallel fan-out and a worker extension for
weight sync. vLLM's server covers all of it, so the app is gone and the command is now a deprecated wrapper that
translates its flags into vllm serve and prints the equivalent command.

Changes

• Server: trl/scripts/vllm_serve.py goes from 1218 to ~130 lines! It adds the three settings TRL needs
  (--weight-transfer-config, --logprobs-mode processed_logprobs, --max-logprobs -1), sets
  VLLM_SERVER_DEV_MODE=1, and forwards any other argument to vllm serve. fastapi, uvicorn and pydantic leave the vllm extra.
• Generation: /v1/completions for token IDs. No OpenAI-compatible endpoint takes token IDs and images, so
  multimodal prompts have their images processed on their own and the resulting features paired with the trainer's
  token IDs on /inference/v1/generate, which keeps multi-turn generation working with images.
• Weight sync: vLLM's NCCL weight-transfer engine. Tensors are announced once and streamed in a single packed broadcast, instead of one HTTP request and one broadcast per tensor.
• Distillation: teacher logprobs come from prompt_logprobs.
• Docs: vllm serve is the documented way to start the server.

No change to training code: vllm_mode, vllm_server_base_url, vllm_group_port and the rest behave as before,
trl vllm-serve still accepts the same flags, and GRPO and RLOO are untouched.

Benchmark

GRPO in server mode on H100s, Qwen2.5-1.5B (text) and Qwen2.5-VL-3B (VLM), same seed and same data, server on its own GPUs and a two-process trainer.

Setup	main (two runs)	this PR (two runs)
Text, TP=1	1.08, 0.91 s/step	0.63, 0.62 s/step	1.59x
Text, TP=2	0.96, 0.96 s/step	0.62, 0.60 s/step	1.57x
VLM, TP=1	2.14, 2.13 s/step	1.48, 1.48 s/step	1.44x

Same behaviour: with a fixed generation seed the two implementations produce bit-identical completions (16/16 over 4 prompts x n=4, logprobs identical to 0.0 over 499 tokens). The training curves still do not lie on top of each
other, for reasons that have nothing to do with this PR.

Why the curves do not overlap exactly

The text curves diverge because each run talks to its own server process. Against the same server, two runs of
the same code are bit-identical, so the trainer itself is deterministic:

# repro_curve_divergence.py — needs two GPUs
#   VLLM_SERVER_DEV_MODE=1 CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-1.5B \
#       --weight-transfer-config '{"backend": "nccl"}' --logprobs-mode processed_logprobs --max-logprobs -1 &
#   CUDA_VISIBLE_DEVICES=1 python repro_curve_divergence.py
from datasets import load_dataset

from trl import GRPOConfig, GRPOTrainer

STEPS = 10


def reward_len(completions, **kwargs):
    return [-abs(40 - len(completion)) for completion in completions]


def run(output_dir, group_port):
    config = GRPOConfig(
        output_dir=output_dir,
        per_device_train_batch_size=8,
        num_generations=8,
        max_completion_length=64,
        max_steps=STEPS,
        use_vllm=True,
        vllm_mode="server",
        vllm_server_base_url="http://127.0.0.1:8000",
        vllm_group_port=group_port,  # one port per run, the first is still bound when the second starts
        generation_kwargs={"seed": 42},  # same sampling seed in both runs
        seed=0,
        data_seed=0,
        temperature=1.0,
        bf16=True,
        logging_steps=1,
        report_to=[],
        save_strategy="no",
    )
    trainer = GRPOTrainer(
        model="Qwen/Qwen2.5-1.5B",
        reward_funcs=reward_len,
        args=config,
        train_dataset=load_dataset("trl-internal-testing/zen", "standard_prompt_only", split="train"),
    )
    trainer.train()
    return [entry["reward"] for entry in trainer.state.log_history if "reward" in entry]


first, second = run("/tmp/repro_run_1", 51216), run("/tmp/repro_run_2", 51217)
differing = [step for step, (a, b) in enumerate(zip(first, second, strict=True)) if abs(a - b) > 1e-9]
print(f"run 1: {[round(value, 2) for value in first]}")
print(f"run 2: {[round(value, 2) for value in second]}")
print(f"identical for the first {differing[0] if differing else STEPS} steps")
print(f"mean |difference|: {sum(abs(a - b) for a, b in zip(first, second, strict=True)) / STEPS:.2f}")

run 1: [-230.88, -224.12, -240.62, -259.38, -203.25, -280.38, -247.0, -232.5, -266.88, -246.25]
run 2: [-230.88, -224.12, -240.62, -259.38, -203.25, -280.38, -247.0, -232.5, -266.88, -246.25]
identical for the first 10 steps
mean |difference|: 0.00

Give each run its own server — which comparing two implementations necessarily does — and they drift apart, because a
fresh engine returns slightly different logits and one flipped token changes every later step. That drift is as large
between two runs of main as it is between main and this PR:

Comparison (text, TP=1, reward over 30 steps)	identical steps	mean abs difference
main vs main, rerun	0	5.99
this PR vs this PR, rerun	0	9.78
main vs this PR	5	6.05

The VLM curves sit apart for a real reason. The server returns the processed image already cast to the model
dtype, while handing the engine the image directly preprocesses in float32. The values are otherwise the same:

# repro_image_dtype.py — needs one GPU
#   VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct &
#   python repro_image_dtype.py
import base64
import io

import requests
import torch
from PIL import Image
from transformers import AutoProcessor
from vllm.entrypoints.scale_out.token_in_token_out.mm_serde import decode_mm_kwargs_item

MODEL = "Qwen/Qwen2.5-VL-3B-Instruct"

image = Image.new("RGB", (64, 64), color=(30, 200, 30))
buffer = io.BytesIO()
image.save(buffer, format="PNG")
url = f"data:image/png;base64,{base64.b64encode(buffer.getvalue()).decode()}"

response = requests.post(
    "http://127.0.0.1:8000/v1/chat/completions/render",
    json={
        "model": MODEL,
        "max_tokens": 1,
        "messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": url}}]}],
    },
    timeout=300,
)
response.raise_for_status()
served = decode_mm_kwargs_item(response.json()["features"]["kwargs_data"]["image"][0])["pixel_values"].data

processor = AutoProcessor.from_pretrained(MODEL)
local = processor(images=[image], text=["<|vision_start|><|image_pad|><|vision_end|>"], return_tensors="pt")[
    "pixel_values"
]

print(f"server: {tuple(served.shape)} {served.dtype}")
print(f"local : {tuple(local.shape)} {local.dtype}")
print(f"max |difference|: {(served.float() - local.float()).abs().max().item():.3e}")
print(f"equal after casting the local one: {torch.equal(served, local.to(served.dtype))}")

server: (16, 1176) torch.bfloat16
local : (16, 1176) torch.float32
max |difference|: 2.747e-03
equal after casting the local one: True

So the image is the same image, rounded to the dtype the model runs in. That moves logprobs by ~0.04 and occasionally flips a greedy argmax: over three greedy generations the two implementations agreed on two and differed by one word on the third, both naming the colour correctly.

Validation

On H100s against vLLM 0.27.1:

• weight sync at TP=1, TP=2 and DP=2: a no-op sync leaves greedy output bit-identical, a perturbation propagates, restoring the weights restores the output
• GRPO, RLOO, VLM GRPO and server distillation trained end-to-end with the server on a separate GPU, plus the client-level weight sync checks (no-op, perturb, restore, single-tensor push, prefix cache)
• multimodal generation continued from token IDs across turns
• tests/test_vllm_client_server.py -m slow: 34 passed

Two fixes came out of this: vllm-project/vllm#52399, where
/inference/v1/generate drops completions when n > 1 (the client sets output_kind explicitly until it lands), and
the client here no longer builds its NCCL communicator on an unindexed device, which on main breaks a
single-process trainer with this nccl communicator is created to work on cuda, but the input tensor is on cuda:0.

---

Note

High Risk
Server-mode training depends on new vLLM HTTP/NCCL paths and required serve flags; regressions would affect generation, weight sync, and distillation across GRPO/RLOO/Online DPO, though trainer config surface is unchanged.

Overview
Replaces TRL’s custom FastAPI vLLM server with upstream vllm serve. trl vllm-serve is deprecated: it only maps TRL flags to vllm serve, sets VLLM_SERVER_DEV_MODE=1, injects TRL-required options (NCCL weight transfer, processed_logprobs, unlimited logprobs), and forwards extra CLI args. The vllm extra drops FastAPI/uvicorn/pydantic.

VLLMClient now talks to vLLM’s HTTP API: text/token prompts via /v1/completions, chat/multimodal via /v1/chat/completions, and token-ID + image features via /inference/v1/generate after image_features (/v1/chat/completions/render). Teacher logprobs for distillation use prompt_logprobs on completions instead of a custom endpoint. Weight sync uses vLLM’s NCCL weight-transfer engine (init_weight_transfer_engine, batched update_weights, optional start_weight_update / finish_weight_update) with a weight_update() context for one sync per optimizer step.

Trainers and VLLMGeneration are updated to match (e.g. Online DPO multimodal → chat, packed param streaming, prefix-cache reset). Docs, examples, and config help text now document vllm serve instead of trl vllm-serve.

^{Reviewed by Cursor Bugbot for commit e48e5a6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 659619393f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e48e5a6. Configure here.}

[cursor]

None-image rows break multimodal chat

Medium Severity

Passing images=None into prepare_multimodal_messages for None-image rows does not work with the earlier training_step rewrite that inserts {"type": "image"} placeholders on every conversational prompt whenever an image column is present. prepare_multimodal_messages then sees one unfilled placeholder and zero images and raises, so Online DPO server-mode VLM batches that mix real images with None fail. The old generate(..., images=...) path tolerated those rows.

Additional Locations (1)

• trl/experimental/online_dpo/online_dpo_trainer.py#L1126-L1141

 

^{Reviewed by Cursor Bugbot for commit e48e5a6. Configure here.}

[qgallouedec]

Merging without review. This one is quite big and important so please feel free to give late review share if anything is missing.