Use vLLM's own server instead of TRL's

docs/source/clis.md

@@ -19,7 +19,7 @@ Currently supported commands are:
 ### Other Commands
 
 - `trl env`: get the system information
-- `trl vllm-serve`: serve a model with vLLM
+- `trl vllm-serve`: serve a model with vLLM (deprecated, use `vllm serve` directly, see [vLLM integration](vllm_integration))
 
 ## Fine-Tuning with the TRL CLI
 

docs/source/distillation_trainer.md

@@ -125,7 +125,10 @@ In this mode, vLLM runs in a separate process (and using separate GPUs) and comm
 1. **Start the vLLM server**:
 
    ```bash
-   trl vllm-serve --model <model_name>
+   VLLM_SERVER_DEV_MODE=1 vllm serve <model_name> \
+       --weight-transfer-config '{"backend": "nccl"}' \
+       --logprobs-mode processed_logprobs \
+       --max-logprobs -1
    ```
 
 2. **Enable server mode in your training script**:

docs/source/grpo_trainer.md

@@ -241,7 +241,10 @@ In this mode, vLLM runs in a separate process (and using separate GPUs) and comm
 1. **Start the vLLM server**:
 
    ```bash
-   trl vllm-serve --model <model_name>
+   VLLM_SERVER_DEV_MODE=1 vllm serve <model_name> \
+       --weight-transfer-config '{"backend": "nccl"}' \
+       --logprobs-mode processed_logprobs \
+       --max-logprobs -1
    ```
 
 2. **Enable server mode in your training script**:
@@ -369,7 +372,10 @@ srun --nodes=4 --ntasks=4 --nodelist="${NODELIST[@]:0:4}" accelerate launch \
      --server_ip $VLLM_NODE &
 
 # Run vLLM server on the 5th node (Group 2)
-srun --nodes=1 --ntasks=1 --nodelist="${NODELIST[4]}" trl vllm-serve --model Qwen/Qwen2.5-72B --tensor_parallel_size 8 &
+srun --nodes=1 --ntasks=1 --nodelist="${NODELIST[4]}" env VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen2.5-72B --tensor-parallel-size 8 \
+    --weight-transfer-config '{"backend": "nccl"}' \
+    --logprobs-mode processed_logprobs \
+    --max-logprobs -1 &
 
 wait
 ```

docs/source/openenv.md

@@ -344,7 +344,10 @@ This runs vLLM in the same process as training, requiring only a single GPU.
 
 ```bash
 # Terminal 1: Start vLLM inference server
-CUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen3-1.7B --host 0.0.0.0 --port 8000
+CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3-1.7B --host 0.0.0.0 --port 8000 \
+    --weight-transfer-config '{"backend": "nccl"}' \
+    --logprobs-mode processed_logprobs \
+    --max-logprobs -1
 
 # Terminal 2: Run GRPO training with OpenEnv
 CUDA_VISIBLE_DEVICES=1 python examples/scripts/openenv/wordle.py --vllm-mode server --vllm-server-url http://localhost:8000

docs/source/rloo_trainer.md

@@ -185,7 +185,10 @@ In this mode, vLLM runs in a separate process (and using separate GPUs) and comm
 1. **Start the vLLM server**:
 
    ```bash
-   trl vllm-serve --model <model_name>
+   VLLM_SERVER_DEV_MODE=1 vllm serve <model_name> \
+       --weight-transfer-config '{"backend": "nccl"}' \
+       --logprobs-mode processed_logprobs \
+       --max-logprobs -1
    ```
 
 2. **Enable server mode in your training script**:
@@ -253,7 +256,10 @@ srun --nodes=4 --ntasks=4 --nodelist="${NODELIST[@]:0:4}" accelerate launch \
      --server_ip $VLLM_NODE &
 
 # Run vLLM server on the 5th node (Group 2)
-srun --nodes=1 --ntasks=1 --nodelist="${NODELIST[4]}" trl vllm-serve --model Qwen/Qwen2.5-72B --tensor_parallel_size 8 &
+srun --nodes=1 --ntasks=1 --nodelist="${NODELIST[4]}" env VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen2.5-72B --tensor-parallel-size 8 \
+    --weight-transfer-config '{"backend": "nccl"}' \
+    --logprobs-mode processed_logprobs \
+    --max-logprobs -1 &
 
 wait
 ```

docs/source/speeding_up_training.md

@@ -19,7 +19,10 @@ pip install trl[vllm]
 First, start a vLLM server by running:
 
 ```bash
-trl vllm-serve --model <model_name>
+VLLM_SERVER_DEV_MODE=1 vllm serve <model_name> \
+    --weight-transfer-config '{"backend": "nccl"}' \
+    --logprobs-mode processed_logprobs \
+    --max-logprobs -1
 ```
 
 Then, run the training script and pass `use_vllm=True` in the training arguments.
@@ -36,7 +39,10 @@ training_args = OnlineDPOConfig(..., use_vllm=True, vllm_mode="server")
 First, start a vLLM server by running:
 
 ```bash
-trl vllm-serve --model <model_name>
+VLLM_SERVER_DEV_MODE=1 vllm serve <model_name> \
+    --weight-transfer-config '{"backend": "nccl"}' \
+    --logprobs-mode processed_logprobs \
+    --max-logprobs -1
 ```
 
 Then, run the training script and pass `use_vllm=True` in the training arguments.
@@ -55,7 +61,10 @@ You can customize the server configuration by passing additional arguments. For
 > Set GPUs **0-3** for vLLM generation:  
 >
 > ```sh
-> CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model <model_name>
+> CUDA_VISIBLE_DEVICES=0,1,2,3 VLLM_SERVER_DEV_MODE=1 vllm serve <model_name> \
+>     --weight-transfer-config '{"backend": "nccl"}' \
+>     --logprobs-mode processed_logprobs \
+>     --max-logprobs -1
 > ```  
 >
 > And GPUs **4-7** for training:
@@ -70,7 +79,10 @@ You can customize the server configuration by passing additional arguments. For
 First, start a vLLM server by running:
 
 ```bash
-trl vllm-serve --model <model_name>
+VLLM_SERVER_DEV_MODE=1 vllm serve <model_name> \
+    --weight-transfer-config '{"backend": "nccl"}' \
+    --logprobs-mode processed_logprobs \
+    --max-logprobs -1
 ```
 
 Then, run the training script and pass `use_vllm=True` in the training arguments.
@@ -89,7 +101,10 @@ You can customize the server configuration by passing additional arguments. For
 > Set GPUs **0-3** for vLLM generation:
 >
 > ```sh
-> CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model <model_name>
+> CUDA_VISIBLE_DEVICES=0,1,2,3 VLLM_SERVER_DEV_MODE=1 vllm serve <model_name> \
+>     --weight-transfer-config '{"backend": "nccl"}' \
+>     --logprobs-mode processed_logprobs \
+>     --max-logprobs -1
 > ```  
 >
 > And GPUs **4-7** for training: