[Feature]Introduce vLLM as a new rollout backend (initial integration)

[CalvinXKY]

Purpose

vLLM is a strong, widely used inference backend (throughput, scheduling, and the OpenAI-compatible serving stack). The goal of this work is to bring that backend into Slime so teams can run the same RL / rollout training loop while choosing vLLM for generation and serving, instead of being limited to a single stack. This PR is the first step: wire vLLM into Slime with baseline engine, rollout, and router integration, and validate it on a representative dense model setup.

What’s included

• vLLM engine: VLLMEngine (Ray actor) to spawn a local vllm serve child process or attach to an external vLLM HTTP server; maps Slime expectations (health, weight version, cache flush, pause/resume, sleep/wake, etc.) onto vLLM’s HTTP control plane where it differs from SGLang.
• vLLM rollout: vllm_rollout (and related wiring) so training can drive rollout through a path parallel to SGLang rollout.
• Router: Support for routing via the vLLM router (replacement / integration path—adjust wording to match your exact flags or env vars if you document them in-repo).
• Megatron / weight sync: Hooks into vLLM’s update_weights, init_weight_transfer_engine, etc., including native vs fallback behavior as implemented on this branch.

Test plan

1. End-to-end smoke on a Qwen dense model under disaggregated deployment
2. Run unchanged Slime + SGLang vs Slime + vLLM with the same setup; compare reward curves to spot regressions from the new backend.

docker env:

docker pull slimerl/slime:latest

docker run -itd --rm --gpus all  --network=host --ipc=host --ulimit memlock=-1 \
--ulimit stack=67108864 -v /data/nfs_87:/data/nfs_87 \
--name slime-dev5 slimerl/slime:latest bash

pkgs install:

pip install vllm-router ray
pip unisntall slime

cd vime
pip install -e . --no-deps

test vllm is ok:

VLLM_SERVER_DEV_MODE=1 python -m vllm.entrypoints.openai.api_server \
  --model /data/nfs_87/xky/models/Qwen3-0.6B \
  --host 127.0.0.1 \
  --port 8000 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
   --enable-sleep-mode \
  --dtype auto

curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/data/nfs_87/xky/models/Qwen3-0.6B",
    "messages": [{"role":"user","content":"你好"}],
    "max_tokens": 32
  }'

Megatron+SGLang E2E (cmp)

#!/usr/bin/env bash
set -euo pipefail

export PYTHONUNBUFFERED=1
export TENSORBOARD_DIR="/data/nfs_87/xky/logs/tb_qwen3_0.6b_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$TENSORBOARD_DIR" "/data/nfs_87/xky/logs"

export PYTHONPATH=/root/Megatron-LM
SCRIPT_DIR="/root/slime/scripts"
source "${SCRIPT_DIR}/models/qwen3-0.6B.sh"
cd /root/slime/
python train.py \
  --train-backend megatron \
  --actor-num-nodes 1 \
  --actor-num-gpus-per-node 4 \
  --rollout-num-gpus 4 \
  --rollout-num-gpus-per-engine 1 \
  ${MODEL_ARGS[@]} \
  \
  --hf-checkpoint /data/nfs_87/xky/models/Qwen3-0.6B \
  --ref-load /data/nfs_87/xky/models/Qwen3-0.6B_torch_dist \
  \
  --prompt-data /data/nfs_87/xky/datasets/dapo-math-17k/dapo-math-17k.jsonl \
  --input-key prompt \
  --label-key label \
  --apply-chat-template \
  --rollout-shuffle \
  --rm-type deepscaler \
  \
  --num-rollout 200 \
  --rollout-batch-size 32 \
  --n-samples-per-prompt 8 \
  --rollout-max-response-len 8192 \
  --rollout-temperature 1.0 \
  --global-batch-size 256 \
  --balance-data \
  \
  --advantage-estimator grpo \
  --use-kl-loss \
  --kl-loss-coef 0.0 \
  --kl-loss-type low_var_kl \
  --entropy-coef 0.0 \
  --eps-clip 0.2 \
  --eps-clip-high 0.28 \
  \
  --optimizer adam \
  --lr 1e-6 \
  --lr-decay-style constant \
  --weight-decay 0.1 \
  --adam-beta1 0.9 \
  --adam-beta2 0.98 \
  \
  --tensor-model-parallel-size 1 \
  --pipeline-model-parallel-size 1 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 1 \
  --expert-tensor-parallel-size 1 \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 8192 \
  \
  --attention-dropout 0.0 \
  --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 \
  --attention-softmax-in-fp32 \
  --attention-backend flash \
  \
  --train-memory-margin-bytes 2147483648 \
  --use-tensorboard \
  2>&1 | tee -a "$LOG_FILE"

Megatron+vLLM E2E (ours)

export PYTHONPATH=/root/Megatron-LM
SCRIPT_DIR="/data/nfs_87/xky/RL/vime/scripts"
source "${SCRIPT_DIR}/models/qwen3-0.6B.sh"
LOG_FILE="/data/nfs_87/xky/logs/train_qwen3_0.6b_vllm_$(date +%Y%m%d_%H%M%S).log"

python train.py \
  --train-backend megatron \
  --actor-num-nodes 1 \
  --actor-num-gpus-per-node 4 \
  --rollout-num-gpus 4 \
  --rollout-num-gpus-per-engine 1 \
  ${MODEL_ARGS[@]} \
  \
  --hf-checkpoint /data/nfs_87/xky/models/Qwen3-0.6B \
  --ref-load /data/nfs_87/xky/models/Qwen3-0.6B_torch_dist \
  \
  --prompt-data /data/nfs_87/xky/datasets/dapo-math-17k/dapo-math-17k.jsonl \
  --input-key prompt \
  --label-key label \
  --apply-chat-template \
  --rollout-shuffle \
  --rm-type deepscaler \
  \
  --rollout-backend vllm \
  --vllm-weight-sync-mode native \
  \
  --num-rollout 200 \
  --rollout-batch-size 32 \
  --n-samples-per-prompt 8 \
  --rollout-max-response-len 8192 \
  --rollout-temperature 1.0 \
  --global-batch-size 256 \
  --balance-data \
  \
  --advantage-estimator grpo \
  --use-kl-loss \
  --kl-loss-coef 0.0 \
  --kl-loss-type low_var_kl \
  --entropy-coef 0.0 \
  --eps-clip 0.2 \
  --eps-clip-high 0.28 \
  \
  --optimizer adam \
  --lr 1e-6 \
  --lr-decay-style constant \
  --weight-decay 0.1 \
  --adam-beta1 0.9 \
  --adam-beta2 0.98 \
  \
  --tensor-model-parallel-size 1 \
  --pipeline-model-parallel-size 1 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 1 \
  --expert-tensor-parallel-size 1 \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 8192 \
  \
  --attention-dropout 0.0 \
  --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 \
  --attention-softmax-in-fp32 \
  --attention-backend flash \
  \
  --train-memory-margin-bytes 2147483648 \
  2>&1 | tee -a "$LOG_FILE"

Test result

ENV: GPU(A100/A800)

• NVIDIA A100-SXM4-80GB
• NVIDIA-SMI 570.172.08
• Driver Version: 570.172.08
• CUDA Version: 13.1

Curve compare:

TPS compare：

vLLM

SGLang

Known issue / follow-up

1. Installing vLLM in the Slime image upgrades flashinfer, which breaks the pinned SGLang combo—SGLang and vLLM are not reliably co-installable in one image:

2. router may log errors like Failed to send typed request … route=/v1/completions / error sending request for url when proxying to a worker. In our runs this is benign noise: training continues normally and outcomes are unaffected. Root cause / cleanup (timeouts, transient worker load, or router–vLLM compatibility) is TBD.

3. TODO: replace inline / hard-coded argument passing with a proper config path in a follow-up.

4. Features not yet supported:

• PD disaggregation: The vLLM engine path drops disaggregation_bootstrap_port and related dist args, treats non-regular workers as warnings only, and registers workers without SGLang-style bootstrap_port for prefill, so prefill/decode split is not wired the same way as SGLang.

• EPD / encoder: Multi-stage rollout still injects encoder_urls and language_only through sglang_overrides for Ray server groups; the vLLM launcher does not mirror that contract, so encoder-disaggregated setups are aligned with SGLang, not guaranteed for vLLM(unless you extend launch config yourself.)

• Offline / disk weights: SGLang updates from disk via a dedicated HTTP API with model_path / load_format; vLLM in slime only triggers collective_rpc reload_weights, which ignores path and format and is closer to a full reload than SGLang’s targeted disk update.

• Update weights via PIC/Colocate mode.

[aoshen02]

check vllm-router

[CalvinXKY]

Done. vllm backend runs with vllm router.

[aoshen02]

We need to ask jason why we need a separate process to do update weights from distributed

[aoshen02]

bug exists.

[aoshen02]

Ao should refactor the arguments.

[CalvinXKY]

Follow up

[aoshen02]

I don't think vllm has prompt_routed_experts as return key.

[CalvinXKY]

Yes. updated.

[aoshen02]

Could you add a list listing the points that we don't support compared with sglang? like encoder-prefill-disaggregation

[aoshen02]

syntax problem

[CalvinXKY]

syntax problem

fixed

[aoshen02]

Codex advice, please check:
High release_memory_occupation / resume_memory_occupation 现在给 /sleep、/wake_up 发的是 JSON body（slime/backends/vllm_utils/vllm_engine.py (line 483)），但 vLLM handler 读的是 query params（vllm/entrypoints/serve/sleep/api_router.py (line 22)），所以 level/tags 会被吞掉

[CalvinXKY]

Codex advice, please check: High release_memory_occupation / resume_memory_occupation 现在给 /sleep、/wake_up 发的是 JSON body（slime/backends/vllm_utils/vllm_engine.py (line 483)），但 vLLM handler 读的是 query params（vllm/entrypoints/serve/sleep/api_router.py (line 22)），所以 level/tags 会被吞掉

fixed

[CalvinXKY]

Could you add a list listing the points that we don't support compared with sglang? like encoder-prefill-disaggregation

See latest pr comment.

[aoshen02]

vllm might not has these keys in sglang.

[aoshen02]

/gemini review

[gemini-code-assist]

Code Review

This PR introduces vLLM as a rollout backend for the Slime framework. Key additions include a VLLMEngine Ray actor for managing vLLM server processes, a vllm_rollout module for handling inference and reward modeling, and support for vllm-router. To prevent NCCL conflicts between vLLM and Megatron, a _NcclBridge subprocess is implemented for weight transfers. Feedback suggests using multiprocessing.get_context("spawn") instead of global set_start_method or default process spawning to ensure safety and isolation. Additionally, bucketing parameters during weight gathering is recommended to alleviate performance bottlenecks in the new weight sync logic.

[gemini-code-assist]

Calling multiprocessing.set_start_method("spawn", force=True) inside a function is discouraged as it globally affects the start method for the entire application, which can conflict with other components (like Ray) or host applications using Slime as a library. It is safer to use multiprocessing.get_context("spawn") to create a local context for spawning processes, ensuring isolation and avoiding global state modification.

Suggested change

	multiprocessing.set_start_method("spawn", force=True)
	p = multiprocessing.Process(target=_exec_vllm_cmd, args=(cmd, env))
	p.start()
	ctx = multiprocessing.get_context("spawn")
	p = ctx.Process(target=_exec_vllm_cmd, args=(cmd, env))
	p.start()

[gemini-code-assist]

Spawning a process with the default multiprocessing.Process can be unsafe on platforms where the default start method is fork, especially when CUDA contexts or multiple threads are present in the parent process. Using an explicit spawn context is more robust and consistent with other parts of the codebase (e.g., in update_weight_from_distributed.py).

Suggested change

	process = multiprocessing.Process(
	target=run_router,
	args=(router_args,),
	args=((impl, router_args),),
	)
	ctx = multiprocessing.get_context("spawn")
	process = ctx.Process(
	target=run_router,
	args=((impl, router_args),),
	)

[gemini-code-assist]

The sequential all_gather_param calls for every parameter in a loop create a significant performance bottleneck, especially for large models. Each call involves a distributed collective operation and synchronization across ranks. Since this PR introduces a 'packed' weight sync mode for efficiency, consider also bucketing these parameters before gathering them to minimize the number of collective operations and synchronization points.

[aoshen02]

Code Review: vLLM 后端集成的若干问题(AI 协助评审)

这条 review 由 AI 协助生成,所有结论我都对照过当前 main 分支的 vLLM / vllm-project/router / sglang 源码进行了验证。已注明引用路径,方便核查。

评审目标:不破坏当前的 megatron + vLLM native 路径(也就是 PR description 里 reward curve 测的那条路径),同时把"非 happy-path 但默认配置会走"的几条死路点出来。

---

Bug #2 ⚠️(高优先级,silent failure,--offload-rollout 路径必踩)

onload_kv() 给 vLLM /wake_up 传 sglang 的 tag "cuda_graph",vLLM 直接 early-return,KV cache 也没被唤醒

slime/ray/rollout.py:373-378:

def onload_kv(self):
    handles = []
    for g in self.server_groups:
        handles.extend(g.onload(tags=[GPU_MEMORY_TYPE_KV_CACHE, GPU_MEMORY_TYPE_CUDA_GRAPH]))

slime.srt.constants 里 GPU_MEMORY_TYPE_CUDA_GRAPH = "cuda_graph",会被 vllm_engine.py:504-518 原样塞进 POST /wake_up?tags=kv_cache&tags=cuda_graph。

vLLM 这边 v1/executor/abstract.py:329-345:

self.sleeping_tags = {"weights", "kv_cache"}    # ← 没有 "cuda_graph"
...
def wake_up(self, tags: list[str] | None = None):
    if tags:
        for tag in tags:
            if tag not in self.sleeping_tags:
                logger.warning("Tag %s is not in sleeping tags %s", tag, self.sleeping_tags)
                return        # ← 整个方法直接 return,kv_cache 也不唤醒

HTTP 仍然返回 200,调用方完全感知不到失败。下一次 generate 请求 → KV cache 没分配 → 挂或 OOM,root cause 要翻 vllm-server log 才能找到 Tag cuda_graph is not in sleeping tags ... warning。

触发场景:train.py 主循环每个 rollout step 后都会执行 onload_kv(),只要打开 --offload-rollout 这条路就坏。PR 自带的 test plan(8 GPU,4+4 不 colocate)没开 offload 所以测不出来。

修法:在 vime engine 里加 sglang→vLLM tag 翻译表;或者干脆 vime 自己定义 weights/kv_cache 两个常量,删掉 cuda_graph(onload_kv 也对应改掉)。

---

Bug #3 🟠(中优先级,silent failure,worker 永远摘不掉)

shutdown() 给 vllm-router 的 DELETE /workers/{url} 路径塞的是 worker["id"](UUID)而不是 url,router 单 router 模式无脑返回 200 success 假装成功

vllm_engine.py:420-435:

for worker in all_workers:
    if worker["url"] == worker_url:
        wid = worker["id"]
        response = requests.delete(
            f"http://{self.router_ip}:{self.router_port}/workers/{wid}",   # ← 应该是 url
            timeout=30,
        )

vllm-router 是 vllm-project/router 这个 Rust 实现(pip install vllm-router 装的就是这个;_vllm_router_args_from_cli import 的 vllm_router.router_args 也是它)。路由声明在 server.rs:747-750:

.route("/workers", post(create_worker))
.route("/workers", get(list_workers_rest))
.route("/workers/{url}", get(get_worker))
.route("/workers/{url}", delete(delete_worker));

delete_worker 取 Path(url) 当 worker URL 用,然后 state.router.remove_worker(&url) 在 registry 里按 URL 字符串 匹配。PR 传的是 UUID,registry 里 w.url() 都是 http://10.0.0.5:15042 这种,永远匹配不上,但 handler 还是返回:

let response = WorkerApiResponse {
    success: true,
    message: format!("Worker {url} removed successfully"),
    worker: None,
};
(StatusCode::OK, Json(response)).into_response()

PR 这边 raise_for_status() 看到 200 不抛,日志一切如常。

后果:每次 engine restart / CI fault injection / dispose() 后,vllm-router 的 worker registry 都不会清理,堆 stale entry。Router routing policy 在下一次 health check(默认 60s 间隔)之前会派请求给已死的 worker → connection refused → 重试 5 次 → 用户看到的就是 PR description 里 "Known issue #2" 那条 Failed to send typed request … route=/v1/completions。那条"benign noise"的 root cause 大概率就是这个。

修法(二选一):

# 方案 A:URL 编码后走 RESTful 路径
from urllib.parse import quote
response = requests.delete(
    f"http://{self.router_ip}:{self.router_port}/workers/{quote(worker_url, safe='')}",
    timeout=30,
)

# 方案 B:走老的 query param 路径(跟 _register_worker_with_router 的 use_slime_router 分支对称)
response = requests.post(
    f"http://{self.router_ip}:{self.router_port}/remove_worker",
    params={"url": worker_url},
    timeout=30,
)

顺手也把 if self.args.rollout_external: return 这条早退里的 deregister 加上 —— 外部 router 生命周期更长,清理更重要。

---

Bug #6 🟡(中优先级,多模型部署下必踩,说明已重写并替换原版)

launch_server_process 永远用 args.hf_checkpoint 起 vLLM,丢掉 ServerGroup.model_path → 多模型部署里 reference / reward 的 vLLM 实例错误加载 actor 的 checkpoint

slime 已有的多模型数据流(--sglang-config YAML 这条)

slime 早就支持一个 trainer 进程后挂多个 inference model(actor + ref + rm),通过 --sglang-config YAML 配:

sglang:
  - name: actor
    model_path: /path/to/actor
    update_weights: true           # 接受 trainer NCCL weight sync
    server_groups:
      - { worker_type: regular, num_gpus: 4 }
  - name: ref
    model_path: /path/to/ref       # ← 跟 actor 不同
    update_weights: false          # ← frozen,不接收 weight sync
    server_groups:
      - { worker_type: regular, num_gpus: 2 }
  - name: rm
    model_path: /path/to/rm
    update_weights: false
    server_groups:
      - { worker_type: regular, num_gpus: 2 }

每个 model 起独立 router(start_rollout_servers 里 force_new=(model_idx > 0))。ModelConfig.model_path 通过 ModelConfig.resolve() 注入到每个 group 的 overrides["model_path"],然后 _make_group 又把它拍到 ServerGroup.model_path 这个 first-class 字段 上:

# slime/ray/rollout.py:_make_group
group = ServerGroup(
    ...,
    sglang_overrides=overrides,
    model_path=overrides.get("model_path", args.hf_checkpoint),     # ← first-class field
    ...
)

同一份 model_path 在 sglang 体系里走了三层(ModelConfig.model_path → g.overrides["model_path"] → ServerGroup.model_path),冗余但语义清晰。

sglang_engine 是怎么用上正确 model_path 的(两步覆盖)

slime/backends/sglang_utils/sglang_engine.py:_compute_server_args 用一个"默认 + override"的两步 pattern:

# 第 1 步:默认值,只是占位
kwargs = {
    "model_path": args.hf_checkpoint,
    ...
}

# (中间从 args.sglang_* 拷一遍字段)

# 第 2 步:从 ServerGroup.sglang_overrides 覆盖,**永远会包含 model_path**,因为 ModelConfig.resolve() 强制注入
if sglang_overrides:
    for key, value in sglang_overrides.items():
        normalized_key = key.replace("-", "_")
        ...
        kwargs[normalized_key] = value             # ← 这里把 model_path 真正定下来

第 1 步那行 "model_path": args.hf_checkpoint 看起来像 ground truth,其实只是 fallback 占位,第 2 步立刻把它盖成 ServerGroup.model_path(= YAML 里写的 actor / ref / rm 各自的路径)。运行时 sglang 实际加载的就是各 model 自己的 checkpoint。

vime 这条 PR 只走了第 1 步,没第 2 步

slime/backends/vllm_utils/vllm_engine.py:82:

def launch_server_process(*, bind_host, server_port, args, rank, visible_devices):
    ...
    model = getattr(args, "vllm_model", None) or args.hf_checkpoint   # ← 只有第 1 步
    ...
    cmd = ["vllm", "serve", str(model), "--tensor-parallel-size", str(tp), ...]

launch_server_process 不接 sglang_overrides,也不接 ServerGroup.model_path,永远拿全局 args.hf_checkpoint 起 vllm。VLLMEngine.__init__ 虽然在 self.sglang_overrides 里保留了那个 dict,但 _init_normal / launch_server_process 链路完全无视它。

后果

只要用户用 --sglang-config 配多模型,ref / rm 这些 group 的 vLLM 实例全部加载 actor 的 args.hf_checkpoint,根本不是它们自己 YAML 里写的 path。具体影响:

1. frozen reference 形同虚设:update_weights: false 这个开关在 RolloutServer.update_weights 一侧能正确不把 weight sync 发给 ref,但 ref vLLM 实例打从启动那一刻就装的是 actor 的初始权重,所以"frozen ref"实际上是"frozen 在 actor 起点的副本"。这跟 RLHF 算 KL penalty 的语义直接冲突 —— KL 项算出来的是"当前 actor vs actor 起点",不是"当前 actor vs ref"。reward 曲线还会跑出来,但语义错。

2. reward model 完全跑错模型:rm 应该是个独立的 score head 模型,被 PR 静默替换成 actor 的 LM,生成的 reward 信号毫无意义。

3. PR 自带 test plan(单模型 actor-only)感知不到这条:Qwen3-0.6B 单模型路径,args.hf_checkpoint 就是唯一的 model,没冲突。所以 PR 描述里的 reward 曲线在那条 setup 上是有效的;但任何打开 --sglang-config 的多模型 setup 都会撞上。

4. 下游 PR(比如要把 sglang 这条腿真正删掉的时候)如果直接照搬 vime engine 的接法,会把这个错误固化进新代码。

推荐修法:用 ServerGroup.model_path 这个 first-class 字段,绕开 sglang_overrides

考虑到这条 PR description 里写了"this project will eventually remove sglang content",修法不应该继续往 sglang_overrides dict 上加新职责(像 sglang_engine 第 2 步那样从 dict 里挖 model_path),否则等 sglang 路径整体下线的时候,vime engine 又要再改一次。

ServerGroup.model_path 已经是 first-class 字段了,直接当独立 kwarg 透传给 actor 就行,跟 sglang_overrides 那条 dead-name 通道完全解耦:

1. slime/ray/rollout.py:ServerGroup.start_engines

 rollout_engine = RolloutRayActor.options(
     num_cpus=num_cpus,
     num_gpus=num_gpus,
     scheduling_strategy=scheduling_strategy,
     runtime_env={"env_vars": env_vars},
 ).remote(
     self.args,
     rank=global_rank,
     worker_type=self.worker_type,
     base_gpu_id=base_gpu_id,
+    model_path=self.model_path,         # ServerGroup 已经算好的值
     sglang_overrides=self.sglang_overrides,
     num_gpus_per_engine=self.num_gpus_per_engine,
 )

2. slime/backends/vllm_utils/vllm_engine.py:VLLMEngine.__init__

 def __init__(
     self,
     args,
     rank: int,
     worker_type: str = "regular",
     base_gpu_id: int | None = None,
+    model_path: str | None = None,
     sglang_overrides: dict | None = None,
     num_gpus_per_engine: int | None = None,
 ):
     ...
+    self.model_path = model_path or args.hf_checkpoint
     ...

3. launch_server_process 改签名,_init_normal 透传

-def launch_server_process(*, bind_host, server_port, args, rank, visible_devices):
+def launch_server_process(*, bind_host, server_port, args, rank, visible_devices, model_path):
     ...
-    model = getattr(args, "vllm_model", None) or args.hf_checkpoint
+    model = model_path
     ...
     cmd = ["vllm", "serve", str(model), ...]

 def _init_normal(self) -> None:
     ...
     self.process = launch_server_process(
         bind_host=bind_host,
         server_port=self.server_port,
         args=self.args,
         rank=self.rank,
         visible_devices=visible_devices,
+        model_path=self.model_path,
     )

(sglang_engine 那条不动 —— 它的两步覆盖在 sglang 下线之前都还要继续用。)

这样三件事同时成立:

• 多模型 YAML 里每个 ModelConfig.model_path 真的能传到对应 vLLM 实例启动命令里。
• update_weights: false 的 frozen 模型不仅在 weight-sync 路径上被跳过,初始加载的也是正确的 checkpoint,frozen 语义恢复正确。
• 当后续 PR 删 sglang 代码时,VLLMEngine.__init__ 里的 sglang_overrides 参数可以整个拿掉,model_path 这条新加的独立 kwarg 完全不受影响,不留尾巴。

加一个 sanity test 防止回归

PR 现在的 test 只跑单模型,这条 bug 测不出来。建议在合入前加一个最小验证(--sglang-config 配 actor + ref 两个不同 path 的小模型,断言 ref vLLM 的 /v1/models 返回的 id 跟 ref 的 path 一致,而不是 actor 的)。或者在 _init_normal 里加一句 assert self.model_path,把 silent 错误转成 startup 时直接挂。

---

Bug #8 🟢(低优先级,external 模式下日志噪音)

_wait_external_config_ready 比较的 body["tensor_parallel_size"] 字段在 vLLM 的 /server_info 响应里根本不在顶层,导致每次外部 engine 启动都会日志 false-positive mismatch warning

vllm_engine.py:244-254:

expect = {"tensor_parallel_size": self.args.rollout_num_gpus_per_engine}
for name, expect_value in expect.items():
    if name not in body and name != "tensor_parallel_size":
        continue
    actual_value = body.get(name)            # ← None
    if actual_value != expect_value and expect_value is not None:
        logger.warning("External vLLM server_info mismatch ...")    # ← 必触发

vLLM 的 /server_info 返回结构是 {"vllm_config": ..., "vllm_env": ..., "system_env": ...}(instrumentator/server_info.py:43-59),tensor_parallel_size 嵌在 vllm_config.parallel_config.tensor_parallel_size。

修法:

parallel_cfg = body.get("vllm_config", {}).get("parallel_config", {})
actual_tp = parallel_cfg.get("tensor_parallel_size")
if actual_tp != self.args.rollout_num_gpus_per_engine:
    logger.warning(...)

或者干脆删掉这个 weak check,它本来就 "non-fatal"。

---

Bug #9 🟢(低优先级,死代码)

flush_cache 里的 except NewConnectionError: raise 永远不会触发,requests 会把 urllib3 的 NewConnectionError 包成 requests.ConnectionError(后者继承自 RequestException/OSError),会被下面的 except Exception 接走重试。如果想保留"连不上立刻 raise"的语义,得 catch requests.ConnectionError:

except requests.ConnectionError:
    raise
except Exception as e:
    logger.info("Error resetting vLLM prefix cache: %s", e)
    time.sleep(1)
    continue

---

---

其他观察(不算 bug,可选 follow-up)

1. NCCL_IB_DISABLE=1 默认开(vllm_engine.py:77):是 setdefault 可被 env 覆盖,但任何多节点部署里这条会让 NCCL bridge 走 TCP 而不是 IB,weight transfer 带宽掉一个数量级以上。建议默认不设。

2. multiprocessing.set_start_method("spawn", force=True) 放进 launch_server_process 里(vllm_engine.py:118):应该在 module 顶层只 set 一次。放在函数里每次调用都 force=True 会强行抢占同进程内其他 multiprocessing 用法的 start method。

3. launch_server_process 用 multiprocessing.Process(target=_exec_vllm_cmd) + os.execvpe:本质是借 spawn 走干净的 fork 再 exec,但用 subprocess.Popen(cmd, env=env, start_new_session=True) 语义更直白,parent 也更好观测子进程退出码。

4. pause_generation 默认 mode="abort" + vLLM 自身 /pause 的 clear_cache=True 默认:同步训练下没关系;但任何 async rollout pipeline(partial_rollout 等)都会丢 in-flight 请求,跟 SGLang pause_generation 的 "wait/keep" 语义不同。建议 default 改成 "keep",或者至少把 clear_cache=False 显式发出。

5. health_generate 只 GET /health:vLLM /health 检查的是 engine loop alive 而不是"能生成"。SGLang /health_generate 会真的跑一次 1-token generation。如果想跟 SGLang 等价,需要发一次 POST /v1/completions with max_tokens=1。否则 GPU 死锁 / 调度卡死场景下健康检查不会标记 unhealthy。

6. simulate_crash 走 shutdown() 而不是 os._exit(1):跟 SGLang engine 行为一致,但严格意义上是 graceful drain,不是 crash 语义。CI fault injection 测的是 "engine 突然失联" 还是 "engine 优雅 drain" 取决于这个区别。

7. init 接受 dist_init_addr/nccl_port/disaggregation_bootstrap_port 然后 del 掉:为了跟 SGLang engine signature 兼容这是合理的,但 PR description 已经说 "this project will eventually remove sglang content",这些 dead arg 在那个阶段建议清掉(包括 init_weights_update_group 的 group_name/backend、sglang_overrides 字段名等)。

8. _register_worker_with_router 的 sglang-router 分支基本是 dead code:slime_validate_args 已经强制 args.use_slime_router = False 并打 deprecation warning,这条 if 分支永远走不到。直接删掉吧,留着只会让读者以为有两条注册路径。

---

---

总结优先级

Bug	严重度	是否在 PR test plan 中触发
#2 cuda_graph wake_up	高	否(test 没 --offload-rollout)
#3 shutdown DELETE 路径	中	触发但 silent
#6 launch_server_process 忽略 model_path	中	否(单模型)
#8 _wait_external_config_ready 字段路径	低	rollout_external 才触发
#9 NewConnectionError dead code	低	无功能影响

#2 / #6 的"silent correctness"性质比一眼可见的 crash 更危险,因为 reward 曲线还能跑出来,但跑出来的是错的。建议合入之前至少补一个端到端 sanity check(对 #6 是验 /v1/models 的 id;对 #2 是开 --offload-rollout 跑两个 step 看 vLLM log 有没有那条 wake_up warning)。

---

注:这是 AI 协助生成的 review,基础的"它存在不存在""它对不对"我都对照源码核过,但建议 maintainer 在合并前对 #2 / #6 自己用一个最小可复现命令验证一遍。

[CalvinXKY]

/gemini review

Changes made：

Review fixes (#2, #3, #6, #8, #9)

• Drop SGLang-only sleep/wake tags (e.g. cuda_graph) before calling vLLM POST /wake_up.
• Deregister vLLM router workers via URL-encoded DELETE /workers/{url} instead of worker UUID.
• Pass ServerGroup.model_path through VLLMEngine into launch_server_process for multi-model YAML setups.
• Read external tensor_parallel_size from vllm_config.parallel_config on /server_info.
• Re-raise requests.ConnectionError in flush_cache instead of catching unused NewConnectionError.

#4 / #7 and small follow-ups

• Default --vllm-weight-sync-mode to native.
• Pass weights_path and is_checkpoint_format in collective_rpc reload_weights kwargs for disk reload.
• Deregister from vLLM router on shutdown() even when rollout_external is set (skip only local process kill).
• Remove deprecated slime-router register/deregister branches; use vLLM router only.
• Default vllm_pause_mode to keep and send clear_cache=false on /pause.
• Stop forcing NCCL_IB_DISABLE=1 in the vLLM server child env (use cluster env if needed).

For Gemini：

• Start vLLM and router child processes with multiprocessing.get_context("spawn") instead of set_start_method(..., force=True).
• Bucket vLLM packed weight sync by update_weight_buffer_size instead of gathering all parameters in one shot.
  Not changed：
• Still launch the vLLM server via multiprocessing.Process + os.execvpe rather than subprocess.Popen with start_new_session=True.

[gemini-code-assist]

Code Review

This pull request adds vLLM as a rollout backend, introducing a VLLMEngine Ray actor and a NcclBridge to prevent NCCL conflicts with the Megatron trainer. It includes a new vllm_rollout module for OpenAI-compatible generation and integrates vllm-router for request routing. Reviewers identified a potential crash in generate_and_rm when processing sample lists and recommended keeping sglang as the default backend to prevent breaking changes. Additionally, it was noted that the current vLLM server launcher lacks support for multi-node tensor parallelism.

[gemini-code-assist]

The generate_and_rm function will crash if the input sample is a list, as it attempts to access sample.response_length and sample.status before performing the type check at line 645. While current callers appear to pass a single Sample, the type hint and the logic later in the function suggest that a list of samples is a supported input/output format (e.g., for custom generation functions that might return multiple samples). The type check should be moved to the beginning of the function to safely handle both cases.

[CalvinXKY]

fix

[gemini-code-assist]

Changing the default value of --rollout-backend to vllm is a breaking change for existing users who rely on the default sglang backend. As noted in the PR description, vLLM and SGLang are not reliably co-installable in the same image due to conflicting dependencies (like flashinfer). This change could cause immediate failures for users in SGLang-only environments. It is recommended to keep sglang as the default and allow users to opt-in to vllm via the command line.

Suggested change

	default="vllm",
	default="sglang",

[CalvinXKY]

Changing the default value of --rollout-backend to vllm is a breaking change for existing users who rely on the default sglang backend. As noted in the PR description, vLLM and SGLang are not reliably co-installable in the same image due to conflicting dependencies (like flashinfer). This change could cause immediate failures for users in SGLang-only environments. It is recommended to keep sglang as the default and allow users to opt-in to vllm via the command line.

Keeping vllm as default for this integration; use --rollout-backend sglang when you need SGLang.

[gemini-code-assist]

The launch_server_process function currently lacks support for multi-node tensor parallelism (TP) when spawning a local vLLM server. It ignores the dist_init_addr and other distributed coordination parameters that the SGLang backend uses to synchronize TP across multiple nodes. If rollout_num_gpus_per_engine exceeds the number of GPUs available on a single node, the vLLM server will fail to launch or operate correctly because it won't be able to coordinate with other nodes. Consider adding support for vLLM's distributed environment variables (e.g., VLLM_HOST_IP, VLLM_PORT) to enable multi-node TP in future iterations.

[CalvinXKY]

Out of scope for this PR—local vllm serve only; multi-node TP / dist_init_addr is a follow-up.

[aoshen02]

LGTM

[gcanlin]

We can make vllm import global and remove all sglang import first, or do we have to install sglang currently?

[hsliuustc0106]

Review: PR #3 vLLM Rollout Backend

Verdict: Approved

Given that SGLang will be removed later, the architecture and default choices are the right forward-looking decisions.

What is good

• NcclBridge is the right isolation strategy for the NCCL conflict (vLLM#5477). Multiprocessing + CUDA IPC avoids GPU→CPU→GPU copies.
• Packed weight sync leverages vLLM's NCCLWeightTransferEngine.trainer_send_weights for dense models — meaningful optimization over per-parameter broadcast.
• _normalize_vllm_wake_tags correctly drops SGLang-only tags before they reach vLLM.
• model_path threading (ServerGroup → VLLMEngine → launch_server_process) correctly supports multi-model YAML configs.
• All issues from the earlier review rounds (Bug #2, #3, #6, #8, #9) are addressed.

Non-blocking observations

1. vllm_gpu_memory_utilization default inconsistency: argparse default is 0.55 but launch_server_process fallback is 0.4. These should agree.

2. health_generate is weaker than SGLang's: GET /health only checks process aliveness — a GPU hang or scheduler deadlock will not be detected. Consider an optional POST /v1/completions with max_tokens=1 behind a flag for stricter health checking.

3. Router subprocess daemon=True: if the main process crashes, orphaned router processes continue running and holding ports. Consider atexit cleanup or non-daemon mode.

4. _restart_local_server on reload fallback: when native weight sync is not available, continue_generation kills and reinitializes the vLLM process — adds full model load time per weight update for large models. Worth documenting the performance implication until native sync is the universal path.

None of these are blockers. The code is solid and ready to merge.