Repository navigation
[rollout, vllm, trainer, ckpt, doc] feat: external vLLM rollout (official vllm serve + verl plugin) - #2
[rollout, vllm, trainer, ckpt, doc] feat: external vLLM rollout (official vllm serve + verl plugin)#2Luosuu wants to merge 10 commits into
Conversation
…replica (plugin, launcher, HTTP registry) - HTTP weight registry: the trainer can serve InMemoryWeightRegistry over HTTP (RegistryServer) and processes outside its Ray job use HttpWeightRegistry. The registry also keeps a replica directory (URL, loaded version, heartbeat). Manifests travel as JSON. - vLLM plugin: verl_transport_pull, a WeightTransferEngine registered through the vllm.general_plugins entry point. It pulls a published version into the worker's host mirror under a lease and loads it as a layerwise reload, starting the reload only after the pull succeeded. - external_launcher: starts an official vllm serve with the plugin and runs a sidecar (HttpReplicaUpdateAgent) that keeps it on a recent version through vLLM's HTTP weight update routes and reports it to the registry. - WeightUpdateAgent gains prefetch/apply/heartbeat hooks for that subclass. - Design doc: docs/advance/external_vllm_rollout.md. Trainer wiring and the HTTP generation client come next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…LM's PATH vLLM's engine core runs ninja for kernel builds; in an unactivated venv it was not on PATH and the engine failed to start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-enable-scale-out) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…M servers - trainer.v1.decoupled_async.external_rollout: the trainer serves the weight registry over HTTP (RegistryServer), points the publisher at it, waits for min_replicas servers started by external_launcher to register, and generates through them. It launches no rollout replicas or update agents. - HttpServerHandle stands in for a vLLM server actor handle: generate.remote() calls vLLM's /inference/v1/generate and returns TokenOutput (abort -> aborted, weight version spans -> global_steps), so the load balancer and FullyAsyncLLMServerClient are unchanged. ExternalServerManager provides the server-manager surface the trainer uses. - PromptDispatcher can gate on loaded versions from the registry's replica directory and fails when a server stops reporting; the trainer waits for versions and checks heartbeats the same way. - The registry reports each replica's age rather than a raw timestamp; the launcher's sidecar waits for the registry, so servers can start first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The vLLM plugin keeps one registry client for the whole run. Its pooled connection sat idle past uvicorn's keep-alive between weight updates, so the next get_manifest failed with ServerDisconnectedError (seen in the external vLLM GRPO run at version 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b8a1aa984
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| manifest = self._run(self._registry.get_manifest(update_info.version)) | ||
| if manifest is None: | ||
| raise RuntimeError(f"Weight version {update_info.version} is not published (or already reclaimed)") | ||
| pulled = self._run(self._puller.fetch(manifest)) |
There was a problem hiding this comment.
If the trainer publishes another version between the sidecar's latest_ready() poll and this fetch, WeightPublisher.publish() retires the selected version and WeightPuller.fetch() raises LeaseError when acquiring its lease. Unlike the existing transport checkpoint engine's prefetch path, this path propagates the error through /update_weights, causing the sidecar to terminate the server and eventually fail training. This is a normal race in asynchronous publication, not necessarily a broken replica. Handle stale-version acquisition at the replica level and retry a newer version consistently across workers rather than treating it as a fatal update failure.
Useful? React with 👍 / 👎.
| deadline = time.monotonic() + timeout | ||
| while min(self._replica_versions(), default=-1) < version: | ||
| self._check_rollout() | ||
| if time.monotonic() > deadline: | ||
| raise TimeoutError(f"Rollout servers did not load version {version} within {timeout}s") | ||
| time.sleep(0.5) |
There was a problem hiding this comment.
When an external launcher uses --min-version-lag greater than one, validation can wait forever for a version the sidecar deliberately declines to load. For example, after loading version 0 with lag 2, the sidecar will not load version 1, but validation after publishing version 1 blocks here until init_timeout; the trainer cannot publish version 2 while blocked. The in-job branch avoids this through wait_for_version(), which sets the agent's _demanded version, whereas this branch only polls. Provide an equivalent demand mechanism for external sidecars, or explicitly reject lag configurations that cannot satisfy validation waits.
Useful? React with 👍 / 👎.
| async def _apply(self, version: int): | ||
| await self._post("pause", params={"mode": "abort", "clear_cache": "true"}) | ||
| await self._post("start_weight_update") | ||
| await self._post("update_weights", {"update_info": {"version": version}}) |
There was a problem hiding this comment.
If a legitimate pull and reload takes longer than external_rollout.heartbeat_timeout (60 seconds by default), the trainer or prompt dispatcher declares the healthy replica dead and fails the job. WeightUpdateAgent.run() awaits the entire _update() before issuing another heartbeat, so this potentially 1800-second HTTP request suppresses all replica reports while the transfer is progressing. Run heartbeats independently of update application so large-model or slower-network transfers do not trigger the missing-heartbeat failure.
Useful? React with 👍 / 👎.
| content = (choice.get("logprobs") or {}).get("content") | ||
| log_probs = [entry["logprob"] for entry in content] if content else None | ||
| extra_fields: dict[str, Any] = {} |
There was a problem hiding this comment.
When score centering is enabled with rollout.topk_log_probs > 0, this converter retains only each sampled token's log probability and discards the returned top-k alternatives. The in-job server supplies response_topk_ids and response_topk_log_probs, which the agent loop converts into rollout_topk_ids and rollout_topk_log_probs; without them, score_centering_logits_processor() directly indexes missing batch fields and training fails. Extract the requested heads into the same extra fields, or reject this configuration before starting external rollout.
Useful? React with 👍 / 👎.
| handles = [ | ||
| HttpServerHandle(url, prompt_length=rollout.prompt_length, response_length=rollout.response_length) | ||
| for url in self.server_addresses | ||
| ] |
There was a problem hiding this comment.
When actor_rollout_ref.rollout.ignore_eos=True, external generation silently ignores that setting: AgentLoopWorkerTQ.generate_sequences() does not include it in the request's sampling parameters, and these handles receive only the configured lengths. The in-job vLLMHttpServer.generate() explicitly injects self.config.ignore_eos, but the external converter has no access to it, so requests instead use the server's default EOS behavior and can terminate before the intended response budget. Pass the configured EOS policy into the handle and apply it as a sampling default.
Useful? React with 👍 / 👎.
| if self._external_rollout: | ||
| self._setup_external_rollout() |
There was a problem hiding this comment.
With external rollout enabled and rollout.nnodes=0 (the default), initialization fails before reaching this branch. The preceding super()._setup() calls PPOTrainer._setup(), which creates an empty LLMServerManager because _enable_hybrid_replicas=False; however, LLMServerManager.create_empty() still invokes its constructor without a worker group, triggering assert worker_group is not None or self.rollout_config.nnodes > 0. Although this change removes the trainer's standalone-GPU assertion for external mode, the downstream assertion still requires users to configure fictitious local rollout nodes. Adjust the empty-manager initialization so external-only training can start with zero local rollout resources.
Useful? React with 👍 / 👎.
…ecture diagrams Replace the ASCII diagrams with Mermaid flowcharts, which GitHub renders in the PR view. verl's Sphinx docs show them as code blocks unless sphinxcontrib-mermaid is enabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Standalone SVGs (explicit colors, white background, system fonts) render the same on GitHub and in verl's Sphinx docs, which has no Mermaid extension. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bbcca331a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ) -> dict[str, Any]: | ||
| """verl's sampling params as vLLM's, following `vLLMHttpServer.generate`.""" | ||
| params = dict(sampling_params) | ||
| max_possible_tokens = prompt_length + response_length - prompt_len |
There was a problem hiding this comment.
When the server's configured context length is smaller than prompt_length + response_length, this calculation permits requests that exceed the model's context window. For example, with max_model_len=2048, prompt length 512 and response budget 2048, the converter requests 2048 new tokens despite only 1536 fitting. Unlike the in-job vLLMHttpServer.generate(), which clamps against self.config.max_model_len - len(prompt_ids), external generation sends the oversized request and propagates the server's rejection as a rollout failure. Pass the actual context limit into the handle and use it for this bound.
Useful? React with 👍 / 👎.
| for url in self.server_addresses | ||
| ] | ||
| self.global_load_balancer = get_router_handle( |
There was a problem hiding this comment.
With rollout.full_determinism=True and stochastic sampling, external rollout enables deterministic routing but never forwards rollout.seed into generation requests. AgentLoopWorkerTQ.generate_sequences() does not supply a seed, and the in-job vLLMHttpServer.generate() normally fills that gap with sampling_params.setdefault("seed", self.replica_rank + self.config.seed). These external handles receive neither setting, so requests use the server's unseeded-request sampling behavior instead, losing the configured per-request reproducibility even when routing and server startup seeds are fixed. Pass the determinism configuration and a stable replica-specific seed into each handle and inject the sampling seed when enabled.
Useful? React with 👍 / 👎.
Review PR for M5 on top of
feat/transport-weight-registry(the branch behind verl-project#8526). Not for merging upstream as-is.What this does
decoupled_asynccan generate on officialvllm serveservers outside the trainer's Ray job. verl contributes a vLLM plugin, a launcher with a sidecar, and an HTTP registry; the vLLM build is unmodified. Design doc:docs/advance/external_vllm_rollout.md.Where to look
verl/checkpoint_engine/transport/http_registry.py:RegistryServer(FastAPI overInMemoryWeightRegistry) and theHttpWeightRegistryclient.registry.pygains a replica directory (report_replica/list_replicas, with each replica's age) and JSON manifest serialization.verl/workers/rollout/vllm_rollout/transport_pull_engine.py: theverl_transport_pullWeightTransferEngine. Pulls a version under a lease, then runs a layerwise reload; the reload starts only after the pull succeeded.vllm_plugin.pyplus thevllm.general_pluginsentry point inpyproject.tomlregister it lazily.verl/workers/rollout/vllm_rollout/external_launcher.py: wrapsvllm serve(setsVLLM_PLUGINS,VLLM_SERVER_DEV_MODE,--weight-transfer-config,--enable-scale-out). ItsHttpReplicaUpdateAgentsidecar drives/pause→/start_weight_update→/update_weights→/finish_weight_update→/resumeand heartbeats to the registry.update_agent.pygains prefetch/apply/heartbeat hooks for it.verl/workers/rollout/external_server.py:HttpServerHandlestands in for a vLLM server actor handle, calling/inference/v1/generateand returningTokenOutput.ExternalServerManagerprovides what the trainer uses ofLLMServerManager.verl/trainer/ppo/v1/trainer_decoupled_async.pyandprompt_dispatcher.py: theexternal_rolloutmode (start the registry server, wait for replicas, gate dispatch and validation on registry versions, fail on missing heartbeats).Tests
tests/trainer/ppo/v1/,tests/checkpoint_engine/*_on_cpu.py,tests/workers/rollout/test_external_server_on_cpu.py)pytest tests/workers/rollout/rollout_vllm/test_vllm_external_launcher.py(officialvllm servefollows versions 0 → 1 → 2; greedy outputs and vLLM'sweight_versionsspans checked)vllm serve(TP=2) on the other 4decoupled_async0.501); 1.0 versions per trajectoryBugs found by the end-to-end runs and fixed here: registry server returning before it accepts connections;
ninjamissing from vLLM's PATH in an unactivated venv;--enable-scale-outneeded for the generate route; the plugin's reused registry connection dropping after idling between updates.Known limitations
No prefetch (the pull runs inside
/update_weights, while paused); fixed replica set; text-only; no pause metrics in the trainer;VLLM_SERVER_DEV_MODEexposes unauthenticated routes, so the rollout servers need an isolated network.AI assistance (Claude Code) was used for the code, tests, experiments and this description.
🤖 Generated with Claude Code