Repository navigation
chore: bump vllm to 0.31.0 - #3888
Merged
Merged
Conversation
vLLM 0.31 moves enable_return_routed_experts from ModelConfig to VllmConfig.aux_output_config; update the NIXL routed-experts patch and PrimeRlServingTokens accordingly. Exempt flashinfer-python from the exclude-newer cooldown since vLLM 0.31 pins 0.7.0.post1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Remove the DeepSeek-V4 request-tools placement patch: fixed upstream in vllm#51856, which ships in 0.31. - Remove the NIXL routed-experts bypass: 0.31 captures routed experts only on the V2 runner via the AuxOutput connector, so on V1 the scheduler asserts on the first output. Reject router replay with disaggregated P/D in config validation instead, and always use V2 for routed experts. - Resync the copied LoRAModel.from_local_checkpoint with 0.31's modules_to_save handling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
samsja
marked this pull request as ready for review
October 6, 2026 04:13
This was referenced Oct 6, 2026
garrett361
approved these changes
Oct 6, 2026
garrett361
left a comment
garrett361
left a comment
Contributor
There was a problem hiding this comment.
LGTM. The CI failures are not related
| Router replay works by capturing the expert routing decisions into a buffer. This buffer then gets sent to the trainer, which can use it instead of re-computing the routing. This lowers the trainer↔inference mismatch by an order of magnitude, resulting in more stable training. | ||
|
|
||
| To enable router replay, you can set `inference.vllm.enable_return_routed_experts = true`. Standard deployments use the V2 model runner. Disaggregated P/D remains on V1 because NIXL routed-expert stitching is V1-only. | ||
| To enable router replay, you can set `inference.vllm.enable_return_routed_experts = true`. Capture runs on vLLM's V2 model runner (forced automatically). Router replay is not supported with disaggregated P/D: vLLM does not capture routed experts across NIXL KV transfer. |
Contributor
There was a problem hiding this comment.
Little confused about the wording here. Is this a new constraint that we can't use router replay and NIXL? Or was this also not possible before?
This was referenced Oct 6, 2026
faresobeid
added a commit
that referenced
this pull request
Oct 7, 2026
… 0.31 (#3913) ## Problem `Integration tests (reverse_text_lora)` has failed since the vLLM 0.31 bump (#3888's own CI run, and main's GPU Tests at b9c0bfe). The first `POST /load_lora_adapter` returns `400 Bad Request` and the orchestrator exits. Root cause: vLLM 0.31 added a check to `OpenAIServingModels._check_load_lora_adapter_request`. It rejects a `lora_name` equal to a served base model name ("conflicts with a served base model", 400). vLLM 0.30 had no such check. prime-rl registers its adapter under the model name (`model.name`) on purpose, because every client (rollouts, evals, the renderer client) sends that name. vLLM serves the base model under the same name by default, so the two collide. ## Change No vLLM patch. With LoRA on, the inference server serves the base model under a different name, and the adapter keeps the name clients already send. - `InferenceConfig.to_namespace` sets `served_model_name = [f"{vllm.model}-base"]` when `vllm.enable_lora` is on, unless the user sets `served_model_name` explicitly. Every launch path builds the vLLM args here: local `rl`/`sft`, standalone `uv run inference`, SLURM single/multi-node, and P/D prefill/decode ranks. - `maybe_check_has_model` (startup readiness) also accepts the name as a model's `root`. Before the first weight sync, a LoRA server lists only `<model>-base`, whose `root` is the model path. - The `load_lora_adapter` docstring, an orchestrator comment, and the LoRA section of `docs/advanced.md` now describe the served name. ## Checks - **Nothing on the policy server asks for the base model by name.** Rollouts, evals, and the renderer/prefill scorer all send `model.name`, which is now the adapter. OPD teachers and frozen reference models are separate endpoints. A LoRA server also cannot serve a frozen reference under `model.name`. Before this change that name would already have resolved to the adapter, and the base model can now be reached as `<model>-base`. Admin routes (`/pause`, `/update_weights`, `/load_lora_adapter`, health, liveness) do not take a model name. vllm-router and llm-d configs do not reference model names. Dynamo workers are external, so this change does not affect them. - **The v0 adapter is loaded before the first request.** The orchestrator and online evals wait for the trainer's startup broadcast (v0, or the resume step). They load it through `/load_lora_adapter` before they dispatch any rollout or eval. Readiness is the only earlier check that uses the model name, and the `root` match covers it. - **`/metrics`:** vLLM's `model_name` label becomes `<model>-base`. The inference-metrics collector and the concurrency controller group samples by the `engine` label only, and the dashboard does not read vLLM's `/metrics`. ## Tests - `configs/ci/integration/reverse-text-lora/start.toml` on one H200 node (vLLM 0.31.0, this commit, patch module absent): all 15 steps run and 16/16 `/load_lora_adapter` calls return 200. There are no 400s or 404s. Reward goes from 0.145 to 0.798, matching the patched version (0.14 to 0.79). Before the first sync, `/v1/models` lists only `...-Reverse-Text-SFT-base` (root `...-Reverse-Text-SFT`). After it, the list also has the adapter `...-Reverse-Text-SFT` (parent `...-SFT-base`). - `reverse-text-sft-lora/start.toml` (5 steps) passes. - `pytest tests/unit/orchestrator tests/unit/inference tests/unit/test_configs.py tests/unit/test_parsers.py -m "not gpu"` (469 passed), `ruff check`, and `ruff format --check` all pass. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes how LoRA vLLM instances advertise models at startup and during readiness checks; client-facing model IDs stay the same, but anything expecting the base under the policy name would need `<model>-base`. > > **Overview** > Fixes LoRA inference startup on **vLLM 0.31**, which returns 400 when an adapter is registered under the same name as a served base model. > > When `enable_lora` is on and `served_model_name` is not set explicitly, **`InferenceConfig.to_namespace`** now serves the base weights as ``<vllm.model>-base`` so the reloaded policy adapter can keep **`model.name`** for all rollout/eval clients. **`maybe_check_has_model`** treats a match on either the model **`id`** or **`root`**, so readiness passes before the first `/load_lora_adapter` when only the `-base` entry is listed. Docs and comments describe the naming split. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 18064f3. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
mikasenghaas
added a commit
that referenced
this pull request
Oct 7, 2026
The rebase's conflict resolution kept both sides' patch calls, but monkey_patch_deepseek_v4_request_tools_placement was deleted upstream in the vLLM 0.31.0 bump (#3888) along with its definition -- vLLM handles the placement natively now. The surviving call raised NameError at inference startup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
flashinfer-python0.7.0.post1 from theexclude-newercooldown).enable_return_routed_expertsfromVllmConfig.aux_output_config(moved in 0.31)._patch_lora_key_prefixwith 0.31 (modules_to_save).Removed patches:
monkey_patch_deepseek_v4_request_tools_placement: fixed upstream (vllm#51856).monkey_patch_return_routed_experts_with_nixl_connector: crashes the engine on 0.31.Tested end to end: multi-node RL on SLURM (1 trainer + 1 inference node), Qwen3-0.6B reverse-text, 10 steps. Eval reward 0.08 → 0.81.
🤖 Generated with Claude Code