[llm] Upgrade vLLM to 0.31.0 by jeffreywang88 · Pull Request #66720 · ray-project/ray

jeffreywang88 · 2026-10-05T09:30:51Z

Description

As title.

Drop Pixtral patch: vLLM 0.31.0 ships the transformers 5.17 Pixtral fix ([CI] Bump Transformers version to 5.17.0 vllm-project/vllm#56108).
KV router tokenizer: forward the new trust_request_mm_kwargs and tool_strict_level frontend args to OnlineRenderer.
Tests: fix OnlineRenderer mocks for 0.31.0; add a direct-streaming release test for DiffusionGemma structured reads ([Core] structured generation mode for DiffusionGemma model (Jev-like) vllm-project/vllm#57250).
Docs: add a diffusion language models section to vLLM compatibility; bump nightly to vLLM 0.31.0.

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

gemini-code-assist

Code Review

This pull request upgrades vLLM from version 0.30.0 to 0.31.0, updating lockfiles, requirements, and Docker configurations accordingly, and removes the Pixtral transformers patch. It also updates the tokenizer initialization to support new vLLM arguments: trust_request_mm_kwargs and tool_strict_level. The feedback recommends defensively mocking the new tool_strict_level attribute on hand-built OnlineRenderer instances in the test files to prevent potential AttributeErrors, similar to how trust_request_mm_kwargs was mocked.

gemini-code-assist · 2026-10-05T09:32:20Z

            online_renderer.OnlineRenderer
        )
        renderer.renderer = Renderer()
+        renderer.trust_request_mm_kwargs = False


Since tool_strict_level is also a new attribute introduced in vLLM 0.31.0's OnlineRenderer, we should defensively mock it on this hand-built OnlineRenderer instance to prevent potential AttributeErrors when rendering chat/tools.

renderer.trust_request_mm_kwargs = False renderer.tool_strict_level = "off"

gemini-code-assist · 2026-10-05T09:32:20Z

            online_renderer.OnlineRenderer
        )
        renderer.renderer = Renderer()
+        renderer.trust_request_mm_kwargs = False


Since tool_strict_level is also a new attribute introduced in vLLM 0.31.0's OnlineRenderer, we should defensively mock it on this hand-built OnlineRenderer instance to prevent potential AttributeErrors when rendering chat/tools.

Suggested change

renderer.trust_request_mm_kwargs = False

renderer.trust_request_mm_kwargs = False

renderer.tool_strict_level = "off"

gemini-code-assist · 2026-10-05T09:32:20Z

    request.build_chat_params = lambda *args: ChatParams()
    renderer = online_renderer.OnlineRenderer.__new__(online_renderer.OnlineRenderer)
    renderer.renderer = Renderer()
+    renderer.trust_request_mm_kwargs = False


Since tool_strict_level is also a new attribute introduced in vLLM 0.31.0's OnlineRenderer, we should defensively mock it on this hand-built OnlineRenderer instance to prevent potential AttributeErrors when rendering chat/tools.

renderer.trust_request_mm_kwargs = False renderer.tool_strict_level = "off"

jeffreywang88 · 2026-10-05T13:24:19Z

KV router: https://buildkite.com/ray-project/release/builds/109488/table

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 · 2026-10-05T19:46:42Z

Running release tests with A10 instead. AWS is running out of L4 capacities.

llm-serve & llm-batch: https://buildkite.com/ray-project/release/builds/109573/table
KV router tests: https://buildkite.com/ray-project/release/builds/109575/table

-> No luck with A10 either :(

jeffreywang88 · 2026-10-05T21:28:47Z

llm-serve H100: https://buildkite.com/ray-project/release/builds/109647/canvas?sid=01a10e46-43dc-4831-8147-fd0aaae48dcb&open=false

jeffreywang88 · 2026-10-06T19:19:57Z

Trying with another H100 pool: https://buildkite.com/ray-project/release/builds/109824/table (can't provision necessary IAM permissions)

jeffreywang88 · 2026-10-07T05:46:23Z

Another try -- llm-serve & llm-batch with concurrency cap 4 and llm-team-queue: https://buildkite.com/ray-project/release/builds/109935/table.

jeffreywang88 · 2026-10-07T17:24:03Z

I ran release tests with the usual setup which acquires on-demand AWS L4 instances and with our H100 pool (but missing some credentials). None of the failures in either run are related to this PR, and the union of passed tests across both runs covers all tests.

Why did I run with two different setups? AWS is suffering from insufficient L4/A10 capacities.

Running tests with usual setup on this PR branch: https://buildkite.com/ray-project/release/builds/109543/canvas:

Running tests with H100 pool on branch vllm-0.31.0-h100: https://buildkite.com/ray-project/release/builds/109935/canvas. None of the failures are related to changes in this PR.
- HF token secret access denied: qwen tp1, qwen tp1 prefix-cache, llama 8B tp2, llama 1B no-accelerator. These tests read llm_release_test_hf_token from AWS Secrets Manager which the pod role has no access to it.
- nginx 413, request too large: llama 1B S3, qwen 1p1d, qwen 1p1d direct-streaming, qwen 2p6d, qwen 2p6d direct-streaming. The test_too_long_completion_request probe sends a ~1.2 MB prompt and expects vLLM to reject it with a 400. The new cluster's nginx has a 1 MB request limit and rejects it first with a 413.
- No free H100s, cluster never started: qwen tp1 direct-streaming, llama 8B tp2 direct-streaming, llama LoRA. All 4 H100 nodes were full. The recent config makes each of these tests hold a 4×H100 node for the orchestrating job even though only its Service needs GPUs.
- Service replica failed to start (likely S3 access denied): llama 1B S3 runai streamer. This test loads the model from s3://air-example-data with the pod's AWS credentials, which the new pod role doesn't allow.

jeffreywang88 · 2026-10-07T20:13:44Z

That said, this PR is ready to merge despite the release test failures. I reran the tests on a separate H100 pool and verified that every test passes in at least one of the two environments (ref).

I haven't run the kv_router tests, which are now set to manual frequency regardless. I'll follow up with another PR to update our release test configuration to ensure sufficient GPU capacity and address any kv_router test failures.

cc: @eicherseiji @elliot-barn

eicherseiji

🐐

eicherseiji · 2026-10-07T23:20:32Z

Should we move this to the Ray Serve LLM index docs page instead

sounds good given that lots of people are asking for it, will do in a separate PR

eicherseiji · 2026-10-07T23:23:32Z

 prometheus-fastapi-instrumentator>=8.0.0  # for starlette v1.0.1+
 # Keep NIXL in sync with vLLM's kv_connectors requirements.
-# https://github.com/vllm-project/vllm/blob/v0.30.0/requirements/kv_connectors.txt#L4
+# https://github.com/vllm-project/vllm/blob/v0.31.0/requirements/kv_connectors.txt#L4


Now it is #L5 haha

apparently claude didn't look at the source code. anyway, I'd rather not rerun premerge since it took 30 hours just to acquire a GPU instance last time...

eicherseiji · 2026-10-07T23:25:42Z

+:::{tab-item} Server
+:sync: server
+
+```python


Is there a blocker to making this a doctest

i'll convert all code snippets in this page to doc tests in a follow-up

[llm] Upgrade vLLM to 0.31.0

9b3f8b0

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 requested review from a team, edoakes and richardliaw as code owners October 5, 2026 09:30

jeffreywang88 added the go add ONLY when ready to merge, run all tests label Oct 5, 2026

gemini-code-assist Bot reviewed Oct 5, 2026

View reviewed changes

ray-gardener Bot added serve Ray Serve Related Issue llm labels Oct 5, 2026

[llm] Add DiffusionGemma structured read test and docs

e384998

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 requested review from a team as code owners October 5, 2026 18:35

eicherseiji approved these changes Oct 7, 2026

View reviewed changes

eicherseiji reviewed Oct 7, 2026

View reviewed changes

elliot-barn approved these changes Oct 8, 2026

View reviewed changes

elliot-barn merged commit 0fe2388 into master Oct 8, 2026
5 of 6 checks passed

elliot-barn deleted the vllm-0.31.0 branch October 8, 2026 03:19

	renderer.trust_request_mm_kwargs = False
	renderer.trust_request_mm_kwargs = False
	renderer.tool_strict_level = "off"

Conversation

jeffreywang88 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

jeffreywang88 commented Oct 5, 2026

Uh oh!

jeffreywang88 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffreywang88 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffreywang88 commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffreywang88 commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffreywang88 commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffreywang88 commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eicherseiji left a comment

Choose a reason for hiding this comment

Uh oh!

eicherseiji Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

jeffreywang88 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

eicherseiji Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

jeffreywang88 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

eicherseiji Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

jeffreywang88 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jeffreywang88 commented Oct 5, 2026 •

edited

Loading

jeffreywang88 commented Oct 5, 2026 •

edited

Loading

jeffreywang88 commented Oct 5, 2026 •

edited

Loading

jeffreywang88 commented Oct 6, 2026 •

edited

Loading

jeffreywang88 commented Oct 7, 2026 •

edited

Loading

jeffreywang88 commented Oct 7, 2026 •

edited

Loading

jeffreywang88 commented Oct 7, 2026 •

edited

Loading