[llm] Upgrade vLLM to 0.31.0

doc/source/serve/llm/troubleshooting.md

@@ -102,7 +102,7 @@ Each Ray release is fully tested with a compatible vLLM version.
 
 | Ray release | vLLM version |
 | ----------- | ------------ |
-| nightly     | 0.30.0       |
+| nightly     | 0.31.0       |
 | 2.59.0      | 0.27.0       |
 | 2.58.0      | 0.26.0       |
 | 2.57.0      | 0.25.1       |

doc/source/serve/llm/user-guides/vllm-compatibility.md

@@ -1,7 +1,7 @@
 ---
 myst:
   html_meta:
-    description: "vLLM features reachable through Ray Serve LLM: embeddings, transcriptions, structured JSON output, and vision language models."
+    description: "vLLM features reachable through Ray Serve LLM: embeddings, transcriptions, structured JSON output, vision language models, and diffusion language models."
 ---
 
 (vllm-compatibility-guide)=
@@ -16,7 +16,7 @@ This compatibility means you can:
 - Switch between `vllm serve` and Ray Serve LLM with no code changes and scale
 - Use Ray Serve's production features such as autoscaling, multi-model serving, and advanced routing
 
-This guide shows how to use vLLM features such as embeddings, structured output, vision language models, and reasoning models with Ray Serve.
+This guide shows how to use vLLM features such as embeddings, structured output, vision language models, diffusion language models, and reasoning models with Ray Serve.
 
 ## Embeddings
 
@@ -357,6 +357,82 @@ for chunk in response:
 
 For a complete list of supported vision models, see the [vLLM multimodal models documentation](https://docs.vllm.ai/en/stable/models/supported_models/#list-of-multimodal-language-models).
 
+## Diffusion language models
+
+Ray Serve LLM supports discrete diffusion models such as DiffusionGemma through vLLM. Set `diffusion_config` in the engine arguments and serve the model with {doc}`direct streaming <direct-streaming>`, so that per-request `vllm_xargs` reach the engine.
+
+### Deploy a diffusion model
+
+::::{tab-set}
+
+:::{tab-item} Server
+:sync: server
+
+```python
+# Run with RAY_SERVE_ENABLE_HA_PROXY=1 and RAY_SERVE_LLM_ENABLE_DIRECT_STREAMING=1.
+from ray import serve
+from ray.serve.llm import LLMConfig, build_openai_app
+
+
+# Configure a diffusion model
+llm_config = LLMConfig(
+    model_loading_config=dict(
+        model_id="diffusiongemma",
+        model_source="google/diffusiongemma-26B-A4B-it",
+    ),
+    engine_kwargs=dict(
+        tensor_parallel_size=4,
+        max_model_len=8192,
+        diffusion_config=dict(canvas_length=64),
+    ),
+)
+
+# Build and deploy the model
+app = build_openai_app({"llm_configs": [llm_config]})
+serve.run(app, blocking=True)
+```
+:::
+
+:::{tab-item} Python Client
+:sync: client
+
+```python
+from openai import OpenAI
+
+# Initialize client
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="fake-key")
+
+# Create and send a chat request
+response = client.chat.completions.create(
+    model="diffusiongemma",
+    messages=[{"role": "user", "content": "Write a haiku about the ocean."}],
+    stream=True,
+)
+
+for chunk in response:
+    if chunk.choices[0].delta.content is not None:
+        print(chunk.choices[0].delta.content, end="", flush=True)
+```
+:::
+
+:::{tab-item} cURL
+:sync: curl
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -H "Authorization: Bearer fake-key" \
+    -d '{
+        "model": "diffusiongemma",
+        "messages": [{"role": "user", "content": "Write a haiku about the ocean."}]
+    }'
+```
+:::
+
+::::
+
+For structured reads and the request options they take, see vLLM's [structured diffusion example](https://github.com/vllm-project/vllm/tree/main/examples/features/structured_diffusion).
+
 ## Reasoning models
 
 Ray Serve LLM supports reasoning models such as DeepSeek-R1 and QwQ through vLLM. These models use extended thinking processes before generating final responses.

docker/ray-llm/Dockerfile

@@ -5,13 +5,12 @@ FROM "$BASE_IMAGE"
 
 COPY python/deplocks/llm/rayllm_*.lock ./
 COPY python/requirements/llm/patches/vllm-device-aware-compile-cache.patch \
-    python/requirements/llm/patches/vllm-preserve-model-eos-tokens.patch \
-    python/requirements/llm/patches/vllm-pixtral-transformers-5.17.patch ./
+    python/requirements/llm/patches/vllm-preserve-model-eos-tokens.patch ./
 COPY python/requirements/llm/nccl_overrides.txt ./
 
 # vLLM version tag to use for EP kernel and DeepGEMM install scripts
 # Keep in sync with vllm version in python/requirements/llm/llm-requirements.txt
-ARG VLLM_SCRIPTS_REF="v0.30.0"
+ARG VLLM_SCRIPTS_REF="v0.31.0"
 
 # Keep in sync with DEEPEP_COMMIT_HASH in vllm's docker/Dockerfile. This is
 # DeepEP V2 ("NCCL Gin"), which needs NCCL >= 2.30.4 at build and run time;
@@ -50,7 +49,6 @@ uv pip install --system --no-cache-dir --no-deps \
 # TODO (jeffreywang): Remove this patch once https://github.com/vllm-project/vllm/pull/38962 lands.
 VLLM_DEVICE_AWARE_COMPILE_CACHE_PATCH="$(pwd)/vllm-device-aware-compile-cache.patch"
 VLLM_PRESERVE_MODEL_EOS_TOKENS_PATCH="$(pwd)/vllm-preserve-model-eos-tokens.patch"
-VLLM_PIXTRAL_TRANSFORMERS_5_17_PATCH="$(pwd)/vllm-pixtral-transformers-5.17.patch"
 VLLM_SITE_PACKAGES="$(python - <<'PY'
 import site
 import sysconfig
@@ -76,8 +74,6 @@ PY
     git apply "${VLLM_DEVICE_AWARE_COMPILE_CACHE_PATCH}"
     # TODO (jeffreywang): Remove this patch when https://github.com/vllm-project/vllm/pull/52153 is merged.
     git apply "${VLLM_PRESERVE_MODEL_EOS_TOKENS_PATCH}"
-    # TODO (jeffreywang): Remove this patch when https://github.com/vllm-project/vllm/pull/56484 is merged.
-    git apply "${VLLM_PIXTRAL_TRANSFORMERS_5_17_PATCH}"
 )
 
 sudo apt-get update -y && sudo apt-get install -y curl kmod pkg-config librdmacm-dev cmake

docker/ray-llm/cuda.wanda.yaml

@@ -6,7 +6,6 @@ srcs:
   - python/deplocks/llm/rayllm_py312_cu130.lock
   - python/requirements/llm/patches/vllm-device-aware-compile-cache.patch
   - python/requirements/llm/patches/vllm-preserve-model-eos-tokens.patch
-  - python/requirements/llm/patches/vllm-pixtral-transformers-5.17.patch
   - python/requirements/llm/nccl_overrides.txt
 build_args:
   - BASE_IMAGE=cr.ray.io/rayproject/ray-py$PYTHON_VERSION-cu$CUDA_VERSION-base$ARCH_SUFFIX