Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 30 additions & 3 deletions container/templates/vllm_runtime.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,13 @@ COPY --chmod=775 --chown=dynamo:0 --from=wheel_builder /opt/dynamo/dist/*.whl /o
{# Inline expression, not a block tag: render.py leaves trim_blocks off, so a tag
on its own line inside the RUN breaks the backslash continuation. #}
{% set vllm_rs_required = "1" if device == "cuda" else "0" %}
{# EngineCore msgpack is positional: an auto-loaded plugin that extends
EngineCoreOutput breaks the `vllm-rs` decoder. See lib/sidecar/vllm/README.md.
Only the targets that install vLLM-Omni carry such a plugin; dev and local-dev
skip that install, so they keep vLLM's own plugin discovery instead of an
allowlist that would name an entry point they never install. #}
{% set vllm_rs_allowlist = "1" if target not in ("dev", "local-dev") else "0" %}
{% set vllm_rs_plugins = "modelexpress" if context.vllm.enable_modelexpress == "true" else "" %}

# The vLLM 0.28.0 release images resolve the unbounded `transformers>=5.5.3`
# requirement to 5.15.1, but vLLM-Omni 0.28.0rc1 caps Transformers below 5.15.
Expand Down Expand Up @@ -532,17 +539,37 @@ if actual != expected:
PY

# `vllm-rs` ships inside the installed `vllm` package, not as a console script;
# linking it keeps the binary at that package's vLLM revision. Fatal on cuda only.
# putting it on PATH keeps the binary at that package's vLLM revision. Fatal on
# cuda only. The shell branches on a rendered flag rather than a `{% raw %}{% if %}{% endraw %}` in the
Comment thread
glamr-agent marked this conversation as resolved.
Resolved
# middle of the `\`-continued command, for the reason given above the vLLM install.
RUN set -eu; \
pkg="$({{ python_executable }} -c 'import os, vllm; print(os.path.dirname(vllm.__file__))')"; \
if [ -f "${pkg}/vllm-rs" ] && [ -x "${pkg}/vllm-rs" ]; then \
ln -sf "${pkg}/vllm-rs" {{ vllm_rs_link }}; \
if [ "{{ vllm_rs_allowlist }}" = "1" ]; then \
printf '%s\n' \
'#!/bin/sh' \
'# vLLM loads every vllm.general_plugins entry point unless VLLM_PLUGINS' \
'# names an allowlist. vLLM-Omni appends three fields to' \
'# vllm.v1.engine.EngineCoreOutput, and EngineCore msgpack is positional,' \
'# so the strict Rust decoder in this binary then rejects every engine' \
'# output as the wrong length. This allowlist is deliberately minimal:' \
'# it excludes every plugin this image does not install on purpose,' \
'# including the LoRA resolvers that vLLM itself ships. An exported' \
'# VLLM_PLUGINS, the empty allowlist included, still wins.' \
'VLLM_PLUGINS="${VLLM_PLUGINS-{{ vllm_rs_plugins }}}"' \
'export VLLM_PLUGINS' \
"exec \"${pkg}/vllm-rs\" \"\$@\"" \
> {{ vllm_rs_link }}; \
chmod 755 {{ vllm_rs_link }}; \
else \
ln -sf "${pkg}/vllm-rs" {{ vllm_rs_link }}; \
fi; \
vllm-rs --help >/dev/null; \
elif [ "{{ vllm_rs_required }}" = "1" ]; then \
echo "ERROR: installed vllm package (${pkg}) ships no executable vllm-rs" >&2; \
exit 1; \
else \
echo "WARNING: installed vllm package (${pkg}) ships no executable vllm-rs; not linking it onto PATH" >&2; \
echo "WARNING: installed vllm package (${pkg}) ships no executable vllm-rs; not putting it onto PATH" >&2; \
Comment thread
glamr-agent marked this conversation as resolved.
Resolved
fi

USER dynamo
Expand Down
16 changes: 15 additions & 1 deletion lib/sidecar/vllm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,20 @@ The official `Qwen/Qwen3-ASR-1.7B` repository currently needs Rust-frontend-comp

The Python `vllm` package and `vllm-rs` must expose compatible EngineCore and gRPC contracts. Prefer artifacts built from the same vLLM source revision; do not combine a Python wheel from one nightly with a `vllm-rs` binary from another. The sidecar's vendored gRPC source revisions are recorded in [`proto/README.md`](proto/README.md).

A matched pair is not sufficient on its own. EngineCore encodes each message as a msgpack array ordered by field position, so a third package that appends fields to `vllm.v1.engine.EngineCoreOutput` makes every element longer than the strict Rust decoder in `vllm-rs` expects, and the engine output fails to decode even though both halves of `vllm` agree. vLLM loads every `vllm.general_plugins` entry point unless `VLLM_PLUGINS` names an allowlist, so such a package reaches the engine processes `vllm-rs` manages without being asked for. vLLM-Omni is one such package: it appends three fields. The Dynamo vLLM runtime image therefore ships `vllm-rs` on `PATH` as a wrapper that sets `VLLM_PLUGINS`.

That allowlist is deliberately minimal: it names only `modelexpress`, the one plugin the image installs on purpose, so it also excludes the two `vllm.general_plugins` entry points vLLM itself ships, `lora_filesystem_resolver` and `lora_hf_hub_resolver`. Those two do not break the decoder, but this protocol does not support LoRA, and `lora_hf_hub_resolver` already declines to register unless it is named in `VLLM_PLUGINS` explicitly. `VLLM_PLUGINS` is also the shared gate for the `vllm.platform_plugins`, `vllm.io_processor_plugins`, `vllm.stat_logger_plugins`, and `vllm.endpoint_plugins` groups, so a plugin you add in any of them needs naming too. In an image built without ModelExpress the wrapper sets `VLLM_PLUGINS` to the empty string, which vLLM reads as an allowlist matching no plugin rather than as unset, so nothing loads. Export `VLLM_PLUGINS` yourself to change any of this; the wrapper keeps whatever you set, including the empty value.
Comment thread
glamr-agent marked this conversation as resolved.
Resolved

The `dev` and `local-dev` images install neither vLLM-Omni nor ModelExpress, so they have no plugin to exclude and nothing to allow. They link `vllm-rs` onto `PATH` directly and leave `VLLM_PLUGINS` unset, which keeps vLLM's default discovery and lets a plugin you install into a dev image load without being named.

Invoking the binary by its path inside the `vllm` package bypasses that wrapper, so set `VLLM_PLUGINS` yourself when you do:

```bash
VLLM_PLUGINS=modelexpress \
"$(python3 -c 'import os, vllm; print(os.path.dirname(vllm.__file__))')/vllm-rs" \
serve Qwen/Qwen3-0.6B --host 127.0.0.1 --grpc-port 50051
```

Start vLLM with its gRPC listener:

```bash
Expand Down Expand Up @@ -180,7 +194,7 @@ command.

The sidecar waits for both the Control and Inference services through the standard gRPC health API before registering the worker. The deployment manifests retain lightweight socket probes for container lifecycle monitoring. The engine image must include a `vllm-rs` build compatible with the vendored protocol.

The Dynamo vLLM CUDA runtime image exposes `vllm-rs` on `PATH`, linked from the `vllm` package that image installs, so `vllm-rs serve` runs there by name; that image fails to build if its `vllm` package ever stops shipping the binary. The XPU and CPU variants build from separate upstream vLLM distributions and link it the same way when it is present, but only warn when it is not, so check `command -v vllm-rs` before relying on it there. A stock upstream engine image ships the same binary inside the package but leaves it off `PATH`; `deploy/agg.yaml` and `deploy/disagg.yaml` run that image and resolve the path out of the package themselves.
The Dynamo vLLM CUDA runtime image exposes `vllm-rs` on `PATH` through the wrapper script described under "Runtime compatibility" above, which resolves the binary out of the `vllm` package that image installs, so `vllm-rs serve` runs there by name; that image fails to build if its `vllm` package ever stops shipping the binary. The XPU and CPU variants build from separate upstream vLLM distributions and install the same wrapper when the binary is present, but only warn when it is not, so check `command -v vllm-rs` before relying on it there. A stock upstream engine image ships the same binary inside the package but leaves it off `PATH`; `deploy/agg.yaml` and `deploy/disagg.yaml` run that image and resolve the path out of the package themselves.

### Prerequisites

Expand Down
Loading