diff --git a/docs/fern/backends/vllm/vllm-config-reference.mdx b/docs/fern/backends/vllm/vllm-config-reference.mdx index ac5f11046c3d..2e1f17e49827 100644 --- a/docs/fern/backends/vllm/vllm-config-reference.mdx +++ b/docs/fern/backends/vllm/vllm-config-reference.mdx @@ -8,7 +8,7 @@ subtitle: Field reference for the Dynamo-specific CLI flags and environment vari `DynamoVllmConfig` holds the Dynamo-specific configuration for the vLLM backend (`python -m dynamo.vllm`). Every field, type, default, and choice on this page comes from the [`DynamoVllmArgGroup` and `DynamoVllmConfig`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/vllm/backend_args.py) definitions. For features and operational details, see the [vLLM Reference Guide](vllm-reference-guide.md). - These are **only** the Dynamo wrapper flags. The vLLM backend also accepts every native vLLM `EngineArgs` argument (`--model`, `--tensor-parallel-size`, `--max-model-len`, and so on) in the same command, plus the cross-cutting [Dynamo Runtime](../../reference/runtime-config-reference.mdx) flags (`--namespace`, `--endpoint`, and others). Except for the KV event interoperability note below, this page covers neither — only the vLLM-specific `DYN_VLLM_*` surface. + These are **only** the Dynamo wrapper flags. The vLLM backend also accepts every native vLLM `EngineArgs` argument (`--model`, `--tensor-parallel-size`, `--max-model-len`, and so on) in the same command, plus the cross-cutting [Dynamo Runtime](../../reference/runtime-config-reference.mdx) flags (`--namespace`, `--endpoint`, and others). Except for the native KV event and KV transfer sections below, this page covers neither — only the vLLM-specific `DYN_VLLM_*` surface. ## How the config is loaded @@ -72,6 +72,31 @@ The `endpoint` value is the base ZeroMQ port. vLLM assigns each data-parallel ra If workers do not publish KV events, configure the frontend with `--no-router-kv-events` for prediction-based KV routing or `--load-aware` for load-only routing. +## Native KV transfer configuration + +`--kv-transfer-config` is a native vLLM engine argument rather than a `DynamoVllmConfig` field, so it has no `DYN_VLLM_*` environment variable. It selects the KV connector that moves cache blocks between prefill and decode workers. + + +A worker started with `--disaggregation-mode prefill` must be passed `--kv-transfer-config` explicitly. Without it, the worker raises a `ValueError` during argument parsing and never starts. All non-prefill modes — `agg`, `pd`, `decode`, and `encode` — do not enforce this check. + + +The value is a JSON object. For NIXL-based prefill/decode disaggregation: + +```bash +python -m dynamo.vllm \ + --model Qwen/Qwen3-0.6B \ + --disaggregation-mode prefill \ + --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' +``` + +Only the prefill worker is required to set it, but both halves of a NIXL pair must agree on a connector for transfers to succeed. Pass the same `--kv-transfer-config` value to the decode worker, as the [disaggregated vLLM launch script](https://github.com/ai-dynamo/dynamo/blob/release/1.4.0/examples/backends/vllm/launch/disagg.sh) does. + +The earlier `--connector` flag is no longer accepted by the vLLM backend. Setting it — on the command line or through the `DYN_CONNECTOR` environment variable — raises a `ValueError` during argument parsing. The message depends on the value: + +- An active connector, such as `--connector nixl` or `DYN_CONNECTOR=nixl`, reports the equivalent `--kv-transfer-config` JSON to use instead. +- `--connector none` or `--connector null` reports that the flag is no longer needed, because no connector is already the default. There is no equivalent value to migrate to, so none is shown. +- `DYN_CONNECTOR` set to an empty or whitespace-only value reports that the variable is no longer supported, without an equivalent value. + ## Worker role and disaggregation These flags control which role this worker plays in a disaggregated deployment. The default when no `--disaggregation-mode` is set is aggregated (`agg`). @@ -79,6 +104,8 @@ These flags control which role this worker plays in a disaggregated deployment. Worker disaggregation mode. `agg` (default when unset) runs a combined aggregated prefill+decode worker. `pd` is a legacy alias for `agg`. `prefill` and `decode` split the pipeline for prefill/decode disaggregation. `encode` starts a multimodal encode-only worker. + `prefill` additionally requires the native `--kv-transfer-config` argument - see [Native KV transfer configuration](#native-kv-transfer-configuration). + Allowed values: pd agg prefill decode encode Environment variable: `DYN_VLLM_DISAGGREGATION_MODE` @@ -204,18 +231,6 @@ These flags control the self-benchmark sweep that runs on startup before the wor These flags are retained for backward compatibility and will be removed in a future release. Each is mapped to its replacement at startup with a deprecation warning. - - **Deprecated** — use `--disaggregation-mode=prefill`. Enable prefill functionality for this worker. - - Environment variable: `DYN_VLLM_IS_PREFILL_WORKER` - - - - **Deprecated** — use `--disaggregation-mode=decode`. Mark this as a decode worker that does not publish KV events. - - Environment variable: `DYN_VLLM_IS_DECODE_WORKER` - - **Deprecated** — accepted for compatibility with older ModelExpress manifests only. The vLLM ModelExpress plugin reads its own configuration. @@ -224,8 +239,6 @@ These flags are retained for backward compatibility and will be removed in a fut ## Validation rules -- `--disaggregation-mode` cannot be combined with `--is-prefill-worker` or `--is-decode-worker`. -- `--is-prefill-worker` and `--is-decode-worker` cannot both be set at the same time. - `--embedding-worker` is only valid with `--disaggregation-mode=agg` (or the default aggregated mode) and cannot be combined with `--enable-multimodal` or `--benchmark-mode`. ## Related pages