From 8915d200b1bcd152ce7d7bdea14df109c99c0e52 Mon Sep 17 00:00:00 2001 From: krishung5 Date: Wed, 16 Sep 2026 13:38:27 -0700 Subject: [PATCH 1/2] docs(mm-routing): document video KV routing Signed-off-by: krishung5 --- .../backends/vllm/multimodal.md | 20 ++-- .../pages/reference/general/compatibility.mdx | 2 +- .../multimodal-kv-routing.md | 105 +++++++++++++++--- 3 files changed, 100 insertions(+), 27 deletions(-) diff --git a/docs/fern/pages/developer-guide/knowledge-base/modular-components/backends/vllm/multimodal.md b/docs/fern/pages/developer-guide/knowledge-base/modular-components/backends/vllm/multimodal.md index e7590e9f3580..d2a93925c33e 100644 --- a/docs/fern/pages/developer-guide/knowledge-base/modular-components/backends/vllm/multimodal.md +++ b/docs/fern/pages/developer-guide/knowledge-base/modular-components/backends/vllm/multimodal.md @@ -63,16 +63,16 @@ The aggregated vLLM worker can load an author-provided vision tower in process, ## Multimodal KV Routing -vLLM supports two multimodal KV-routing paths. Both give the router and vLLM the same image identity so requests can be placed on workers that already cache the image's KV blocks. +vLLM supports two multimodal KV-routing paths. Both give the router and vLLM the same media identity so requests can be placed on workers that already cache the image or video's KV blocks. ### Choose a Routing Path | Consideration | Default Rust Frontend | Python Chat Processor | |---------------|-----------------------|-----------------------| -| Frontend work | Hashes media and calculates the routing token layout | Runs vLLM's full Hugging Face multimodal processor | +| Frontend work | Hashes media and calculates the routing token layout; exact video routing also samples and decodes the video | Runs vLLM's full Hugging Face multimodal processor | | Worker work | Processes the original multimodal input | Consumes processed `mm_kwargs` when transfer succeeds | | Model coverage | Models registered in Dynamo's `llm-multimodal` registry | Models supported by vLLM's multimodal processor | -| Data sent to worker | Original media reference plus `mm_hashes` | Processed inputs over shared memory or NIXL, with media fallback when available | +| Data sent to worker | Original image reference or frontend-decoded media, plus modality-grouped hashes | Processed inputs over shared memory or NIXL, with media fallback when available | | Best fit | Lowest frontend overhead for a registered model | Broader model coverage or centralized preprocessing | Start with the default path when the model is in Dynamo's Rust registry. It avoids running the full Hugging Face processor in the frontend and does not require a processed-input transfer channel. @@ -83,14 +83,16 @@ Use the Python chat processor when the Rust registry does not recognize the mode The default path keeps multimodal processing on the worker: -1. The frontend computes an `mm_hash` for each image. -2. A model-specific processor specification resolves the image placeholder and calculates its expanded token count. +1. The frontend computes an `mm_hash` for each supported media object. +2. A model-specific processor specification resolves the media placeholder and calculates its expanded token layout. 3. The frontend expands the placeholder in a routing-only token view and builds per-block multimodal metadata. 4. The KV router credits that overlap in its combined prefill-and-decode cost and selects the lowest-cost eligible worker. -5. The frontend forwards `mm_hashes`, which the worker passes to vLLM as `multi_modal_uuids`. +5. The frontend forwards image hashes through `mm_hashes` and grouped image/video hashes through `mm_hashes_by_modality`. The worker passes them to vLLM as `multi_modal_uuids`. By default, the frontend hashes the exact full URI: the complete `data:` URI string or the HTTP URL including its query string. Set `--frontend-decoding` on the worker to register frontend media decoding and use decoded image content as the hash input. Content-addressed hashing lets different URLs for identical image bytes share a routing key. +Exact video routing on this path requires `--frontend-decoding`. The frontend hashes sampled RGB frames and their model-visible metadata, then uses a model adapter to reproduce the worker's prompt expansion. Supported adapters cover Qwen3-VL, Qwen3.5, and Nemotron 3 Nano Omni. See [Multimodal KV Routing](../../../../../use-cases/multimodal-serving/multimodal-kv-routing.md#video-identity-and-token-layout) for setup, verification, and fallback behavior. + Launch the default path: ```bash @@ -106,18 +108,18 @@ Key settings: | `NUM_WORKERS` | `2` | Number of backend workers | | `BLOCK_SIZE` | `16` | Shared frontend and worker KV block size | | `GPU_MEMORY_UTILIZATION` | `0.20` | Per-worker GPU memory fraction | -| `VLLM_EXTRA_ARGS` | unset | Additional worker flags; set `--frontend-decoding` for content-addressed image hashes | +| `VLLM_EXTRA_ARGS` | unset | Additional worker flags; set `--frontend-decoding` for content-addressed image hashes and exact video routing | ### Python Chat Processor -Use the Python path when the model is supported by vLLM but not by the Rust model registry, or when the frontend should preprocess images: +Use the Python path when the model is supported by vLLM but not by the Rust model registry, or when the frontend should preprocess media: ```bash cd $DYNAMO_HOME bash examples/backends/vllm/launch/agg_multimodal_router_chat_processor.sh ``` -This launcher sets `--dyn-chat-processor vllm`. The frontend runs vLLM's Hugging Face processor, extracts hashes and expanded multimodal inputs, builds routing metadata, and transfers processed `mm_kwargs` to the selected worker. This path supports any VLM handled by vLLM's multimodal processor. +This launcher sets `--dyn-chat-processor vllm`. The frontend runs vLLM's Hugging Face processor, extracts hashes and expanded multimodal inputs for images and videos, builds routing metadata, and transfers processed `mm_kwargs` to the selected worker. This path supports models handled by vLLM's multimodal processor. `DYNAMO_MM_TRANSFER` selects the transfer mechanism: diff --git a/docs/fern/pages/reference/general/compatibility.mdx b/docs/fern/pages/reference/general/compatibility.mdx index fe63aea4f3ac..d1b3346dacd9 100644 --- a/docs/fern/pages/reference/general/compatibility.mdx +++ b/docs/fern/pages/reference/general/compatibility.mdx @@ -56,7 +56,7 @@ vLLM offers the broadest feature coverage in Dynamo, with full support for disag | **KV-Aware Routing** | ✓ | | | **SLA-Based Planner** | ✓ | | | **KV Block Manager** | ✓ | | -| **Multimodal** | ✓ | Image + video; audio experimental (Qwen2-Audio). With KV-aware routing, image-aware routing on documented paths ([Source][mm-kv-routing]) | +| **Multimodal** | ✓ | Image + video; audio experimental (Qwen2-Audio). With KV-aware routing, exact image routing and model-specific video routing on documented paths ([Source][mm-kv-routing]) | | **Request Migration** | ✓ | | | **Request Cancellation** | ✓ | | | **LoRA** | ✓ | Dynamic load/unload; KV-aware routing supports adapter affinity | diff --git a/docs/fern/pages/use-cases/multimodal-serving/multimodal-kv-routing.md b/docs/fern/pages/use-cases/multimodal-serving/multimodal-kv-routing.md index 6da6bcdb8815..43c7f6d1b700 100644 --- a/docs/fern/pages/use-cases/multimodal-serving/multimodal-kv-routing.md +++ b/docs/fern/pages/use-cases/multimodal-serving/multimodal-kv-routing.md @@ -2,14 +2,14 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 title: Multimodal KV Routing -subtitle: Include multimodal identity in the router's combined cache-and-load cost +subtitle: Include image and video identity in the router's combined cache-and-load cost --- ## Overview -Multimodal KV routing extends Dynamo's KV-aware router to account for image content when calculating cache overlap. The frontend assigns each image a stable hash and includes its identity in the routing view of the prompt. +Multimodal KV routing extends Dynamo's KV-aware router to account for image or video content when calculating cache overlap. The frontend assigns each supported media object a stable hash and includes its identity in the routing view of the prompt. -When an image appears again, its identity contributes to each worker's KV overlap. The router balances that cache credit against projected prefill and decode load, so the worker with the largest overlap does not necessarily win when it is busy. Cache-aware placement increases prefix reuse without abandoning load balancing. +When media appears again, its identity contributes to each worker's KV overlap. The router balances that cache credit against projected prefill and decode load, so the worker with the largest overlap does not necessarily win when it is busy. Cache-aware placement increases prefix reuse without abandoning load balancing. > [!IMPORTANT] > The KV cache stores attention key/value state so a worker can skip repeated prefill work. The embedding cache stores vision encoder outputs so the encoder can skip repeated image processing. You can use both features together. See [Embedding Cache](embedding-cache.md). @@ -19,18 +19,20 @@ When an image appears again, its identity contributes to each worker's KV overla Use multimodal KV routing when: - Multiple backend workers serve multimodal requests. -- Images repeat across requests, such as product photos or shared reference images. +- Images or videos repeat across requests, such as product photos, shared reference images, or several questions about the same video. - You want to maximize KV cache reuse for multimodal content. -Single-worker deployments do not need routing, and workloads with entirely unique images receive little image-specific cache benefit. +Single-worker deployments do not need routing, and workloads with entirely unique media receive little media-specific cache benefit. Exact video routing decodes and samples the video in the frontend before worker selection, so use it when the expected KV reuse offsets that frontend work. ## How It Works The routing flow in general has three steps: -1. The frontend computes a stable identity for each image. -2. The frontend represents the image in a routing-only token view that matches the backend's cache identity. -3. The KV router includes that overlap in its combined cache-and-load score, selects the lowest-cost eligible worker, and forwards the same image identity. +1. The frontend computes a stable identity for each supported media object. +2. The frontend represents the media in a routing-only token view that matches the backend's cache identity and token layout. +3. The KV router includes that overlap in its combined cache-and-load score, selects the lowest-cost eligible worker, and forwards the same media identity. + +### Image Identity By default, an HTTP image's identity hashes the exact URL bytes, including its query string. Reusing the identical URL produces the same identity, but two URLs @@ -39,27 +41,37 @@ content-stable identity across URLs; the frontend then hashes decoded image content. Data URLs follow the same rule: the default path hashes the full data URI string, while frontend decoding hashes the decoded bytes. +### Video Identity and Token Layout + +Exact video routing requires frontend decoding. The frontend samples the video and computes an XXH3-64 identity over the sampled RGB frames, decoded shape and data type, and video metadata such as the source frame rate and sampled timestamps. Two URLs that decode to the same sampled content and metadata receive the same identity. + +Video prompt expansion is model-specific. The frontend uses a model adapter to reproduce the worker's resize, patch, temporal grouping, delimiter, timestamp, and pruning behavior in a routing-only token sequence. At worker startup, vLLM reports the installed processor behavior that affects this sequence. Dynamo disables exact video routing if the reported contract does not match the frontend adapter. + +The selected worker receives the same per-video hash, and its KV events use that identity. This keeps the frontend's routing blocks aligned with the blocks published by vLLM without running the vision model in the frontend. + - vLLM provides two routing paths. Use the default Rust frontend for supported model families when you want minimal frontend processing. Use the Python chat processor when you need vLLM's broader model support or want the frontend to preprocess images and transfer the processed inputs to workers. + vLLM provides two routing paths. Use the default Rust frontend for supported model families when you want minimal frontend processing. Use the Python chat processor when you need vLLM's broader model support or want the frontend to preprocess media and transfer the processed inputs to workers. **Default Rust frontend** - The frontend calculates only the image identity and routing token layout. The selected worker still runs the model's multimodal processor. This path has lower frontend overhead, but multimodal routing depends on the model being registered with Dynamo's Rust processor registry. + For images, the frontend calculates only the identity and routing token layout unless frontend decoding is enabled. For videos, exact routing requires `--frontend-decoding` so the frontend can sample the video and build its model-visible token layout. The selected worker still runs the model's multimodal processor. This path depends on the model being registered with Dynamo's Rust processor registry. **Alternative: Python chat processor** - With `--dyn-chat-processor vllm`, the frontend runs vLLM's full multimodal processor. It supports models known to vLLM without requiring a Dynamo Rust processor specification and can transfer processed inputs through shared memory or NIXL. This shifts preprocessing and transfer work to the frontend. + With `--dyn-chat-processor vllm`, the frontend runs vLLM's full multimodal processor. It derives routing positions from vLLM's processed image or video features without requiring a Dynamo Rust processor specification and can transfer processed inputs through shared memory or NIXL. This shifts preprocessing and transfer work to the frontend. The frontend hashes each image and converts that hash into the same `pad_value` token that SGLang RadixAttention uses for its prefix-cache key. The matching token view lets the router measure image overlap before selecting a worker. - Dynamo's SGLang image includes the required hash-forwarding support. + Dynamo's SGLang image includes the required hash-forwarding support. Exact video routing is not available on this path; video requests use text-prefix routing. The frontend hashes each image, represents that identity in its routing token view, and forwards the hash as `multi_modal_uuids`. TensorRT-LLM workers publish matching KV events so the router can identify cached image blocks. + Exact video routing is not available on this path; video requests use text-prefix routing. + @@ -76,6 +88,18 @@ URI string, while frontend decoding hashes the decoded bytes. bash examples/backends/vllm/launch/agg_multimodal_router.sh ``` + To enable exact video routing on the default path, add frontend decoding: + + ```bash + cd $DYNAMO_HOME + MODEL=Qwen/Qwen3-VL-2B-Instruct \ + NUM_WORKERS=2 \ + VLLM_EXTRA_ARGS="--frontend-decoding" \ + bash examples/backends/vllm/launch/agg_multimodal_router.sh + ``` + + `DYN_MM_VIDEO_NUM_FRAMES` controls the maximum number of sampled frames and defaults to `32`. Keep this setting consistent across workers. The frontend passes the decoded frames and sampling metadata to the selected worker, so the worker processes the same view used for routing. + **Alternative: Python chat processor** ```bash @@ -107,13 +131,60 @@ URI string, while frontend decoding hashes the decoded bytes. +## Send a Video Request + +The standard OpenAI chat payload uses a `video_url` content part: + +```bash +VIDEO_URL="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/draw.mp4" + +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d @- <` placeholder for image and video features, so its exact path currently supports one video-only media object per request; mixed image-and-video or multiple-video Nemotron requests fall back. + ## Support Matrix | Backend | Routing Path | Status | Notes | |---------|--------------|--------|-------| -| [vLLM](../../developer-guide/knowledge-base/modular-components/backends/vllm/multimodal.md#multimodal-kv-routing) | Rust frontend (default) | Yes | Supported families include Qwen2-VL, Qwen2.5-VL, Qwen3-VL, LLaVA 1.5, LLaVA-NeXT, Llama 4, Kimi K2.5/K2.6, Qwen3.5, Qwen3.6, and Nemotron 3 Nano Omni. The rest use text-prefix-only routing. | -| [vLLM](../../developer-guide/knowledge-base/modular-components/backends/vllm/multimodal.md#multimodal-kv-routing) | Python chat processor | Yes | Uses vLLM’s own multimodal processor — supports any VLM that vLLM supports. | -| [SGLang](../../developer-guide/knowledge-base/modular-components/backends/sglang/multimodal.md#multimodal-kv-routing) | Rust frontend (default) | Yes | Hash forwarding is upstream in SGLang 0.5.13+; Dynamo pins 0.5.19. Older custom installations need the upstream patch. | -| [TensorRT-LLM](../../developer-guide/knowledge-base/modular-components/backends/tensorrt-llm/multimodal.md#multimodal-kv-routing) | Rust frontend (default) | Yes | Supported model scope is the Qwen2-VL family (Qwen2-VL / Qwen2.5-VL / Qwen3-VL) and Kimi (Kimi-K2.5 / Kimi-K2.6). Other multimodal models fall back to text-prefix routing. | +| [vLLM](../../developer-guide/knowledge-base/modular-components/backends/vllm/multimodal.md#multimodal-kv-routing) | Rust frontend (default) | Yes | Exact image routing supports Qwen2-VL, Qwen2.5-VL, Qwen3-VL, LLaVA 1.5, LLaVA-NeXT, Llama 4, Kimi K2.5/K2.6, Qwen3.5, Qwen3.6, and Nemotron 3 Nano Omni. Exact video routing requires frontend decoding and supports Qwen3-VL, Qwen3.5, and Nemotron 3 Nano Omni. Other media layouts use text-prefix routing. | +| [vLLM](../../developer-guide/knowledge-base/modular-components/backends/vllm/multimodal.md#multimodal-kv-routing) | Python chat processor | Yes | Uses vLLM's multimodal processor to derive image and video feature hashes and positions for models supported by vLLM. | +| [SGLang](../../developer-guide/knowledge-base/modular-components/backends/sglang/multimodal.md#multimodal-kv-routing) | Rust frontend (default) | Images | Hash forwarding is upstream in SGLang 0.5.13+; Dynamo pins 0.5.19. Video requests use text-prefix routing. | +| [TensorRT-LLM](../../developer-guide/knowledge-base/modular-components/backends/tensorrt-llm/multimodal.md#multimodal-kv-routing) | Rust frontend (default) | Images | Exact image routing supports the Qwen2-VL family and Kimi K2.5/K2.6. Video requests and other models use text-prefix routing. | -Nemotron 3 Nano Omni exact image routing currently requires the vLLM backend. Other backends fall back to text-prefix routing for this model. +Nemotron 3 Nano Omni exact image and video routing currently requires the vLLM backend. Other backends fall back to text-prefix routing for this model. From 593045c50bd99c5ee3c5fa64a1d483f489b15bc6 Mon Sep 17 00:00:00 2001 From: krishung5 Date: Wed, 16 Sep 2026 15:02:14 -0700 Subject: [PATCH 2/2] docs(mm-routing): correct video codec guidance Signed-off-by: krishung5 --- .../use-cases/multimodal-serving/multimodal-kv-routing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/fern/pages/use-cases/multimodal-serving/multimodal-kv-routing.md b/docs/fern/pages/use-cases/multimodal-serving/multimodal-kv-routing.md index 43c7f6d1b700..7f5315d196e2 100644 --- a/docs/fern/pages/use-cases/multimodal-serving/multimodal-kv-routing.md +++ b/docs/fern/pages/use-cases/multimodal-serving/multimodal-kv-routing.md @@ -136,7 +136,7 @@ The selected worker receives the same per-video hash, and its KV events use that The standard OpenAI chat payload uses a `video_url` content part: ```bash -VIDEO_URL="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/draw.mp4" +VIDEO_URL="https://interactive-examples.mdn.mozilla.net/media/cc0-videos/flower.webm" curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ @@ -155,7 +155,7 @@ curl http://localhost:8000/v1/chat/completions \ JSON ``` -Send the request twice to populate and then reuse the worker's KV cache. Video decode requires H.264 or H.265 input and an NVIDIA GPU with a video decode engine. See [Video Decode GPU Requirements](video-decode-gpu-requirements.md). +Send the request twice to populate and then reuse the worker's KV cache. Exact routing through the default Rust frontend supports VP8 or VP9 video in MP4, WebM, or MKV containers. The in-tree frontend decoder does not support H.264 or H.265. Re-encode those inputs to VP9, or disable frontend decoding and let the backend decode them through NVDEC or an installed software decoder; with backend-only decoding, video identity does not contribute to exact routing. See [Additional Media Decoders](additional-media-decoders.md) and [Video Decode GPU Requirements](video-decode-gpu-requirements.md). ## Verify Video Routing