fix(vllm): support unified multimodal routing

[maheshrbapatu]

Summary

Support modality-aware multimodal routing metadata for Dynamo's vLLM backend path.

Gemma 4 unified multimodal models can carry image, video, and audio features through the same request path. The previous forwarding path treated multimodal cache identity as image-only, which could produce incorrect KV-cache routing and backend metadata.

• Group frontend multimodal routing metadata by modality.
• Keep forwarded multimodal UUID/hash values canonical; modality is represented by the multi_modal_uuids dictionary key.
• Consume modality-keyed UUIDs and received multimodal kwargs in the vLLM backend while preserving the legacy image-only fallback.
• Keep computed fallback UUIDs limited to aggregated mode; EPD requests continue without computed UUIDs.
• Handle Gemma 4's is_embed feature mask in the shared routing utility and make the multimodal router E2E model-independent.
• Add focused coverage for modality grouping, canonical hashes, EPD/aggregated behavior, Gemma feature masks, and backend video routing.

Validation

• Ruff checks: passed.
• Black checks: passed.
• git diff --check: passed.
• Focused common routing, frontend processor, and vLLM handler suite: 33 passed.
• google/gemma-4-12B-it router E2E, test_vllm_mm_overlap_all[shm]: 1 passed with all 29 overlap observations succeeding.

Where should the reviewer start?

• components/src/dynamo/common/multimodal/routing_utils.py
• components/src/dynamo/frontend/vllm_processor.py
• components/src/dynamo/vllm/handlers.py
• tests/mm_router/test_vllm_mm_router_e2e.py

Related Issues

• Fixes [CONTRIBUTION]: Support Gemma4-12B unified multimodal routing in vLLM #10374

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi maheshrbapatu! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

Linter diff in the way? Review this PR in Change Stack to focus on meaningful changes and expand context only when needed.

Walkthrough

This PR adds multimodal feature metadata grouping and single-modality transfer selection to the frontend, and backend support for consuming grouped metadata with fallback UUID namespacing. Frontend vLLM processor extracts per-modality hashes/placeholders and gates mm_kwargs transfer to single-modality requests. Backend handlers normalize modality keys, parse modality-specific values from extra_args, reconstruct fallback UUIDs with namespace prefixes, and validate placeholder embed-mask correctness.

Changes

Multimodal Feature Metadata Extraction and Transfer Selection

Layer / File(s)	Summary
Frontend metadata helpers components/src/dynamo/frontend/vllm_processor.py	New internal helpers extract feature modality (defaulting to "image"), serialize placeholders with is_embed tensor-to-list normalization, group hashes/placeholders by modality with legacy flat-image-only fallback, and detect single transfer modality from feature set.
Frontend metadata grouping tests components/src/dynamo/frontend/tests/test_vllm_processor_unit.py	Unit tests validate _group_mm_feature_metadata output (flat vs grouped), placeholder serialization including is_embed, feature skipping when hash missing, and _single_transfer_modality rejection of mixed modalities.
Frontend routing and transfer selection components/src/dynamo/frontend/vllm_processor.py	_prepare_mm_routing populates extra_args with modality-grouped metadata (mm_hashes_by_modality, mm_placeholders_by_modality); mm_kwargs transfer selects transfer_modality and only initializes sender for single-modality requests (mixed modalities skip transfer).
Frontend routing integration test components/src/dynamo/frontend/tests/test_vllm_processor_unit.py	Async test verifies _prepare_mm_routing with mixed image/audio features skips mm_kwargs transfer and populates both flat and grouped metadata in extra_args.
Backend multimodal helpers components/src/dynamo/vllm/handlers.py	New helpers normalize modality keys for unified-vision-chunk models, build forwarded UUID/hash maps from extra_args, namespace fallback UUIDs with "dynamo-backend-fallback:" prefix, extract modality-specific values (grouped/flat fallback), and parse placeholder ranges with is_embed bitmask validation.
Backend receive path modality handling components/src/dynamo/vllm/handlers.py	_receive_mm_kwargs computes backend_modality, sources mm_hashes/mm_placeholders from modality-specific extra_args keys, parses placeholders via new parser supporting tuple and dict styles with is_embed validation, and keys mm_hashes_dict by backend_modality.
Backend fallback prompt construction components/src/dynamo/vllm/handlers.py	_build_prompt_from_request fallback path builds forwarded MM UUIDs and namespaces them before passing multi_modal_uuids to vLLM (replaces direct mm_hashes mapping for mixed-modality scenarios).
Backend handler tests components/src/dynamo/vllm/tests/test_vllm_video_handler.py	New tests validate fallback namespace grouping without cache-key collisions, legacy flat metadata fallback treats hashes as image-only (audio/video return None), and placeholder parsing preserves is_embed bitmask (embed count and sub-range extraction).

---

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.38% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The PR covers the main changes, but it does not follow the required template and the Related Issues section is not in the prescribed format.	Restructure it to match the template headings and replace Related Issues with either 'Closes #10374' or the confirmed no-issue checkbox path.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title 'fix(vllm): support unified multimodal routing' accurately captures the main change: adding support for modality-aware multimodal routing in the vLLM backend path.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

components/src/dynamo/frontend/tests/test_vllm_processor_unit.py (1)

144-216: ⚡ Quick win

Move the new vllm_processor imports to the top of the file.

The added tests introduce several unguarded method-local imports of dynamo.frontend.vllm_processor. These are not the optional-dependency pattern called out in repo learnings, so they should follow the normal module-level import rule instead.

As per coding guidelines, "Keep all imports at the top of the file; flag any import inside functions/classes/methods."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py` around
lines 144 - 216, Multiple tests perform local imports of
dynamo.frontend.vllm_processor (e.g., _group_mm_feature_metadata,
_single_transfer_modality) inside test methods; move these imports to the module
level at the top of
components/src/dynamo/frontend/tests/test_vllm_processor_unit.py, replacing the
per-test "from dynamo.frontend.vllm_processor import _group_mm_feature_metadata"
(and any other similar lines) with a single top-of-file import listing the
needed symbols, and remove the in-function imports so tests use the module-level
names.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py`:
- Around line 130-132: The default argument for data in _feature is using
object() at import time (Ruff B008); replace it with a module-level sentinel
(e.g., _SENTINEL = object()) and change the signature to use that sentinel, then
inside _feature treat _SENTINEL as “no value” and set the intended default
runtime value. Also consolidate repeated local imports from
dynamo.frontend.vllm_processor used in TestMultimodalFeatureMetadata (and other
tests) by moving them to the top of the file so tests reuse the top-level
imports instead of importing inside individual test methods.

In `@components/src/dynamo/frontend/vllm_processor.py`:
- Around line 76-77: The logging in _prepare_mm_routing uses f.modality directly
which can raise AttributeError for legacy MM features; update
_prepare_mm_routing to call the helper _mm_feature_modality(feature) (or getattr
fallback) instead of accessing f.modality, and use that returned string in the
log and any subsequent logic (replace uses of f.modality with the result of
_mm_feature_modality for features named e.g., f or feature).

In `@components/src/dynamo/vllm/handlers.py`:
- Around line 2058-2061: The branch currently only calls
_compute_mm_uuids(multi_modal_data) when self.embedding_loader is None, which
incorrectly assumes presence of an embedding_loader means the request didn't
include raw MM payload; change the logic so that when forwarded_mm_uuids is
falsy you always compute mm_uuids from the actual payload (multi_modal_data)
regardless of self.embedding_loader (i.e., replace the conditional on
self.embedding_loader with an unconditional else that calls
_compute_mm_uuids(multi_modal_data)), ensuring
_extract_multimodal_data/image_loader paths still produce router-aligned MM
cache identities.

---

Nitpick comments:
In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py`:
- Around line 144-216: Multiple tests perform local imports of
dynamo.frontend.vllm_processor (e.g., _group_mm_feature_metadata,
_single_transfer_modality) inside test methods; move these imports to the module
level at the top of
components/src/dynamo/frontend/tests/test_vllm_processor_unit.py, replacing the
per-test "from dynamo.frontend.vllm_processor import _group_mm_feature_metadata"
(and any other similar lines) with a single top-of-file import listing the
needed symbols, and remove the in-function imports so tests use the module-level
names.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c4217d9f-1c48-4345-a475-785f65f38df7

📥 Commits

Reviewing files that changed from the base of the PR and between b5ff7e2 and b0a45db.

📒 Files selected for processing (4)

• components/src/dynamo/frontend/tests/test_vllm_processor_unit.py
• components/src/dynamo/frontend/vllm_processor.py
• components/src/dynamo/vllm/handlers.py
• components/src/dynamo/vllm/tests/test_vllm_video_handler.py

[rmccorm4]

@krishung5 to help review

[Copilot]

Pull request overview

This PR fixes Dynamo’s vLLM multimodal routing/forwarding logic to support unified multimodal models (e.g., Gemma4-12B) by preserving canonical multimodal hash strings while carrying modality information via modality-keyed metadata, with a legacy image-only fallback.

Changes:

• Frontend: groups forwarded multimodal hashes/placeholders by modality (while keeping legacy flat mm_hashes/mm_placeholders for image-only requests).
• Backend: consumes modality-keyed multimodal UUIDs/hashes and normalizes the unified vision-chunk modality mapping (image → vision_chunk when configured).
• Tests: adds focused unit coverage for grouping behavior, legacy fallback behavior, unified vision-chunk remapping, and placeholder is_embed preservation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
components/src/dynamo/frontend/vllm_processor.py	Builds and forwards modality-keyed multimodal metadata; skips mm_kwargs transfer for mixed-modality requests.
components/src/dynamo/vllm/handlers.py	Adds helpers to normalize modality and consume grouped multimodal hashes/placeholders in prompt building and mm_kwargs reception.
components/src/dynamo/frontend/tests/test_vllm_processor_unit.py	Adds unit tests for modality grouping, legacy fallback behavior, and mixed-modality transfer behavior.
components/src/dynamo/vllm/tests/test_vllm_video_handler.py	Adds unit tests validating backend prompt UUID construction, unified vision-chunk fallback, image UUID computation, and placeholder is_embed handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[krishung5]

/ok to test d4a2807

[krishung5]

Thanks for the PR! Left some comments.

[datadog-official]

⚠️ Warnings

🚦 2 Pipeline jobs failed

Post-Merge CI Pipeline | vllm-runtime / Test cuda13.0, amd64     

Post-Merge CI Pipeline | vllm-xpu / test-xpu     

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f34688e | Docs | Give us feedback!}

[krishung5]

/ok to test 120b344

[krishung5]

/ok to test b01e7c0

[maheshrbapatu]

@krishung5 The branch is now synced with current ai-dynamo/main at f34688ea68, including #11117 (the staged DGD admission-path migration fix). Could you please run /ok to test f34688ea68 for the refreshed head?

[krishung5]

/ok to test f34688e

[dynamo-ops]

Code LGTM. The only changes since last approval are merge-from-main commits — no functional code changes. CI is green across all 64 checks.

[maheshrbapatu]

Thanks so much for all the help with the PR, @krishung5!
Just wanted to check if it's ready to be merged now, or if there's anything else we're waiting on.

[krishung5]

Thanks for the contribution!