feat(vllm): support frontend image decoding with EPD

[Chokoyo]

Summary

Enable opt-in frontend image decoding for vLLM multimodal E/PD and E/P/D deployments. The existing backend-decoding path remains the default.

Prefill/PD forwards only the decoded-image descriptor. On a cache miss, the Encode worker reads RGB pixels directly from the frontend-owned buffer with one NIXL transfer; pixels are not relayed through Prefill/PD.

NIXL for decoded pixels transfers only ONCE from frontend to Encoder worker on EC miss.

sequenceDiagram
    participant F as Frontend
    participant P as Prefill / PD
    participant E as Encode

    F->>F: Decode and register RGB
    F->>P: Request + NIXL descriptor

    alt Prefill cache hit
        P->>P: Reuse embeddings
    else Prefill cache miss
        P->>E: Forward descriptor only
        alt Encode cache miss
            F-->>E: RGB via one direct NIXL transfer
            E->>E: Vision encode
        end
        E-->>P: Embeddings
    end

    P->>P: Prefill and continue to Decode

Loading

Details

• Forward frontend-decoded image descriptors from Prefill/PD to the Encode worker.
• Read decoded RGB pixels directly from frontend memory through NIXL.
• Reuse the canonical content_hash for Prefill and Encode embedding-cache keys.
• Validate URL and decoded-image variants at dispatch and encode boundaries.
• Bypass caching when a canonical content hash is unavailable.
• Add opt-in --frontend-decoding support to the E/PD and E/P/D launchers.
• Preserve existing URL-image behavior and process videos on Prefill/PD.
• Add unit coverage for E/PD and E/P/D serving cases.

Where Should the Reviewer Start?

• components/src/dynamo/vllm/multimodal_utils/request_processor.py
• components/src/dynamo/vllm/multimodal_utils/prefill_worker_utils.py
• components/src/dynamo/vllm/multimodal_handlers/encode_worker_handler.py
• examples/backends/vllm/launch/disagg_multimodal_e_pd.sh
• examples/backends/vllm/launch/disagg_multimodal_epd.sh

Validation

• Targeted unit tests: 12 passed
• Pre-commit on affected files: passed
• Local E/PD E2E: two base64 frontend-decoded requests returned Green; the second request hit the embedding cache.
• Local E/P/D E2E: mm_epd_qwen3-vl-2b_b64_frontend_decoding passed in 91.71 seconds and returned green.
• Added E/PD and E/P/D cases to post-merge CI.

Related Issues

• Based on feat(sglang): support frontend image decoding in multimodal EPD #11880
• DIS-2451

Summary by CodeRabbit

• New Features
  • Added optional frontend image decoding for multimodal inference.
  • Supports both image URLs and frontend-decoded image data in requests.
  • Added --frontend-decoding options to multimodal launch scripts.
  • Enables content-based caching for decoded images when valid identifiers are available.
• Bug Fixes
  • Added validation for malformed, unsupported, conflicting, or empty image inputs.
  • Prevents invalid images from being dispatched or cached.
• Tests
  • Added coverage for decoded-image processing, caching, validation, and launch topologies.

[coderabbitai]

Walkthrough

Changes

Frontend-decoded image processing

Layer / File(s)	Summary
Image contracts and embedding loader components/src/dynamo/vllm/multimodal_utils/protocol.py, components/src/dynamo/vllm/multimodal_utils/prefill_worker_utils.py, components/src/dynamo/vllm/tests/multimodal_utils/test_vllm_embedding_loader.py	Image inputs now support URL and decoded descriptors. The loader parses both forms and caches decoded images by validated content hash.
Request processor routing components/src/dynamo/vllm/multimodal_utils/request_processor.py, components/src/dynamo/vllm/tests/multimodal_utils/test_vllm_request_processor.py	The request processor forwards supported image items to the embedding loader and keeps UUID-only variants on local loading.
Encode-worker decoding and cache handling components/src/dynamo/vllm/multimodal_handlers/encode_worker_handler.py, components/src/dynamo/vllm/worker_factory.py, components/src/dynamo/vllm/tests/test_vllm_encode_worker_decoded.py	The encode worker validates image sources, batches URL and decoded image loading, bypasses caching for unkeyed items, and receives the frontend-decoding setting.
Launch configuration and topology coverage examples/backends/vllm/launch/disagg_multimodal_e_pd.sh, examples/backends/vllm/launch/disagg_multimodal_epd.sh, tests/serve/multimodal_profiles/vllm.py	Launch scripts propagate --frontend-decoding. The e_pd and epd profiles add base64 image cases.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: frontend image decoding support for vLLM EPD deployments.
Description check	✅ Passed	The description covers the changes, reviewer entry points, validation, and related issue references, but does not follow the exact required issue-link format.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

components/src/dynamo/vllm/multimodal_utils/prefill_worker_utils.py (1)

292-292: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use lazy logger formatting.

Line 292 formats the message before logger.debug() checks the log level. Pass the format string and values separately.

Proposed fix

-                logger.debug(f"[{request_id}] Cache hit for image index {idx}")
+                logger.debug("[%s] Cache hit for image index %d", request_id, idx)

As per coding guidelines, use lazy formatting for log messages.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/vllm/multimodal_utils/prefill_worker_utils.py` at line
292, Update the debug log in the cache-hit path around the image-index handling
to use logger lazy formatting: pass the request ID and index as separate
arguments alongside a format string, rather than interpolating them in the
f-string before logger.debug executes.

Sources: Coding guidelines, Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/serve/multimodal_profiles/vllm.py`:
- Around line 250-264: Add a measured profiled_vram_gib value to the epd
TopologyConfig after profiling it with tests/utils/profile_pytest.py, while
retaining the existing requested_vllm_kv_cache_bytes marker. If the measured
peak exceeds 24 GiB, move the frontend-decoding MmCase to a CI tier that
supports the requirement, and ensure the topology remains compliant with both
VRAM and framework-specific KV markers.

---

Nitpick comments:
In `@components/src/dynamo/vllm/multimodal_utils/prefill_worker_utils.py`:
- Line 292: Update the debug log in the cache-hit path around the image-index
handling to use logger lazy formatting: pass the request ID and index as
separate arguments alongside a format string, rather than interpolating them in
the f-string before logger.debug executes.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9b1cf578-b52b-44b6-b28c-197fae35bd6f

📥 Commits

Reviewing files that changed from the base of the PR and between e026f9a and 365fa09.

📒 Files selected for processing (11)

• components/src/dynamo/vllm/multimodal_handlers/encode_worker_handler.py
• components/src/dynamo/vllm/multimodal_utils/prefill_worker_utils.py
• components/src/dynamo/vllm/multimodal_utils/protocol.py
• components/src/dynamo/vllm/multimodal_utils/request_processor.py
• components/src/dynamo/vllm/tests/multimodal_utils/test_vllm_embedding_loader.py
• components/src/dynamo/vllm/tests/multimodal_utils/test_vllm_request_processor.py
• components/src/dynamo/vllm/tests/test_vllm_encode_worker_decoded.py
• components/src/dynamo/vllm/worker_factory.py
• examples/backends/vllm/launch/disagg_multimodal_e_pd.sh
• examples/backends/vllm/launch/disagg_multimodal_epd.sh
• tests/serve/multimodal_profiles/vllm.py

[devin-ai-integration]

Devin Review found 2 potential issues.

[KrishnanPrash]

Left a series of AI generated comments. Let me know what you think 👍

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rmccorm4]

/ok to test 1a48ec9

[rmccorm4]

/ok to test 1635ddb