feat(vllm): add external encoder handoff

[furionw]

Why

Dynamo already maintains the VisionEncoderBackend framework, but applications that place the encoder outside the aggregated vLLM worker still had to reimplement image extraction, encoder driving, tensor packing, request sanitization, and the decoder-side prompt reconstruction contract.

This PR makes that handoff a vLLM-owned contract while keeping GenerateRequest.encoder_result opaque in dynamo.common.

User contract

handoff = ExternalEncoderHandoff(backend)
handoff.load(model_id)
prepared = await handoff.prepare_request(request, target_model=generator_model)
handoff.shutdown()

Users implement VisionEncoderBackend; Dynamo owns the rest of the external handoff.

What changed

• Add public ExternalEncoderHandoff lifecycle and request-preparation API.
• Move the concrete versioned wire format under the vLLM custom-encoder component.
• Share one linear prompt builder between inline and external encoder paths.
• Sanitize source-side multimodal state before forwarding to stock aggregated dynamo.vllm.
• Keep decoder-side conflict validation and off-event-loop tensor reconstruction.

Version 0 remains intentionally narrow: image inputs, ordered two-dimensional CPU linear embeddings, MsgPack request-plane transport, token-in/token-out, and a text-only aggregated vLLM worker with --enable-prompt-embeds.

The thin application-owned example is #14859.

Test plan

• Unit coverage for payload validation and round trips, artifact ordering, request sanitization, lifecycle, malformed inputs, and inline/external prompt parity.
• Existing handler coverage for aggregated, text-mode rejection, and conditional-bypass behavior.
• Pre-commit and Python syntax compilation pass locally.

Full PR sequence

1. #13293 — encoder handoff (merged)
2. #15195 — unary endpoint helpers (merged)
3. #15552 — model and client utilities
4. #14859 — application-owned orchestrator example
5. #15200 — draft end-to-end verification

The three open PRs form GitHub stack #15572. The first two remain in completed stack #15197.

Summary by CodeRabbit

• New Features
  • Added support for passing image embeddings from an external encoder to an aggregated worker, which reconstructs the prompt for generation.
  • External encoder results are supported in token-in/token-out mode on aggregated workers.
• Bug Fixes
  • Requests with incompatible multimodal inputs or invalid external encoder results now return errors instead of proceeding.
• Documentation
  • Clarified when applications and request routing can provide external encoder results.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Walkthrough

The change adds an external vision encoder handoff for aggregated vLLM requests. It serializes image embeddings into encoder_result and adds request validation and prompt reconstruction in the decode handler. The shared prompt builder supports both inline adapter use and external results.

Changes

External Encoder Handoff

Layer / File(s)	Summary
Shared prompt construction components/src/dynamo/vllm/multimodal_utils/custom_encoder/adapter/linear.py, components/src/dynamo/vllm/multimodal_utils/custom_encoder/backend/base.py, components/src/dynamo/common/backend/engine.py	Adds LinearEmbedsPromptBuilder and delegates prompt construction from LinearEmbedsAdapter. Updates documentation for encoder_result and encoder placement.
External result handoff components/src/dynamo/vllm/multimodal_utils/custom_encoder/handoff.py, components/src/dynamo/vllm/multimodal_utils/custom_encoder/__init__.py, components/src/dynamo/vllm/tests/multimodal_utils/custom_encoder/test_vllm_external_handoff.py	Adds tensor serialization, versioned encoder results, and request preparation that encodes image URLs and removes source-side multimodal data. Tests cover serialization, validation, and downstream request contents.
Aggregated worker prompt assembly components/src/dynamo/vllm/multimodal_utils/custom_encoder/external.py, components/src/dynamo/vllm/handlers.py, components/src/dynamo/vllm/tests/test_vllm_external_encoder.py	Adds prompt reconstruction from external results and handler support for aggregated token-mode requests. Tests cover reconstruction, validation errors, and mode selection.

Priority: ⬆️ High

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🔵 Low · up to bc735

The external encoder flow lacks an end-to-end assertion for forwarding its reconstructed prompt to generation. Add that focused integration test before relying on this coverage for future changes.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.94% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 68 functions across 9 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The pull request title 'feat(vllm): add external encoder handoff' directly corresponds to the main changeset objective. The title clearly identifies the primary feature addition: a new external encode…
Description check	✅ Passed	The pull request description covers the required template sections with substantial detail. The 'Overview' section is implicit in the 'Why' heading, which explains the motivation: Dynamo maintains `Vi…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b99cd151-207c-4d05-b606-932f43329d98

📥 Commits

Reviewing files that changed from the base of the PR and between 0ab7d48 and bc73513.

📒 Files selected for processing (9)

• components/src/dynamo/common/backend/engine.py
• components/src/dynamo/vllm/handlers.py
• components/src/dynamo/vllm/multimodal_utils/custom_encoder/__init__.py
• components/src/dynamo/vllm/multimodal_utils/custom_encoder/adapter/linear.py
• components/src/dynamo/vllm/multimodal_utils/custom_encoder/backend/base.py
• components/src/dynamo/vllm/multimodal_utils/custom_encoder/external.py
• components/src/dynamo/vllm/multimodal_utils/custom_encoder/handoff.py
• components/src/dynamo/vllm/tests/multimodal_utils/custom_encoder/test_vllm_external_handoff.py
• components/src/dynamo/vllm/tests/test_vllm_external_encoder.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.