[praveingk]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Cache-aware routers (llm-d, Ray's kv-aware router, SGLang's router) render every prompt to pick an endpoint, then forward the original messages and the engine renders again. On text prompts, kv_transfer_params["prompt_token_ids"] (#48145) already lets the engine skip that second render. On multimodal prompts, #55771 deliberately drops the forwarded IDs because they carry no multimodal data — the engine must render messages to process the images.

This PR closes that gap by letting the router forward the multimodal render result alongside the IDs, through the same kv_transfer_params channel:

{
  "messages": [{"role": "user", "content": [
    {"type": "text", "text": "What is in this image?"},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
  ]}],
  "kv_transfer_params": {
    "prompt_token_ids": [151644, 872, ...],
    "prompt_features": {
      "mm_hashes": {"image": ["<content-hash>"]},
      "mm_placeholders": {"image": [{"offset": 57, "length": 192}]},
      "kwargs_data": {"image": ["<base64-encoded MultiModalKwargsItem>"]},
      "mm_metadata": null
    }
  }
}

prompt_features uses the same MultiModalFeatures schema that /inference/v1/generate already accepts (the token-in/token-out format from the scale-out work). When both keys are present, the chat path's ID-reuse hook builds the engine input with the generate path's existing builder (mm_kwargs_from_features → MultiModalKwargsItems → mm_input) — no new engine semantics, just connecting two existing code paths.

Why this addresses the #55771 guard's concern: the guard exists because "the ids stand in for the rendered prompt, but they carry no multimodal data or hashes." With prompt_features forwarded, the IDs are no longer standing in alone — they come with the complete render result (hashes for prefix-cache disambiguation, processed tensors, placeholder layout). Without prompt_features, the guard fires exactly as today.

Changes:

• vllm/renderers/online_renderer.py: _reused_prompt_token_ids returns (ids, features) and pops both keys; the chat-path call site gains a multimodal reuse branch that mirrors the generate path's input builder; the Harmony call site unpacks the tuple (features unsupported there)
• vllm/entrypoints/openai/chat_completion/protocol.py: drop_prompt_token_ids_with_media only pops the IDs when prompt_features is absent

No new API surface. Both keys travel through the existing kv_transfer_params dict. The feature degrades gracefully: without prompt_features, behavior is identical to main.

Validation (DeepSeek-V4.1-Flash, 2×8 H100, vision tower loaded, on-cluster):

Sentinel proof that the engine uses the forwarded payload:

blue image, no forwarding              → "Blue"  (engine renders from messages)
blue messages + RED image's payload    → "Red"   (engine used the FORWARDED IDs + features)

Full-path regression (18 tests, all pass): text chat (streaming, non-streaming, multi-turn, tools, reasoning), completions, Messages, Responses, images (short, long/forwarded, streaming), and edge cases (tool_choice=none, reasoning_effort=medium, empty content).

Which issue(s) this PR fixes:

Addresses the multimodal gap in #55770 and the router use case in #54937.

Special notes for your reviewer:

The mm_input(...) call in the new multimodal branch is byte-identical to the generate path's usage in token_in_token_out/serving.py. The lazy imports (mm_kwargs_from_features, mm_input, MultiModalKwargsItems, PlaceholderRange) keep the entrypoints layering acyclic. The router-side implementation is llm-d-router #3706.

Release note:

The chat completions path now accepts kv_transfer_params.prompt_features alongside prompt_token_ids, letting a router that already rendered a multimodal prompt forward the processed result so the engine skips templating, tokenization, and multimodal preprocessing. Uses the same MultiModalFeatures schema as /inference/v1/generate.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[DarkLight1337]

cc @hickeyma

[shimib]

The new chat branch rebuilds the engine input from MultiModalFeatures without the validations the generate path runs on the same payload. Findings in priority order.

Blocking

1. SHM cache handles are forwarded unauthenticated (vllm/renderers/online_renderer.py:839)
token_in_token_out/serving.py runs _validate_mm_cache_handles / cache.validate_input_item before kwargs reach the engine. The chat branch skips it. With --mm-processor-cache-type shm, a client can put forged address / monotonic_id / signature fields in prompt_features.kwargs_data and the engine dereferences an unauthenticated shared-memory handle. That is the hole #59357 closed on generate, reopened on chat.

2. Metadata-only items accepted without ec_transfer_params (online_renderer.py:839)
No equivalent of GenerateRequest._require_ec_for_metadata_only. A null kwargs_data (whole or per item) with mm_metadata set builds MultiModalKwargsItems with no pixel_values; the request passes the frontend and dies in engine core instead of returning the 400 generate gives.

3. Placeholder ranges are not bounds-checked (online_renderer.py:832)
No equivalent of _validate_multimodal_feature_bounds. prompt_token_ids of length 100 with mm_placeholders: {"image": [{"offset": 200, "length": 64}]} is accepted; the scheduler never reaches the placeholder and the request produces garbage or hangs rather than a 400.

4. Harmony regression (online_renderer.py:563)
The protocol change keeps prompt_token_ids for media requests when prompt_features is present, but _make_request_with_harmony takes the ids and drops the features. A GPT-OSS chat request with an image plus forwarded ids now submits placeholder-bearing ids with no mm_kwargs. Before this PR the messages were re-rendered. Either thread the features through the Harmony path or keep dropping the ids there.

5. mypy no-redef (online_renderer.py:831 and :858)
conversation: list[ConversationMessage] = [] is annotated in both sibling branches. pre-commit run mypy-3.12 --all-files --hook-stage manual fails with Name "conversation" already defined on line 831. The pre-commit CI job was skipped by the pre-run check, which is why it is green here. Declare it once above the if.

Should fix

• docs/features/disagg_prefill.md:80 still says prompt ids are always ignored when messages has non-text content, and kv_transfer_params["prompt_features"] and its schema are undocumented. Router authors will not find this feature.
• online_renderer.py:129: prompt_features is popped and silently discarded when prompt_token_ids is absent (hides a router bug; at least log it), and it is validated before the echo early return, so a malformed prompt_features 400s a request that was going to ignore it anyway.

Suggestion that covers 1–3 at once

The PlaceholderRange conversion + mm_kwargs_from_features + mm_input block is a byte-for-byte copy of token_in_token_out/serving.py:269-290. If you lift that block and its three validations into one helper next to mm_kwargs_from_features in mm_features.py (something like engine_input_from_features(token_ids, features, cache_salt, mm_cache)) and call it from both endpoints, the missing checks come for free and the two paths cannot drift again.

Minor: the function-local vllm.inputs / vllm.multimodal.inputs imports at online_renderer.py:825 are not needed for layering (vllm.inputs is already imported at module top); only the token_in_token_out imports need to be lazy. A TYPE_CHECKING import of MultiModalFeatures would also let reuse_features be typed instead of Any.

[praveingk]

@shimib Thanks for the review. Let me fix your comments.

[praveingk]

Thanks @shimib — excellent review. All findings addressed in a1aad6f (hardening) + b1d347b (tests):

Blocking

1. SHM cache handles — the engine-input block is now lifted into engine_input_from_features() in mm_features.py, which runs mm_cache.validate_input_item(item, mm_hash) on every deserialized item when a processor cache is configured; the chat branch passes renderer.mm_processor_cache. Forged address/monotonic_id/signature fields now fail with a 400 before the engine sees the request.
2. Metadata-only without ec_transfer_params — validate_metadata_only() shared helper; GenerateRequest._require_ec_for_metadata_only now delegates to it (lazy import to break the protocol ↔ mm_features cycle) so the parse-time gate and the chat path cannot drift.
3. Placeholder bounds — validate_feature_bounds(), same delegation pattern from _validate_multimodal_feature_bounds.
4. Harmony regression — _make_request_with_harmony now requires reuse_features is None for the fast path; with features forwarded the ids are dropped and messages are re-rendered, matching pre-feature behavior.
5. mypy no-redef — conversation is declared once above the reuse branches.

Should fix

• docs/features/disagg_prefill.md documents kv_transfer_params["prompt_features"] with the full MultiModalFeatures schema table, the render → forward round trip, the validation guarantees, and the features-without-ids / Harmony caveats; the stale "ids are always ignored for non-text content" claim is corrected.
• prompt_features without prompt_token_ids now logs a warning instead of being silently discarded; feature validation moved after the echo / non-text early returns so a malformed payload no longer 400s a request that was going to ignore it.

The covering refactor — done as you suggested: engine_input_from_features(token_ids, features, cache_salt=, mm_cache=, ec_transfer_params=) is the single builder called by both serve_tokens and the chat reuse branch, with all three validations inside; _validate_mm_cache_handles is deleted. One deviation: the two GenerateRequest parse-time validators are kept (delegating to the shared helpers) so generate errors still surface at parse time — a comment in the helper notes the intentional double-validation. Also note MultiModalFeatures/PlaceholderRangeInfo stay runtime imports in mm_features.py since extract_mm_features instantiates them; only BaseMultiModalProcessorCache is TYPE_CHECKING-only, and the reuse_features return type is now typed via a TYPE_CHECKING import.

Tests (b1d347b) — new test_chat_mm_feature_reuse.py covers the chat side: forged-handle 400 with the (item, hash) call shape, out-of-bounds placeholders, metadata-only/EC, _reused_prompt_token_ids ordering/warning, preprocess_chat integration (renders skipped, multimodal input built), and the Harmony re-render. 13/13 new + the existing token_in_token_out unit suite pass locally.

[mergify]

Documentation preview: https://vllm--64721.org.readthedocs.build/en/64721/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @praveingk.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[shimib]

Fixes look right, tests and mypy pass locally. pre-commit run --files <changed> still fails: F823 at online_renderer.py:174 (the lazy MultiModalFeatures import shadows the annotation), SIM102 at chat_completion/protocol.py:1080, and ruff-format touches three files. CI skips pre-commit on this PR, so it won't surface there.

[praveingk]

@shimib I have pushed the fixes for the lint errors.

[DarkLight1337]

cc @gty111 @Isotr0py

[DarkLight1337]

Could you elaborate on why couldn't you pass the rendered token_ids and features into /inference/v1/generate to achieve this?

[gty111]

Two issues with the forwarded-feature path:

[gty111]

Could we preserve PlaceholderRange.is_embed through serialization and reconstruction? Models such as Gemma 3 include text/structural tokens inside the placeholder range. Restoring only offset and length treats every position as an embedding position, causing a mismatch with the encoder output. The new chat fast path inherits this existing Generate API limitation; previously chat re-rendered the messages and retained the mask.

[gty111]

Could malformed serialized items be converted to a validation error? A base64-encoded MessagePack payload such as {"pixel_values": {}} passes MultiModalFeatures validation but raises KeyError('data') during deserialization. Since only ValueError is caught, the request returns HTTP 500 instead of 400.

[praveingk]

Could you elaborate on why couldn't you pass the rendered token_ids and features into /inference/v1/generate to achieve this?

@DarkLight1337 Fair question — I initially did consider this. Mechanically it works: GenerateRequest already accepts token_ids + features, and after this PR both entry points share engine_input_from_features(). I kept the chat path as the data plane for four reasons:

1. The response contract. Clients speak OpenAI chat: streaming SSE with content/reasoning deltas, tool_call fragments, usage. The chat endpoint produces that directly; through /inference/v1/generate a separate derender frontend has to reconstruct it, with the caller orchestrating render → generate → derender and carrying stream state across calls. If the engine's chat stack ends up shaping the response anyway, forwarding there is the smaller change.

2. No topology change. The forwarding rides kv_transfer_params on the chat body the EPP already routes (llm-d-router#3706). The generate alternative means a tokens-only engine, a separate derender frontend, and caller-orchestrated response shaping — a different serving architecture for the same one-render result.

3. Feature state stays in one place. On the chat path, adjust_request runs engine-side with the real tokenizer, so tool/reasoning decisions and structured-output grammar are gated in one place, consistently with the client's request. The generate path projects each onto separate side-channels — grammar pre-baked into sampling_params at /render, reasoning gating via reasoning_ended/reasoning_parser_kwargs, parse context via parse_context — workable, but every chat feature must be mapped into that contract and kept in sync.

4. Scope. Same endpoint, same request/response contract, no new components; the chat path falls back to rendering whenever features are absent. The shared builder keeps the two paths from drifting on multimodal engine-input construction, so the token-in-token-out option stays fully open — for --tokens-only deployments it remains the right architecture, and nothing here conflicts with it.

[praveingk]

Two issues with the forwarded-feature path:

Thanks @gty111 — both addressed in dc2cc52:

1. is_embed preservation. PlaceholderRangeInfo now carries is_embed: list[bool] | None (validated against length when set), the render-side serialization emits it, and the shared engine_input_from_features reconstructs the torch bool mask — so both the generate path and the chat fast path keep the sparse-embedding mask end to end. Round-trip tested: the mask survives (get_num_embeds() returns the masked count), None for plain ranges stays None (backward compatible), and a mismatched length is rejected at validation.

2. Malformed items → 400. The decode is now wrapped: bad base64 (binascii.Error, already a ValueError subclass) passes through, and any failure from MultiModalKwargsItem reconstruction — including the KeyError('data') your {"pixel_values": {}} example triggers — is re-raised as ValueError("malformed multimodal item payload: …"), which both the generate and chat error handlers map to HTTP 400. Tested with the exact payload from your comment.

[hickeyma]

@DarkLight1337 sorry for the slow reply. Part of the reasoning above has changed but I still think there's a case for the chat path. With #63103 merged, output_mode: "derender" streams parsed output from generate which means the caller doesn't carry stream state or need a separate derender frontend. It does need the tokenizer and parsers on the generate server. What generate doesn't return is OpenAI chat chunks, so the router has to rebuild them. llm-d-router#3656 went with forwarded IDs on the client's route as its default which keeps the native response.

The bigger concern for me is that I don't see a gate. This accepts client supplied tensors on /v1/chat/completions with no KV connector or --enable-scale-out check which means it's open on a plain vllm serve. The blue/red test in the description shows the model answering from something other than messages which is #62307 with images. I think this should follow whatever we decide on #62333.

@praveingk one question on payload. Forwarded tensors are about 8 KB per image token unless they hit the cache and the original image stays in messages. llm-d-router#3656 lists no rendered tensors across the gateway in its acceptance and llm-d-router#3706 needs vllm.omitMMKwargs: false to send them. How do you see those fitting together?