[roytman]

What type of PR is this?
/kind performance

What this PR does / why we need it:

Define the coordinator render step mm_payload parameter, which selects what the render response carries for multimodal entries. Two values:

• tensors (default): the processed multimodal tensors (kwargs_data, mm_metadata), the behavior before this PR.
• none (opt-in): only token ids, mm_hashes and mm_placeholders. Generate-format encode requests (use_openai_format: false) carry the client's original media as content_parts on /inference/v1/generate. The generate-format prefill request carries features with mm_metadata when every entry has metadata (from the render response, or rebuilt from the encode response's EC descriptors) and EC coverage, and content_parts otherwise.

none is opt-in because its content_parts form requires every placeholder span to be one repeated token. Checked with each model's processor (one 640x448 image, eight video frames), only Qwen-VL images have that shape (<|image_pad|> x N). Gemma3, Pixtral, Idefics3/SmolVLM and Qwen3-VL video interleave other tokens in the span, and on those models the request fails with an error naming mm_payload none instead of sending a prompt the worker would expand differently. Once vllm-project/vllm#54548 carries the is_embed mask in the render response, none can become the default.

For none the render request carries mm_payload: "none" plus return_mm_kwargs: false as a deprecated alias (vLLM has no metadata-only render payload; #3273 made the same request from the EPP token producer). tensors sends no field. A renderer without the field ignores it and returns the tensors, and the request takes the features path. A body that sets mm_processor_kwargs or media_io_kwargs is rendered with the tensors under either value, because the content_parts path preprocesses media at the deployment defaults.

Each content_parts item is {"type": "image_url" | "audio_url" | "video_url", "url": ..., "uuid": <render mm_hash>}. The uuid keeps the item keyed by the render hash, so the encode response's ec_transfer_params matches what prefill looks up. An inline input_audio part is sent as a data: URL. features is omitted on the same request, because vLLM rejects content_parts and features together. vLLM expands the placeholders of a content_parts prompt itself, so each placeholder span in token_ids is collapsed to one token: encode sends BOS plus one placeholder token, prefill sends the collapsed prompt with one part per entry.

The metadata rebuild covers batched int64 keep_on_cpu fields, the image_grid_thw / video_grid_thw shape the Qwen-VL family uses, and is byte-identical to the item vLLM's own render produces. A request with any entry whose metadata has another shape, or no EC descriptor, takes the content_parts form for the whole request, and prefill logs the entry and the reason.

Operating none:

• With ec_connector: ec-shared-storage the encode response carries no EC descriptors, so every prefill request takes the content_parts form. The fallback cost in the table below was measured with ec-nixl; ec-shared-storage was not measured.
• The renderer and the encode and prefill workers must run the same multimodal processor configuration (--mm-processor-kwargs, media IO defaults such as max_pixels), since each worker preprocesses the client's media again. prompt_token_parity: warn (feat(coordinator): check prompt-token parity between prefill and decode #3751) detects a mismatch and is worth leaving on when enabling none.
• A generate-route decode leg (feat(coordinator): put the decode leg on the tokens-in generate format #3705) needs the same content_parts form under none; whichever of the two PRs lands second carries the adaptation.

Changed on the tensors path:

• The generate-format encode sub-request prompt is BOS plus the entry's placeholder span copied from the full prompt. Before, it was BOS plus the span's first token repeated, which differs from what the prefiller expands on models whose span mixes tokens (Gemma3, Pixtral, Idefics3, Qwen3-VL video). A span outside token_ids fails the request; before, it produced a prompt of token 0.
• The sidecar's EC Mooncake connector uses the shared reqcommon.FieldUUID and reqcommon.FieldECDescriptorMetadata constants in place of its local copies (no behavior change).

Unchanged paths:

• Decode, and encode and prefill with use_openai_format: true, already forward the client body with its media parts.
• Client requests to /inference/v1/generate carry their own features and skip the render call.

Measured on the coord-disaggregation guide: Qwen3-VL-32B, one TP=2 pod each for encode, prefill and decode, vLLM nightly 5d35ce3934, freshly generated 1920x1080 JPEGs so no request hit the encoder or prefix cache. Chat requests through the full pipeline, one at a time, 20 measured per row after 3 warmups, max_tokens: 32, all rows on the same coordinator build at log level 2. Medians in ms, step times from the coordinator's step timings. The two none rows are the same mm_payload setting with the prefill features form and its content_parts fallback; they were measured as separate metadata and none values on a build where the values were still separate:

Images	mm_payload / prefill form	render	encode	prefill	decode	Total
1	tensors	152	206	153	392	1,402
1	none, features with metadata	99	110	155	413	1,298
1	none, content_parts fallback	93	104	275	404	1,371
4	tensors	624	516	551	533	2,763
4	none, features with metadata	358	259	552	558	2,290
4	none, content_parts fallback	330	255	934	545	2,603

Every row returned identical output token counts, with prompt_tokens 2,057 (one image) and 8,183 (four images) matching the rendered prompt.

Render response size, median of 5 calls from a pod in the cluster: 16,727,450 bytes with tensors vs 15,402 with the tensors omitted for one image, and 66,906,869 vs 58,575 for four.

With the tensors omitted, render and encode take about half the time. The content_parts fallback regresses prefill, because the prefill worker re-runs the image processor on the client's media before it loads the encoder output from the EC connector. The features form avoids that re-run, and the per-request fallback picks it whenever every entry has metadata and EC coverage. The features form is within 2 ms of tensors on prefill and 7% (one image) to 17% (four images) faster than tensors end to end.

Requires vLLM with mm_payload / return_mm_kwargs on the render endpoints (vllm-project/vllm#61427, vllm-project/vllm#60728) and content_parts on /inference/v1/generate (vllm-project/vllm#51478, in v0.29 and later). The metadata-only prefill form follows the mm_metadata + ec_transfer_params contract of vllm-project/vllm#54659. The coordinator-side metadata rebuild is temporary: vLLM nulls mm_metadata together with kwargs_data when the tensors are omitted, although mm_metadata is tiny. Once vLLM keeps mm_metadata in that response (vllm-project/vllm#64997, or a mm_payload: "metadata" value, discussed in vllm-project/vllm#46722 and vllm-project/vllm#65018), the rebuild is dropped and the render response's metadata is used directly.

Which issue(s) this PR fixes:
Fixes #3750
Part of #3656

Test plan

• mm_payload defaults to tensors and sends the render request without the field; none sends mm_payload: "none" with the return_mm_kwargs: false alias; an invalid value (including metadata) is rejected; neither field leaks into the client body forwarded downstream
• A body with mm_processor_kwargs or media_io_kwargs is rendered with the tensors under none, an explicit JSON null counts as unset, and a renderer that returns no kwargs_data for such a body fails the request
• The rebuilt mm_metadata item is byte-identical to vLLM's msgpack output for image_grid_thw; non-integer, out-of-range (2^63) or non-batched values are rejected
• With the tensors omitted and every entry covered, encode sends content_parts capped to one token and prefill sends features with rebuilt mm_metadata, null kwargs_data and the expanded token_ids; mm_metadata from the render response is sent as-is without EC metadata
• An entry without encodable metadata, or without an EC descriptor even when render carried its mm_metadata, takes the content_parts form for the whole request
• A renderer that ignores mm_payload keeps the features path; a text-only request leaves the path unchanged
• Media URLs are read from chat image_url, Responses input_image and Messages base64 image parts; a missing part, a part without a URL, an unsupported audio format, or an unknown modality fails the request
• OpenAI-format prefill forwards the client body unchanged
• Placeholder collapse handles unsorted spans and rejects overlapping, out-of-range, or non-uniform spans; encode rejects a non-uniform span before sending any sub-request
• The encode sub-request prompt carries the entry's placeholder span as it sits in the full prompt; a negative length, an overflowing span, or a span past the end of token_ids fails the request
• e2e: with a renderer that returns tensors anyway (the simulator), encode and prefill send features and no content_parts. The simulator cannot omit tensors, so the content_parts and metadata-rebuild paths are covered by the unit tests and the GPU run; Honor mm_payload / return_mm_kwargs on the render endpoints llm-d-inference-sim#850 tracks simulator support
• go test -race ./pkg/coordinator/..., make lint and make lint-security
• On a GPU cluster with real vLLM: chat requests with one and four images return 200 in every form with identical prompt_tokens

Release note (write NONE if no user-facing change):

Add `mm_payload` to the coordinator render step: `tensors` (default) or `none`. With `none`, render requests ask vLLM to leave processed multimodal tensors out of the response, generate-format encode requests send the client's original media as `content_parts` on `/inference/v1/generate`, and prefill sends `features` with `mm_metadata` rebuilt from the encode response when every entry is covered, else `content_parts`. `none` requires vLLM v0.29 or later and a model whose placeholder spans are one repeated token (Qwen-VL images); other models fail the request. Generate-format encode requests send the entry's placeholder span as it appears in the prompt, fixing encoder input on models whose placeholder span is not one repeated token.

[roytman]

Read it against the generate contract and the vLLM placeholder model; the design and the numbers hold, and the metadata rebuild is careful. One correctness item on the content_parts fallback (inline), one heads-up for #3705, and two small notes.

Release-note nit: content_parts on /inference/v1/generate is in every release since v0.29, so "v0.29 or later" says more than "a version that accepts content_parts".

One plus worth keeping in mind for the guide: under none the render tier and the worker preprocess independently, which is exactly what prompt_token_parity: warn (#3751) watches for, so it is worth leaving on when enabling none.

Thanks for the careful read. The release note now names vLLM v0.29 or later. The prompt_token_parity point is in the description: under none the render tier and the encode and prefill workers preprocess the media independently, so warn is worth leaving on when enabling none. The uniformity check, the e2e disclosure and the simulator issue are addressed in the inline replies.

[ahg-g]

The render savings look good. Requesting changes on the generate-format path: defaulting to none breaks models whose placeholder spans mix tokens and slows prefill with ec-shared-storage. Details inline.

[ahg-g]

Document that ec-shared-storage always gets the content_parts fallback under none, or allow the metadata form when the connector has no descriptors.

Why: sharedStorageEC.PreparePrefillECParams returns an empty map and MergeEncodeResponse drops the descriptor, so entriesWithECMetadata never succeeds. Every request then re-runs the image processor on prefill: +120 ms for one image, +380 ms for four in the PR's own table. ec-shared-storage is the default ec_connector in config/coordinator/coordinator.yaml, so a default generate-format deployment gets the slow prefill on every request.

[roytman]

With tensors as the default (see 1), a default deployment no longer reaches this path, ec-shared-storage included. Deployments that opt into none with ec-shared-storage hit a gap: the connector publishes no EC descriptors (PreparePrefillECParams returns an empty map and MergeEncodeResponse is a no-op), so the coordinator has nothing to rebuild mm_metadata from, and prefill always takes the content_parts fallback. That is now documented in docs/communication.md.

The metadata form can't be allowed without descriptors. Prefill needs mm_metadata (the grid) to expand placeholders and compute mRoPE after the EC load, and only the descriptor or the render response can supply it. The fix that works for every connector is vllm-project/vllm#64997, which keeps mm_metadata in the render response when the tensors are omitted.

For reference, the fallback's cost was measured with ec-nixl. I haven't measured ec-shared-storage, which has different load costs.

[ahg-g]

Replace MMPayload string with TensorsOmitted bool.

Why: the field only ever holds "" or "none", and every reader checks != "". A string suggests more states than exist and makes readers look up what "" means.

[roytman]

Kept the string for now: the field mirrors the wire enum, and there is a discussion in vllm-project/vllm#64997 and vllm-project/vllm/#65018 about adding a metadata value or extending none. At which point the string holds a third state a bool cannot.
Easy to switch if you'd rather not wait for that.

[ahg-g]

Compute partsByMod once per request (e.g. cache it on RequestContext during encode) instead of calling clientMediaParts again on the prefill fallback.

Why: promptItems decodes messages from json.RawMessage on every call. With inlined data: media that is megabytes of base64 decoded and allocated twice per request on the fallback path.

[roytman]

I don't think the double decode happens on this path. replace-media-urls runs before encode in every shipped pipeline, decodes items once, and writes the decoded slice back into the body when it finds media refs, which happens on every multimodal request. Encode's and prefill's promptItems calls then hit the []any case: a type assertion and a walk over part maps that alias the same objects, so no media bytes are decoded or copied twice. Encode also skips generate-path requests entirely, the only ones whose body would stay raw.
A pipeline configured without replace-media-urls would decode per call; if that configuration matters, I'm happy to cache partsByMod on the request context as a follow-up.

[ahg-g]

Thanks for the updates; defaulting to tensors and checking span uniformity on encode resolve the main blockers. A few remaining items inline (notably json.RawMessage("null") handling in bodySetsProcessorKwargs and duplicate constants across packages), plus the PR title, description, test plan, and release-note block still describe none as the default.