[vMaroon]

Overview

Add return_mm_kwargs to render requests. When false, the render response leaves out the processed multimodal tensors (kwargs_data) and mm_metadata, for callers that need only token ids, item hashes and placeholder ranges, such as cache-aware routers. Part of #46722.

Claims

• return_mm_kwargs: false on the chat, completions, Messages and Responses render endpoints returns kwargs_data and mm_metadata as null. Token ids, mm_hashes and mm_placeholders are unchanged. The default (true) keeps current behavior.
• For four 1080p images the response drops from 66.91 MB to 0.058 MB, and render time from 166-189 ms to 38 ms with a warm processor cache, or from 194 to 74 ms cold.
• Tensor serialization no longer holds the API server's event loop: /health p99 during back-to-back renders goes from 114.5 to 2.0 ms.

Validation

pytest tests/entrypoints/scale_out/token_in_token_out/test_mm_serde.py tests/entrypoints/scale_out/token_in_token_out/test_protocol.py: 45 passed, including the new test_extract_without_kwargs_keeps_layout.

End to end on this head: vllm launch render Qwen/Qwen3-VL-8B-Instruct on CPU (M4 Pro, OMP_NUM_THREADS=1), four 1920x1080 PNGs by URL, median of 3 after a warm-up:

Endpoint	Default	return_mm_kwargs: false	Token ids, hashes, placeholders equal
chat	166.0 ms, 66.91 MB	37.9 ms, 0.058 MB	yes
Messages	172.2 ms, 66.91 MB	38.1 ms, 0.058 MB	yes
Responses	188.9 ms, 66.91 MB	37.5 ms, 0.058 MB	yes

Measured with the same change on main 1afb029839, processor cache off (--mm-processor-cache-gb 0): render 193.9 to 73.6 ms. The time removed is json.dumps of the response (102 ms) and msgpack plus base64 (4.8 ms), both on the event loop, plus 14 ms of transfer. With renders back to back, /health p99 goes from 114.5 to 2.0 ms and max from 117.4 to 4.1 ms.

On GPUs: 2 x H100 serving Qwen3-VL-8B (nightly 81198e97 with this change laid over the installed package) behind llm-d-router with llm-d/llm-d-router#3273, closed loop, four 1080p images by URL per request, 60 s per step, flag off, on, off. At 16 concurrent clients throughput went from 2.6 to 8.3 req/s, client p95 from 15.2 to 3.0 s, and engine /health p99 from 1,177 to 11 ms. With the flag on, the router's predicted cached tokens equal the engine's cached tokens (6 of 6 repeats, chat and Messages).

Not run: the GPU vllm serve render tests (tests/entrypoints/scale_out/render/test_render_multimodal.py). No model eval: the change affects only render responses.

Details

• Not a duplicate of WIP: Reduce multimodal payload in token-in/token-out by deferring pixel_va… #53169: its skip_pixel_values replaces the tensors with the original image bytes so /inference/v1/generate can rebuild them. A router discards those bytes as well (about 33 MB of base64 for the images above). If both land, they could share one field with three values: tensors, source bytes, none.
• Independent of [Perf][Frontend] Share orjson responses across render and pooling #58660, which makes serialization cheaper for callers that need the tensors.
• A response rendered with the flag must not be sent to /inference/v1/generate, which reads a null kwargs_data as every item being cached. The field description and renderer.md say so.
• Render still runs the multimodal processor; moving it out is the open question in [RFC]: Reduce multimodal payload in token-in/token-out by deferring pixel_values preprocessing to /generate #46722 and out of scope here.
• Messages render converts the request to a chat request, so the handler copies the field across.

AI assistance (Claude) was used to write this change and run the validation above.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[mergify]

Documentation preview: https://vllm--60728.org.readthedocs.build/en/60728/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[sagearc]

great! left 2 style nits though

[sagearc]

nit: more python-idiomatic way

Suggested change

	if include_kwargs and (raw_mm_kwargs := mm_engine_input.get("mm_kwargs")):
	if include_kwargs:
	raw_mm_kwargs = mm_engine_input.get("mm_kwargs")
	if raw_mm_kwargs:
	kwargs_data, mm_metadata = _encode_mm_kwargs_with_metadata(
	raw_mm_kwargs,
	metadata_fields_for=metadata_fields_for,
	)

[vMaroon]

Done in 8b8c271.

[sagearc]

Could we rename include_kwargs to include_mm_kwargs in both helpers? It would make the scope explicit and align it with return_mm_kwargs.

[vMaroon]

Done in 8b8c271, in both helpers and the test.

[vMaroon]

/ci run

[github-actions]

❌ This PR is 19 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

[vMaroon]

/ci run

[github-actions]

✅ Triggered Buildkite CI #93789 for commit bfc70e0ee2d9.

[hickeyma]

LGTM, thanks @vMaroon