Support multiple image/audio embeddings per requests

[jeremyteboul]

This enables the Chat Completions API to leverage the model's existing capability for multiple embeddings, previously only accessible through the direct LLM inference API.

• Remove limitation that only allowed one message with image_embeds/audio_embeds

• Update MultiModalItemTracker and AsyncMultiModalItemTracker to treat embeddings as lists

• Embeddings now behave consistently with regular images/audios

• Validation via existing validate_num_items() against --limit-mm-per-prompt

Test Plan

• Add unit tests for multiple image embeddings support:
  • test_parse_chat_messages_multiple_image_embeds
  • test_parse_chat_messages_multiple_image_embeds_with_uuids
  • test_parse_chat_messages_multiple_image_embeds_async

Test Result

passed

[gemini-code-assist]

Code Review

This pull request successfully enables support for multiple image and audio embeddings per request by updating the MultiModalItemTracker to handle lists of embeddings. The changes in vllm/entrypoints/chat_utils.py are correct and effectively remove the previous limitation. The new unit tests are also well-designed to cover these new capabilities. However, this change to the data structure for embeddings (from a single item to a list) breaks several existing unit tests that were written with the single-embedding assumption. I've identified these failing tests and provided suggestions to update them. Addressing these test failures is critical for merging this PR.

[chatgpt-codex-connector]

💡 Codex Review

https://github.com/vllm-project/vllm/blob/c4e242c86de0203e5a2a137cce1cacda6e226705/vllm/entrypoints/chat_utils.py#L725-L727
Keep single image_embeds output shape backward compatible

Mapping image_embeds directly to mm_inputs["image"] now returns a list even when only one embedding is provided. Existing callers/tests (e.g., test_parse_chat_messages_empty_image_embeds_with_uuid, lines 829–858) relied on mm_data["image"] being a lone tensor/None for a single embed; after this change they receive [tensor] or [None], breaking those assertions and changing the public return contract despite the commit claiming backward compatibility.

---

https://github.com/vllm-project/vllm/blob/c4e242c86de0203e5a2a137cce1cacda6e226705/vllm/entrypoints/chat_utils.py#L730-L732
Audio embeddings wrapped in lists stop being parsed as embeddings

Setting mm_inputs["audio"] = audio_embeds_lst means a single audio embedding now surfaces as a list. Downstream parsing treats embeddings only when it receives a tensor or a list of 2D tensors (MultiModalDataParser.is_embeddings, vllm/multimodal/parse.py:383-390); a list containing the previous 3D tensor (see test_parse_chat_messages_audio_embeds_with_string, lines 893–939) is no longer recognized as an embedding and is processed as raw audio instead, breaking existing tests and single-embed requests.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[DarkLight1337]

Thanks, can you update the Multimodal Inputs documentation page accordingly as well?

[mergify]

Documentation preview: https://vllm--29988.org.readthedocs.build/en/29988/

[jeremyteboul]

Thanks, can you update the Multimodal Inputs documentation page accordingly as well?

Here is the doc !

[DarkLight1337]

#29970 just got merged, can you update your code to use the tensor2base64 convenience function?

[DarkLight1337]

This section is under offline inference so I think it's not related?

[DarkLight1337]

Thanks, LGTM then

[DarkLight1337]

I have fixed DCO for you, next time please sign-off your commits.