[Frontend][EPD] Use JSON arrays for multimodal metadata - #56090
Merged
Merged
Conversation
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Tianyu Guo <guoty@inferact.ai>
gty111
requested review from
AndreasKaratzas,
DarkLight1337,
NickLucche,
aarnphm and
robertgshaw2-redhat
as code owners
September 9, 2026 13:49
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
|
Documentation preview: https://vllm--56090.org.readthedocs.build/en/56090/ |
mergify
Bot
added
documentation
Member
Isotr0py
approved these changes
Sep 9, 2026
Isotr0py
left a comment
Isotr0py
left a comment
Member
There was a problem hiding this comment.
Overall look reasonable, leave a nit.
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Member
|
/ci run |
|
✅ Triggered Buildkite CI #88038 for commit |
Isotr0py
enabled auto-merge (squash)
September 10, 2026 03:52
Contributor
|
I’ve reviewed this and it looks good to me. |
ItsRoy69
pushed a commit
to ItsRoy69/vllm
that referenced
this pull request
Sep 10, 2026
…#56090) Signed-off-by: Tianyu Guo <guoty@inferact.ai> Signed-off-by: Jyotirmoy Roy <jyotirmoyroy649@gmail.com>
pctablet505
added a commit
to pctablet505/vllm
that referenced
this pull request
Sep 10, 2026
Clears the `needs-rebase` conflict. The only conflict was the `vllm.entrypoints.chat_utils` import block in tests/entrypoints/unit_tests/test_chat_utils.py: vllm-project#56090 (479611c, "[Frontend][EPD] Use JSON arrays for multimodal metadata") added `_load_embeds_dict` and `_parse_metadata_array` at the same spot where this branch added `_parse_chat_message_content`. All three are kept, in isort order. Nothing was dropped from either side. Upstream's new multimodal-metadata tests and this branch's reasoning round-trip tests are unrelated, and the behaviour this branch covers (the `reasoning` -> `reasoning_content` mirror in `_parse_chat_message_content`, and `include_reasoning` gating the message field but not the usage counter) is unchanged on upstream main. Signed-off-by: pctablet505 <pctablet505@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jiaran-king
added a commit
to jiaran-king/vllm
that referenced
this pull request
Sep 15, 2026
Merge upstream main at 3192898. Include the EPD proxy serialization optimization (vllm-project#56657), JSON-array multimodal metadata (vllm-project#56090), and media processing options forwarding (vllm-project#56786). Git merged the overlapping Mooncake KV worker and tests without conflicts; no manual code adjustments were needed. Validation: 61 EC/cache/runner/proxy CPU tests and 171 Mooncake KV Store worker CPU tests passed. PR-file pre-commit, including mypy 3.10, passed. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Send EPD placeholder metadata as numeric JSON arrays instead of torch/base64 tensors. Accept arrays only for model-declared metadata fields, preserving legacy base64 embedding inputs and the proxy's raw-media path.
Python-only prerequisite extracted from #55842, which now contains only Rust frontend changes. No scheduler, model runner, or EC transport changes. Duplicate checks found no other PR implementing this Python contract. Related to #52409.
Motivation
torch.save/torch.loadand base64 encoding/decoding for small metadata arrays. The MUIRBench comparison below observed 2.59% higher throughput with JSON, with no aggregate accuracy regression; different node pairs prevent attributing the entire gain to this change.Validation
.venv/bin/python -m pytest tests/entrypoints/unit_tests/test_chat_utils.py -k 'image_embeds or json_metadata or json_arrays' -q .venv/bin/python -m pytest tests/v1/ec_connector/unit/test_epd_proxy_round_robin.py -q17 + 11 tests passed; applicable pre-commit hooks passed.
MUIRBench, Qwen3.5-35B-A3B + Mooncake RDMA, 4E+4PD, V2, concurrency 96, CUDA graphs through 16384 tokens (no eager): 200 warmup questions, then five repetitions of the first 1200/2600 questions.
Base64 → JSON: accuracy 59.02% → 59.10%, mean throughput 60.48 → 62.05 req/s; both 0/6000 failures. Sequential runs on different node pairs, so the throughput difference is not an isolated causal estimate.
Prepared with AI assistance (OpenAI Codex).