Conversation
vllm-project#50915) Signed-off-by: harjoth <harjoth.khara@gmail.com>
…roject#50879) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com> Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
…0867) Signed-off-by: Anuj Bolewar <anujbolewar@gmail.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Anuj Bolewar <anujbolewar@gmail.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…oject#49558) Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com> Co-authored-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…lm-project#50157) Signed-off-by: amitz-nv <203509407+amitz-nv@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>
…9919) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: lkchen <anthropic@lkchen.net> Co-authored-by: lkchen <anthropic@lkchen.net>
…6.98 GiB memory/GPU saved (vllm-project#50912) Signed-off-by: yewentao256 <zhyanwentao@126.com>
…-project#50911) Signed-off-by: Shreyas Misra <shreyasm@nvidia.com> Co-authored-by: OpenAI Codex <codex@openai.com>
…t collective (vllm-project#50697) Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>
Signed-off-by: 墨楼 <huangzhilin.hzl@antgroup.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: khluu <khluu000@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com>
…t#51079) Signed-off-by: Kevin H. Luu <khluu000@gmail.com>
Signed-off-by: Peiyuan Zhou <peiyuanzhou1994@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
…t#50323) Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com>
…ndex (vllm-project#48061) Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: khluu <khluu000@gmail.com> Co-authored-by: OpenAI Codex <noreply@openai.com>
…oject#50607) Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Signed-off-by: Rohan138 <rohanpotdar138@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…llm-project#51050) Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
…lel,test_async_tp]` (vllm-project#51068) Signed-off-by: mgoin <mgoin64@gmail.com>
…inism tests (vllm-project#50905) Signed-off-by: Divakar Verma <divakar.verma@amd.com>
…ct#50404) Signed-off-by: Tasos Varoudis <varoudis@archtech.gr> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lai, Yejing <yejing.lai@intel.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com> Co-authored-by: Li, Jiang <jiang1.li@intel.com>
Address reviewer feedback: use the flat MediaContentPart schema
({"type": "image_url", "url": "..."}) instead of the nested OpenAI
chat format. This eliminates the ChatContentPart intermediate type
and the content_parts_to_media_parts conversion function in Rust.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Allen Shen <aoshen@inferact.ai>
aoshen02
force-pushed
the
feat/generate-raw-multimodal
branch
from
August 10, 2026 02:06
c643af4 to
3c77b55
Compare
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com> Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
…ct#51379) Signed-off-by: jiang1.li <jiang1.li@intel.com>
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com> Co-authored-by: Li, Jiang <jiang1.li@intel.com>
…nd per-expert checkpoint mapping (vllm-project#51419) Signed-off-by: Isotr0py <Isotr0py@outlook.com>
…51265) Signed-off-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…duler (vllm-project#49579) Signed-off-by: omerpaz95 <omerpaz95@gmail.com> Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com> Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
vllm-project#48171) Signed-off-by: Zetian Li <804561096@qq.com> Co-authored-by: Claude (Anthropic) <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com> Signed-off-by: Micah Williamson <micah.williamson@amd.com> Co-authored-by: Micah Williamson <micah.williamson@amd.com>
…ad (vllm-project#48414) Signed-off-by: Itay Etelis <itay.etelis@ibm.com> Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
- Use MEDIA_CONNECTOR_REGISTRY + fetch_*_async instead of chat_utils - Remove skip_mm_cache so RL multi-epoch rollouts benefit from cache - Add model_validator to reject content_parts + features together Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Allen Shen <aoshen@inferact.ai>
…ct#51603) Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com> Co-authored-by: Akshat Anand <40275336+cipheraxat@users.noreply.github.com> Co-authored-by: Codex <codex@openai.com>
aoshen02
force-pushed
the
feat/generate-raw-multimodal
branch
from
August 10, 2026 09:05
166c624 to
6938a7e
Compare
Switch back to AsyncMultiModalItemTracker per Cyrus's feedback. This reuses the same connector setup as chat completions, including SSRF protections (allowed_media_domains, etc.). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Allen Shen <aoshen@inferact.ai>
aoshen02
force-pushed
the
feat/generate-raw-multimodal
branch
from
August 10, 2026 09:13
6938a7e to
0e2c69c
Compare
…ses in Intel GPU CI (vllm-project#51604) Signed-off-by: zengxian <xiangdong.zeng@intel.com>
Signed-off-by: Emmanuel Acheampong <achampion.emma@gmail.com>
- Add missing content_parts: None in render.rs GenerateRequest init - Fix import ordering in types.rs (rustfmt) - Fix chain call formatting in generate.rs (rustfmt) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Allen Shen <aoshen@inferact.ai>
…ct#50734) Signed-off-by: efschu <51944948+efschu@users.noreply.github.com>
…in YAML config (vllm-project#51573) Signed-off-by: Raj Firke <79653531+rajfirke@users.noreply.github.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…llm-project#51635) Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com> Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
…oject#51657) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Merge into test_serving_multimodal_tokens.py to share the server fixture and avoid launching a second Qwen3-VL instance in CI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Allen Shen <aoshen@inferact.ai>
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
content_partsfield toGenerateRequestfor raw multimodal input on the/inference/v1/generateendpointtoken_ids+ raw media (URLs/base64) in a single requestMotivation: RL workloads
RL frameworks (e.g. prime-rl) need token-level inference with multimodal inputs. Today's options:
/v1/completions/v1/chat/completionstoken_ids/render→/generate/generate+content_partsUsage
Design
image_url,input_audio,video_url, etc.) — same schema as chat completions, consistent across Rust and Python frontendscontent_parts→content_parts_to_media_parts()→ChatLlm::prepare_media()→MmFeaturesonTextRequestcontent_parts→AsyncMultiModalItemTracker→TokensPrompt→OnlineRendererpipelinefeatures(pre-processed path from/render)Test Plan
content_parts_to_media_partsconversioncontent_parts(image URL → generation)content_partsResolves vllm-project#51472
🤖 Generated with Claude Code