Skip to content

[Frontend] Add content_parts to /inference/v1/generate for raw multimodal input - #12

Open
aoshen02 wants to merge 324 commits into
mainfrom
feat/generate-raw-multimodal
Open

aoshen02 wants to merge 324 commits into
mainfrom
feat/generate-raw-multimodal

Conversation

@aoshen02

@aoshen02 aoshen02 commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add content_parts field to GenerateRequest for raw multimodal input on the /inference/v1/generate endpoint
  • Both Rust and Python frontends resolve media server-side — no pixel data transfer needed
  • Enables RL frameworks to pass token_ids + raw media (URLs/base64) in a single request

Motivation: RL workloads

RL frameworks (e.g. prime-rl) need token-level inference with multimodal inputs. Today's options:

Path Problem
/v1/completions No multimodal support in OpenAI spec
/v1/chat/completions Requires messages + chat template — RL callers already have token_ids
/render/generate Pixel data round-trip (~MB per image, see RFC vllm-project#46722)
/generate + content_parts Token IDs + raw media → one request, zero overhead

Usage

POST /inference/v1/generate
{
  "token_ids": [1, 32010, 32044, ...],
  "content_parts": [
    {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
  ],
  "sampling_params": {"max_tokens": 64, "temperature": 0}
}

Design

  • Reuses OpenAI content part format (image_url, input_audio, video_url, etc.) — same schema as chat completions, consistent across Rust and Python frontends
  • Rust path: content_partscontent_parts_to_media_parts()ChatLlm::prepare_media()MmFeatures on TextRequest
  • Python path: content_partsAsyncMultiModalItemTrackerTokensPromptOnlineRenderer pipeline
  • Mutually exclusive with features (pre-processed path from /render)
  • 6 files changed, ~125 lines added

Test Plan

  • Unit test: Rust content_parts_to_media_parts conversion
  • E2E test: Python TITO endpoint with content_parts (image URL → generation)
  • E2E test: Rust frontend with content_parts

Resolves vllm-project#51472

🤖 Generated with Claude Code

harjothkhara and others added 30 commits August 4, 2026 14:56
…roject#50879)

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
…0867)

Signed-off-by: Anuj Bolewar <anujbolewar@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Anuj Bolewar <anujbolewar@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…oject#49558)

Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com>
Co-authored-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…lm-project#50157)

Signed-off-by: amitz-nv <203509407+amitz-nv@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
…9919)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: lkchen <anthropic@lkchen.net>
Co-authored-by: lkchen <anthropic@lkchen.net>
…6.98 GiB memory/GPU saved (vllm-project#50912)

Signed-off-by: yewentao256 <zhyanwentao@126.com>
…-project#50911)

Signed-off-by: Shreyas Misra <shreyasm@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
…t collective (vllm-project#50697)

Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>
Signed-off-by: 墨楼 <huangzhilin.hzl@antgroup.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Peiyuan Zhou <peiyuanzhou1994@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
…t#50323)

Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
…ndex (vllm-project#48061)

Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
…oject#50607)

Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
…lel,test_async_tp]` (vllm-project#51068)

Signed-off-by: mgoin <mgoin64@gmail.com>
…inism tests (vllm-project#50905)

Signed-off-by: Divakar Verma <divakar.verma@amd.com>
…ct#50404)

Signed-off-by: Tasos Varoudis <varoudis@archtech.gr>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lai, Yejing <yejing.lai@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
Address reviewer feedback: use the flat MediaContentPart schema
({"type": "image_url", "url": "..."}) instead of the nested OpenAI
chat format. This eliminates the ChatContentPart intermediate type
and the content_parts_to_media_parts conversion function in Rust.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Allen Shen <aoshen@inferact.ai>
@aoshen02
aoshen02 force-pushed the feat/generate-raw-multimodal branch from c643af4 to 3c77b55 Compare August 10, 2026 02:06
Srinivasoo7 and others added 14 commits August 10, 2026 05:23
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
…nd per-expert checkpoint mapping (vllm-project#51419)

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
…51265)

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…duler (vllm-project#49579)

Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
vllm-project#48171)

Signed-off-by: Zetian Li <804561096@qq.com>
Co-authored-by: Claude (Anthropic) <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
…ad (vllm-project#48414)

Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
- Use MEDIA_CONNECTOR_REGISTRY + fetch_*_async instead of chat_utils
- Remove skip_mm_cache so RL multi-epoch rollouts benefit from cache
- Add model_validator to reject content_parts + features together

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Allen Shen <aoshen@inferact.ai>
…ct#51603)

Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
Co-authored-by: Akshat Anand <40275336+cipheraxat@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
@aoshen02
aoshen02 force-pushed the feat/generate-raw-multimodal branch from 166c624 to 6938a7e Compare August 10, 2026 09:05
Switch back to AsyncMultiModalItemTracker per Cyrus's feedback.
This reuses the same connector setup as chat completions,
including SSRF protections (allowed_media_domains, etc.).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Allen Shen <aoshen@inferact.ai>
@aoshen02
aoshen02 force-pushed the feat/generate-raw-multimodal branch from 6938a7e to 0e2c69c Compare August 10, 2026 09:13
zxd1997066 and others added 11 commits August 10, 2026 17:32
…ses in Intel GPU CI (vllm-project#51604)

Signed-off-by: zengxian <xiangdong.zeng@intel.com>
Signed-off-by: Emmanuel Acheampong <achampion.emma@gmail.com>
- Add missing content_parts: None in render.rs GenerateRequest init
- Fix import ordering in types.rs (rustfmt)
- Fix chain call formatting in generate.rs (rustfmt)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Allen Shen <aoshen@inferact.ai>
…ct#50734)

Signed-off-by: efschu <51944948+efschu@users.noreply.github.com>
…in YAML config (vllm-project#51573)

Signed-off-by: Raj Firke <79653531+rajfirke@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…llm-project#51635)

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
…oject#51657)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Merge into test_serving_multimodal_tokens.py to share the server
fixture and avoid launching a second Qwen3-VL instance in CI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Allen Shen <aoshen@inferact.ai>
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Multimodal RL inputs for /inference/v1/generate