Skip to content

[Core] Enable inputs_embeds_size separate from hidden_size#29741

Merged
DarkLight1337 merged 13 commits intovllm-project:mainfrom
DarkLight1337:inputs-embeds-size
Nov 30, 2025
Merged

[Core] Enable inputs_embeds_size separate from hidden_size#29741
DarkLight1337 merged 13 commits intovllm-project:mainfrom
DarkLight1337:inputs-embeds-size

Conversation

@DarkLight1337
Copy link
Copy Markdown
Member

@DarkLight1337 DarkLight1337 commented Nov 30, 2025 •
edited by github-actions bot
Loading

Purpose

  • Add a new method ModelConfig.get_inputs_embeds_size to distinguish the hidden dimension of inputs_embeds from ModelConfig.get_hidden_size.
  • Update model runners to use ModelConfig.get_inputs_embeds_size. (Notable for OOT model runners)
  • Update CLIP and SigLIP to handle the case when get_inputs_embeds_size and get_hidden_size return different values.

FIX #27566 (comment)

cc @piood @patrickvonplaten @wangxiyuan

Test Plan

Add google/siglip2-giant-opt-patch16-384 to SigLIP tests

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@DarkLight1337
[Core] Enable inputs_embeds_size separate from hidden_size
fbcf6ae 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Add test
dd70d6b 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 30, 2025
@chatgpt-codex-connector
Copy link
Copy Markdown

chatgpt-codex-connector bot commented Nov 30, 2025

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

@mergify mergify bot added multi-modality Related to multi-modality (#4194) speculative-decoding v1 tpu Related to Google TPUs labels Nov 30, 2025
@DarkLight1337 DarkLight1337 mentioned this pull request Nov 30, 2025
Merged
5 tasks
@DarkLight1337
Comment
1dba085 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
gemini-code-assist[bot]
gemini-code-assist bot reviewed Nov 30, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for models where inputs_embeds_size is different from hidden_size, which is a great addition for supporting more multimodal models. The changes are mostly about propagating this new configuration. However, I found a critical issue in the implementation of get_inputs_embeds_size which could lead to incorrect behavior for the intended models. My review includes a fix for this issue.

@DarkLight1337
Patch siglip
c5f1b91 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Patch
ee6bf12 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Comment
e489b0e 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Wrong dim
9c3d65e 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337 DarkLight1337 removed the ready ONLY add when PR is ready to merge/full CI is needed label Nov 30, 2025
@DarkLight1337
Fix
0842458 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Update
9b894a8 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Reword
20b5d56 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Fixes
e521f58 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Comment
5972e28 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 30, 2025
@DarkLight1337
Copy link
Copy Markdown
Member Author

DarkLight1337 commented Nov 30, 2025

OK the test passes locally now.

@DarkLight1337 DarkLight1337 moved this to In Progress in Multi-modality Core Nov 30, 2025
DarkLight1337
DarkLight1337 commented Nov 30, 2025
View reviewed changes
Copy link
Copy Markdown
Member Author

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/gemini review

@DarkLight1337
Explicit unsupported case
504fe44 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337 DarkLight1337 merged commit 64bc09b into vllm-project:main Nov 30, 2025
57 checks passed
@DarkLight1337 DarkLight1337 deleted the inputs-embeds-size branch November 30, 2025 09:31
@github-project-automation github-project-automation bot moved this from In Progress to Done in Multi-modality Core Nov 30, 2025
@wangxiyuan
Copy link
Copy Markdown
Contributor

wangxiyuan commented Dec 1, 2025

Thanks for reminding. @Potabk Please consider this change when upgrading vLLM

@piood
Copy link
Copy Markdown
Contributor

piood commented Dec 1, 2025

Thanks for fix that!

kitaekatt pushed a commit to kitaekatt/vllm that referenced this pull request Dec 1, 2025
@DarkLight1337 @kitaekatt
[Core] Enable inputs_embeds_size separate from hidden_size (vllm-…
785c9fe 
…project#29741)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
amd-hhashemi pushed a commit to amd-hhashemi/vllm that referenced this pull request Dec 2, 2025
@DarkLight1337 @amd-hhashemi
[Core] Enable inputs_embeds_size separate from hidden_size (vllm-…
5590253 
…project#29741)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
dsuhinin pushed a commit to dsuhinin/vllm that referenced this pull request Jan 21, 2026
@DarkLight1337 @dsuhinin
[Core] Enable inputs_embeds_size separate from hidden_size (vllm-…
6fe20ac 
…project#29741)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: dsuhinin <suhinin.dmitriy@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Isotr0py Isotr0py Isotr0py approved these changes

@noooop noooop Awaiting requested review from noooop noooop is a code owner

@NickLucche NickLucche Awaiting requested review from NickLucche NickLucche is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@benchislett benchislett Awaiting requested review from benchislett benchislett is a code owner

@luccafong luccafong Awaiting requested review from luccafong luccafong is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@houseroad houseroad Awaiting requested review from houseroad houseroad is a code owner

@hmellor hmellor Awaiting requested review from hmellor hmellor is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

@ProExpertProg ProExpertProg Awaiting requested review from ProExpertProg ProExpertProg is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

multi-modality Related to multi-modality (#4194) ready ONLY add when PR is ready to merge/full CI is needed speculative-decoding tpu Related to Google TPUs v1

Projects

Multi-modality Core
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@DarkLight1337 @wangxiyuan @piood @Isotr0py