Skip to content

[5/N] Initialize MM components in context managers (Q-Z)#32695

Merged
DarkLight1337 merged 2 commits intovllm-project:mainfrom
DarkLight1337:mm-ctx-5
Jan 20, 2026
Merged

[5/N] Initialize MM components in context managers (Q-Z)#32695
DarkLight1337 merged 2 commits intovllm-project:mainfrom
DarkLight1337:mm-ctx-5

Conversation

@DarkLight1337
Copy link
Member

@DarkLight1337 DarkLight1337 commented Jan 20, 2026

Purpose

Part of #32631

  • Skipped Qwen-VL and Whisper as the MM and LM parts are in the same child module.
  • Skipped Terratorch impl as there is only MM component but not LM component.
  • Skipping Transformers impl for now; we may revisit this after adding video support ([Model] Add video input support for transformers modeling backend #30680) and getting a better idea of how to target the modules to skip.

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 20, 2026
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@mergify mergify bot added the qwen Related to Qwen models label Jan 20, 2026
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the initialization of multimodal components across several models (Qwen2-Audio, Qwen3-Omni-MoE-Thinker, Siglip, SkyworkR1V, Tarsier, Ultravox, Voxtral) by introducing _mark_tower_model and _mark_language_model context managers. This change enhances modularity and allows for more flexible management of model sub-components, particularly for scenarios like multimodal encoder-only modes where certain parts of the model might be skipped. The removal of explicit get_language_model methods and some assert statements indicates a shift towards a more streamlined and potentially more robust initialization pattern, where the presence of components is either guaranteed by design or handled gracefully within the new context-managed blocks. The additions of getattr checks for deepstack_input_embeds in qwen3_omni_moe_thinker.py are good for ensuring robustness when components might be conditionally initialized.

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) January 20, 2026 16:54
@DarkLight1337 DarkLight1337 merged commit 193069d into vllm-project:main Jan 20, 2026
62 checks passed
@DarkLight1337 DarkLight1337 deleted the mm-ctx-5 branch January 20, 2026 19:10
gopalsarda pushed a commit to gopalsarda/vllm that referenced this pull request Jan 20, 2026
…t#32695)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
dsuhinin pushed a commit to dsuhinin/vllm that referenced this pull request Jan 21, 2026
…t#32695)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: dsuhinin <suhinin.dmitriy@gmail.com>
monajafi-amd pushed a commit to monajafi-amd/vllm that referenced this pull request Jan 23, 2026
…t#32695)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mohammad najafi <mohammad.najafi@amd.com>
lapy pushed a commit to lapy/vllm that referenced this pull request Jan 27, 2026
…t#32695)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
ItzDEXX pushed a commit to ItzDEXX/vllm that referenced this pull request Feb 19, 2026
…t#32695)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

qwen Related to Qwen models ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants