[Models]: Make Multimodal config implicit in ViT implementation by Isotr0py · Pull Request #31972 · vllm-project/vllm

Isotr0py · 2026-01-08T13:20:23Z

Purpose

Currently, we are passing MultimodalConfig through whole MMEncoder only to enable data parallel and attention backend overrides
This PR makes it implicit through get_current_vllm_config.

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

gemini-code-assist

Code Review

This pull request refactors the codebase to make the MultimodalConfig implicit in ViT implementations by using get_current_vllm_config. The changes are generally in the right direction, but I've identified a couple of critical issues. There's a bug in the new helper function is_vit_use_data_parallel that could lead to incorrect behavior with data parallelism. Additionally, the refactoring is incomplete for Qwen2_5_VisionMLP in qwen2_5_vl.py, which would also break data parallelism. I've provided detailed comments and suggestions to address these issues.

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify · 2026-01-09T21:24:40Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…-config Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify · 2026-01-20T06:48:24Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

…-config Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify · 2026-01-24T03:01:28Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…-config Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

…-project#31972) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: 陈建华 <1647430658@qq.com>

### What this PR does / why we need it? This PR upgrades the vLLM dependency from `v0.14.1` to `v0.15.0`. This involves: - Updating the `VLLM_TAG` in all `Dockerfile`. - Updating the vLLM version in `docs/source/conf.py`. - Removing conditional code paths specific to `v0.14.1` across the codebase, which simplifies maintenance. - Fix `TypeError: MMEncoderAttention.__init__() got an unexpected keyword argument 'multimodal_config'` due to vllm-project/vllm#31972. - Fix `_shared_experts: 'NoneType' object is not callable` due to vllm-project/vllm#32082 by #6335. - Fix `ReshapeAndCacheOperation setup failed!` due to vllm-project/vllm#25954 by overriding attention metadata slots. This upgrade is necessary to keep the project aligned with the latest features, bug fixes, and API changes in the vLLM project. ### Does this PR introduce _any_ user-facing change? No, this is an internal dependency update and does not introduce any user-facing changes. ### How was this patch tested? CI is expected to pass with these changes, ensuring that all existing tests are successful with the new vLLM version. - vLLM version: v0.14.1 - vLLM main: vllm-project/vllm@dc917cc co-authored-by: shen-shanshan <467638484@qq.com> --------- Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

### What this PR does / why we need it? This PR upgrades the vLLM dependency from `v0.14.1` to `v0.15.0`. This involves: - Updating the `VLLM_TAG` in all `Dockerfile`. - Updating the vLLM version in `docs/source/conf.py`. - Removing conditional code paths specific to `v0.14.1` across the codebase, which simplifies maintenance. - Fix `TypeError: MMEncoderAttention.__init__() got an unexpected keyword argument 'multimodal_config'` due to vllm-project/vllm#31972. - Fix `_shared_experts: 'NoneType' object is not callable` due to vllm-project/vllm#32082 by vllm-project#6335. - Fix `ReshapeAndCacheOperation setup failed!` due to vllm-project/vllm#25954 by overriding attention metadata slots. This upgrade is necessary to keep the project aligned with the latest features, bug fixes, and API changes in the vLLM project. ### Does this PR introduce _any_ user-facing change? No, this is an internal dependency update and does not introduce any user-facing changes. ### How was this patch tested? CI is expected to pass with these changes, ensuring that all existing tests are successful with the new vLLM version. - vLLM version: v0.14.1 - vLLM main: vllm-project/vllm@dc917cc co-authored-by: shen-shanshan <467638484@qq.com> --------- Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

### What this PR does / why we need it? This PR upgrades the vLLM dependency from `v0.14.1` to `v0.15.0`. This involves: - Updating the `VLLM_TAG` in all `Dockerfile`. - Updating the vLLM version in `docs/source/conf.py`. - Removing conditional code paths specific to `v0.14.1` across the codebase, which simplifies maintenance. - Fix `TypeError: MMEncoderAttention.__init__() got an unexpected keyword argument 'multimodal_config'` due to vllm-project/vllm#31972. - Fix `_shared_experts: 'NoneType' object is not callable` due to vllm-project/vllm#32082 by vllm-project#6335. - Fix `ReshapeAndCacheOperation setup failed!` due to vllm-project/vllm#25954 by overriding attention metadata slots. This upgrade is necessary to keep the project aligned with the latest features, bug fixes, and API changes in the vLLM project. ### Does this PR introduce _any_ user-facing change? No, this is an internal dependency update and does not introduce any user-facing changes. ### How was this patch tested? CI is expected to pass with these changes, ensuring that all existing tests are successful with the new vLLM version. - vLLM version: v0.14.1 - vLLM main: vllm-project/vllm@dc917cc co-authored-by: shen-shanshan <467638484@qq.com> --------- Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com> Signed-off-by: momochenchuw <chenchuw@huawei.com>

### What this PR does / why we need it? This PR upgrades the vLLM dependency from `v0.14.1` to `v0.15.0`. This involves: - Updating the `VLLM_TAG` in all `Dockerfile`. - Updating the vLLM version in `docs/source/conf.py`. - Removing conditional code paths specific to `v0.14.1` across the codebase, which simplifies maintenance. - Fix `TypeError: MMEncoderAttention.__init__() got an unexpected keyword argument 'multimodal_config'` due to vllm-project/vllm#31972. - Fix `_shared_experts: 'NoneType' object is not callable` due to vllm-project/vllm#32082 by vllm-project#6335. - Fix `ReshapeAndCacheOperation setup failed!` due to vllm-project/vllm#25954 by overriding attention metadata slots. This upgrade is necessary to keep the project aligned with the latest features, bug fixes, and API changes in the vLLM project. ### Does this PR introduce _any_ user-facing change? No, this is an internal dependency update and does not introduce any user-facing changes. ### How was this patch tested? CI is expected to pass with these changes, ensuring that all existing tests are successful with the new vLLM version. - vLLM version: v0.14.1 - vLLM main: vllm-project/vllm@dc917cc co-authored-by: shen-shanshan <467638484@qq.com> --------- Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com> Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>

### What this PR does / why we need it? This PR upgrades the vLLM dependency from `v0.14.1` to `v0.15.0`. This involves: - Updating the `VLLM_TAG` in all `Dockerfile`. - Updating the vLLM version in `docs/source/conf.py`. - Removing conditional code paths specific to `v0.14.1` across the codebase, which simplifies maintenance. - Fix `TypeError: MMEncoderAttention.__init__() got an unexpected keyword argument 'multimodal_config'` due to vllm-project/vllm#31972. - Fix `_shared_experts: 'NoneType' object is not callable` due to vllm-project/vllm#32082 by vllm-project#6335. - Fix `ReshapeAndCacheOperation setup failed!` due to vllm-project/vllm#25954 by overriding attention metadata slots. This upgrade is necessary to keep the project aligned with the latest features, bug fixes, and API changes in the vLLM project. ### Does this PR introduce _any_ user-facing change? No, this is an internal dependency update and does not introduce any user-facing changes. ### How was this patch tested? CI is expected to pass with these changes, ensuring that all existing tests are successful with the new vLLM version. - vLLM version: v0.14.1 - vLLM main: vllm-project/vllm@dc917cc co-authored-by: shen-shanshan <467638484@qq.com> --------- Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

### What this PR does / why we need it? This PR upgrades the vLLM dependency from `v0.14.1` to `v0.15.0`. This involves: - Updating the `VLLM_TAG` in all `Dockerfile`. - Updating the vLLM version in `docs/source/conf.py`. - Removing conditional code paths specific to `v0.14.1` across the codebase, which simplifies maintenance. - Fix `TypeError: MMEncoderAttention.__init__() got an unexpected keyword argument 'multimodal_config'` due to vllm-project/vllm#31972. - Fix `_shared_experts: 'NoneType' object is not callable` due to vllm-project/vllm#32082 by vllm-project#6335. - Fix `ReshapeAndCacheOperation setup failed!` due to vllm-project/vllm#25954 by overriding attention metadata slots. This upgrade is necessary to keep the project aligned with the latest features, bug fixes, and API changes in the vLLM project. ### Does this PR introduce _any_ user-facing change? No, this is an internal dependency update and does not introduce any user-facing changes. ### How was this patch tested? CI is expected to pass with these changes, ensuring that all existing tests are successful with the new vLLM version. - vLLM version: v0.14.1 - vLLM main: vllm-project/vllm@dc917cc co-authored-by: shen-shanshan <467638484@qq.com> --------- Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com> Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>

### What this PR does / why we need it? This PR upgrades the vLLM dependency from `v0.14.1` to `v0.15.0`. This involves: - Updating the `VLLM_TAG` in all `Dockerfile`. - Updating the vLLM version in `docs/source/conf.py`. - Removing conditional code paths specific to `v0.14.1` across the codebase, which simplifies maintenance. - Fix `TypeError: MMEncoderAttention.__init__() got an unexpected keyword argument 'multimodal_config'` due to vllm-project/vllm#31972. - Fix `_shared_experts: 'NoneType' object is not callable` due to vllm-project/vllm#32082 by vllm-project#6335. - Fix `ReshapeAndCacheOperation setup failed!` due to vllm-project/vllm#25954 by overriding attention metadata slots. This upgrade is necessary to keep the project aligned with the latest features, bug fixes, and API changes in the vLLM project. ### Does this PR introduce _any_ user-facing change? No, this is an internal dependency update and does not introduce any user-facing changes. ### How was this patch tested? CI is expected to pass with these changes, ensuring that all existing tests are successful with the new vLLM version. - vLLM version: v0.14.1 - vLLM main: vllm-project/vllm@dc917cc co-authored-by: shen-shanshan <467638484@qq.com> --------- Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

Isotr0py added 2 commits January 8, 2026 20:32

clean

0d0ce5d

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

refine

7294fa6

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot added the qwen Related to Qwen models label Jan 8, 2026

gemini-code-assist Bot reviewed Jan 8, 2026

View reviewed changes

Comment thread vllm/model_executor/models/qwen2_5_vl.py

Comment thread vllm/model_executor/models/vision.py Outdated

fix

4c02e0e

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot added the needs-rebase label Jan 9, 2026

Merge remote-tracking branch 'upstream/main' into implicit-multimodal…

1cd3664

…-config Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot removed the needs-rebase label Jan 10, 2026

Isotr0py added 7 commits January 18, 2026 22:15

Merge branch 'main' into implicit-multimodal-config

8e465ea

update dots ocr

b79b456

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

update

c93d07e

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean

58c1d17

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean clip

f9b2ab5

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean clip

32cfdbf

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean deep ocr

20eb759

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot added the deepseek Related to DeepSeek models label Jan 19, 2026

Isotr0py added 9 commits January 19, 2026 16:48

clean

ed6289a

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

update ernie45_vl

ba4e16b

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean glm4_1v

dc2102c

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean

b46db51

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

further clean hunyuan

4360ba0

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

moonvit, pixtral and siglip2navit

9b65faf

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

update paddle ocr

06da65e

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean qwen3 and interface

a2b409c

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean _init_tower

a5b63b7

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot added the needs-rebase label Jan 20, 2026

Isotr0py added 2 commits January 20, 2026 14:50

final clean

c85d43d

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Merge remote-tracking branch 'upstream/main' into implicit-multimodal…

16f9278

…-config Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Isotr0py requested a review from sighingnow as a code owner January 22, 2026 14:22

mergify Bot removed the needs-rebase label Jan 22, 2026

fix

103b649

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Isotr0py requested a review from DarkLight1337 January 22, 2026 14:32

Isotr0py added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 22, 2026

miss

f0d7a8f

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot added the speculative-decoding label Jan 23, 2026

DarkLight1337 approved these changes Jan 24, 2026

View reviewed changes

mergify Bot added the needs-rebase label Jan 24, 2026

Merge remote-tracking branch 'upstream/main' into implicit-multimodal…

7b2ac61

…-config Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot removed the needs-rebase label Jan 24, 2026

Isotr0py merged commit 9ad7f89 into vllm-project:main Jan 24, 2026
58 checks passed

Isotr0py deleted the implicit-multimodal-config branch January 24, 2026 12:34

DarkLight1337 mentioned this pull request Jan 25, 2026

[GLM-OCR] GLM-OCR with MTP Support #33005

Merged

cwazai pushed a commit to cwazai/vllm that referenced this pull request Jan 25, 2026

[Models]: Make Multimodal config implicit in ViT implementation (vllm…

bc2a887

…-project#31972) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: 陈建华 <1647430658@qq.com>

LucasWilkinson mentioned this pull request Jan 25, 2026

[CI] Fix MHA attention test failure (AttributeError when model_config is None in ViT attention backend) #33033

Merged

shen-shanshan mentioned this pull request Jan 27, 2026

[Main2Main] Upgrade vllm commit to v0.15.0rc0 vllm-project/vllm-ascend#6304

Closed

wangxiyuan mentioned this pull request Feb 2, 2026

[Main2Main][Deps][Misc] Upgrade vLLM to v0.15.0 vllm-project/vllm-ascend#6470

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Models]: Make Multimodal config implicit in ViT implementation#31972

[Models]: Make Multimodal config implicit in ViT implementation#31972
Isotr0py merged 27 commits intovllm-project:mainfrom
Isotr0py:implicit-multimodal-config

Isotr0py commented Jan 8, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented Jan 9, 2026

Uh oh!

mergify Bot commented Jan 20, 2026

Uh oh!

mergify Bot commented Jan 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Isotr0py commented Jan 8, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented Jan 9, 2026

Uh oh!

mergify Bot commented Jan 20, 2026

Uh oh!

mergify Bot commented Jan 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Isotr0py commented Jan 8, 2026 •

edited by github-actions Bot

Loading