Skip to content

Fix pipeline parallel with multimodal models with the Transformers modelling backend#37057

Merged
DarkLight1337 merged 1 commit intovllm-project:mainfrom
hmellor:fix-pp-with-mm-models
Mar 16, 2026
Merged

Fix pipeline parallel with multimodal models with the Transformers modelling backend#37057
DarkLight1337 merged 1 commit intovllm-project:mainfrom
hmellor:fix-pp-with-mm-models

Conversation

@hmellor
Copy link
Copy Markdown
Member

@hmellor hmellor commented Mar 14, 2026

In Transformers v4 tie_word_embeddings is found in the text model config, but in v5 it's found in the top level config. The Transformers modelling backend now checks both places.

Base.pipeline_parallel was not able to apply the pipeline parallel plan to multimodal models because the plan referred to nested modules (text_model.layers for example). This Transformers modelling backend now uses attrgetter (and a custom inverse attrsetter) to make sure that nested modules can still be accessed.

…delling backend

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces two main improvements. First, it adds support for pipeline parallelism in multimodal models by using attrgetter and a new attrsetter helper to handle nested modules, which is a clean and effective solution. Second, it refactors the logic for checking tie_word_embeddings to support both Transformers v4 and v5 configurations, centralizing this logic in a new _get_tie_word_embeddings method. The changes are well-implemented, correctly address the described issues, and improve the overall robustness and feature set of the Transformers modeling backend. The code is clear and the approach is solid.

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) March 16, 2026 08:12
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 16, 2026
@DarkLight1337 DarkLight1337 merged commit 122f75d into vllm-project:main Mar 16, 2026
54 checks passed
@github-project-automation github-project-automation bot moved this from Todo to Done in Transformers backend Mar 16, 2026
@hmellor hmellor deleted the fix-pp-with-mm-models branch March 16, 2026 10:32
Lucaskabela pushed a commit to Lucaskabela/vllm that referenced this pull request Mar 17, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
wendyliu235 pushed a commit to wendyliu235/vllm-public that referenced this pull request Mar 18, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
fxdawnn pushed a commit to fxdawnn/vllm that referenced this pull request Mar 19, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
khairulkabir1661 pushed a commit to khairulkabir1661/vllm that referenced this pull request Mar 27, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Monishver11 pushed a commit to Monishver11/vllm that referenced this pull request Mar 27, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com>
JiantaoXu pushed a commit to JiantaoXu/vllm that referenced this pull request Mar 28, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
vrdn-23 pushed a commit to vrdn-23/vllm that referenced this pull request Mar 30, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Vinay Damodaran <vrdn@hey.com>
EricccYang pushed a commit to EricccYang/vllm that referenced this pull request Apr 1, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: EricccYang <yangyang4991@gmail.com>
liuchenbing2026 pushed a commit to liuchenbing2026/vllm that referenced this pull request Apr 4, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
big-yellow-duck pushed a commit to EmbeddedLLM/vllm that referenced this pull request Apr 8, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
mtparet pushed a commit to blackfuel-ai/vllm that referenced this pull request Apr 9, 2026
…delling backend (vllm-project#37057)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants