Add pipeline parallel support to `TransformersModel` by hmellor · Pull Request #12832 · vllm-project/vllm

hmellor · 2025-02-06T14:15:16Z

Merge after:

Tested with Llama 3.1 70B in pp=4 and pp2,tp2 configurations.

The following command can be used for testing:

CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Llama-3.2-1B-Instruct --disable-log-requests --model-impl transformers --gpu-memory-utilization 0.25 --pipeline-parallel-size 2

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

github-actions · 2025-02-06T14:15:28Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

vllm/model_executor/models/transformers.py

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify · 2025-02-12T11:58:40Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

hmellor · 2025-02-12T12:01:06Z

Can only be merged after Transformers v4.49, but marking as ready for review so this can be merged soon after.

andoorve

Comments addressed, approved!

Isotr0py

Overall LGTM! Just a nit, PTAL!

vllm/model_executor/models/transformers.py

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify · 2025-02-13T21:29:35Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify · 2025-02-14T11:25:38Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

DarkLight1337 · 2025-02-22T13:55:30Z

Let's update this after #13634 to ensure that nothing is broken.

mergify · 2025-03-02T17:24:26Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor · 2025-03-24T20:21:32Z

The test passes locally, but I've commented it out so that we are not blocked by #13905

I'd be keen to get this merged because it makes quite a few changes to the TransformersModel without breaking it for non-PP cases. Merging would allow me to commit smaller QoL changes and pave the way for VLM support.

DarkLight1337

Sure, sounds good.

) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <2037008807@qq.com>

) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <2037008807@qq.com> Signed-off-by: Wes Medford <wryanmedford@gmail.com>

) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <2037008807@qq.com> Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>

) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <2037008807@qq.com>

) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <2037008807@qq.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

hmellor added 3 commits February 6, 2025 09:14

Add pipeline parallel support to TransformersModel

04a93de

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Simplify weight loading

099b3e0

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Only allocate tensors for current pipeline stage

8b1ea1d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Don't set buffers to empty right after initialising them

ec626e8

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

andoorve reviewed Feb 7, 2025

View reviewed changes

vllm/model_executor/models/transformers.py Outdated Show resolved Hide resolved

vllm/model_executor/models/transformers.py Outdated Show resolved Hide resolved

vllm/model_executor/models/transformers.py Show resolved Hide resolved

hmellor added 4 commits February 7, 2025 19:07

Update to use _pp_plan and _tp_plan

8eafc6d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Respond to comment

87c395c

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Add docstring to init_buffers

3498758

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Update some comments

e652d50

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor marked this pull request as ready for review February 12, 2025 11:58

mergify bot added the needs-rebase label Feb 12, 2025

Merge branch 'main' into pipeline-parallel

e756f27

mergify bot removed the needs-rebase label Feb 12, 2025

hmellor requested a review from andoorve February 12, 2025 15:47

andoorve approved these changes Feb 12, 2025

View reviewed changes

Isotr0py approved these changes Feb 13, 2025

View reviewed changes

vllm/model_executor/models/transformers.py Outdated Show resolved Hide resolved

vllm/model_executor/models/transformers.py Show resolved Hide resolved

hmellor added 2 commits February 13, 2025 21:44

Rename inner tensor parallel method

95cdfcf

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Add test

14e84e8

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor requested a review from youkaichao as a code owner February 13, 2025 21:28

mergify bot added the needs-rebase label Feb 13, 2025

Merge branch 'main' into pipeline-parallel

ce42695

mergify bot removed the needs-rebase label Feb 13, 2025

Use fallback model for tests instead

a591f44

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify bot added the needs-rebase label Feb 14, 2025

Resintate weight renaming in load_weights

fd0445a

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Merge branch 'main' into pipeline-parallel

4f414be

hmellor added 2 commits February 24, 2025 11:40

Merge branch 'main' into pipeline-parallel

9f60911

Merge branch 'main' into pipeline-parallel

3e476ee

mergify bot added the needs-rebase label Mar 2, 2025

hmellor mentioned this pull request Mar 17, 2025

[Model] Support VLMs with transformers backend #13754

Closed

hmellor added 5 commits March 19, 2025 15:25

Merge branch 'main' into pipeline-parallel

54f9928

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Stop passing intermediate_tensors to the HF model

3ef3fe2

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Revert requirements changes

cadfbd2

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Disable test

b84a845

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Update docs

0216271

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify bot added documentation Improvements or additions to documentation and removed needs-rebase labels Mar 24, 2025

DarkLight1337 approved these changes Mar 25, 2025

View reviewed changes

DarkLight1337 merged commit 97cfa65 into vllm-project:main Mar 25, 2025
59 checks passed

hmellor deleted the pipeline-parallel branch March 25, 2025 10:41

XuehaiPan mentioned this pull request Apr 16, 2025

[RFC][Feature][Model] Add templated fallback HF transformers model backend in SRT sgl-project/sglang#5471

Closed

6 tasks

ckhordiasma mentioned this pull request Apr 17, 2025

[do not merge] pr test for nm changes into 2.20 red-hat-data-services/vllm#107

Closed

hmellor added this to Transformers backend Sep 24, 2025

hmellor moved this to Done in Transformers backend Sep 24, 2025

Uh oh!

Conversation

hmellor commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Feb 12, 2025

Uh oh!

hmellor commented Feb 12, 2025

Uh oh!

andoorve left a comment

Choose a reason for hiding this comment

Uh oh!

Isotr0py left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Feb 13, 2025

Uh oh!

mergify bot commented Feb 14, 2025

Uh oh!

DarkLight1337 commented Feb 22, 2025

Uh oh!

mergify bot commented Mar 2, 2025

Uh oh!

hmellor commented Mar 24, 2025

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hmellor commented Feb 6, 2025 •

edited

Loading