Make the Transformers modeling backend as fast as native vLLM by hmellor · Pull Request #47187 · vllm-project/vllm

hmellor · 2026-06-30T17:14:00Z

Introduction

For the Transformers modelling backend this PR:

enables the use of MergedColumnParallelLinear and QKVParallelLinear
fully leverages MoERunner to enable:
- async shared experts
- DBO
- Sequence parallel
tp_plan becomes optional because the fusion passes know correct the TP orientations of the resulting fusions
pp_plan becomes optional because vLLM now checks for the nn.ModuleList in the text model and splits that
Fixes a TP bug with OLMoE style models by adding gather/scatter around full width norms

It does this by using torch.fx to detect patterns in the original model and then editing the model instance in place using ast so nothing gets baked in for downstream torch.compile calls.

Logs indicate when and where these graph manipulations are happening, e.g.:

INFO [moe.py:281] Fused: gate + experts (Qwen3MoeSparseMoeBlock) -> FusedMoE (internal routing)
...
INFO [base.py:505] Fused: q_proj + k_proj + v_proj (self_attn: Qwen3MoeAttention) -> qkv_proj (QKVParallelLinear)
INFO [base.py:505] Fused: q_norm (Qwen3MoeRMSNorm) -> RMSNorm (CustomOp)
INFO [base.py:505] Fused: k_norm (Qwen3MoeRMSNorm) -> RMSNorm (CustomOp)
INFO [base.py:505] Fused: input_layernorm (Qwen3MoeRMSNorm) -> RMSNorm (CustomOp)
INFO [base.py:505] Fused: post_attention_layernorm (Qwen3MoeRMSNorm) -> RMSNorm (CustomOp)
INFO [base.py:505] Fused: norm (Qwen3MoeRMSNorm) -> RMSNorm (CustomOp)

Benchmarks

The Transformers modelling backend is now as fast or faster than vLLM implementations for dense/sliding attention and MoE models across all concurrencies tested.

Hybrid full/sliding attention

Run on 1xH100

vllm serve google/gemma-2-2b-it

FP8 MoE

Run on 8xH100

vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --enable-expert-parallel --tensor-parallel-size 8

Future work

Identify low-medium usage models which can be migrated to the Transformers modelling backend to reduce maintenance burden for vLLM
MLA

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify · 2026-07-02T10:17:00Z

Documentation preview: https://vllm--47187.org.readthedocs.build/en/47187/

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Isotr0py

Great idea about layer fusion through fx graph!

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…roject#47187) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

hmellor added 5 commits June 30, 2026 15:41

More sensible name for vllm_attention_forward

9254562

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Add fusers

54dd407

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Use fusers as part of recursive_replace

5a67279

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Check for fusions in general transformers backend tests

d9ec10e

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fuser tests

b7180c2

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

github-project-automation Bot added this to Transformers modeling backend Jun 30, 2026

github-project-automation Bot moved this to Todo in Transformers modeling backend Jun 30, 2026

hmellor added 2 commits July 1, 2026 09:15

rework a bit

4293df9

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Actually run the new fusion tests

2772c10

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify Bot added the ci/build label Jul 1, 2026

hmellor added 4 commits July 1, 2026 13:47

Refactor so each fuser gets its own file and fx utils are separated

6ef790d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Use Fuser for RMSNorm too

f12eb72

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

update tests

16ff001

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Log fusion info

6c4debd

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor changed the title ~~Enable merged linear layers in the Transformers modeling backend~~ Make the Transformers modeling backend as fast as native vLLM Jul 1, 2026

hmellor added 4 commits July 1, 2026 19:52

Use fusion mechanism for MoE and RMSNorm

7d3a751

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Update perf expectations in doc

83da8be

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Support torch.nn.RMSNorm

f9a601b

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Update tests

0cc2803

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify Bot added the documentation Improvements or additions to documentation label Jul 2, 2026

hmellor added 8 commits July 2, 2026 10:30

Concise RMSNorm comments

b6b12dd

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

renaming

9c408ba

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Consistency pass

126e5df

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

remove print

6faa797

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Attempt replacement of down_proj and o_proj

409db2d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Make pp_plan optional

df4bb49

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Small fix

d90bcd9

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Update tests

a113e2d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Isotr0py reviewed Jul 3, 2026

View reviewed changes

Comment thread tests/models/transformers/fusers/test_linear.py

Comment thread vllm/model_executor/models/transformers/fusers/qkv.py

Update tests

6069c68

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor marked this pull request as ready for review July 3, 2026 13:05

hmellor requested review from AndreasKaratzas, DarkLight1337, Harry-Chen, khluu and ywang96 as code owners July 3, 2026 13:05

claude Bot reviewed Jul 3, 2026

View reviewed changes

Isotr0py approved these changes Jul 4, 2026

View reviewed changes

Isotr0py added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 4, 2026

mergify Bot and others added 4 commits July 4, 2026 02:28

Merge branch 'main' into transformers-fusions

8ecd677

Fix tests

23b265e

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Merge branch 'main' into transformers-fusions

d70a5a7

can't pickle local function

702e5a2

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor merged commit 5bce653 into vllm-project:main Jul 6, 2026
81 checks passed

hmellor deleted the transformers-fusions branch July 6, 2026 15:59

github-project-automation Bot moved this from Todo to Done in Transformers modeling backend Jul 6, 2026

vllm-agent mentioned this pull request Jul 7, 2026

Revert "Make the Transformers modeling backend as fast as native vLLM" (#47187) #47798

Closed

AndreasKaratzas mentioned this pull request Jul 7, 2026

[CI][LoRA] Preserve QKV fuser split sizes through LoRA wrappers #47804

Closed

hmellor mentioned this pull request Jul 7, 2026

[CI] Fix Transformers modeling backend LoRA test #47832

Merged

This was referenced Jul 28, 2026

[Model][Transformers backend] Fuse q/k/v and gate/up linear groups #45284

Closed

[Performance] Measure Transformers backend startup time vs native #50128

Open

hmellor mentioned this pull request Sep 1, 2026

[transformer] RMSNorm matching for alternative rsqrt #54461

Merged

3 of 4 tasks

Navigation Menu

Repository navigation

Make the Transformers modeling backend as fast as native vLLM - #47187

Make the Transformers modeling backend as fast as native vLLM#47187
hmellor merged 37 commits into
vllm-project:mainfrom
hmellor:transformers-fusions

hmellor commented Jun 30, 2026 •

edited

Loading

Uh oh!

mergify Bot commented Jul 2, 2026

Uh oh!

Isotr0py left a comment

Uh oh!

Uh oh!

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

hmellor commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Introduction

Benchmarks

Hybrid full/sliding attention

FP8 MoE

Future work

Uh oh!

mergify Bot commented Jul 2, 2026

Uh oh!

Isotr0py left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hmellor commented Jun 30, 2026 •

edited

Loading