Skip to content

[MM] Keep device input normalization fused when encoder compilation is enabled - #59195

Merged
Isotr0py merged 3 commits into
vllm-project:mainfrom
jzakrzew:fix/pixtral-mm-input-norm-compile
Sep 30, 2026
Merged

[MM] Keep device input normalization fused when encoder compilation is enabled#59195
Isotr0py merged 3 commits into
vllm-project:mainfrom
jzakrzew:fix/pixtral-mm-input-norm-compile

Conversation

@jzakrzew

@jzakrzew jzakrzew commented Sep 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose

Keep device input normalization on its fused kernel when compile_mm_encoder=True.

Encoder compilation changes the default custom_ops setting to none. Pixtral’s pixtral_patch_embed() runs outside the compiled transformer blocks, so Inductor cannot fuse its normalization call. Without this change, FusedMMInputNorm falls back to the eager PyTorch path.

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@jzakrzew
Keep Pixtral input normalization fused with encoder compilation
d4fbdf0 
Assisted-by: OpenAI Codex

Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>
claude[bot]
claude Bot reviewed Sep 29, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@jzakrzew

jzakrzew commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

@noooop Can you take a look? This is something I missed in #57531

@mergify mergify Bot added the mistral Related to Mistral models label Sep 29, 2026
@noooop
noooop requested a review from Isotr0py September 29, 2026 09:54
Isotr0py
Isotr0py reviewed Sep 29, 2026
View reviewed changes
Comment thread vllm/model_executor/layers/fusion/mm_input_norm.py Outdated
Comment on lines +230 to +233
*,
enforce_enable: bool = False,
):
super().__init__()
super().__init__(enforce_enable=enforce_enable)

@Isotr0py Isotr0py Sep 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's always enable fusion for this ops directly like mhc:

vllm/vllm/model_executor/layers/mhc.py

Lines 118 to 120 in 998490c

@classmethod
def enabled(cls) -> bool:
return True

@jzakrzew jzakrzew Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, modified this PR to enable it unconditionally

@jzakrzew @codex
Always enable fused multimodal input normalization
23d27b9 
Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>
@jzakrzew jzakrzew changed the title [MM] Keep Pixtral device input normalization fused when encoder compilation is enabled [MM] Keep device input normalization fused when encoder compilation is enabled Sep 29, 2026
@Isotr0py
Isotr0py enabled auto-merge (squash) September 29, 2026 13:15
@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 29, 2026
@Isotr0py

Isotr0py commented Sep 29, 2026

Copy link
Copy Markdown
Member

/ci run

@Isotr0py

Isotr0py commented Sep 29, 2026

Copy link
Copy Markdown
Member

/ci run --allow-stale

@github-actions

github-actions Bot commented Sep 29, 2026

Copy link
Copy Markdown

❌ This PR is 9 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

@github-actions

github-actions Bot commented Sep 29, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #91856 for commit 23d27b97c31f.

⚠️ This PR is 9 commits behind upstream main. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest main, then rerun /ci run on the latest PR commit.

@jzakrzew

jzakrzew commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

github-actions Bot commented Sep 30, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #92103 for commit 77b1df6ce35d.

@Isotr0py
Isotr0py merged commit 5a4351a into vllm-project:main Sep 30, 2026
172 checks passed
@microslaw microslaw mentioned this pull request Oct 1, 2026
Merged
5 of 6 tasks
@yma11 yma11 mentioned this pull request Oct 3, 2026
Merged
6 tasks
@jzakrzew
jzakrzew deleted the fix/pixtral-mm-input-norm-compile branch October 12, 2026 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments
@noooop noooop noooop approved these changes
@Isotr0py Isotr0py Isotr0py approved these changes
@patrickvonplaten patrickvonplaten Awaiting requested review from patrickvonplaten

Assignees

No one assigned

Labels

mistral Related to Mistral models ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jzakrzew @Isotr0py @noooop