Fix Transformers backend FP8 MoE and remove some boilerplate by hmellor · Pull Request #46820 · vllm-project/vllm

hmellor · 2026-06-26T10:23:33Z

The Transformers backend was eagerly storing expert_weights before quantization. This meant that when using FP8 models a stale reference to the unquantized expert weights was kept and the memory usage is higher than expected.

This PR fixes the issue by removing the eager population of self.expert_weights and removing the set_eplb_state method now that an implementation has been added to MixtureOfExperts.

self.expert_weights is only set in set_eplb_state so it's not necessary to instantiate an empty list in every model, we can just set it to an empty list at the start of set_eplb_state.

This PR removes self.expert_weights = [] from all models.

This PR also updates the type hint of MixtureOfExperts.moe_layers to Iterable[MoERunner] which is more specific and correct.

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor · 2026-06-26T11:08:35Z

@abmfy could you please take a look at this from an EPLB perspective? I think my change should be a nice cleanup but you are more familiar with the mechanism

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…oject#46820) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

ehfd · 2026-06-28T11:46:28Z

#46948 is caused by this PR, preventing Gemma4 Unified models from loading at all.

hmellor · 2026-06-28T12:24:12Z

Thanks for the report, I will forward fix this

hmellor · 2026-06-28T13:57:06Z

Fix is in #46956

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…oject#46820) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

## Summary ### 1. CVE-2026-41523: Replace assert with proper exceptions in pooling Cherry-pick fix for [CVE-2026-41523](GHSA-q8gq-377p-jq3r) (GHSA-q8gq-377p-jq3r, HIGH severity, CVSS 7.5). An `assert`-based security check in vLLM's activation function loading can be bypassed when Python runs in optimized mode (`python -O` or `PYTHONOPTIMIZE=1`), allowing arbitrary code execution via a malicious HuggingFace model. **Changes:** - `vllm/model_executor/layers/pooler/activations.py`: Replace `assert` with `ValueError` for activation function security check; replace `assert callable(act_fn)` with `TypeError` in `resolve_classifier_act_fn()` - `vllm/pooling_params.py`: Replace 3 `assert` statements with `ValueError` in `_merge_default_parameters()`, `_verify_valid_parameters()`, and `__post_init__()` **Upstream fix:** vllm-project/vllm@b3c7ffc #### Cherry-pick conflicts & resolution A direct `git cherry-pick 7fa2c781da` produced two conflicts, so the fix was applied manually: 1. **`vllm/model_executor/layers/pooler.py`** — **modify/delete conflict**: The upstream commit patched `get_cross_encoder_activation_function()` in this file, but it no longer exists on nm-vllm-ent `main`. The code was refactored into `vllm/model_executor/layers/pooler/activations.py` (function renamed to `get_act_fn()`). - **Resolution:** Applied the same `assert` → `if not … raise ValueError` fix to `vllm/model_executor/layers/pooler/activations.py` at the equivalent location. 2. **`vllm/pooling_params.py`** — **content conflict**: Surrounding code differences between upstream and nm-vllm-ent (e.g. different error message formatting, `self.task!r` quoting style). - **Resolution:** Manually replaced all 3 `assert` statements with proper `if … raise ValueError(...)`: - `_merge_default_parameters()`: `assert self.task is not None` - `_verify_valid_parameters()`: `assert self.task is not None` - `__post_init__()`: `assert self.output_kind == RequestOutputKind.FINAL_ONLY` 3. **`v1/pool/metadata.py`** — **not applicable**: The upstream commit also converted asserts in this file, but those are internal consistency checks (list length invariants, initialization guards), not security boundaries exploitable via malicious model configs. Skipped as not security-relevant. --- ### 2. INFERENG-8366: Auto-raise max_num_batched_tokens for prefix-LM multimodal models Cherry-pick upstream PR [#43051](vllm-project/vllm#43051) (commit `84e351555a`). Fixes Gemma4 12B-it crash on A100 where `max_tokens_per_mm_item` (2496) exceeds the default `max_num_batched_tokens` (2048) when `disable_chunked_mm_input` is forced `True` for prefix-LM multimodal models. The fix adds logic to `EngineArgs` to automatically detect when a prefix-LM multimodal model's largest modality item exceeds `max_num_batched_tokens`, and bumps it up to fit. **Changes:** - `vllm/engine/arg_utils.py`: Add `_get_min_mm_batched_tokens()` static method and auto-raise logic in `_set_default_max_num_seqs_and_batched_tokens_args()` - `tests/v1/engine/test_engine_args.py`: Add regression test `test_mm_prefix_lm_raises_batched_tokens_floor()` **JIRA:** [INFERENG-8366](https://redhat.atlassian.net/browse/INFERENG-8366) **Upstream PR:** vllm-project/vllm#43051 --- ### 3. INFERENG-8290: Bump tilelang 0.1.9 -> 0.1.11 tilelang 0.1.10+ includes PR [tile-ai/tilelang#2135](tile-ai/tilelang#2135) which disables Cython PEP-489 multi-phase init, fixing the `_xxsubinterpreters` `ModuleNotFoundError` on UBI Python 3.12 that crashes DeepSeek V4. **Changes:** - `requirements/cuda.txt`: Bump `tilelang` from `0.1.9` to `0.1.11` **JIRA:** [INFERENG-8290](https://redhat.atlassian.net/browse/INFERENG-8290) --- ### 4. Add recursive_replace_linear for Gemma4 vision tower quantization The Gemma4 Unified cherry-pick (`a248b45d05`) introduced calls to `recursive_replace_linear()` for replacing vision/audio tower `nn.Linear` layers with vLLM-native quantized equivalents. The function definition and import were missing, causing `NameError` on model load for all Gemma4 and Gemma4 FP8 models. Ports the function from upstream PR [#46820](vllm-project/vllm#46820) into `transformers/utils.py` and adds the import in `gemma4_mm.py`. **Changes:** - `vllm/model_executor/models/transformers/utils.py`: Add `recursive_replace_linear()` function - `vllm/model_executor/models/gemma4_mm.py`: Add missing import --- ## Test plan - [x] Verify `pre-commit run --all-files` passes - [x] Verify pooling/cross-encoder models still work (CVE fix) - [ ] Verify that loading a model with a non-`torch.nn.modules.*` activation function raises `ValueError` - [ ] Verify passing a non-callable to `resolve_classifier_act_fn` raises `TypeError` - [x] Verify Gemma4 12B-it starts successfully on A100 with default `max_num_batched_tokens` - [ ] Verify DeepSeek V4 loads without `_xxsubinterpreters` crash (tilelang bump) - [x] Verify Gemma4 / Gemma4 FP8 model load succeeds without `NameError` on `recursive_replace_linear` - [x] Run accept-sync CI with `google/gemma-4-12B-it` in model list [INFERENG-8366]: https://redhat.atlassian.net/browse/INFERENG-8366?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

…oject#46820) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…oject#46820) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…oject#46820) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor added 5 commits June 26, 2026 09:56

Clean up some boilerplate

a38371f

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

More standard naming in the Transformers MoE code

b230c31

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Delete method that's no longer needed

4fc741d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

typo

8e12458

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Simpler override

76ad2b4

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor requested review from luccafong, patrickvonplaten, sighingnow, tomeras91 and vadiklyutiy as code owners June 26, 2026 10:23

github-project-automation Bot added this to Transformers modeling backend Jun 26, 2026

github-project-automation Bot moved this to Todo in Transformers modeling backend Jun 26, 2026

claude Bot reviewed Jun 26, 2026

View reviewed changes

mergify Bot added deepseek Related to DeepSeek models llama Related to Llama models mistral Related to Mistral models qwen Related to Qwen models labels Jun 26, 2026

hmellor added 2 commits June 26, 2026 11:04

Fix step3p5

4161f3f

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

don't undertscore op name

1ba3e1f

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Isotr0py approved these changes Jun 26, 2026

View reviewed changes

Isotr0py added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 26, 2026

hmellor added 4 commits June 26, 2026 13:45

fix forward references

6903409

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Revert TransformersMoERunner.load_weights change

a217b4e

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Merge branch 'main' into transformers-moe-fixes

2836a72

fix tests

df43400

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor merged commit d8eb734 into vllm-project:main Jun 26, 2026
83 checks passed

hmellor deleted the transformers-moe-fixes branch June 26, 2026 23:16

github-project-automation Bot moved this from Todo to Done in Transformers modeling backend Jun 26, 2026

WindChimeRan pushed a commit to WindChimeRan/vllm that referenced this pull request Jun 27, 2026

Fix Transformers backend FP8 MoE and remove some boilerplate (vllm-pr…

9383331

…oject#46820) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

ehfd mentioned this pull request Jun 28, 2026

[Bug]: On nightly 'Gemma4ForCausalLM' object has no attribute 'expert_weights' #46948

Closed

1 task

hmellor mentioned this pull request Jun 28, 2026

Remove boilerplate missed by #46820 #46956

Merged

jeejeelee pushed a commit that referenced this pull request Jun 29, 2026

Remove boilerplate missed by #46820 (#46956)

311ad68

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

wincent8 pushed a commit to wincent8/vllm that referenced this pull request Jun 29, 2026

Fix Transformers backend FP8 MoE and remove some boilerplate (vllm-pr…

b5aed86

…oject#46820) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

wincent8 pushed a commit to wincent8/vllm that referenced this pull request Jun 29, 2026

Remove boilerplate missed by vllm-project#46820 (vllm-project#46956)

64c7c3a

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

rjrock pushed a commit to rjrock/vllm that referenced this pull request Jul 1, 2026

Remove boilerplate missed by vllm-project#46820 (vllm-project#46956)

ef3035d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Dao007forever pushed a commit to Dao007forever/vllm that referenced this pull request Jul 18, 2026

Fix Transformers backend FP8 MoE and remove some boilerplate (vllm-pr…

6b3f825

…oject#46820) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Dao007forever pushed a commit to Dao007forever/vllm that referenced this pull request Jul 18, 2026

Remove boilerplate missed by vllm-project#46820 (vllm-project#46956)

113375a

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026

Remove boilerplate missed by vllm-project#46820 (vllm-project#46956)

6a5c19a

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026

Fix Transformers backend FP8 MoE and remove some boilerplate (vllm-pr…

c431916

…oject#46820) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026

Remove boilerplate missed by vllm-project#46820 (vllm-project#46956)

d55ba84

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Navigation Menu

Uh oh!

Uh oh!

Fix Transformers backend FP8 MoE and remove some boilerplate - #46820

Fix Transformers backend FP8 MoE and remove some boilerplate#46820
hmellor merged 11 commits into
vllm-project:mainfrom
hmellor:transformers-moe-fixes

hmellor commented Jun 26, 2026

Uh oh!

claude Bot left a comment

Uh oh!

hmellor commented Jun 26, 2026

Uh oh!

Uh oh!

ehfd commented Jun 28, 2026

Uh oh!

hmellor commented Jun 28, 2026

Uh oh!

hmellor commented Jun 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

hmellor commented Jun 26, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

hmellor commented Jun 26, 2026

Uh oh!

Uh oh!

ehfd commented Jun 28, 2026

Uh oh!

hmellor commented Jun 28, 2026

Uh oh!

hmellor commented Jun 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants