Skip to content

[CI][Bugfix] Skip the SmolVLM HF comparison on Transformers 5.20 - #62620

Merged
vllm-bot merged 2 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_smolvlm_hf_transformers_520
Oct 20, 2026
Merged

vllm-bot merged 2 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_smolvlm_hf_transformers_520

Conversation

@AndreasKaratzas

@AndreasKaratzas AndreasKaratzas commented Oct 20, 2026 •

Copy link
Copy Markdown
Member
  • registry.py: cap the SmolVLM HF reference at transformers<=5.19 with an hf reason, so test_common.py skips the HF-vs-vLLM comparison while vLLM-side tests keep running.

#61983 bumped Transformers to 5.20.2, and since then Multimodal Models (Extended Generation 2) shards 1 and 4 fail in build 96024. The HF reference SmolVLMModel.inputs_merger now splits image features into blocks of vision_config.patch_size (14) instead of the image token count (81 for SmolVLM2-2.2B), so the HF forward raises shape '[-1, 14, 2048]' is invalid; vLLM's SmolVLM is not involved. The Qwen2-VL / Qwen2.5-VL failure in each shard is the next test starting while the failed HF model still holds GPU memory (27.4 of 32.5 GiB free vs 29.9 GiB needed), and the test after it gets the memory back. huggingface/transformers#49725 fixed it on Transformers main on Oct 19, after 5.20.2 was cut, so the bound can go when vLLM moves to a release that includes it.

On MI355 with Transformers 5.20.2, pytest models/multimodal/generation/test_common.py on the two SmolVLM cases and the two Qwen cases that follow them in CI order: without the change the SmolVLM case fails the same way; with it, both SmolVLM cases skip and both Qwen cases pass in 12 runs. Five other runs failed only because another job on the shared node took up to 88 GB of the GPU.

Not a duplicate: no open PR touches SmolVLM or this failure.

AI assistance (Claude Code) was used to draft this change. Every line was reviewed by the submitter.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

Transformers 5.20 splits SmolVLM image features into blocks of the vision
patch size instead of the image token count, so the HF reference forward
fails for SmolVLM2-2.2B. Fixed upstream in huggingface/transformers#49725,
which is not in 5.20.2.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@mergify mergify Bot added the bug Something isn't working label Oct 20, 2026
@AndreasKaratzas
AndreasKaratzas marked this pull request as ready for review October 20, 2026 05:29

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #96077 for commit 0691f1322329.

@AndreasKaratzas
AndreasKaratzas enabled auto-merge (squash) October 20, 2026 05:59
@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 20, 2026
@vllm-bot
vllm-bot merged commit f8e1e87 into vllm-project:main Oct 20, 2026
67 of 69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants