[albertvillanova]

What does this PR do?

Fixes #49724

#49557 made get_image_features return a list of flattened per-sample features, and changed SmolVLMModel.inputs_merger to split them into blocks of self.config.vision_config.patch_size. That is the pixel patch size of the vision encoder, not the number of image tokens per image, so the forward pass with images fails whenever the number of image tokens per image is not a multiple of it, e.g. HuggingFaceTB/SmolVLM2-2.2B-Instruct (81 tokens per image, patch size 14):

RuntimeError: shape '[-1, 14, 16]' is invalid for input of size 22032

With the features already flattened, the block indexing only fills the image tokens in order, so this PR removes inputs_merger and merges the image features directly in forward with get_placeholder_mask + masked_scatter, as in llava (also in ModernVBertModel, which inherits from SmolVLMModel). I did not compute the tokens per image from the config (as suggested in the issue) because the ModernVBert config has no scale_factor.

The token/feature count check of get_placeholder_mask exposed two test bugs, fixed here: the resize tests set model.image_token_id instead of model.config.image_token_id (so they ran with 0 image tokens), and the ModernVBert test_resize_tokens_embeddings reused the inputs_dict mutated by the previous model class.

The tester image_size goes from 16 to 24 (9 image tokens per image, patch size 4), so the common tests cover this case: without the fix, 62 SmolVLM tests fail. On the repro from the issue, the logits are identical to v5.20.0.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline and the Pull Request checks?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes according to the guidelines?
• Did you write any new necessary tests?

Who can review?

@zucchini-nlp

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

Huh i remember arguing that masked scatter will give same output back in the past, nice, thanks for verifying and fixing
Lets just move it to forward as it's a single line now and use get_placehlder_mask as in llava

[zucchini-nlp]

can get rid of inputs merger in that case and just merge inside a froward call

[albertvillanova]

Done: inputs_merger is removed, and forward (in SmolVLM and in ModernVBert, which inherits it) now uses get_placeholder_mask + masked_scatter, as in llava. Its token/feature count check exposed two test bugs, also fixed: the resize tests set model.image_token_id instead of model.config.image_token_id (so they ran with 0 image tokens), and the ModernVBert one reused inputs_dict mutated by the previous model class.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: modernvbert, smolvlm

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 39213668028:1
Result: success | Jobs: 5 | Tests: 700 | Failures: 0 | Duration: 2m 54s

[zucchini-nlp]

thanks! Would be good to make idefics3 follow same style and make smolVLM copy it all, but fine for now. I will do a multimdal mixin class some time in coming months

[albertvillanova]

Thanks for the review and merge! @Cyrilvallez @zucchini-nlp, would it be possible to include this fix in the next patch release? The regression came with v5.20.1 (#49557), and v5.20.2 does not include the fix, so for instance HuggingFaceTB/SmolVLM2-2.2B-Instruct fails on any forward pass with images on both patch releases.

[zucchini-nlp]

@albertvillanova it might get lost in GH, pls tag core maintainers in slack. I will add for patch label as well