[Bugfix] Guard mm_token_type_ids kwarg in get_mrope_input_positions

[AndreasKaratzas]

#35097 introduced passing mm_token_type_ids to model.get_rope_index() in the transformers multimodal backend. However, not all models accept this parameter. For example, Qwen2_5_VLModel.get_rope_index() does not have mm_token_type_ids in its signature nor **kwargs, causing a TypeError at inference time.

TypeError: Qwen2_5_VLModel.get_rope_index() got an unexpected keyword argument 'mm_token_type_ids'

Reproduced via:

pytest -s -v tests/models/multimodal/generation/test_common.py::test_single_image_models[qwen2_5_vl-transformers-test_case53]

Bisected to fd6de37.

Fix

Before passing mm_token_type_ids as a kwarg, inspect the signature of model.get_rope_index() to check whether it actually accepts the parameter (either explicitly or via **kwargs). If it doesn't, skip it.

cc @kenroche

[gemini-code-assist]

Code Review

This pull request correctly fixes a TypeError by guarding the mm_token_type_ids keyword argument before passing it to model.get_rope_index(). The fix involves inspecting the function's signature to ensure it can accept the argument. I've suggested a performance improvement to cache the result of the signature inspection, as it's a costly operation that doesn't need to be repeated on every call.

[gemini-code-assist]

While this fix is correct, using inspect.signature on every call to get_mrope_input_positions can introduce a performance overhead, especially in a hot path. Since the signature of self.model.get_rope_index is not expected to change at runtime, this check can be performed once and the result cached. A simple way to achieve this is to use lazy initialization with hasattr to cache the check result on the first call.

            if not hasattr(self, "_get_rope_index_accepts_mm_token_type_ids"):
                import inspect

                sig = inspect.signature(self.model.get_rope_index)
                params = sig.parameters
                self._get_rope_index_accepts_mm_token_type_ids = "mm_token_type_ids" in params or any(
                    p.kind == inspect.Parameter.VAR_KEYWORD for p in params.values()
                )

            if self._get_rope_index_accepts_mm_token_type_ids:
                kwargs["mm_token_type_ids"] = torch.cat(mm_token_type_ids)

[AndreasKaratzas]

Fair suggestion. Done :)

[DarkLight1337]

@hmellor can you review this?

[hmellor]

Thanks for the fix @AndreasKaratzas, I think the intention in the original PR was that mm_token_type_ids would not be part of kwargs if Transformers v4 (as in CI) is installed.

I'll reproduce locally to see what options we have for fixes.

[AndreasKaratzas]

Thanks for the fix @AndreasKaratzas, I think the intention in the original PR was that mm_token_type_ids would not be part of kwargs if Transformers v4 (as in CI) is installed.

I'll reproduce locally to see what options we have for fixes.

@hmellor Would it be possible to temporarily go with this patch? We would like this test to soon be in the gating set of AMD tests :)

[hmellor]

Yeah this will work for now! I'm excited for the Transformers modelling backend to be tested more on AMD!

[hmellor]

(I cancelled the build because it looks like it needs a rebase/merge and I can't do it myself because the PR comes form an org)