Support mm_processor_cache in the Transformers multimodal backend

[harshaljanjani]

What does this PR do?

→ Follow-up to the Transformers audio backend, dropping the return_mm_token_type_ids request to the processor and the cache-bypassing apply() override so mm_processor_cache works with the backend. _get_prompt_updates now locates placeholders purely from the processor's return_text_replacement_offsets, and both the prompt-update target and the embed token are derived from those offsets, so the per-model _get_{audio,image}_token_id lookups are gone.
→ Requires the companion Transformers PR for Gemma3, Granite Speech and VibeVoice, whose processors only emit offsets once migrated to the base __call__. AudioFlamingo3 and GLM-ASR already emit them.
→ Not supported: Passing pre-computed embeddings through this backend. The offsets come from the HF processor, which is skipped entirely when embeddings are passed, so the extra fields the backend needs are never produced. Documented next to the enable_mm_embeds warning in multimodal_inputs.md.

Verified no regressions:
→ tests/models/multimodal/processing/test_transformers_image.py and test_transformers_audio.py → 9 passed, 1 xfailed (the Voxtral xfail, tracked in #49958)
→ tests/models/multimodal/generation/test_transformers_audio.py → 4 passed
→ tests/models/multimodal/generation/test_common.py -k transformers → 4 passed, 1 skipped
→ Models: LLaVA-OneVision and Gemma3 placeholders are identical (Gemma3 stays 260 tokens with 256 embed positions), and Granite Speech, AudioFlamingo3, VibeVoice and GLM-ASR all generate matching HF

cc: @hmellor

Before

After

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[zucchini-nlp]

hey @harshaljanjani !

Discussed internally with Harry so sharing with you

TBH this approach seems to have a few edge cases with multi-image inputs. I merged just yesterday a PR on transformer that updated all VLMs to enable returning placeholder_metadata.
Since I was already planning to update things on vllm after that PR, imo it'd be easier for us to try enable mm_processor_caching. The idea is to remove mm-token-type ids and instead ask for replacement offsets dict (return_text_replacement_offsets=True). It will be a dict of this format for each image/video/audio.

{
                        "type": "image",
                        "span": (start, end),
                        "new_span": (start_with_offset, start_with_offset + len(replacement_text)), # char span, not token id
                        "text": m.group(),
                        "replacement_text": replacement_text,
                    }

The major issue with caching was that the backend didn't have a working self.get_prompt_updates. I haven't tried anything yet myself, so it is a very raw idea. I am planning to start working on it next week but if you want to take this further, please feel free to check out to transformers main and draft a PoC for "mm_processor_cache support in Transformers backend". I will be able to hop in and help next week 😄

[harshaljanjani]

Thanks for taking the time to flesh out the discussion and what you had in mind @zucchini-nlp :)
I've written the first draft on the vLLM side; we'd need to add return_text_replacement_offsets: bool | None here, but I wanted to get your opinion on 36ce01 before I raised any Transformers PRs regarding whether I understood what you were going for correctly and got the direction right. I've accounted for quite a few edge cases as well that I encountered along the way, looking forward to the reviews, on my side all local tests pass.

[zucchini-nlp]

@harshaljanjani wow, that was fast! Correct, we will need to allow the arg as TextKwargs for all models. You can open a PR on transformers and share your draft on vLLM as well, so we can discuss further with Harry :)

[harshaljanjani]

@zucchini-nlp I've raised huggingface/transformers#47614, thanks!
@hmellor The vLLM-side draft is in this PR itself. I've refactored and retitled the PR accordingly, and edited the PR description, will await the reviews from the both of you :)

[zucchini-nlp]

Left a few questions, though the main one is why we need to re-purpose offsets as a new mm-field and can we instead rely on raw hf_processor_mm_kwargs when updating prompt replacements? I was hoping to reduce "magically inferred stuff" by introducing this metadata 😅

I haven't been following vLLM for a while, so cmiiw anywhere and cc @hmellor for another pair of eyes to review

[zucchini-nlp]

Oke, to summarize and align on next steps:

1. We have a few models with old-format processing __call__ in transformers, which I can update (or if you wish, go ahead pls). After we update those 3 or more processors, we can delete fallback path
2. Deleting a fallback path mean we also delete magical fn like run_token_length
3. The promp update fn doesn't get raw processor output but rather vllm-format mm-kwargs. So we can either keep tokenized prompt updates as {modality}_placeholder_ids and save in mm-kwargs, or we can directly save offset_dict as a new field mm-kwargs?

LMK if first two points make sense, that would clear up a lot of "magical" code imo. The last point is more about format/style, and not so much magically inferred

[harshaljanjani]

@zucchini-nlp Yep makes sense, went ahead and did it on both sides, pushed to here 1fbbf5 and Transformers faeea1 so you can look directly 🤗

→ We have a few models with old-format
Gemma3, VibeVoice and Granite are all migrated. A bit worth your eyes, Granite only has audio_processor but the base gated audio on feature_extractor so I've changed it in my PR so that the base now resolves either name rather than renaming anything, going by what was discussed in this PR I raised a while back and where huggingface/transformers#44394 is headed, probably need to coordinate with Eustache as well for this :)
→ Deleting a fallback path mean we also delete magical fn like run_token_length
Agreed and nothing depends on it anymore so it's gone in this PR :)
→ The promp update fn doesn't
I was wondering that {modality}_placeholder_ids is really the only part that's both prompt-independent and tensor-shaped. The spans are derived from the prompt itself so it's added to the prompt update step and as of today tmk the dict wouldn't round-trip in full without tokenizing it anyway. Happy to switch if you see it differently though, please do let me know!

Looking forward to your thoughts!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @harshaljanjani.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

Documentation preview: https://vllm--50107.org.readthedocs.build/en/50107/

[zucchini-nlp]

Amazing work, thanks for iterating on this! LGTM, maybe we can add more tests with different ckpt but it's up to vllm maintainers

One thing I wanted to ask, I am seeing that you deleted the tokenizer.decode when ids are passed. IIUC now we take base class'es apply_text_only/mm_only path which allows us to support tokenized inputs more seamlessly? Did you by chance test it, if not dw :)

[zucchini-nlp]

i don't like having this which forces us to hold an unused helper in transformers, but it's not on you. @hmellor , I will keep this in mind and try to delete in the future

[hmellor]

Which helper are you referring to?

I see lots of references to num_image_patches/get_number_of_image_patches in Transformers main

[zucchini-nlp]

I meant calling processor._get_num_multimodal_tokens which iiuc is not used outside vllm. The number of patches is used by CB atm as well since our pixels are shaped differently
I guess we either call directly get_number_of_image_patches in vllm or try to set easier way to infer/split total pixels per input image

[harshaljanjani]

One thing I wanted to ask, I am seeing that you deleted the tokenizer.decode when ids are passed. IIUC now we take base class'es apply_text_only/mm_only path which allows us to support tokenized inputs more seamlessly? Did you by chance test it, if not dw :)

Yep that's exactly it, and it's covered by an existing test which doesn't regress with this PR, and across all models (except the Voxtral XFAIL being worked on rn) the backend tests cover the placeholders are identical either way

[harshaljanjani]

Actually qq, won't we need a check-version on Transformers here because the offsets PR isn't in a release yet? A lot of models would fail outright, though I'm not sure a version gate cuts it either given some processors already emit offsets and others don't, would love to know your thoughts or if I'm missing something.

[zucchini-nlp]

ah forgot we didn't release it yet. Then we'll need to wait a bit until the next transformers release which should be around next week and until vllm bumps that version

[hmellor]

First pass from me, could we update the PR description too to accurately represent the current state

[harshaljanjani]

Could we update the PR description too to accurately represent the current state

@hmellor Done, rewrote and pushed the changes.

[molbap]

pulling this out of offline discussions:

I think chat-template prompts stay broken after this PR, #50899 patches them in the backend text path, which this PR removes.

logiflow is, IIRC: the template renders bos itself. The renderer then encodes the rendered string with add_special_tokens=True. This happens for offline LLM.chat (mm default since #34510 ) and for pre-rendered strings passed to generate. The online endpoint passes False and is unaffected. in #50899 I put test_renderer_chat_template_prompt_matches_hf which asserts engine ids == processor.apply_chat_template(..., tokenize=True). It does not depend on the mechanism (like with this PR the processor only receives ids)

So the fix must be on the renderer, I think in _tokenize_prompt, where you can add back add_special_tokens=False for the mm chat default. And mirror the bos fallback from processor.apply_chat_template at the encode, for pre-rendered strings like the test does.

[harshaljanjani]

Can confirm the double BOS comes from _tokenize_prompt, so fixing the renderer seems like the right approach. But an important thing I found is that of the models this backend covers, I could find only Gemma 3's template actually renders <bos>; LLaVA-1.5's doesn't. For Gemma 3 it's off by 2 tokens, not 1 (107/108 here vs 109 in HF), because the replacement's \n\n merges with the template's newline. So add_special_tokens=False alone still wouldn't fix the test. prefers_prompt_text is what fixes both cases, since the processor then tokenizes the prompt string only once. The remaining problem is that going through the text path currently fails because it finds 0 placeholders, where main goes through fine. It looks like _apply_prompt_updates already falls back to string matching when tokenization splits the search text (here), and there's also a segmented version added for MiniCPM-V that does the same (here). But the find path doesn't have the same fallback (here). So yes, the BOS issue belongs in the renderer, but the placeholder-finding failure looks like a general issue. Happy to fix that here if you both think the reasoning is sound, or if that's not within scope as well would like to know.

[hmellor]

With this PR we would rely 100% on return_text_replacement_offsets, which was first released in Transformers v5.10.0. So we should add a self.check_version in MultiModalMixin.__init__ so users on older versions don't end up with strange errors

[harshaljanjani]

Done!

[hmellor]

A few more problems:

• return_text_replacement_offsets was only added to TextKwargs in feat[vLLM]: Support text replacement offsets in the remaining old-format processors huggingface/transformers#47614. This means that, for all versions prior to that PR, ProcessorMixin._merge_kwargs drops it and the processor doesn't return the text replacement offsets
• Most processors that override ProcessorMixin.__call__ in Transformers never call super().__call__(). This means that return_text_replacement_offsets has no effect on them
• _get_mm_fields_config loops over all hf_inputs for each modality. This means that for models with both audio and image support:
  • All image inputs have flat_from_sizes("audio", num_audio_tokens) assigned
  • All audio inputs have flat_from_sizes("image", num_image_patches) assigned, which clobbers the num_audio_tokens previously set for the audio inputs
• Retokenizing offset["replacement"] and offset["text"] is not safe, as we have previously discovered. Why can we not use span and new_span directly?

[hmellor]

in the meantime I've just pushed a small commit with some naming improvements so it's easier to understand what's going on

[zucchini-nlp]

@harshaljanjani hey, thanks a lot for your help ❤️
We are discovering possible conflicts with such an abrupt change so I will take over the PR and think what we can do with BC for remote code, and the pesky special token edge cases

[harshaljanjani]

Thank you @hmellor and @zucchini-nlp, and noted on the takeover :)
Happy to help if anything comes up down the line, and looking forward to seeing how this changes!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @harshaljanjani.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork