Fix custom processors that use deleted behaviour for Transformers v5

[hmellor]

Some remote code processors may define optional_attributes in their ProcessorMixin subclass, and then pass these arbitrary attributes directly to ProcessorMixin.__init__, which is no longer allowed in Transformers v5. For backward compatibility, we intercept these optional attributes and set them on the processor instance before calling the original ProcessorMixin.__init__.

In vLLM CI, one architecture which does this is Molmo2ForConditionalGeneration. If we upstream this architecture to Transformers, we can remove this backward compatibility patch in vLLM.

[hmellor]

(this will have merge conflicts with #35101 so it must be merged after)

[gemini-code-assist]

Code Review

This pull request introduces a compatibility layer for Transformers v4 processors in vLLM to address a breaking change in Transformers v5. It ensures that processors defining optional_attributes in their ProcessorMixin subclasses can still be initialized correctly. The solution involves intercepting these optional attributes and setting them on the processor instance before calling the original ProcessorMixin.__init__. This change is crucial for maintaining backward compatibility, particularly for models like Molmo2ForConditionalGeneration. The code introduces a new function _transformers_v4_compatibility_from_pretrained to apply this patch.

[DarkLight1337]

Please fix docs though

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork