Image + text + audio uniform processors

[molbap]

What does this PR do?

This PR is a stab at uniformizing the processors across all transformers models. If we are happy with the design, I'll expand it to all existing models. It only touches on some text + audio and text + images models as experiment subjects. Linked with #28711 which has a larger scope, and several previous discussions with team members.

Usage

As before, kwargs that are passed to processors at __call__ time take priority. However, per-modality processors can be instantiated with their own kwargs, and if they are not overriden at call time, they will serve as defaults.

Type hinting of kwargs is preserved if they are passed as structured dictionary entries

It also works with kwargs passed without nesting:

Merging of kwargs and handling priority order is done in processing_utils through a dedicated method.
The order of operations is as follows:

1. kwargs passed as before have highest priority to preserve BC.

high_priority_kwargs = {"crop_size" = (224, 224), "padding" = "max_length"}
processor(..., **high_priority_kwargs)

2. kwargs passed as modality-specific kwargs have second priority. This is the recommended API.

processor(..., text_kwargs={"padding": "max_length"}, images_kwargs={"crop_size": (224, 224)}})

3. kwargs passed during instantiation of a modality processor have fourth priority.

tokenizer = tokenizer_class(..., {"padding": "max_length"})
image_processor = image_processor_class(...)
processor(tokenizer, image_processor) # will pass max_length unless overriden by kwargs at call

4. defaults kwargs specified at processor level have lowest priority.

class MyProcessingKwargs(ProcessingKwargs, CommonKwargs, TextKwargs, ImagesKwargs, total=False):
    _defaults = {
        "text_kwargs": {
            "padding": "max_length",
            "max_length": 64,
        },
    }

What changes:

• ~~There is now an attribute in the constructor that stores the processing kwargs needed to be passed to various encoders down the line. ~~ Not anymore, see next point
• There is now a base ProcessingKwargs TypedDict of kwargs that inherits from ImagesKwargs, TextKwargs, and so on.
• These nested attributes (one dictionary for text, one for images, one for audio, one for video) are typed with a TypedDict that does not need to be total. We can expand this typed dict (in processing_utils) as processors get uniformized.
• The processors are called with the same kwargs signature, text, images, audio, videos. kwargs set by a user will always override default processing kwargs.
• Slicing of positional args is removed and replaced by named kwargs corresponding to the modality. Order of kwargs is not constant as a consequence to preserve BC.
• Inputs (text, images, audio, videos) are now always typed in the call.

What that allows:

• We know each model has its own processing logic/way to mix inputs. This way of typing both the modalities sent to the processors AND the processing arguments allow a faster design of future processors, hence faster review, and faster merge, better usage.
• The reason for that is that we push away from the function call the actual mixing of modalities and their specific processing, which can be handled explicitly with the kwargs passed.
• With TypedDict, type hints are preserved even in the nesting. I hesitated with pydantic/dataclasses and opted for TypedDict because it is less flexible, and we want to enforce a standard.

Limitations:

• This still relies on kwargs for the processing.
• Few models tested on that PR - more will follow in other PRs.

Tests pass (I think). What's missing:

• A consistent way (with maybe logger.warning_once) to warn a user they are using a deprecated call.
• Typing of input modalities need to be improved and be more flexible especially for audio&videos
• A test in the CI to check that the enforced designed is respected for a new model added. This can be done in a separate PR.
• All models PRs:
  • Initial design + Align model add initial design for uniform processors + align model #31197
  • Altclip add uniform processors for altclip + chinese_clip #31198
  • Blip Uniformize model processors #31368
  • Blip2 Uniformize model processors #31368
  • Bridgetower Uniformize model processors #31368
  • Chinese_clip add uniform processors for altclip + chinese_clip #31198
  • Donut
  • wav2vec2

Who can review?

Models:

• vision models: @amyeroberts
• speech models: @sanchit-gandhi

[gafrom]

Always true? (same in other places)