Trainer - deprecate tokenizer for processing_class

[amyeroberts]

For reviewers: most files touched here are just updates to the documentation. Unfortunately, a lot of the diff is just stripping of unnecessary whitespaces (which my editor does automatically).

The most important changes are in:

• src/transformers/trainer.py
• src/transformers/trainer_callback.py
• src/transformers/trainer_seq2seq.py

---

What does this PR do?

At the moment, if we wish for a processing class e.g. an image processor, to be saved alongside the model e.g. when pushing to the hub, we have to do the following:

trainer = Trainer(
    model, 
    args, 
    tokenizer=image_processor, 
    ...
)

This causes a lot of confusion for users (an image processor is not a tokenizer).

Previous efforts were made to add individual classes to the trainer s.t. one could do: trainer = Trainer(model, args, image_processor=image_processor, ...), however, this has some drawbacks:

• It creates ambiguity when a model has a processor class: should we be able to do Trainer(model, args, processor=processor) or should it be Trainer(model, args, tokenizer=tokenizer, image_processor=image_processor)?
• It adds new arguments to handle for every processing class, meaning more work if future classes are added

This PR therefore just deprecates tokenizer in favour of a more general processing_class argument, which can be any class with a from_pretrained method.

Relevant PRs and discussions:

• [Trainer] Allow passing image processor #29896
• [Trainer] Undo #29896 #30129
• [Trainer] Rename tokenizer to processor, add deprecation #30102
• [trainer] allow processor instead of tokenizer #30864

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SangbumChoi]

Love this (Always confused about writing tokenizer=image_processor)

[amyeroberts]

This file contains the most important logic changes for review

[amyeroberts]

I directly just swapped the names here, as the docstring says the object is just an internal class

[amyeroberts]

@molbap If you have capacity - do you think you could do a first review of this? Arthur and Zack are super busy so would be good to have someone who knows the processors well to review in their stead

[molbap]

Checked out the rework, nice - saw that now you can't pass both tokenizer and processing_class, it will rightfully raise an error, much better. lgtm!

[amyeroberts]

Asking for review from @SunMarc for Trainer as Zach is off. As this is a very public API I'd like to get a few eyes on it, especially from people who know it well, to make sure it's OK!

[ArthurZucker]

Clean, nothing to say! Thanks for ths!

[SunMarc]

Thanks for the PR @amyeroberts ! LGTM !

[SunMarc]

Thanks for adding these nice tests !

[goea-shuhei]

_pad_tensors_to_max_len in Seq2SeqTrainer still uses tokenizer instead of processing_class