[Trainer] Rename tokenizer to processor, add deprecation

[NielsRogge]

What does this PR do?

This PR updates the tokenizer argument which people can pass to the Trainer and TrainerCallback classes to processor instead. This allows for people to pass tokenizers, image processors, feature extractors or multimodal, which it will then automatically save along the model when training.

Follow-up of #29896

To do:

• perhaps use preprocessor instead of processor to avoid confusion with multimodal processors like CLIPProcessor?
• add deprecation message for tokenizer attribute

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Thanks for updating - looks a lot better with processor!

• Other examples which have tokenizer=tokenizer will also need to be updated if they exist
• Question over the deprecation cycle - v5 means it won't be remove in a long time - but as this is a fairly large change from a commonly used API I think it's OK

[younesbelkada]

I think we should be careful when manipulating the init signature, see: #30126 - for new args we should put them at the end

[NielsRogge]

Thanks, I opened #30129 as a better quick fix cause there are a huge amount of files having tokenizer=tokenizer and I'm not sure it's worth the effort (would require some offline discussion).

[amyeroberts]

@NielsRogge Although #30129 proposes a solution for the default data collator, we should still add this update. tokenizer=image_processor is a confusing API (as highlighted by a few issues from users) and considering we have more and more multimodal, audio and vision models increasingly out-of-date

Replacing tokenizer=tokenizer with processor=tokenizer might touch a lot of files but I'd expect to be fairly straightforward with some grep and find/replace.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sanchit-gandhi]

Thanks for pointing me in the direction of this PR @amyeroberts! I personally don't think replacing tokenizer by processor is a good idea in the context of NLP users: many NLP users won't know the notion of a processor, since they only use the tokenizer to pre- and post-process their data. To me, having them set processor=tokenizer is just as confusing as having multimodal users set tokenizer=processor.

I would be more in favour of a design where allow NLP users to pass tokenizer=tokenizer, and multimodal users to pass processor=processor, as suggested in #30864 (comment) -> to me, this is the cleanest and most intuitive API. Yes, it adds some complexity under-the-hood in the Trainer, since we now have to handle both object, but this is a valid burden to make the API as intuitive as possible for the user.

[amyeroberts]

@sanchit-gandhi I think what's being suggested is actually a flag more like preprocessor or processing_class which would replace tokenizer. As Trainer is such a commonly used object, in reality we're unlikely to fully remove or deprecate tokenizer anytime soon, but I don't think this means we have to add a whole new argument for every processing class.

[sanchit-gandhi]

I believe this PR is proposing deprecating tokenizer @amyeroberts! (as per this line and the title) I think preprocessor is limited in the sense that we don't just want to save the pre-processing class (e.g. image processor or feature extractor), but also the post-processing class (e.g. the tokenizer) => for this reason I think passing the processor is cleanest here (for multimodal models)

[amyeroberts]

Ah, yes. Perhaps I wasn't clear. What I meant was we would introduce an new argument, which should be used in preference to tokenizer which would accept all of the processing classes. However we wouldn't fully deprecate tokenizer for a long time, and still allow it to be used

[sanchit-gandhi]

Great thanks for the clarification - on the same page here! Happy to update the PR #30864 accordingly, unless you want to see this one to completion @NielsRogge?