[Zero-shot image classification pipeline] Remove tokenizer_kwargs

[NielsRogge]

What does this PR do?

This PR is a follow-up of #29261, namely the tokenizer_kwargs argument is unnecessary, one can just update the model_input_names attribute of the tokenizer.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Thanks for opening this PR.

We can't nor shouldn't remove tokenizer_kwargs from the pipeline. It is both a breaking change in terms of the pipeline's functionality and will break for all existing Blip2 checkpoints saved with a bert tokenizer

[amyeroberts]

I don't think this should be removed. tokenizer_kwargs is a fairly standard input to _sanitize_parameters as a way to control tokenizer behaviour. It would also be breaking for anyone using this in their pipelines.

We might be able to remove in the tests but it should stay here

[NielsRogge]

Hmm could you clarify? tokenizer_kwargs was added in #29261 which is not yet in a stable release. Hence removing this argument wouldn't break anything

[amyeroberts]

Ah, OK, in this case we can remove!

[amyeroberts]

This is pretty hacky. It should at the very least only be done in the BertTokenizer branch to make explicit why this is needed (including an explanatory comment)

[NielsRogge]

I don't see what's hacky about this, model_input_names is a genuine attribute of PretrainedTokenizer that has been there since BERT/GPT-2 came out (it's not used a lot). As can be seen here, a tokenizer will return token_type_ids if it's present in the model_input_names.

A less "hacky" way would be to pass model_input_names directly in the from_pretrained method, which the Transformers library also supports.

[NielsRogge]

What I'm trying to say is that one can avoid lines like these by appropriately setting the model_input_names attribute of the tokenizer, which is something we can still do for the Blip2ImageTextRetrieval models on the hub as they are just added and not yet in a stable release.

[amyeroberts]

Ah, yes, I see. Although both modifying model_input_names and return_token_ids are somewhat hacky: they modify a class attribute which might be depended upon for other behaviours / assumed to have certain properties. Passing in model_input_names to the init is better, as this is more likely to correctly propagate any changes to any other dependant attributes / variables.

[NielsRogge]

It is both a breaking change in terms of the pipeline's functionality and will break for all existing Blip2 checkpoints saved with a bert tokenizer

The Blip2 checkpoints compatible with this pipeline are the following:

• https://huggingface.co/Salesforce/blip2-itm-vit-g
• https://huggingface.co/Salesforce/blip2-itm-vit-g-coco

The pipeline was updated in a just-merged PR (which also pushed those 2 checkpoints). My point would be to just set the model_input_names of the tokenizer of those 2 repos appropriately which would ensure the tokenizer_kwargs isn't needed - I also don't see it present in any other pipeline

[amyeroberts]

OK, as tokenizer_kwargs were just added for a model on main then we can remove. Could you open PRs on the BLIP2 checkpoints to update to make sure they are compatible?

[NielsRogge]

It seems the model_input_names attribute doesn't get serialized in tokenizer_config.json (as it's a class attribute), cc @ArthurZucker. So I'm not sure it's possible to remove this.

[amyeroberts]

@NielsRogge In this case, we should just keep the tokenizer_kwargs in the pipeline, as they're a fairly standard input anyways. The issue regarding having to pass {return_token_ids: "False"} for BLIP will hopefully be something we can resolve once we can load processors in pipelines - #32514, as tokenizer.return_token_ids will be automatically set.

What should be done is document that we need to pass in {return_token_ids: "False"} to use these blip2 checkpoints in the pipeline

[ArthurZucker]

I think model_input_names should be serialized! (some model like mamba use GPTNeoXTokenizerFast, but don't use token_type_ids for example)