Fix whisper kwargs and generation config

[zucchini-nlp]

What does this PR do?

Whisper code was changed recently to use prepare_generation_config from the GenerationMixin here, which started causing some bugs. See failing test in this PR.

Before, Whisper generation config did not do any generation_config.update and manually updated kwargs along the way. After the above change, kwargs cannot be used anymore because it contains unused kwargs from generation config.

This PR replaces everywhere we had kwargs.pop() with generaion_config. All tests are passing, except for some slow tests. But the slow tests are not passing in main also, so should be unrelated.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gante]

LGTM, thank you for fixing 💛

(I am surprised that no failures were raised in the previous PR's CI 👀 )

[ydshieh]

likely test fetcher didn't find it ...

[ydshieh]

well, i see , whisper is not in IMPORTANT_MODELS

IMPORTANT_MODELS = [
"auto",
# Most downloaded models
"bert",
"clip",
"t5",
"xlm-roberta",
"gpt2",
"bart",
"mpnet",
"gpt-j",
"wav2vec2",
"deberta-v2",
"layoutlm",
"llama",
"opt",
"longformer",
"vit",
# Pipeline-specific model (to be sure each pipeline has one model in this list)
"tapas",
"vilt",
"clap",
"detr",
"owlvit",
"dpt",
"videomae",
]

should we put it ... 😄 @sanchit-gandhi @ArthurZucker ?

[sanchit-gandhi]

Awesome - thanks for the fix @zucchini-nlp! Requesting final review from @ArthurZucker

[sanchit-gandhi]

I would be in favour of adding Whisper and possible Wav2Vec2 as well to this list as the two most used audio models @ydshieh

[ydshieh]

@sanchit-gandhi wav2vec2 is already in that list.

[ArthurZucker]

Could you resolve merge conflicts since #29225 was merged!

[zucchini-nlp]

Done!

[ArthurZucker]

Thanks

[ydshieh]

@zucchini-nlp Do you have the merge permission (i.e. click the squash and merge button above)?

[zucchini-nlp]

yep, merged!