[whisper] small changes for faster tests#38236

gante merged 2 commits intohuggingface:mainhuggingface/transformers:mainfrom

gante:faster_whisper_testsgante/transformers:faster_whisper_testsCopy head branch name to clipboard