Add Unispeech & Unispeech-SAT

[patrickvonplaten]

What does this PR do?

This PR adds UniSpeech from Microsoft: https://github.com/microsoft/UniSpeech

TODOS:

• Run UniSpeech models and verify that HF forward pass yields same output

• Add UniSpeech checkpoints: https://huggingface.co/microsoft/unispeech-large-1500h-cv

• Run UniSpeech-SAT and verify that HF forward pass yields same output (blocked by: Access Required for UniSpeech-SAT models microsoft/UniSpeech#4)

• Add UniSpeech-SAT checkpoints

• Add UniSpeech vocab and preprocessing (verify with Microsoft)

• Add UniSpeech vocab and preprocessing (verify wiht Microsoft)

• Verify naming with Microsoft & make README.md's pretty

• Clean PR and add tests

• Verify fine-tuning works

Future PR:

• Correct pretraining loss

[patrickvonplaten]

Wait until #13877 is merged

[patrickvonplaten]

PR is good for review IMO:

• PreTrained Unispeech checkpoints: https://huggingface.co/models?other=unispeech
• PreTrained Unispeech-SAT checkpoints: https://huggingface.co/models?other=unispeech-sat

[patrickvonplaten]

I think we can merge the pretrained models now. To make them "promotable" we should still do 2 things:

• Unispeech: Add phoneme <-> text tokenizer, need some feedback here from the authors
• Unispeech-SAT: the model should work very well for speaker-verification and speaker-diarization. We should add those two tasks and then promote the model on it as it performs very well

[sgugger]

Thanks a lot for adding those two models!

[anton-l]

Looks good, thanks a lot for debugging the original models!

[anton-l]

Doesn't UniSpeech use the same masking strategy for quantization as Wav2Vec? Or did you remove masking just for debugging purposes?

[patrickvonplaten]

pretraining is quite different and not implemented yet really - this code should not be used yet