wav2vec-u GAN training not converging on Korean dataset

[KimJeongSun]

❓ Questions and Help

Before asking:

1. search the issues.
2. search the docs.

What is your question?

we are trying to train wav2vec-u on Korean data but our GAN model does not converge well. Are the parameters used to train the model presented in the paper the same as those included in the repo? Also, are the parameters for the multilingual GAN the same as the English model?

Code

What have you tried?

We successfully replicated the Librispeech experiment with English data. We prepared a Korean wav2vec 2.0 model with the public AI hub Korean dataset, but the GAN model does not converge when trained on this same data.
( it is not decreasing until 150K)

We also tried training the GAN using the XLRS model as an encoder using English Librispeech 100 dataset, and it also does not converge well (with hyper parameter range in paper and recipe). The valididation weighted PPL does not fall below ~60.
( it is not decreasing until 150K)

What's your environment?

• fairseq Version (e.g., 1.0 or main): 1.0.0a0+741fd13
• PyTorch Version (e.g., 1.0): 1.8.0+cu111
• OS (e.g., Linux): ubuntu 20.04
• How you installed fairseq (pip, source): git clone https://github.com/pytorch/fairseq.git && cd fairseq && git reset --hard 741fd13
• Build command you used (if compiling from source): python3 -m pip install --editable ./
• Python version: 3.8.10
• CUDA/cuDNN version: cuda_11.1
• GPU models and configuration: A100 x 8
• Any other relevant information:

[lsrami]

@KimJeongSun Hello, I have the same problem with wav2vec-u experiment on the Mandarin data set. I guess it may be related to GAN's training hyperparameter settings, but I have adjusted the hyperparameters many times and have not achieved reasonable results. If you have new progress, please share it.

[stale]

This issue has been automatically marked as stale. If this issue is still affecting you, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!

[stale]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue with up-to-date information. Thank you!

[XR1988]

Thanks for your work. What's the current status? I'm not getting good results; my UER is stuck around 90.
I've cloned this repo (it might have environment problems new): https://github.com/oneapi-src/ai-transcribe
Others cloned it with a virtual environment: https://github.com/voidful/wav2vec-u-exp
I'm having trouble with this: #5572
@KimJeongSun