About the exact iteration number of LJSpeech pretrained model

[Interfish]

Hi all:

I download pretrained model 20180510_mixture_lj_checkpoint_step000320000_ema.pth. As it's name suggested, it is trained 320k steps. I tried synthesising audios from it, and the quality is very good.

Then I train my own model on LJSpeech using the same preset as the author's. It is now running over 620k but there are little noise in the background. Here is my log:

So I am wondering what's wrong with my trainning. After some reading, I found some contradictory explaination in README.md.
For example,

It says this model is trained over 1000k, but its filename suggests that it's trained only 320k.
And in #1 (comment) , the author also mentioned he trained the model over 1000k.

SO, what is the exact iteration number of this 20180510_mixture_lj_checkpoint_step000320000_ema.pth ?.
This is important cause I am trying to figure out if there is something wrong with my own training. If the pretrained model is 320k and my trainning is definately wrong so I can start debugging.

[r9y9]

Sorry for the confusion, but the name doesn't suggest the model was trained for 320k steps in total. I had noted for this before:

Note: As for the pretrained model for LJSpeech, the model was fine-tuned multiple times and trained for more than 1000k steps in total. Please refer to the issues (#1, #75, #45) to know how the model was trained.

Ref: #129

I initially thought the notice was enough for the explanation but it seems not true as I got the same issue twice. I will update the filename of the pretrained model to avoid the confusion.

[Interfish]

Thanks for the quick response and hard work. Now I know i have to keep on trainning. Hope anyone who share the same confusion with me can see this post!

[puppylpg]

Sorry for the confusion, but the name doesn't suggest the model was trained for 320k steps in total. I had noted for this before:

Note: As for the pretrained model for LJSpeech, the model was fine-tuned multiple times and trained for more than 1000k steps in total. Please refer to the issues (#1, #75, #45) to know how the model was trained.

Ref: #129

I initially thought the notice was enough for the explanation but it seems not true as I got the same issue twice. I will update the filename of the pretrained model to avoid the confusion.

Thanks for this explanation. However I'm still puzzled after load 20180510_mixture_lj_checkpoint_step000320000_ema.pth as checkpoint. The code output suggests that:

• checkpoint["global_step"] = 320000
• checkpoint["global_epoch"] = 51
• checkpoint["global_test_step"] = 14739

and If I train a new modle base on 20180510_mixture_lj_checkpoint_step000320000_ema.pth, a model with 320k iteration will be saved as a checkpoint. Then 330k and so on.

Don't these mean that 20180510_mixture_lj_checkpoint_step000320000_ema.pth is actuall a model with only 320k iterations rather than more than 1000k?
Again, thanks for this great work.

[r9y9]

wavenet_vocoder/train.py

Line 13 in 474e411

--reset-optimizer Reset optimizer.

--reset-optimizer will clear the number of steps in the checkpoint, and start training from 0-th step.

[puppylpg]

wavenet_vocoder/train.py

Line 13 in 474e411

--reset-optimizer Reset optimizer.

--reset-optimizer will clear the number of steps in the checkpoint, and start training from 0-th step.

Emm...Maybe I didn't express myself clearly. I just wanna confirm why the step continues at 320k rather than 1000k, since you said it was trained over 1000k steps...

[r9y9]

For example,

• model 1: a model trained for 700k steps
• model 2: a fine-tuned model from model 1 with --reset-optimizer for 300k steps.

Then, model 2 was trained in total for 1000k steps, but the checkpoint only keep the last number of iterations (i.e. 300 ksteps)

[puppylpg]

For example,

• model 1: a model trained for 700k steps
• model 2: a fine-tuned model from model 1 with --reset-optimizer for 300k steps.

Then, model 2 was trained in total for 1000k steps, but the checkpoint only keep the last number of iterations (i.e. 300 ksteps)

Gotcha! Thanks for your patient reply.