Transitioning to the PyTorch version with Tensorflow-trained models

[StElysse]

Hello, I've just discovered that the repo has changed over to the PyTorch, tensorflow-less version.

I still have old models trained with tensorflow that I wish to use for the creative project I've been working on. I've been following the process located in #437 for a couple of weeks now, and have produced very satisfactory results for a number of single-speaker models.

To what extent is this possible in the PyTorch version of the repo? Also, are my old tensorflow models in anyway compatible with the new repo? If so, how can I do that? If not, will I have to train my models again? (That's ok with me)

I took a glance at the code and apparently synthesizer_train no longer allows us to set global train steps? (I like to use this to fix a certain amount of steps that the model trains at a single session And also, there's a line of code that indicates the lack of an ability to train saved models?

parser.add_argument("-f", "--force_restart", action="store_true", help= \ "Do not load any saved model and restart from scratch.")

If this is the case, is there any way I can save and restart model training with this repo?

If I must keep using the tensorflow version to keep proceeding in my project, I'm ok with that too.

Thanks!

[ghost]

Also, are my old tensorflow models in anyway compatible with the new repo? If so, how can I do that? If not, will I have to train my models again?

Realistically speaking, you will need to start over. The TF taco2 and the PT taco1 model architectures are far too different to transfer pretrained weights.

For completeness, I will add that with knowledge of ML and a lot of patience, it is possible to write an equivalent PT taco2 model in such a way that allows conversion of pretrained weights between the old and new repos.

And also, there's a line of code that indicates the lack of an ability to train saved models?

You can resume training on saved models with the new repo. If the specified model name exists, it will automatically continue the training. --force_restart is used to overwrite existing models and start from scratch.

[StElysse]

Thank you again for the insight.

I got the new repo to work with the num_workers=0 workaround from #669, because I was encountering the 'pickle' error.

My step time seems to hover around 0.3-4 steps per second. Do you know what steps I could take to improve my training time other than reducing max_mel_frames for preprocessing? The sounds in my dataset range from 0-11 seconds, and I would like to see if it is possible to keep the longer sounds while increasing my training time.

I have an i7-7700 CPU, and a GTX 1080 GPU, if that is helpful.

[chankl3579]

I am still using the old Tensorflow repo.
Does it mean that if I change to the PyTorch version, all saved models, including pretrained models, are no longer usable?
Also if I want to do the single speaker finetune in #437 , do I need to train new "pretrained models" so that I can use them as a starting point of finetune?

[ghost]

@StElysse Slow training on Windows is also reported in #700. Are you also using anaconda?

@chankl3579 We provide a pretrained model for pytorch. Instructions for getting it are in the readme.

[StElysse]

I'm not sure if I am using anaconda or not. How would I find this out? If so, how do I disable it/run the program without anaconda?

[ghost]

It's a matter of how you installed Python. The download from python.org provides a normal installation. Another way to install Python is through Anaconda (anaconda.com). Some users experience problems when using an Anaconda-based Python with this repo, which is why I ask.

To find out, type this in the command prompt and see if the output contains "Anaconda".

python -c "import sys; print sys.version"

[StElysse]

The output does not contain "Anaconda".

I also clearly recall installing Python from python.org. The Anaconda.com site is completely new to me.

[MGSousa]

I've got the same problem as @StElysse even without Anaconda, using Python v3.8.
With RTX 2080, only 0.42 steps per second...

[Ca-ressemble-a-du-fake]

Same problem here too, on Ubuntu 20.04 without Anaconda and on RTX 3070 under Python 3.8.8. Around 0.52-0.54 step / s.