Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Questions about training time #5

Closed
xw1324832579 opened this issue Dec 6, 2019 · 4 comments
Closed

Questions about training time #5

xw1324832579 opened this issue Dec 6, 2019 · 4 comments
Labels
bug Something isn't working

Comments

@xw1324832579
Copy link

@begeekmyfriend Thank you for your implementation. Recently, i train multi-speaker tts model with your code, and my training process is too slow.Details are as follows.
batch_size 32
iter_time:3.292461
How about your training speed? By the way ,I didn't train the model in docker container.Will this affect my training speed significantly?

@begeekmyfriend
Copy link
Owner

The same as yours. I also doubt why the training speed is so slow on PyTorch compared with that on Tensorflow. I have used kernprof to generate the profile and find out that the major bottleneck lies on decoder_lstm and location_layer which are both APIs provided by PyTorch. I still have no idea about the reason. What is more, the bigger the batch size the less the training speed. That would not happen on Tensorflow 1.x

@begeekmyfriend begeekmyfriend added the question Further information is requested label Dec 6, 2019
@begeekmyfriend
Copy link
Owner

Sorry for my misusage of LSTM in PyTorch. There is a loop in mel spectrogram decoding and
on each iteration it is a hop size but not the whole sequence length instead. Therefore we shall apply LSTMCell in such loop rather than LSTM. Now the training speed would be normal for you. d131f46

@begeekmyfriend
Copy link
Owner

By the way, if we want to use LSTM we should apply it outside the decoding loop to feed the whole sequences. LSTM would utilize less GPU memory.

@begeekmyfriend begeekmyfriend added bug Something isn't working and removed question Further information is requested labels Dec 9, 2019
@begeekmyfriend
Copy link
Owner

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants