-
Notifications
You must be signed in to change notification settings - Fork 217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Heavily CPU Dependent #19
Comments
Side note, this seems to be cpu time spent loading the .wav files. |
Note: It runs at about twice the speed per batch with Librosa caching enabled. |
@Teravus you could consider rewriting their dataloader to NVIDIA DALI framework, which can decode wav files and convert them to spectograms on GPU instead of CPU. |
@Teravus THANKS for that hint, the Librosa Cache is for me a HUGE improvement, since im learning on an Jetson AGX Xavier, where the CPU is a lot weaker it is more than twice as fast. Now the GPU is almost all the time running since it has the cache where it can fastly reload and not recalcualting the stuff all the time ms/batch is now around 1200-1000 and 15 Seconds to generate the samples |
I fixed this by loading all audio in memory. Depending on your input dataset, this might be feasible. In # Load all audio files into memory
self.audio_data = []
for audio_file in self.audio_files:
audio, _ = self.load_wav_to_torch(audio_file)
self.audio_data.append(audio) Then in # Read audio
filename = self.audio_files[index]
audio, sampling_rate = self.load_wav_to_torch(filename) With this: # Get audio from memory
audio = self.audio_data[index] |
I'm not sure if this would be a bug or an implementation detail... and training using the train.py script seems to be very heavily CPU dependent. I checked that it is processing some parts on the GPU. It goes through 16 seconds of processing on the CPU and then a tiny amount of CUDA activity per step for me. This has essentially CPU bound the training for me. I've been training one for 14 days on a NVidia GeForce GTX 2080 ti and it just reached epoch 76,000
The text was updated successfully, but these errors were encountered: