Replies: 2 comments
-
|
>>> erogol |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
>>> Sadam1195 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
>>> Sadam1195
[June 8, 2020, 5:59am]
I have some custom dataset which is about 40 hours voice data. slash
Some utterances are larger (1 and half mint, there are many which are
longer than 40 seconds) which I think is causing the issue but I need
your comments on that and some are of very short time duration (1 sec, 2
sec, etc). slash
I tried to train it on different VM machines but on every machine I get
the same error 'CUDA out of memory'. slash
The maximum memory I had was 16 GB on Tesla V100 machine. slash
I used 'use_phonemes': true, 'phoneme_language': 'it'. slash
I tried smaller batch sizes but it didn't work either. slash
I am attaching my config.json and logs of few vm machines which failed
to do the training. slash
I used CUDA_VISIBLE_DEVICES='0,1,3,4,5,6,7' python distribute.py
slash --config_path config.json after few steps it failed it gave me the
error.
Please review these files and help me sort this out and also what is the
best way to estimate the GPU memory required to train on a dataset, is
there any way to calculate that?
I can't upload the files 'new user'
here is the output of my config.json
> { slash
> 'model': 'Tacotron2', slash
> 'run_name': 'ljspeech', slash
> 'run_description': 'tacotron2',
// AUDIO PARAMETERS
'audio':{
// stft parameters
'num_freq': 513, // number of stft frequency levels. Size of the linear spectogram frame.
'win_length': 1024, // stft window length in ms.
'hop_length': 256, // stft window hop-lengh in ms.
'frame_length_ms': null, // stft window length in ms.If null, 'win_length' is used.
'frame_shift_ms': null, // stft window hop-lengh in ms. If null, 'hop_length' is used.
// Audio processing parameters
'sample_rate': 22050, // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
'preemphasis': 0.0, // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
'ref_level_db': 20, // reference level db, theoretically 20db is the sound of air.
// Silence trimming
'do_trim_silence': true,// enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
'trim_db': 60, // threshold for timming silence. Set this according to your dataset.
// Griffin-Lim
'power': 1.5, // value to sharpen wav signals after GL algorithm.
'griffin_lim_iters': 60,// #griffin-lim iterations. 30-60 is a good range. Larger the value, slower the generation.
// MelSpectrogram parameters
'num_mels': 80, // size of the mel spec frame.
'mel_fmin': 0.0, // minimum freq level for mel-spec.
50 for male and
TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash95 for female voices. Tune for dataset!!
','mel_fmax': 8000.0, // maximum freq level for mel-spec. Tune for dataset!!
// Normalization parameters
'signal_norm': true, // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
'min_level_db': -100, // lower bound for normalization
'symmetric_norm': true, // move normalization to range [-1, 1]
'max_norm': 4.0, // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
'clip_norm': true, // clip normalized values into the range.
'stats_path': null // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
},
// VOCABULARY PARAMETERS
// if custom character set is not defined,
// default set in symbols.py is used
// 'characters':{
// 'pad': '_',
// 'eos': '
// 'bos': '^',
// 'characters': 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!'(),-.:;? ',
// 'punctuations':'!'(),-.:;? ',
// 'phonemes':'iyɨʉɯuɪʏʊeøɘəɵɤoɛœɜɞʌɔæɐaɶɑɒᵻʘɓǀɗǃʄǂɠǁʛpbtdʈɖcɟkɡqɢʔɴŋɲɳnɱmʙrʀⱱɾɽɸβfvθðszʃʒʂʐçʝxɣχʁħʕhɦɬɮʋɹɻjɰlɭʎʟˈˌːˑʍwɥʜʢʡɕʑɺɧɚ˞ɫ'
// },
// DISTRIBUTED TRAINING
'distributed':{
'backend': 'nccl',
'url': 'tcp: slash / slash /localhost:54321'
},
'reinit_layers': [], // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.
// TRAINING
'batch_size': 32, // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.
'eval_batch_size':16,
'r': 7, // Number of decoder frames to predict per iteration. Set the initial values if gradual training is enabled.
'gradual_training': [[0, 7, 64], [1, 5, 64], [50000, 3, 32], [130000, 2, 32], [290000, 1, 32]], //set gradual training steps [first_step, r, batch_size]. If it is null, gradual training is disabled. For Tacotron, you might need to reduce the 'batch_size' as you proceeed.
'loss_masking': true, // enable / disable loss masking against the sequence padding.
'ga_alpha': 10.0, // weight for guided attention loss. If > 0, guided attention is enabled.
// VALIDATION
'run_eval': true,
'test_delay_epochs': 10, //Until attention is aligned, testing only wastes computation time.
'test_sentences_file': null, // set a file to load sentences to be used for testing. If it is null then we use default english sentences.
// OPTIMIZER
'noam_schedule': false, // use noam warmup and lr schedule.
'grad_clip': 1.0, // upper limit for gradients for clipping.
'epochs': 1000, // total number of epochs to train.
'lr': 0.0001, // Initial learning rate. If Noam decay is active, maximum learning rate.
'wd': 0.000001, // Weight decay weight.
'warmup_steps': 4000, // Noam decay steps to increase the learning rate from 0 to 'lr'
'seq_len_norm': false, // Normalize eash sample loss with its length to alleviate imbalanced datasets. Use it if your dataset is small or has skewed distribution of sequence lengths.
// TACOTRON PRENET
'memory_size': -1, // ONLY TACOTRON - size of the memory queue used fro storing last decoder predictions for auto-regression. If < 0, memory queue is disabled and decoder only uses the last prediction frame.
'prenet_type': 'original', // 'original' or 'bn'.
'prenet_dropout': true, // enable/disable dropout at prenet.
// ATTENTION
'attention_type': 'original', // 'original' or 'graves'
'attention_heads': 4, // number of attention heads (only for 'graves')
'attention_norm': 'sigmoid', // softmax or sigmoid. Suggested to use softmax for Tacotron2 and sigmoid for Tacotron.
'windowing': false, // Enables attention windowing. Used only in eval mode.
'use_forward_attn': false, // if it uses forward attention. In general, it aligns faster.
'forward_attn_mask': false, // Additional masking forcing monotonicity only in eval mode.
'transition_agent': false, // enable/disable transition agent of forward attention.
'location_attn': true, // enable_disable location sensitive attention. It is enabled for TACOTRON by default.
'bidirectional_decoder': false, // use https://arxiv.org/abs/1907.09006. Use it, if attention does not work well with your dataset.
// STOPNET
'stopnet': true, // Train stopnet predicting the end of synthesis.
'separate_stopnet': true, // Train stopnet seperately if 'stopnet==true'. It prevents stopnet loss to influence the rest of the model. It causes a better model, but it trains SLOWER.
// TENSORBOARD and LOGGING
'print_step': 25, // Number of steps to log traning on console.
'print_eval': false, // If True, it prints intermediate loss values in evalulation.
'save_step': 10000, // Number of training steps expected to save traninpg stats and checkpoints.
'checkpoint': true, // If true, it saves checkpoints per 'save_step'
'tb_model_param_stats': false, // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.
// DATA LOADING
'text_cleaner': 'phoneme_cleaners',
'enable_eos_bos_chars': false, // enable/disable beginning of sentence and end of sentence chars.
'num_loader_workers': 4, // number of training data loader processes. Don't set it too big. 4-8 are good values.
'num_val_loader_workers': 4, // number of evaluation data loader processes.
'batch_group_size': 0, //Number of batches to shuffle after bucketing.
'min_seq_len': 6, // DATASET-RELATED: minimum text length to use in training
'max_seq_len': 153, // DATASET-RELATED: maximum text length
// PATHS
'output_path': '/home/sadam/Projects/Italian_TTS/TTS/Models/LJSpeech/',
// PHONEMES
'phoneme_cache_path': '/home/sadam/Projects/Italian_TTS/TTS/mozilla_us_phonemes_3', // phoneme computation is slow, therefore, it caches results in the given folder.
'use_phonemes': true, // use phonemes instead of raw characters. It is suggested for better pronounciation.
'phoneme_language': 'it', // depending on your target language, pick one from https://github.com/bootphon/phonemizer#languages
// MULTI-SPEAKER and GST
'use_speaker_embedding': false, // use speaker embedding to enable multi-speaker learning.
'style_wav_for_test': null, // path to style wav file to be used in TacotronGST inference.
'use_gst': false, // TACOTRON ONLY: use global style tokens
// DATASETS
'datasets': // List of datasets. They all merged and they get different speaker_ids.
[
{
'name': 'ljspeech',
'path': '/home/sadam/Projects/Italian_TTS/LJSpeech-1.1/',
'meta_file_train': 'metadata.csv',
'meta_file_val': null
}
]
}
Here is the text from logs of different vm machines.
log_1 slash _tesla_v slash _100
> > Using CUDA: True slash
> > Number of GPUs: 1 slash
> > Git Hash: 8f8ba5e slash
> > Experiment folder:
> > .../LJSpeech/ljspeech-June-07-2020_06+46PM-8f8ba5e slash
> > Setting up Audio Processor... slash
> > slash | slash > sample_rate:22050 slash
> > slash | slash > num_mels:80 slash
> > slash | slash > min_level_db:-100 slash
> > slash | slash > frame_shift_ms:None slash
> > slash | slash > frame_length_ms:None slash
> > slash | slash > ref_level_db:20 slash
> > slash | slash > num_freq:513 slash
> > slash | slash > power:1.5 slash
> > slash | slash > preemphasis:0.0 slash
> > slash | slash > griffin_lim_iters:60 slash
> > slash | slash > signal_norm:True slash
> > slash | slash > symmetric_norm:True slash
> > slash | slash > mel_fmin:0 slash
> > slash | slash > mel_fmax:8000.0 slash
> > slash | slash > max_norm:4.0 slash
> > slash | slash > clip_norm:True slash
> > slash | slash > do_trim_silence:True slash
> > slash | slash > trim_db:60 slash
> > slash | slash > do_sound_norm:False slash
> > slash | slash > stats_path:None slash
> > slash | slash > hop_length:256 slash
> > slash | slash > win_length:1024 slash
> > slash | slash > n_fft:1024 slash
> > Using model: Tacotron2 slash
> > slash | slash > Num output units : 513
> Model has 28921234 parameters
> EPOCH: 0/1000
> Number of output frames: 7
> DataLoader initialization slash
> slash | slash > Use phonemes: True slash
> slash | slash > phoneme language: it slash
> slash | slash > Number of instances : 14820 slash
> slash | slash > Max length sequence: 1029 slash
> slash | slash > Min length sequence: 2 slash
> slash | slash > Avg length sequence: 113.39838056680162 slash
> slash | slash > Num. instances discarded by max-min (max=153, min=6) seq limits:
> 4153 slash
> slash | slash > Batch group size: 0.
> TRAINING (2020-06-07 18:46:37)
slash | slash > decoder_loss: 6.44258 (7.48856) slash
slash | slash > postnet_loss: 8.24267 (9.43440) slash
slash | slash > stopnet_loss: 0.75217 (0.80312) slash
slash | slash > ga_loss: 0.13037 (0.22320) slash
slash | slash > loss: 14.81562 slash
slash | slash > align_error: 0.95799 (0.93276) slash
slash | slash > avg_spec_len: 173.09375 slash
slash | slash > avg_text_len: 21.90625 slash
slash | slash > step_time: 0.30 slash
slash | slash > loader_time: 19.86 slash
slash | slash > lr: 0.00010
slash | slash > decoder_loss: 5.46517 (6.67045) slash
slash | slash > postnet_loss: 5.34141 (8.11494) slash
slash | slash > stopnet_loss: 0.65614 (0.77784) slash
slash | slash > ga_loss: 0.07665 (0.16223) slash
slash | slash > loss: 10.88323 slash
slash | slash > align_error: 0.97988 (0.94869) slash
slash | slash > avg_spec_len: 256.21875 slash
slash | slash > avg_text_len: 37.78125 slash
slash | slash > step_time: 0.63 slash
slash | slash > loader_time: 1.59 slash
slash | slash > lr: 0.00010
slash | slash > decoder_loss: 4.78313 (6.11033) slash
slash | slash > postnet_loss: 3.83975 (6.78688) slash
slash | slash > stopnet_loss: 0.58747 (0.74448) slash
slash | slash > ga_loss: 0.05462 (0.12980) slash
slash | slash > loss: 8.67750 slash
slash | slash > align_error: 0.97914 (0.95827) slash
slash | slash > avg_spec_len: 340.5625 slash
slash | slash > avg_text_len: 54.28125 slash
slash | slash > step_time: 0.52 slash
slash | slash > loader_time: 0.62 slash
slash | slash > lr: 0.00010
slash | slash > decoder_loss: 2.77750 (5.47478) slash
slash | slash > postnet_loss: 2.61901 (5.85372) slash
slash | slash > stopnet_loss: 0.40204 (0.68065) slash
slash | slash > ga_loss: 0.03796 (0.10869) slash
slash | slash > loss: 5.43447 slash
slash | slash > align_error: 0.98513 (0.96466) slash
slash | slash > avg_spec_len: 431.40625 slash
slash | slash > avg_text_len: 77.59375 slash
slash | slash > step_time: 0.51 slash
slash | slash > loader_time: 0.01 slash
slash | slash > lr: 0.00010 slash
slash [WARNING slash ] fount 1 utterances containing language switches on lines 2 slash
slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
slash [WARNING slash ] language switch flags have been kept (applying 'keep-flags'
policy)
slash | slash > decoder_loss: 2.40896 (4.89162) slash
slash | slash > postnet_loss: 2.30191 (5.19040) slash
slash | slash > stopnet_loss: 0.33852 (0.61896) slash
slash | slash > ga_loss: 0.02859 (0.09361) slash
slash | slash > loss: 4.73946 slash
slash | slash > align_error: 0.98797 (0.96909) slash
slash | slash > avg_spec_len: 599.25 slash
slash | slash > avg_text_len: 103.390625 slash
slash | slash > step_time: 0.70 slash
slash | slash > loader_time: 19.06 slash
slash | slash > lr: 0.00010 slash
slash [WARNING slash ] fount 1 utterances containing language switches on lines 1 slash
slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
slash [WARNING slash ] language switch flags have been kept (applying 'keep-flags'
policy) slash
slash [WARNING slash ] fount 1 utterances containing language switches on lines 2 slash
slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
slash [WARNING slash ] language switch flags have been kept (applying 'keep-flags'
policy) slash
slash [WARNING slash ] fount 1 utterances containing language switches on lines 4 slash
slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
slash [WARNING slash ] language switch flags have been kept (applying 'keep-flags'
policy)
slash | slash > decoder_loss: 1.87681 (4.43319) slash
slash | slash > postnet_loss: 1.90342 (4.68345) slash
slash | slash > stopnet_loss: 0.31595 (0.57050) slash
slash | slash > ga_loss: 0.02243 (0.08223) slash
slash | slash > loss: 3.80266 slash
slash | slash > align_error: 0.99110 (0.97259) slash
slash | slash > avg_spec_len: 727.28125 slash
slash | slash > avg_text_len: 133.703125 slash
slash | slash > step_time: 0.80 slash
slash | slash > loader_time: 3.95 slash
slash | slash > lr: 0.00010 slash
slash [WARNING slash ] fount 1 utterances containing language switches on lines 2 slash
slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
slash [WARNING slash ] language switch flags have been kept (applying 'keep-flags'
policy)
slash | slash > avg_postnet_loss: 4.38260 slash
slash | slash > avg_decoder_loss: 4.15887 slash
slash | slash > avg_stopnet_loss: 0.54385 slash
slash | slash > avg_align_error: 0.97455 slash
slash | slash > avg_step_time: 0.47402 slash
slash | slash > avg_loader_time: 6.36835 slash
slash | slash > avg_ga_loss: 0.07595
> EVALUATION
warning: audio amplitude out of range, auto clipped. slash
slash | slash > avg_postnet_loss: 1.50400 (+0.00000) slash
slash | slash > avg_decoder_loss: 1.80960 (+0.00000) slash
slash | slash > avg_stopnet_loss: 0.60253 (+0.00000) slash
slash | slash > avg_align_error: 0.97442 (+0.00000) slash
slash | slash > avg_ga_loss: 0.06078 (+0.00000)
> BEST MODEL :
> .../LJSpeech/ljspeech-June-07-2020_06+46PM-8f8ba5e/best_model.pth.tar
> EPOCH: 1/1000
> Number of output frames: 5
> TRAINING (2020-06-07 19:06:01)
slash | slash > decoder_loss: 2.15284 (2.45591) slash
slash | slash > postnet_loss: 1.97614 (2.40361) slash
slash | slash > stopnet_loss: 0.79609 (0.77545) slash
slash | slash > ga_loss: 0.20827 (0.29093) slash
slash | slash > loss: 4.33725 slash
slash | slash > align_error: 0.91573 (0.89624) slash
slash | slash > avg_spec_len: 122.21875 slash
slash | slash > avg_text_len: 11.84375 slash
slash | slash > step_time: 0.25 slash
slash | slash > loader_time: 0.01 slash
slash | slash > lr: 0.00010
slash | slash > decoder_loss: 1.29097 (1.77009) slash
slash | slash > postnet_loss: 1.19053 (1.82301) slash
slash | slash > stopnet_loss: 0.56630 (0.69001) slash
slash | slash > ga_loss: 0.08262 (0.16353) slash
slash | slash > loss: 2.56412 slash
slash | slash > align_error: 0.93850 (0.92108) slash
slash | slash > avg_spec_len: 193.15625 slash
slash | slash > avg_text_len: 26.0625 slash
slash | slash > step_time: 0.35 slash
slash | slash > loader_time: 0.01 slash
slash | slash > lr: 0.00010
slash | slash > decoder_loss: 1.07999 (1.50720) slash
slash | slash > postnet_loss: 1.01930 (1.60808) slash
slash | slash > stopnet_loss: 0.45162 (0.60346) slash
slash | slash > ga_loss: 0.05914 (0.12355) slash
slash | slash > loss: 2.15844 slash
slash | slash > align_error: 0.96196 (0.93652) slash
slash | slash > avg_spec_len: 259.546875 slash
slash | slash > avg_text_len: 42.546875 slash
slash | slash > step_time: 0.40 slash
slash | slash > loader_time: 0.01 slash
slash | slash > lr: 0.00010
slash | slash > decoder_loss: 1.02187 (1.36912) slash
slash | slash > postnet_loss: 1.02076 (1.47320) slash
slash | slash > stopnet_loss: 0.34571 (0.53457) slash
slash | slash > ga_loss: 0.04714 (0.10205) slash
slash | slash > loss: 2.08977 slash
slash | slash > align_error: 0.97285 (0.94658) slash
slash | slash > avg_spec_len: 377.828125 slash
slash | slash > avg_text_len: 61.109375 slash
slash | slash > step_time: 0.51 slash
slash | slash > loader_time: 0.01 slash
slash | slash > lr: 0.00010
slash | slash > decoder_loss: 1.05496 (1.28802) slash
slash | slash > postnet_loss: 1.08451 (1.38476) slash
slash | slash > stopnet_loss: 0.22061 (0.47793) slash
slash | slash > ga_loss: 0.03556 (0.08775) slash
slash | slash > loss: 2.17503 slash
slash | slash > align_error: 0.98056 (0.95409) slash
slash | slash > avg_spec_len: 490.6875 slash
slash | slash > avg_text_len: 84.625 slash
slash | slash > step_time: 0.85 slash
slash | slash > loader_time: 0.02 slash
slash | slash > lr: 0.00010
slash | slash > decoder_loss: 0.96331 (1.23226) slash
slash | slash > postnet_loss: 0.92254 (1.31572) slash
slash | slash > stopnet_loss: 0.23428 (0.43323) slash
slash | slash > ga_loss: 0.02817 (0.07714) slash
slash | slash > loss: 1.91402 slash
slash | slash > align_error: 0.98439 (0.95954) slash
slash | slash > avg_spec_len: 610.546875 slash
slash | slash > avg_text_len: 111.125 slash
slash | slash > step_time: 0.84 slash
slash | slash > loader_time: 0.02 slash
slash | slash > lr: 0.00010 slash
Traceback (most recent call last): slash
File 'train.py', line 676, in slash
main(args) slash
File 'train.py', line 591, in main slash
global_step, epoch) slash
File 'train.py', line 170, in train slash
text_input, text_lengths, mel_input, speaker_ids=speaker_ids) slash
File
'/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
line 550, in call slash
result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts
File
'/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8f8ba5e-py3.7.egg/TTS/models/tacotron2.py',
line 76, in forward slash
postnet_outputs = self.postnet(decoder_outputs) slash
File
'/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
line 550, in call slash
result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash
File
'/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8f8ba5e-py3.7.egg/TTS/layers/tacotron2.py',
line 49, in forward slash
o = layer(o) slash
File
'/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
line 550, in call slash
result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash
File
'/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8f8ba5e-py3.7.egg/TTS/layers/tacotron2.py',
line 27, in forward slash
o = self.convolution1d(x) slash
File
'/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
line 550, in call slash
result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash
File '/opt/conda/lib/python3.7/site-packages/torch/nn/modules/conv.py',
line 208, in forward slash
self.padding, self.dilation, self.groups) slash
RuntimeError: CUDA out of memory. Tried to allocate 250.00 MiB (GPU 0;
15.78 GiB total capacity; 14.32 GiB already allocated; 197.88 MiB free;
14.60 GiB reserved in total by PyTorch) slash
! Run is kept in .../LJSpeech/ljspeech-June-07-2020_06+46PM-8f8ba5e
log_4 slash _RTX_2080_Ti
> > Using CUDA: True slash
> > Number of GPUs: 4 slash
> > Git Hash: 8ec568b slash
> > Experiment folder:
> > .../LJSpeech/ljspeech-June-07-2020_05+54PM-8ec568b slash
> > Setting up Audio Processor... slash
> > slash | slash > sample_rate:22050 slash
> > slash | slash > num_mels:80 slash
> > slash | slash > min_level_db:-100 slash
> > slash | slash > frame_shift_ms:None slash
> > slash | slash > frame_length_ms:None slash
> > slash | slash > ref_level_db:20 slash
> > slash | slash > num_freq:513 slash
> > slash | slash > power:1.5 slash
> > slash | slash > preemphasis:0.0 slash
> > slash | slash > griffin_lim_iters:60 slash
> > slash | slash > signal_norm:True slash
> > slash | slash > symmetric_norm:True slash
> > slash | slash > mel_fmin:0 slash
> > slash | slash > mel_fmax:8000.0 slash
> > slash | slash > max_norm:4.0 slash
> > slash | slash > clip_norm:True slash
> > slash | slash > do_trim_silence:True slash
> > slash | slash > trim_db:60 slash
> > slash | slash > do_sound_norm:False slash
> > slash | slash > stats_path:None slash
> > slash | slash > hop_length:256 slash
> > slash | slash > win_length:1024 slash
> > slash | slash > n_fft:1024 slash
> > Using model: Tacotron2 slash
> > slash | slash > Num output units : 513
> Model has 28921234 parameters
> EPOCH: 0/1000
> Number of output frames: 7
> DataLoader initialization slash
> slash | slash > Use phonemes: True slash
> slash | slash > phoneme language: it slash
> slash | slash > Number of instances : 14820 slash
> slash | slash > Max length sequence: 1029 slash
> slash | slash > Min length sequence: 2 slash
> slash | slash > Avg length sequence: 113.39838056680162 slash
> slash | slash > Num. instances discarded by max-min (max=153, min=6) seq limits:
> 4153 slash
> slash | slash > Batch group size: 0.
> TRAINING (2020-06-07 17:54:21) slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash ' slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash ' slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash ' slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash '
slash | slash > decoder_loss: 4.34630 (5.94781) slash
slash | slash > postnet_loss: 6.14363 (7.89377) slash
slash | slash > stopnet_loss: 0.70103 (0.75914) slash
slash | slash > ga_loss: 0.03984 (0.10998) slash
slash | slash > loss: 10.52977 slash
slash | slash > align_error: 0.98900 (0.96387) slash
slash | slash > avg_spec_len: 438.640625 slash
slash | slash > avg_text_len: 77.015625 slash
slash | slash > step_time: 6.44 slash
slash | slash > loader_time: 33.55 slash
slash | slash > lr: 0.00010 slash
slash [WARNING slash ] fount 1 utterances containing language switches on lines 1 slash
slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
slash [WARNING slash ] language switch flags have been kept (applying 'keep-flags'
policy) slash
slash [WARNING slash ] fount 1 utterances containing language switches on lines 2 slash
slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
slash [WARNING slash ] language switch flags have been kept (applying 'keep-flags'
policy) slash
slash [WARNING slash ] fount 1 utterances containing language switches on lines 2 slash
slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
slash [WARNING slash ] language switch flags have been kept (applying 'keep-flags'
policy) slash
slash [WARNING slash ] fount 1 utterances containing language switches on lines 4 slash
slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
slash [WARNING slash ] language switch flags have been kept (applying 'keep-flags'
policy) slash
Traceback (most recent call last): slash
File 'train.py', line 676, in slash
main(args) slash
File 'train.py', line 591, in main slash
global_step, epoch) slash
File 'train.py', line 170, in train slash
text_input, text_lengths, mel_input, speaker_ids=speaker_ids) slash
File
'/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
line 550, in call slash
result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash
File
'/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8ec568b-py3.7.egg/TTS/models/tacotron2.py',
line 75, in forward slash
encoder_outputs, mel_specs, mask) slash
File
'/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
line 550, in call slash
result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash
File
'/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8ec568b-py3.7.egg/TTS/layers/tacotron2.py',
line 264, in forward slash
decoder_output, attention_weights, stop_token = self.decode(memory) slash
File
'/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8ec568b-py3.7.egg/TTS/layers/tacotron2.py',
line 226, in decode slash
self.processed_inputs, self.mask) slash
File
'/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
line 550, in call slash
result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash
File
'/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8ec568b-py3.7.egg/TTS/layers/common_layers.py',
line 335, in forward slash
query, processed_inputs) slash
File
'/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8ec568b-py3.7.egg/TTS/layers/common_layers.py',
line 275, in get_location_attention slash
torch.tanh(processed_query + processed_attention_weights + slash
RuntimeError: CUDA out of memory. Tried to allocate 18.00 MiB (GPU 2;
10.76 GiB total capacity; 9.86 GiB already allocated; 17.56 MiB free;
9.96 GiB reserved in total by PyTorch)
6_RTX_2080_Ti
> Using CUDA: True slash
> Number of GPUs: 6 slash
> Git Hash: 8f8ba5e slash
> Experiment folder: .../LJSpeech/ljspeech-June-07-2020_09+10PM-8f8ba5e slash
> Setting up Audio Processor... slash
> slash | slash > sample_rate:22050 slash
> slash | slash > num_mels:80 slash
> slash | slash > min_level_db:-100 slash
> slash | slash > frame_shift_ms:None slash
> slash | slash > frame_length_ms:None slash
> slash | slash > ref_level_db:20 slash
> slash | slash > num_freq:513 slash
> slash | slash > power:1.5 slash
> slash | slash > preemphasis:0.0 slash
> slash | slash > griffin_lim_iters:60 slash
> slash | slash > signal_norm:True slash
> slash | slash > symmetric_norm:True slash
> slash | slash > mel_fmin:0 slash
> slash | slash > mel_fmax:8000.0 slash
> slash | slash > max_norm:4.0 slash
> slash | slash > clip_norm:True slash
> slash | slash > do_trim_silence:True slash
> slash | slash > trim_db:60 slash
> slash | slash > do_sound_norm:False slash
> slash | slash > stats_path:None slash
> slash | slash > hop_length:256 slash
> slash | slash > win_length:1024 slash
> slash | slash > n_fft:1024 slash
> Using model: Tacotron2 slash
> slash | slash > Num output units : 513
> Model has 28921234 parameters
> EPOCH: 0/1000
> Number of output frames: 7
> DataLoader initialization slash
> slash | slash > Use phonemes: True slash
> slash | slash > phoneme language: it slash
> slash | slash > Number of instances : 14820 slash
> slash | slash > Max length sequence: 1029 slash
> slash | slash > Min length sequence: 2 slash
> slash | slash > Avg length sequence: 113.39838056680162 slash
> slash | slash > Num. instances discarded by max-min (max=153, min=6) seq limits:
> 4153 slash
> slash | slash > Batch group size: 0.
> TRAINING (2020-06-07 21:10:31) slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash ' slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash ' slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash ' slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash ' slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash ' slash
> /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:102:
> UserWarning: torch.distributed.reduce_op is deprecated, please use
> torch.distributed.ReduceOp instead slash
> warnings.warn( slash 'torch.distributed.reduce_op is deprecated, please use
> slash ' slash
> slash [WARNING slash ] fount 1 utterances containing language switches on lines
> 2 slash
> slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
> slash [WARNING slash ] language switch flags have been kept (applying
> 'keep-flags' policy) slash
> slash [WARNING slash ] fount 1 utterances containing language switches on lines
> 1 slash
> slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
> slash [WARNING slash ] language switch flags have been kept (applying
> 'keep-flags' policy) slash
> slash [WARNING slash ] fount 1 utterances containing language switches on lines
> 4 slash
> slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
> slash [WARNING slash ] language switch flags have been kept (applying
> 'keep-flags' policy) slash
> slash [WARNING slash ] fount 1 utterances containing language switches on lines
> 2 slash
> slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
> slash [WARNING slash ] language switch flags have been kept (applying
> 'keep-flags' policy) slash
> Traceback (most recent call last): slash
> File 'train.py', line 676, in slash
> main(args) slash
> File 'train.py', line 591, in main slash
> global_step, epoch) slash
> File 'train.py', line 170, in train slash
> text_input, text_lengths, mel_input, speaker_ids=speaker_ids) slash
> File
> '/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
> line 550, in call slash
> result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash
> File
> '/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8f8ba5e-py3.7.egg/TTS/models/tacotron2.py',
> line 75, in forward slash
> encoder_outputs, mel_specs, mask) slash
> File
> '/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
> line 550, in call slash
> result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash
> File
> '/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8f8ba5e-py3.7.egg/TTS/layers/tacotron2.py',
> line 264, in forward slash
> decoder_output, attention_weights, stop_token = self.decode(memory) slash
> File
> '/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8f8ba5e-py3.7.egg/TTS/layers/tacotron2.py',
> line 226, in decode slash
> self.processed_inputs, self.mask) slash
> File
> '/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py',
> line 550, in call slash
> result = self.forward( slash *input, slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *kwargs) slash
> File
> '/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8f8ba5e-py3.7.egg/TTS/layers/common_layers.py',
> line 335, in forward slash
> query, processed_inputs) slash
> File
> '/opt/conda/lib/python3.7/site-packages/TTS-0.0.2+8f8ba5e-py3.7.egg/TTS/layers/common_layers.py',
> line 276, in get_location_attention slash
> processed_inputs)) slash
> RuntimeError: CUDA out of memory. Tried to allocate 18.00 MiB (GPU 0;
> 10.76 GiB total capacity; 9.87 GiB already allocated; 10.25 MiB free;
> 9.97 GiB reserved in total by PyTorch) slash
> ! Run is removed from
> .../LJSpeech/ljspeech-June-07-2020_09+10PM-8f8ba5e slash
> slash [WARNING slash ] fount 1 utterances containing language switches on lines
> 2 slash
> slash [WARNING slash ] extra phones may appear in the 'it' phoneset slash
> slash [WARNING slash ] language switch flags have been kept (applying
> 'keep-flags' policy)
[This is an archived TTS discussion thread from discourse.mozilla.org/t/runtimeerror-cuda-out-of-memory]
Beta Was this translation helpful? Give feedback.
All reactions