-
Notifications
You must be signed in to change notification settings - Fork 5.4k
Nnet3 dropout: code for Fast LSTM and scripts for AMI etc #1537
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
13e8bed
[src,scripts,egs] nnet3,fast-lstm: changes to support separate per-fr…
danpovey 863534b
[egs] Small fixes/additions in Swbd/s5c chain scripts
danpovey 8384eae
Merge branch 'shortcut' into shortcut-dropout
danpovey eb0f458
[src,egs,scripts] Modifying dropout in LSTM to be on (i,f,o) gates no…
danpovey 96d92d7
Merge remote-tracking branch 'upstream/shortcut' into shortcut-dropout
danpovey 19af8ca
Merge remote-tracking branch 'upstream/shortcut' into shortcut-dropout
danpovey 4d2f00e
Merge branch 'shortcut' into shortcut-dropout
danpovey 6582acf
[scripts] Update example scripts for dropout on Tedlium s5_r2
danpovey a406d0f
Merge branch 'shortcut' into shortcut-dropout
danpovey eb94ffd
for ref
GaofengCheng b9c3e20
merge fast lstm dropout
GaofengCheng 9afaf39
delete temporary tuning sdripts in tedlium
GaofengCheng e9ac4e2
delete irrelevant file
GaofengCheng 638f083
delete exclusive option in fast lstm code
GaofengCheng 49c4558
solve some cuda-kernel line mismatch problem
GaofengCheng 05fc6d2
small bug fix
GaofengCheng 90df5d7
small fix
GaofengCheng 1a58236
update scripts for tdnn-(fast)lstm of AMI-IHM
GaofengCheng 69a36e4
change scripts comment style and RESULTS
GaofengCheng d03be0f
adding SDM results
GaofengCheng 07d6774
Merge branch 'master' of https://github.com/kaldi-asr/kaldi into nnet…
GaofengCheng 936863e
adding SWBD (parts of all) scripts with dropout
GaofengCheng f51fb75
small fix
GaofengCheng 139f412
update tdnn-blstm with dropout in SWBD
GaofengCheng 9a8b81c
update tdnn+regular-LSTM(4epoch) in SWBD
GaofengCheng 48f41a7
adding tedlium scripts
GaofengCheng 62fee2b
small fix
GaofengCheng File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,248 @@ | ||
| #!/bin/bash | ||
|
|
||
| # 6l is same as 6k, but with the per-frame dropout | ||
| # location4 as paper : http://www.danielpovey.com/files/2017_interspeech_dropout.pdf | ||
|
|
||
| # local/chain/compare_wer_general.sh blstm_6k_sp blstm_6l_sp | ||
| # attention: the blatm_6k_sp result here is far better than the updated | ||
| # result (14.5 vs 14.1), this may due to noise | ||
|
|
||
| # System blstm_6k_sp blstm_6l_sp | ||
| # WER on train_dev(tg) 13.30 13.06 | ||
| # WER on train_dev(fg) 12.34 12.16 | ||
| # WER on eval2000(tg) 15.5 15.2 | ||
| # WER on eval2000(fg) 14.1 13.8 | ||
| # Final train prob -0.052 -0.065 | ||
| # Final valid prob -0.090 -0.093 | ||
| # Final train prob (xent) -0.743 -0.831 | ||
| # Final valid prob (xent) -0.9579 -0.9821 | ||
|
|
||
| # exp/chain/blstm_6k_sp/: num-iters=327 nj=3..16 num-params=41.2M dim=40+100->6074 combine=-0.069->-0.069 xent:train/valid[217,326,final]=(-0.849,-0.748,-0.743/-1.04,-0.959,-0.958) logprob:train/valid[217,326,final]=(-0.065,-0.053,-0.052/-0.096,-0.090,-0.090) | ||
| # exp/chain/blstm_6l_sp/: num-iters=327 nj=3..16 num-params=41.2M dim=40+100->6074 combine=-0.084->-0.082 xent:train/valid[217,326,final]=(-1.45,-0.840,-0.831/-1.58,-0.994,-0.982) logprob:train/valid[217,326,final]=(-0.110,-0.066,-0.065/-0.132,-0.094,-0.093) | ||
| set -e | ||
|
|
||
| # configs for 'chain' | ||
| stage=12 | ||
| train_stage=-10 | ||
| get_egs_stage=-10 | ||
| speed_perturb=true | ||
| dir=exp/chain/blstm_6l # Note: _sp will get added to this if $speed_perturb == true. | ||
| decode_iter= | ||
| decode_dir_affix= | ||
|
|
||
| # training options | ||
| leftmost_questions_truncate=-1 | ||
| chunk_width=150 | ||
| chunk_left_context=40 | ||
| chunk_right_context=40 | ||
| xent_regularize=0.025 | ||
| self_repair_scale=0.00001 | ||
| label_delay=0 | ||
| dropout_schedule='0,0@0.20,0.1@0.50,0' | ||
|
|
||
| # decode options | ||
| extra_left_context=50 | ||
| extra_right_context=50 | ||
| frames_per_chunk= | ||
|
|
||
| remove_egs=false | ||
| common_egs_dir= | ||
|
|
||
| affix= | ||
| # End configuration section. | ||
| echo "$0 $@" # Print the command line for logging | ||
|
|
||
| . ./cmd.sh | ||
| . ./path.sh | ||
| . ./utils/parse_options.sh | ||
|
|
||
| if ! cuda-compiled; then | ||
| cat <<EOF && exit 1 | ||
| This script is intended to be used with GPUs but you have not compiled Kaldi with CUDA | ||
| If you want to use GPUs (and have them), go to src/, and configure and make on a machine | ||
| where "nvcc" is installed. | ||
| EOF | ||
| fi | ||
|
|
||
| # The iVector-extraction and feature-dumping parts are the same as the standard | ||
| # nnet3 setup, and you can skip them by setting "--stage 8" if you have already | ||
| # run those things. | ||
|
|
||
| suffix= | ||
| if [ "$speed_perturb" == "true" ]; then | ||
| suffix=_sp | ||
| fi | ||
|
|
||
| dir=$dir${affix:+_$affix} | ||
| if [ $label_delay -gt 0 ]; then dir=${dir}_ld$label_delay; fi | ||
| dir=${dir}$suffix | ||
| train_set=train_nodup$suffix | ||
| ali_dir=exp/tri4_ali_nodup$suffix | ||
| treedir=exp/chain/tri5_7d_tree$suffix | ||
| lang=data/lang_chain_2y | ||
|
|
||
|
|
||
| # if we are using the speed-perturbed data we need to generate | ||
| # alignments for it. | ||
| local/nnet3/run_ivector_common.sh --stage $stage \ | ||
| --speed-perturb $speed_perturb \ | ||
| --generate-alignments $speed_perturb || exit 1; | ||
|
|
||
|
|
||
| if [ $stage -le 9 ]; then | ||
| # Get the alignments as lattices (gives the CTC training more freedom). | ||
| # use the same num-jobs as the alignments | ||
| nj=$(cat exp/tri4_ali_nodup$suffix/num_jobs) || exit 1; | ||
| steps/align_fmllr_lats.sh --nj $nj --cmd "$train_cmd" data/$train_set \ | ||
| data/lang exp/tri4 exp/tri4_lats_nodup$suffix | ||
| rm exp/tri4_lats_nodup$suffix/fsts.*.gz # save space | ||
| fi | ||
|
|
||
|
|
||
| if [ $stage -le 10 ]; then | ||
| # Create a version of the lang/ directory that has one state per phone in the | ||
| # topo file. [note, it really has two states.. the first one is only repeated | ||
| # once, the second one has zero or more repeats.] | ||
| rm -rf $lang | ||
| cp -r data/lang $lang | ||
| silphonelist=$(cat $lang/phones/silence.csl) || exit 1; | ||
| nonsilphonelist=$(cat $lang/phones/nonsilence.csl) || exit 1; | ||
| # Use our special topology... note that later on may have to tune this | ||
| # topology. | ||
| steps/nnet3/chain/gen_topo.py $nonsilphonelist $silphonelist >$lang/topo | ||
| fi | ||
|
|
||
| if [ $stage -le 11 ]; then | ||
| # Build a tree using our new topology. | ||
| steps/nnet3/chain/build_tree.sh --frame-subsampling-factor 3 \ | ||
| --leftmost-questions-truncate $leftmost_questions_truncate \ | ||
| --context-opts "--context-width=2 --central-position=1" \ | ||
| --cmd "$train_cmd" 7000 data/$train_set $lang $ali_dir $treedir | ||
| fi | ||
|
|
||
| if [ $stage -le 12 ]; then | ||
| echo "$0: creating neural net configs using the xconfig parser"; | ||
|
|
||
| num_targets=$(tree-info $treedir/tree |grep num-pdfs|awk '{print $2}') | ||
| [ -z $num_targets ] && { echo "$0: error getting num-targets"; exit 1; } | ||
| learning_rate_factor=$(echo "print 0.5/$xent_regularize" | python) | ||
|
|
||
| lstm_opts="decay-time=20 dropout-proportion=0.0" | ||
|
|
||
| mkdir -p $dir/configs | ||
| cat <<EOF > $dir/configs/network.xconfig | ||
| input dim=100 name=ivector | ||
| input dim=40 name=input | ||
|
|
||
| # please note that it is important to have input layer with the name=input | ||
| # as the layer immediately preceding the fixed-affine-layer to enable | ||
| # the use of short notation for the descriptor | ||
| fixed-affine-layer name=lda input=Append(-2,-1,0,1,2,ReplaceIndex(ivector, t, 0)) affine-transform-file=$dir/configs/lda.mat | ||
|
|
||
| # the first splicing is moved before the lda layer, so no splicing here | ||
|
|
||
| # check steps/libs/nnet3/xconfig/lstm.py for the other options and defaults | ||
| fast-lstmp-layer name=blstm1-forward input=lda cell-dim=1024 recurrent-projection-dim=256 non-recurrent-projection-dim=256 delay=-3 $lstm_opts | ||
| fast-lstmp-layer name=blstm1-backward input=lda cell-dim=1024 recurrent-projection-dim=256 non-recurrent-projection-dim=256 delay=3 $lstm_opts | ||
|
|
||
| fast-lstmp-layer name=blstm2-forward input=Append(blstm1-forward, blstm1-backward) cell-dim=1024 recurrent-projection-dim=256 non-recurrent-projection-dim=256 delay=-3 $lstm_opts | ||
| fast-lstmp-layer name=blstm2-backward input=Append(blstm1-forward, blstm1-backward) cell-dim=1024 recurrent-projection-dim=256 non-recurrent-projection-dim=256 delay=3 $lstm_opts | ||
|
|
||
| fast-lstmp-layer name=blstm3-forward input=Append(blstm2-forward, blstm2-backward) cell-dim=1024 recurrent-projection-dim=256 non-recurrent-projection-dim=256 delay=-3 $lstm_opts | ||
| fast-lstmp-layer name=blstm3-backward input=Append(blstm2-forward, blstm2-backward) cell-dim=1024 recurrent-projection-dim=256 non-recurrent-projection-dim=256 delay=3 $lstm_opts | ||
|
|
||
| ## adding the layers for chain branch | ||
| output-layer name=output input=Append(blstm3-forward, blstm3-backward) output-delay=$label_delay include-log-softmax=false dim=$num_targets max-change=1.5 | ||
|
|
||
| # adding the layers for xent branch | ||
| # This block prints the configs for a separate output that will be | ||
| # trained with a cross-entropy objective in the 'chain' models... this | ||
| # has the effect of regularizing the hidden parts of the model. we use | ||
| # 0.5 / args.xent_regularize as the learning rate factor- the factor of | ||
| # 0.5 / args.xent_regularize is suitable as it means the xent | ||
| # final-layer learns at a rate independent of the regularization | ||
| # constant; and the 0.5 was tuned so as to make the relative progress | ||
| # similar in the xent and regular final layers. | ||
| output-layer name=output-xent input=Append(blstm3-forward, blstm3-backward) output-delay=$label_delay dim=$num_targets learning-rate-factor=$learning_rate_factor max-change=1.5 | ||
|
|
||
| EOF | ||
| steps/nnet3/xconfig_to_configs.py --xconfig-file $dir/configs/network.xconfig --config-dir $dir/configs/ | ||
| fi | ||
|
|
||
| if [ $stage -le 13 ]; then | ||
| if [[ $(hostname -f) == *.clsp.jhu.edu ]] && [ ! -d $dir/egs/storage ]; then | ||
| utils/create_split_dir.pl \ | ||
| /export/b0{5,6,7,8}/$USER/kaldi-data/egs/swbd-$(date +'%m_%d_%H_%M')/s5c/$dir/egs/storage $dir/egs/storage | ||
| fi | ||
|
|
||
| steps/nnet3/chain/train.py --stage $train_stage \ | ||
| --cmd "$decode_cmd" \ | ||
| --feat.online-ivector-dir exp/nnet3/ivectors_${train_set} \ | ||
| --feat.cmvn-opts "--norm-means=false --norm-vars=false" \ | ||
| --chain.xent-regularize $xent_regularize \ | ||
| --chain.leaky-hmm-coefficient 0.1 \ | ||
| --chain.l2-regularize 0.00005 \ | ||
| --chain.apply-deriv-weights false \ | ||
| --chain.lm-opts="--num-extra-lm-states=2000" \ | ||
| --trainer.num-chunk-per-minibatch 64 \ | ||
| --trainer.frames-per-iter 1200000 \ | ||
| --trainer.max-param-change 2.0 \ | ||
| --trainer.num-epochs 4 \ | ||
| --trainer.optimization.shrink-value 0.99 \ | ||
| --trainer.optimization.num-jobs-initial 3 \ | ||
| --trainer.optimization.num-jobs-final 16 \ | ||
| --trainer.optimization.initial-effective-lrate 0.001 \ | ||
| --trainer.optimization.final-effective-lrate 0.0001 \ | ||
| --trainer.optimization.momentum 0.0 \ | ||
| --trainer.deriv-truncate-margin 8 \ | ||
| --egs.stage $get_egs_stage \ | ||
| --egs.opts "--frames-overlap-per-eg 0" \ | ||
| --egs.chunk-width $chunk_width \ | ||
| --egs.chunk-left-context $chunk_left_context \ | ||
| --egs.chunk-right-context $chunk_right_context \ | ||
| --trainer.dropout-schedule $dropout_schedule \ | ||
| --egs.dir "$common_egs_dir" \ | ||
| --cleanup.remove-egs $remove_egs \ | ||
| --feat-dir data/${train_set}_hires \ | ||
| --tree-dir $treedir \ | ||
| --lat-dir exp/tri4_lats_nodup$suffix \ | ||
| --dir $dir || exit 1; | ||
| fi | ||
|
|
||
| if [ $stage -le 14 ]; then | ||
| # Note: it might appear that this $lang directory is mismatched, and it is as | ||
| # far as the 'topo' is concerned, but this script doesn't read the 'topo' from | ||
| # the lang directory. | ||
| utils/mkgraph.sh --self-loop-scale 1.0 data/lang_sw1_tg $dir $dir/graph_sw1_tg | ||
| fi | ||
|
|
||
| decode_suff=sw1_tg | ||
| graph_dir=$dir/graph_sw1_tg | ||
| if [ $stage -le 15 ]; then | ||
| [ -z $extra_left_context ] && extra_left_context=$chunk_left_context; | ||
| [ -z $extra_right_context ] && extra_right_context=$chunk_right_context; | ||
| [ -z $frames_per_chunk ] && frames_per_chunk=$chunk_width; | ||
| iter_opts= | ||
| if [ ! -z $decode_iter ]; then | ||
| iter_opts=" --iter $decode_iter " | ||
| fi | ||
| for decode_set in train_dev eval2000; do | ||
| ( | ||
| steps/nnet3/decode.sh --acwt 1.0 --post-decode-acwt 10.0 \ | ||
| --nj 50 --cmd "$decode_cmd" $iter_opts \ | ||
| --extra-left-context $extra_left_context \ | ||
| --extra-right-context $extra_right_context \ | ||
| --frames-per-chunk "$frames_per_chunk" \ | ||
| --online-ivector-dir exp/nnet3/ivectors_${decode_set} \ | ||
| $graph_dir data/${decode_set}_hires \ | ||
| $dir/decode_${decode_set}${decode_dir_affix:+_$decode_dir_affix}_${decode_suff} || exit 1; | ||
| if $has_fisher; then | ||
| steps/lmrescore_const_arpa.sh --cmd "$decode_cmd" \ | ||
| data/lang_sw1_{tg,fsh_fg} data/${decode_set}_hires \ | ||
| $dir/decode_${decode_set}${decode_dir_affix:+_$decode_dir_affix}_sw1_{tg,fsh_fg} || exit 1; | ||
| fi | ||
| ) & | ||
| done | ||
| fi | ||
| wait; | ||
| exit 0; |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At this level, please just include the results for the "recommended" system which is 1m.
You should put all the comparative results in the individual scripts inside local/chain/tuning.
Use the standard compare_wer.sh script, whatever it's called, and also include the output
of chain_dir_info.pl from each of those scripts, in a comment in that script.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@danpovey OK, will do