Skip to content

[egs] Add improved TDNN-F example for Switchboard. #2291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danpovey merged 1 commit into from
Mar 18, 2018
Merged

[egs] Add improved TDNN-F example for Switchboard. #2291

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion egs/swbd/s5c/local/chain/run_tdnn.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
tuning/run_tdnn_7n.sh
tuning/run_tdnn_7o.sh
297 changes: 297 additions & 0 deletions egs/swbd/s5c/local/chain/tuning/run_tdnn_7o.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,297 @@
#!/bin/bash


# 7o is as 7n but with a bunch of tuning changes affecting both the structure
# and the learning rates/l2 regularization. Structurally the main change is
# that we also do splicing via an extra layer whose input and output are in the
# "small" dim (256); this increases the left and right context. We also change
# the orthonormal-constraint to be "floating" meaning it doesn't constrain the
# size of the matrix (the value orthonormal-constraint=-1 is interpreted
# specially by the code), which means we can control how fast these constrained
# layers learn layers via l2, just like the unconstrained layers. Also the l2
# values were increased and the learning rates were decreased; there are
# more epochs (6->8); and the dimension of some of the layers (the ones that
# are subsampled and which don't receive skip-splicing) was increased from
# 1280 to 1536. The config is a bit messy and I'd like to find a way to
# encapsulate things a bit better; treat this as a work in progress.
#
#
#
# local/chain/compare_wer_general.sh --rt03 tdnn7n_sp tdnn7m26o_sp
# System tdnn7n_sp tdnn7m26j_sp
# WER on train_dev(tg) 12.18 11.74
# WER on train_dev(fg) 11.12 10.69
# WER on eval2000(tg) 14.9 14.6
# WER on eval2000(fg) 13.5 13.1
# WER on rt03(tg) 18.4 17.5
# WER on rt03(fg) 16.2 15.4
# Final train prob -0.077 -0.070
# Final valid prob -0.093 -0.084
# Final train prob (xent) -0.994 -0.883
# Final valid prob (xent) -1.0194 -0.9110
# Num-parameters 20111396 22865188


# exp/chain/tdnn7o_sp: num-iters=525 nj=3..16 num-params=22.9M dim=40+100->6034 combine=-0.074->-0.073 (over 7) xent:train/valid[348,524,final]=(-1.20,-0.884,-0.883/-1.24,-0.918,-0.911) logprob:train/valid[348,524,final]=(-0.100,-0.071,-0.070/-0.115,-0.086,-0.084)

set -e

# configs for 'chain'
stage=0
train_stage=-10
get_egs_stage=-10
speed_perturb=true
affix=7o
suffix=
$speed_perturb && suffix=_sp
if [ -e data/rt03 ]; then maybe_rt03=rt03; else maybe_rt03= ; fi

dir=exp/chain/tdnn${affix}${suffix}
decode_iter=
decode_nj=50

# training options
frames_per_eg=150,110,100
remove_egs=false
common_egs_dir=
xent_regularize=0.1
dropout_schedule='0,0@0.20,0.5@0.50,0'

test_online_decoding=false # if true, it will run the last decoding stage.

# End configuration section.
echo "$0 $@" # Print the command line for logging

. ./cmd.sh
. ./path.sh
. ./utils/parse_options.sh

if ! cuda-compiled; then
cat <<EOF && exit 1
This script is intended to be used with GPUs but you have not compiled Kaldi with CUDA
If you want to use GPUs (and have them), go to src/, and configure and make on a machine
where "nvcc" is installed.
EOF
fi

# The iVector-extraction and feature-dumping parts are the same as the standard
# nnet3 setup, and you can skip them by setting "--stage 8" if you have already
# run those things.

train_set=train_nodup$suffix
ali_dir=exp/tri4_ali_nodup$suffix
treedir=exp/chain/tri5_7d_tree$suffix
lang=data/lang_chain_2y


# if we are using the speed-perturbed data we need to generate
# alignments for it.
local/nnet3/run_ivector_common.sh --stage $stage \
--speed-perturb $speed_perturb \
--generate-alignments $speed_perturb || exit 1;


if [ $stage -le 9 ]; then
# Get the alignments as lattices (gives the LF-MMI training more freedom).
# use the same num-jobs as the alignments
nj=$(cat exp/tri4_ali_nodup$suffix/num_jobs) || exit 1;
steps/align_fmllr_lats.sh --nj $nj --cmd "$train_cmd" data/$train_set \
data/lang exp/tri4 exp/tri4_lats_nodup$suffix
rm exp/tri4_lats_nodup$suffix/fsts.*.gz # save space
fi


if [ $stage -le 10 ]; then
# Create a version of the lang/ directory that has one state per phone in the
# topo file. [note, it really has two states.. the first one is only repeated
# once, the second one has zero or more repeats.]
rm -rf $lang
cp -r data/lang $lang
silphonelist=$(cat $lang/phones/silence.csl) || exit 1;
nonsilphonelist=$(cat $lang/phones/nonsilence.csl) || exit 1;
# Use our special topology... note that later on may have to tune this
# topology.
steps/nnet3/chain/gen_topo.py $nonsilphonelist $silphonelist >$lang/topo
fi

if [ $stage -le 11 ]; then
# Build a tree using our new topology. This is the critically different
# step compared with other recipes.
steps/nnet3/chain/build_tree.sh --frame-subsampling-factor 3 \
--context-opts "--context-width=2 --central-position=1" \
--cmd "$train_cmd" 7000 data/$train_set $lang $ali_dir $treedir
fi

if [ $stage -le 12 ]; then
echo "$0: creating neural net configs using the xconfig parser";

num_targets=$(tree-info $treedir/tree |grep num-pdfs|awk '{print $2}')
learning_rate_factor=$(echo "print 0.5/$xent_regularize" | python)
opts="l2-regularize=0.004 dropout-proportion=0.0 dropout-per-dim=true dropout-per-dim-continuous=true"
linear_opts="orthonormal-constraint=-1.0 l2-regularize=0.004"
output_opts="l2-regularize=0.002"

mkdir -p $dir/configs

cat <<EOF > $dir/configs/network.xconfig
input dim=100 name=ivector
input dim=40 name=input

# please note that it is important to have input layer with the name=input
# as the layer immediately preceding the fixed-affine-layer to enable
# the use of short notation for the descriptor
fixed-affine-layer name=lda input=Append(-1,0,1,ReplaceIndex(ivector, t, 0)) affine-transform-file=$dir/configs/lda.mat

# the first splicing is moved before the lda layer, so no splicing here
relu-batchnorm-dropout-layer name=tdnn1 $opts dim=1280
linear-component name=tdnn2l0 dim=256 $linear_opts input=Append(-1,0)
linear-component name=tdnn2l dim=256 $linear_opts input=Append(-1,0)
relu-batchnorm-dropout-layer name=tdnn2 $opts input=Append(0,1) dim=1280
linear-component name=tdnn3l dim=256 $linear_opts input=Append(-1,0)
relu-batchnorm-dropout-layer name=tdnn3 $opts dim=1280 input=Append(0,1)
linear-component name=tdnn4l0 dim=256 $linear_opts input=Append(-1,0)
linear-component name=tdnn4l dim=256 $linear_opts input=Append(0,1)
relu-batchnorm-dropout-layer name=tdnn4 $opts input=Append(0,1) dim=1280
linear-component name=tdnn5l dim=256 $linear_opts
relu-batchnorm-dropout-layer name=tdnn5 $opts dim=1280 input=Append(0, tdnn3l)
linear-component name=tdnn6l0 dim=256 $linear_opts input=Append(-3,0)
linear-component name=tdnn6l dim=256 $linear_opts input=Append(-3,0)
relu-batchnorm-dropout-layer name=tdnn6 $opts input=Append(0,3) dim=1536
linear-component name=tdnn7l0 dim=256 $linear_opts input=Append(-3,0)
linear-component name=tdnn7l dim=256 $linear_opts input=Append(0,3)
relu-batchnorm-dropout-layer name=tdnn7 $opts input=Append(0,3,tdnn6l,tdnn4l,tdnn2l) dim=1280
linear-component name=tdnn8l0 dim=256 $linear_opts input=Append(-3,0)
linear-component name=tdnn8l dim=256 $linear_opts input=Append(0,3)
relu-batchnorm-dropout-layer name=tdnn8 $opts input=Append(0,3) dim=1536
linear-component name=tdnn9l0 dim=256 $linear_opts input=Append(-3,0)
linear-component name=tdnn9l dim=256 $linear_opts input=Append(-3,0)
relu-batchnorm-dropout-layer name=tdnn9 $opts input=Append(0,3,tdnn8l,tdnn6l,tdnn5l) dim=1280
linear-component name=tdnn10l0 dim=256 $linear_opts input=Append(-3,0)
linear-component name=tdnn10l dim=256 $linear_opts input=Append(0,3)
relu-batchnorm-dropout-layer name=tdnn10 $opts input=Append(0,3) dim=1536
linear-component name=tdnn11l0 dim=256 $linear_opts input=Append(-3,0)
linear-component name=tdnn11l dim=256 $linear_opts input=Append(-3,0)
relu-batchnorm-dropout-layer name=tdnn11 $opts input=Append(0,3,tdnn10l,tdnn9l,tdnn7l) dim=1280
linear-component name=prefinal-l dim=256 $linear_opts

relu-batchnorm-layer name=prefinal-chain input=prefinal-l $opts dim=1536
linear-component name=prefinal-chain-l dim=256 $linear_opts
batchnorm-component name=prefinal-chain-batchnorm
output-layer name=output include-log-softmax=false dim=$num_targets $output_opts

relu-batchnorm-layer name=prefinal-xent input=prefinal-l $opts dim=1536
linear-component name=prefinal-xent-l dim=256 $linear_opts
batchnorm-component name=prefinal-xent-batchnorm
output-layer name=output-xent dim=$num_targets learning-rate-factor=$learning_rate_factor $output_opts
EOF
steps/nnet3/xconfig_to_configs.py --xconfig-file $dir/configs/network.xconfig --config-dir $dir/configs/
fi

if [ $stage -le 13 ]; then
if [[ $(hostname -f) == *.clsp.jhu.edu ]] && [ ! -d $dir/egs/storage ]; then
utils/create_split_dir.pl \
/export/b0{5,6,7,8}/$USER/kaldi-data/egs/swbd-$(date +'%m_%d_%H_%M')/s5c/$dir/egs/storage $dir/egs/storage
fi

# --cmd "queue.pl --config /home/dpovey/queue_conly.conf" \


steps/nnet3/chain/train.py --stage $train_stage \
--feat.online-ivector-dir exp/nnet3/ivectors_${train_set} \
--feat.cmvn-opts "--norm-means=false --norm-vars=false" \
--chain.xent-regularize $xent_regularize \
--chain.leaky-hmm-coefficient 0.1 \
--chain.l2-regularize 0.0 \
--chain.apply-deriv-weights false \
--chain.lm-opts="--num-extra-lm-states=2000" \
--trainer.dropout-schedule $dropout_schedule \
--trainer.add-option="--optimization.memory-compression-level=2" \
--egs.dir "$common_egs_dir" \
--egs.stage $get_egs_stage \
--egs.opts "--frames-overlap-per-eg 0" \
--egs.chunk-width $frames_per_eg \
--trainer.num-chunk-per-minibatch 128 \
--trainer.frames-per-iter 1500000 \
--trainer.num-epochs 8 \
--trainer.optimization.num-jobs-initial 3 \
--trainer.optimization.num-jobs-final 16 \
--trainer.optimization.initial-effective-lrate 0.0005 \
--trainer.optimization.final-effective-lrate 0.00005 \
--trainer.max-param-change 2.0 \
--cleanup.remove-egs $remove_egs \
--feat-dir data/${train_set}_hires \
--tree-dir $treedir \
--lat-dir exp/tri4_lats_nodup$suffix \
--dir $dir || exit 1;

fi

if [ $stage -le 14 ]; then
# Note: it might appear that this $lang directory is mismatched, and it is as
# far as the 'topo' is concerned, but this script doesn't read the 'topo' from
# the lang directory.
utils/mkgraph.sh --self-loop-scale 1.0 data/lang_sw1_tg $dir $dir/graph_sw1_tg
fi


graph_dir=$dir/graph_sw1_tg
iter_opts=
if [ ! -z $decode_iter ]; then
iter_opts=" --iter $decode_iter "
fi
if [ $stage -le 15 ]; then
rm $dir/.error 2>/dev/null || true
for decode_set in train_dev eval2000 $maybe_rt03; do
(
steps/nnet3/decode.sh --acwt 1.0 --post-decode-acwt 10.0 \
--nj $decode_nj --cmd "$decode_cmd" $iter_opts \
--online-ivector-dir exp/nnet3/ivectors_${decode_set} \
$graph_dir data/${decode_set}_hires \
$dir/decode_${decode_set}${decode_iter:+_$decode_iter}_sw1_tg || exit 1;
if $has_fisher; then
steps/lmrescore_const_arpa.sh --cmd "$decode_cmd" \
data/lang_sw1_{tg,fsh_fg} data/${decode_set}_hires \
$dir/decode_${decode_set}${decode_iter:+_$decode_iter}_sw1_{tg,fsh_fg} || exit 1;
fi
) || touch $dir/.error &
done
wait
if [ -f $dir/.error ]; then
echo "$0: something went wrong in decoding"
exit 1
fi
fi

if $test_online_decoding && [ $stage -le 16 ]; then
# note: if the features change (e.g. you add pitch features), you will have to
# change the options of the following command line.
steps/online/nnet3/prepare_online_decoding.sh \
--mfcc-config conf/mfcc_hires.conf \
$lang exp/nnet3/extractor $dir ${dir}_online

rm $dir/.error 2>/dev/null || true
for decode_set in train_dev eval2000 $maybe_rt03; do
(
# note: we just give it "$decode_set" as it only uses the wav.scp, the
# feature type does not matter.

steps/online/nnet3/decode.sh --nj $decode_nj --cmd "$decode_cmd" \
--acwt 1.0 --post-decode-acwt 10.0 \
$graph_dir data/${decode_set}_hires \
${dir}_online/decode_${decode_set}${decode_iter:+_$decode_iter}_sw1_tg || exit 1;
if $has_fisher; then
steps/lmrescore_const_arpa.sh --cmd "$decode_cmd" \
data/lang_sw1_{tg,fsh_fg} data/${decode_set}_hires \
${dir}_online/decode_${decode_set}${decode_iter:+_$decode_iter}_sw1_{tg,fsh_fg} || exit 1;
fi
) || touch $dir/.error &
done
wait
if [ -f $dir/.error ]; then
echo "$0: something went wrong in decoding"
exit 1
fi
fi


exit 0;
1 change: 1 addition & 0 deletions egs/wsj/s5/steps/libs/nnet3/xconfig/parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@
'norm-pgru-layer' : xlayers.XconfigNormPgruLayer,
'norm-opgru-layer' : xlayers.XconfigNormOpgruLayer,
'renorm-component': xlayers.XconfigRenormComponent,
'batchnorm-component': xlayers.XconfigBatchnormComponent,
'no-op-component': xlayers.XconfigNoOpComponent,
'linear-component': xlayers.XconfigLinearComponent
}
Expand Down
Loading