Nnet3 dropout: code for Fast LSTM and scripts for AMI etc by GaofengCheng · Pull Request #1537 · kaldi-asr/kaldi

GaofengCheng · 2017-04-09T07:18:24Z

This PR will include：

code for fast lstm dropout (based on dpovey's PR1387)
scripts with dropout for AMI-IHM, AMI-SDM, SWBD and TEdlium (tdnn-(fast)lstms and b(fast)lstms)

Time schedule:

fast lstm dropout code
scripts for
AMI-IHM-> (done)
AMI-SDM-> (done)
SWBD-> (done)
TEdlium-> (done)
(follow the time axis)

…ame dropout masks on i and f gates. Old dropout method not supported in this branch.

…t just (i,f); test on tedlium.

danpovey · 2017-04-11T01:35:10Z

egs/ami/s5b/RESULTS_ihm

 %WER 20.8 | 13098 94489 | 82.0 10.0 8.0 2.8 20.8 53.2 | -0.096 | exp/ihm/chain_cleaned/tdnn_lstm1i_sp_bi_ld5/decode_dev/ascore_11/dev_hires.ctm.filt.sys
 %WER 20.7 | 12643 89980 | 81.7 11.5 6.8 2.5 20.7 51.8 | 0.015 | exp/ihm/chain_cleaned/tdnn_lstm1i_sp_bi_ld5/decode_eval/ascore_11/eval_hires.ctm.filt.sys

+# local/chain/tuning/run_tdnn_lstm_1l.sh --mic ihm  --train-set train_cleaned  --gmm tri3_cleaned


At this level, please just include the results for the "recommended" system which is 1m.
You should put all the comparative results in the individual scripts inside local/chain/tuning.
Use the standard compare_wer.sh script, whatever it's called, and also include the output
of chain_dir_info.pl from each of those scripts, in a comment in that script.

@danpovey OK, will do

danpovey · 2017-04-11T02:00:51Z

egs/ami/s5b/local/chain/tuning/run_tdnn_lstm_1l.sh

 #System               tdnn_lstm1i_sp_bi_ld5 tdnn_lstm1l_sp_bi_ld5
 #WER on dev        20.6      19.8
 #WER on eval        20.1      19.2
-#Final train prob      -0.045 -0.067


please give a bit more context in all of these scripts, at the top e.g. in this case where it says
# same as 1i but with per-frame dropout on LSTM layer
about what this is, e.g....
# This (1l.sh) is thesame as 1i but with per-frame dropout on LSTM layer
# It is a regular (not-fast) LSTM with per-frame dropout on [which gates?].
And explain how it relates to the paper, e.g. is it "place4" in the paper?
You can send me the final pdf to put on my publications page if it's not already there,
and include a link from the script to the paper.

@danpovey I have emailed the final pdf to you, later will go on updating this PR

vijayaditya

just some minor comments.

vijayaditya · 2017-04-11T05:50:28Z

egs/ami/s5b/RESULTS_ihm

 %WER 20.8 | 13098 94489 | 82.0 10.0 8.0 2.8 20.8 53.2 | -0.096 | exp/ihm/chain_cleaned/tdnn_lstm1i_sp_bi_ld5/decode_dev/ascore_11/dev_hires.ctm.filt.sys
 %WER 20.7 | 12643 89980 | 81.7 11.5 6.8 2.5 20.7 51.8 | 0.015 | exp/ihm/chain_cleaned/tdnn_lstm1i_sp_bi_ld5/decode_eval/ascore_11/eval_hires.ctm.filt.sys

+# local/chain/tuning/run_tdnn_lstm_1m.sh --mic ihm  --train-set train_cleaned  --gmm tri3_cleaned


Does 1i already account for all the new changes like fast-lstm component. If not please specify what all things changed between 1i and 1m.

vijayaditya · 2017-04-11T05:51:06Z

egs/ami/s5b/local/chain/tuning/run_tdnn_lstm_1l.sh

+#Final train prob (xent)     -0.722765 -0.915559
+#Final valid prob (xent)      -1.03985  -1.09907
+
+# steps/info/chain_dir_info.pl exp/ihm/chain_cleaned/tdnn_lstm1i_sp_bi_ld5/ exp/ihm/chain_cleaned/tdnn_lstm1l_sp_bi_ld5/


Please specify the set of flags to be used to recreate these results.

vijayaditya · 2017-04-11T05:51:52Z

egs/ami/s5b/local/chain/tuning/run_tdnn_lstm_1m.sh

+#Final train prob (xent)     -0.683776 -0.884698
+#Final valid prob (xent)      -1.05254  -1.09002
+
+# steps/info/chain_dir_info.pl exp/ihm/chain_cleaned/tdnn_lstm1j_sp_bi_ld5/ exp/ihm/chain_cleaned/tdnn_lstm1m_sp_bi_ld5/


Please specify the flags to be specified to recreate these results.

vijayaditya · 2017-04-11T05:52:49Z

egs/wsj/s5/steps/libs/nnet3/xconfig/lstm.py

        configs.append("# Input = (i_part, f_part, c_part, o_part, c_{t-1}), output = (c_t, m_t)")
        configs.append("# See cu-math.h:ComputeLstmNonlinearity() for details.")
-        configs.append("component name={0}.lstm_nonlin type=LstmNonlinearityComponent cell-dim={1} {2}".format(name, cell_dim, lstm_str))
+        configs.append("component name={0}.lstm_nonlin type=LstmNonlinearityComponent cell-dim={1} "


using named arguments is preferable when you have so many fields.

@vijayaditya Sorry Vijay, I'm not quite clear how to use named arguments here, can you give some detailed examples?

vijayaditya · 2017-04-11T05:54:09Z

src/cudamatrix/cu-kernels.cu

-      const Real i_t = 1 / (1 + exp(-i_part - w_ic * c_prev));
-      const Real f_t = 1 / (1 + exp(-f_part - w_fc * c_prev));
+
+      const Real i_scale = (have_dropout_mask ?


NIT: has_dropout_mask might be easier to follow than have_dropout_mask.

danpovey · 2017-04-12T02:09:31Z

What he means is something like ``` "component name={name}.lstm_nonlin... ".format(name=name, ...) ```

…

On Tue, Apr 11, 2017 at 6:47 PM, Gaofeng Cheng ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In egs/wsj/s5/steps/libs/nnet3/xconfig/lstm.py <#1537 (comment)>: > @@ -833,14 +831,16 @@ def generate_lstm_config(self): configs.append("# The core LSTM nonlinearity, implemented as a single component.") configs.append("# Input = (i_part, f_part, c_part, o_part, c_{t-1}), output = (c_t, m_t)") configs.append("# See cu-math.h:ComputeLstmNonlinearity() for details.") - configs.append("component name={0}.lstm_nonlin type=LstmNonlinearityComponent cell-dim={1} {2}".format(name, cell_dim, lstm_str)) + configs.append("component name={0}.lstm_nonlin type=LstmNonlinearityComponent cell-dim={1} " @vijayaditya <https://github.com/vijayaditya> Sorry Vijay, I'm not quite clear how to use named arguments here, can you give some detailed examples? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1537 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVuyZnCR9mizJttpAp3Gi04X3JU7Lrks5rvC0-gaJpZM4M38oj> .

GaofengCheng · 2017-04-12T14:03:10Z

@danpovey PR for SWBD and Tedlium will take about 2~3 days

…3_dropout

danpovey · 2017-04-17T17:06:35Z

@GaofengCheng, how much longer till this is ready to merge?

5epoch is on the way

also SWBD RESULTS updated

@osadjadi

* 'master' of https://github.com/kaldi-asr/kaldi: (21 commits) [egs] bug-fix in egs/ami/s5/run_ihm.sh (kaldi-asr#1577) [src] Minor bug-fixes in compute-wer-bootci and WSJ run.sh. Thanks: @osadjadi [egs] Add soft link for mini-librispeech setup [egs] adding results and cleanup in mini-librispeech [egs] Add mini-librispeech example scripts [intended as a sanity-checker/tutorial setup] (kaldi-asr#1566) [src] Fix to testing code signal-test.cc, change threshold to resolve failure (kaldi-asr#1565) [src] Add documentation for dropout function. [src,scripts,egs] Add dropout for nnet3 LSTMs, with recipes. (kaldi-asr#1537) [src] nnet3 online silence weighting - adding frame subsampling factor (kaldi-asr#1559) [doc] Small edit to hmm.dox, clarifying something [egs] Added check for kaldi_lm being installed in fisher_swbd recipe. (kaldi-asr#1558) Update travis.yml so PRs to kaldi_52 are built [srcipts] steps/nnet3/report/generate_plots.py: plot 5,50,95th percentile of value and derivative instead of mean+-stddev (kaldi-asr#1472) [egs] AMI TDNN Results Update (kaldi-asr#1545) [src] add template instantiations for ConvertStringToReal, address issue kaldi-asr#1544 [egs,scripts,src] SID and LID tools and scripts: cosmetic improvements, better error-handling, and various minor fixes; results unchanged. (kaldi-asr#1543) [src] Change ConvertStringToReal to be locale-independent (i.e. always-US). Fixes android issue. (kaldi-asr#1513) [scripts] nnet3 : fix issue where LDA estimation failed for LSTMs with label delay (kaldi-asr#1540) [scripts] fix to get_egs_targets.sh (thanks: David Pye) [src] Fix copy-feats for using the --write-num-frames and --compress true flags at the same time (kaldi-asr#1541) ...

…sr#1537) See also http://www.danielpovey.com/files/2017_interspeech_dropout.pdf this improves on the best recipes.

danpovey and others added 18 commits January 30, 2017 20:41

[src,scripts,egs] nnet3,fast-lstm: changes to support separate per-fr…

13e8bed

…ame dropout masks on i and f gates. Old dropout method not supported in this branch.

[egs] Small fixes/additions in Swbd/s5c chain scripts

863534b

Merge branch 'shortcut' into shortcut-dropout

8384eae

[src,egs,scripts] Modifying dropout in LSTM to be on (i,f,o) gates no…

eb0f458

…t just (i,f); test on tedlium.

Merge remote-tracking branch 'upstream/shortcut' into shortcut-dropout

96d92d7

Merge remote-tracking branch 'upstream/shortcut' into shortcut-dropout

19af8ca

Merge branch 'shortcut' into shortcut-dropout

4d2f00e

[scripts] Update example scripts for dropout on Tedlium s5_r2

6582acf

Merge branch 'shortcut' into shortcut-dropout

a406d0f

for ref

eb94ffd

merge fast lstm dropout

b9c3e20

delete temporary tuning sdripts in tedlium

9afaf39

delete irrelevant file

e9ac4e2

delete exclusive option in fast lstm code

638f083

solve some cuda-kernel line mismatch problem

49c4558

small bug fix

05fc6d2

small fix

90df5d7

update scripts for tdnn-(fast)lstm of AMI-IHM

1a58236

danpovey reviewed Apr 11, 2017

View reviewed changes

change scripts comment style and RESULTS

69a36e4

danpovey reviewed Apr 11, 2017

View reviewed changes

vijayaditya reviewed Apr 11, 2017

View reviewed changes

adding SDM results

d03be0f

GaofengCheng added 3 commits April 13, 2017 10:47

Merge branch 'master' of https://github.com/kaldi-asr/kaldi into nnet…

07d6774

…3_dropout

adding SWBD (parts of all) scripts with dropout

936863e

small fix

f51fb75

update tdnn-blstm with dropout in SWBD

139f412

GaofengCheng added 3 commits April 18, 2017 12:07

update tdnn+regular-LSTM(4epoch) in SWBD

9a8b81c

5epoch is on the way

adding tedlium scripts

48f41a7

also SWBD RESULTS updated

small fix

62fee2b

danpovey merged commit d8be99a into kaldi-asr:master Apr 20, 2017

Skaiste pushed a commit to Skaiste/idlak that referenced this pull request Sep 26, 2018

[src,scripts,egs] Add dropout for nnet3 LSTMs, with recipes. (kaldi-a…

0ca6fe5

…sr#1537) See also http://www.danielpovey.com/files/2017_interspeech_dropout.pdf this improves on the best recipes.

Conversation

GaofengCheng commented Apr 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danpovey Apr 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vijayaditya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danpovey commented Apr 12, 2017 via email

Uh oh!

GaofengCheng commented Apr 12, 2017

Uh oh!

danpovey commented Apr 17, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GaofengCheng commented Apr 9, 2017 •

edited

Loading

danpovey Apr 11, 2017 •

edited

Loading