WIP: Clean version of semi-supervised PR #14

vimalmanohar · 2017-09-15T22:52:58Z

No description provided.

…sfer-learning-wsj-rm Conflicts: egs/wsj/s5/steps/nnet3/xconfig_to_configs.py

…vised Travis was failing to compile(not sure why)-- I used the "Update Branch" button

vimalmanohar · 2017-12-07T20:46:54Z

@hhadian Could you please review this PR?

hhadian · 2017-12-07T22:45:54Z

Sure, I will do it.

pegahgh · 2017-12-07T22:56:06Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_100k_a.sh

+# Copyright 2017  Vimal Manohar
+# Apache 2.0
+
+# This is fisher chain recipe for training a model on a subset of around 100 hours.


Does this script use 100hrs supervised training data? add better description e.g. this script uses 100hrs supervised data

pegahgh · 2017-12-07T22:57:25Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_100k_a.sh

+exp=exp/semisup_100k
+gmm=tri4a
+xent_regularize=0.1
+hidden_dim=725


Isn't it large? we use 625 for 300hrs swbd data.

pegahgh · 2017-12-07T22:57:41Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_100k_a.sh

+num_epochs=4
+remove_egs=false
+common_egs_dir=
+minibatch_size=128


fix it in the script.

pegahgh · 2017-12-07T23:04:14Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_100k_a_oracle.sh

+set -e
+
+# This is an oracle experiment using oracle transcription of 250 hours of 
+# unsupervised data, along with 100 hours of supervised data.


I think you can easily use run_tdnn_100k_a.sh with new combined dataset, I am not sure why do you need two separate scripts?

I agree. In the inital PR, there was only one TDNN recipe which was
called with different training data sets during semi-supervised training.
Separating the scripts might be clearer but it will add too many very similar scripts.

pegahgh · 2017-12-07T23:06:53Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_15k_i.sh

+exp=exp/semisup_15k
+gmm=tri3
+xent_regularize=0.1
+hidden_dim=500


Did you try to reduce it to smaller size or reducing number of layers?

I guess I tried smaller sizes but it did not help much.

I think the Fisher dev and test are more similar to training compared to eval2000 and swbd. So it might be ok for a larger network.

pegahgh · 2017-12-07T23:23:37Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_15k_semisupervised_conf_am.sh

+
+# Semi-supervised options
+comb_affix=comb1am  # affix for new chain-model directory trained on the combined supervised+unsupervised subsets
+supervision_weights=1.0,1.0


add description

pegahgh · 2017-12-07T23:24:28Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_15k_semisupervised_conf_am.sh

+# Neural network opts
+apply_deriv_weights=true
+xent_regularize=0.1
+hidden_dim=725


Did you try to tune it? my guess is that it is large hidden dim.

pegahgh · 2017-12-07T23:27:14Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_15k_semisupervised_conf_am.sh

+apply_deriv_weights=true
+xent_regularize=0.1
+hidden_dim=725
+minibatch_size="150=128/300=64"


Is it better than using minibatch_size=150,300?

pegahgh · 2017-12-07T23:30:48Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_15k_semisupervised_conf_am.sh

+  relu-batchnorm-layer name=prefinal-xent input=tdnn6 dim=$hidden_dim target-rms=0.5
+  output-layer name=output-xent dim=$num_targets learning-rate-factor=$learning_rate_factor max-change=1.5
+
+  output name=output-0 input=output.affine skip-in-init=true


Why do you have two separate output nodes? Do you use weighted training? supervision weights were the same in the script.

what is skip-in-init option? You can add a single line comment about output in config file.

skip-in-init is added to prevent output line (trivial output layer) from being printed in init.config.
Do you know why the trivial output layers are needed in init.config?

I think we just use init.config for training lda matrix and we don't need to add other outputs. Probably, we can modify the xconfig to not print trivial outputs in init.config (we just need to print output-node name=output).

pegahgh · 2017-12-07T23:37:08Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_15k_semisupervised_conf_am.sh

+               --lattice-prune-beam "$lattice_prune_beam" \
+               --phone-insertion-penalty "$phone_insertion_penalty" \
+               --deriv-weights-scp $chaindir/best_path_${unsupervised_set}${decode_affix}/weights.scp \
+               --online-ivector-dir $exp/nnet3${nnet3_affix}/ivectors_${semisup_train_set}_sp_hires \


Do you use supervised data for ivector training?

hhadian

I was under the impression that this PR could be wrapped up in 20-30 changed files (new or modified)
Not sure but I feel 70 changed files is too many.

hhadian · 2017-12-08T20:48:24Z

egs/fisher_english/s5/local/fisher_train_lms.sh


 train_lm.sh --arpa --lmtype 3gram-mincount $dir || exit 1;

+train_lm.sh --arpa --lmtype 4gram-mincount $dir || exit 1;


If you are using pocolm, it might be better to leave this script unchanged (to make the PR smaller)

Ok makes sense.

hhadian · 2017-12-08T20:54:53Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_100k_a_oracle.sh

+set -e
+
+# This is an oracle experiment using oracle transcription of 250 hours of 
+# unsupervised data, along with 100 hours of supervised data.


I agree. In the inital PR, there was only one TDNN recipe which was
called with different training data sets during semi-supervised training.
Separating the scripts might be clearer but it will add too many very similar scripts.

hhadian · 2017-12-08T20:56:20Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_15k_i.sh

+exp=exp/semisup_15k
+gmm=tri3
+xent_regularize=0.1
+hidden_dim=500


I guess I tried smaller sizes but it did not help much.

hhadian · 2017-12-08T20:57:27Z

egs/fisher_english/s5/local/semisup/chain/tuning/run_tdnn_15k_i.sh

@@ -0,0 +1,201 @@
+#!/bin/bash


I guess it would be nicer to keep only one version of everything (even though it's in tuning) so no _i, _a, etc. That's because there are too many files in this PR.

hhadian · 2017-12-08T21:18:09Z

src/online2bin/extend-wav-with-silence.cc


-    std::string wav_rspecifier = po.GetArg(1);
-    std::string wav_wspecifier = po.GetArg(2);
+    if (ClassifyRspecifier(po.GetArg(1), NULL, NULL) != kNoRspecifier) {


This change was for perturb_to_allowed_lengths.py. I guess you are not using
that script (i.e. non-split training) so it might be better to leave this file unchanged.

hhadian · 2017-12-08T21:18:55Z

src/nnet3/nnet-chain-example.cc

+    if (token == "<DW>")
+      ReadVectorAsChar(is, binary, &deriv_weights);
+    else
+      deriv_weights.Read(is, binary);


how is <DW2> diffrent from <DW>?

DW reads only 0 and 1. DW2 reads and writes as float, which is needed for this.

hhadian · 2017-12-08T21:26:09Z

BTW, I noticed your master is not up-to-date with upstream kaldi.
Some of the changes in this PR might be already in kaldi master.

vimalmanohar and others added 30 commits March 24, 2016 21:37

chain-smbr: Adding chain-smbr denominator

4d2cde9

long_utts: Minor fix

58fcd61

added Transfer learning setup using nnet3+chain+tdnn for WSJ->RM.

d97edb6

Merge branch 'master' of https://github.com/kaldi-asr/kaldi into tran…

1c25c88

…sfer-learning-wsj-rm Conflicts: egs/wsj/s5/steps/nnet3/xconfig_to_configs.py

Merge branch 'master' of github.com:kaldi-asr/kaldi

df90bba

Merge branch 'master' of github.com:kaldi-asr/kaldi

ff4ac04

Merge branch 'master' of github.com:kaldi-asr/kaldi

29ced2a

Merge branch 'master' of github.com:kaldi-asr/kaldi

0f69bbd

fixed some issues w.r.t comments.

b5fe795

Merge branch 'master' of github.com:kaldi-asr/kaldi

143877b

merged with kaldi_52 and fixed some incompatibility issues.

947f05c

Merge branch 'master' of github.com:kaldi-asr/kaldi

acf4c14

Merge branch 'master' of github.com:kaldi-asr/kaldi

f887582

Merge branch 'master' of github.com:kaldi-asr/kaldi

a5d4ca8

Merge branch 'master' of github.com:kaldi-asr/kaldi

ea930eb

Merge branch 'master' of github.com:kaldi-asr/kaldi

8f05cdf

[WIP] Add chain semi-supervised script + src changes

55cc6f9

Minor fixes

b91276b

Add more options to run_semisupervised.sh

ba26120

Merge branch 'master' of github.com:kaldi-asr/kaldi

1eb41ee

Merge branch 'master' of github.com:kaldi-asr/kaldi

79e15ad

Add a check in supervision code

477bdf3

Merge branch 'master' of github.com:kaldi-asr/kaldi

61e94b2

Merge branch 'master' of github.com:kaldi-asr/kaldi

d45ecb9

Some fixes + new options

f55f686

Merge branch 'master' into semi_supervised

c6ffb15

Merge branch 'master' of github.com:kaldi-asr/kaldi

7b01bb0

Add nnet3, chain, and semi_sepervised scripts for fisher english

403e3e2

Merge remote-tracking branch 'origin/semi_supervised' into semi_super…

0c8974e

…vised Travis was failing to compile(not sure why)-- I used the "Update Branch" button

Merge branch 'master' of github.com:kaldi-asr/kaldi

e1de4e4

pegahgh reviewed Dec 7, 2017

View reviewed changes

hhadian reviewed Dec 8, 2017

View reviewed changes

vimalmanohar added 6 commits December 8, 2017 18:15

Merge branch 'master' of github.com:kaldi-asr/kaldi

75fbde4

semisup: Making changes based on comments

e62dac0

Merging from kaldi master

ac5da45

semisup: Minor fixes

5d1f4c9

semisup: Minor fixes

f3fd4a9

semisup: Re-organizing some scripts

0a69689

vimalmanohar force-pushed the master branch from b4b42cc to a5561c3 Compare December 27, 2017 17:39

vimalmanohar added 16 commits December 28, 2017 02:02

Reverting discriminative changes for now

cce099f

Revert some changes not required now

ed5efd6

Merging from golden

5cafdc5

Minor fix

7a1ff5c

Removingfew files from the PR

e2c6603

Clean the recipe

2c06bf5

Added some checks

9933ea7

Remove truncate-deriv-weights

db0bc54

Remove some unused binaries in chainbin get-egs

c71cf88

Remove mkgraph.sh changes

6d8350e

Merge branch 'master' of github.com:kaldi-asr/kaldi into semisup-clean

ef4750a

Remove some tuning scripts

0ee0075

Add recipe for build tree multiple sources

4908983

Remove some lattice function changes

85780b1

Rename some scripts

842dce9

semisup: Reduce the number of scripts

926dc3a

vimalmanohar mentioned this pull request Jan 10, 2018

Semi-supervised training on Fisher English kaldi-asr/kaldi#2140

Merged

vimalmanohar closed this Jan 10, 2018


		train_lm.sh --arpa --lmtype 3gram-mincount $dir \|\| exit 1;

		train_lm.sh --arpa --lmtype 4gram-mincount $dir \|\| exit 1;

WIP: Clean version of semi-supervised PR #14

WIP: Clean version of semi-supervised PR #14

Uh oh!

Conversation

vimalmanohar commented Sep 15, 2017

Uh oh!

vimalmanohar commented Dec 7, 2017

Uh oh!

hhadian commented Dec 7, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hhadian left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hhadian commented Dec 8, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants