Update-LID using DNN #999

s-mousmita · 2016-08-23T15:35:32Z

@david-ryan-snyder @danpovey Adding update.
Changes are made with reference to the old pull request.

david-ryan-snyder · 2016-08-23T15:48:28Z

@s-mousmita, thanks for the update.

Could you do the following:

Make egs/lre07/v2/lid a symbolic link to egs/lre07/v1/lid instead of a copy
Add your new DNN-based LID scripts to egs/lre07/v1/lid

david-ryan-snyder · 2016-08-23T15:50:45Z

The point is that we don't want to be maintaining two copies of lid/, one in egs/lre07/v1 and one in egs/lre07/v2 .

See what we did in egs/sre10/v1/sid and egs/sre10/v2/sid. They both point to egs/sre08/v1/sid .

s-mousmita · 2016-08-23T18:02:49Z

Yes, sorry but script name?

david-ryan-snyder · 2016-08-23T18:13:37Z

egs/lre07/README.txt

+ Language Evaluation.  The subdirectory v1 demonstrates the standard LID
+ system, which is an I-Vector based recipe using full covarience GMM-UBM and logistic regression model.  
+ The subdirectory v2 demonstrates the LID sysem using a Time Delay Deep Neural Network based UBM 
+ which is used to replace the GMM-UBM of v1. The DNN is trained using about 1800 hours of the english portion of Fisher.


Please be mindful of typos here. E.g., covarience -> covariance, and sysem - > system.

Also, please format the text so that lines don't go over 80 columns.

david-ryan-snyder · 2016-08-23T19:41:05Z

@s-mousmita, please let me know when you've addressed the comments

s-mousmita · 2016-08-23T20:10:11Z

Thanks @david-ryan-snyder .Please check updated scripts.

david-ryan-snyder · 2016-08-24T13:44:00Z

egs/lre07/v1/lid/init_full_ubm_from_dnn.sh

+
+nnet_feats="ark,s,cs:apply-cmvn-sliding --center=true scp:$sdata_dnn/JOB/feats.scp ark:- |"
+
+## Set up SDC features.


Could you remove the extra '#'?

danpovey · 2016-09-03T00:04:09Z

I just had a look at it...
I think it would be better to move the scripts that are currently in local/dnn/ -- at least the scripts that were copied from steps/nnet2-- to be moved to lid/nnet2/. That way, those scripts won't have to be duplicated if someone wants to create another setup using the same methods.

Also be careful that sometimes the program pathname appears in usage messages (it's wrong in some cases currently)-- grepping for steps/nnet2 and local/dnn in those scripts should find all the places you'd need to change the path.

s-mousmita · 2016-09-03T15:32:34Z

I have removed local/dnn and created lid/nnet2. Paths local/dnn and steps/nnet2 at various places of the scripts have been changed to lid/nnet2.

david-ryan-snyder · 2016-09-03T16:05:00Z

I don't think we should move everything from local/dnn to lid/nnet2. My suggestion:

Recipe specific things go into local. This includes scripts that refer to datasets, or specific instantiations of nnet configurations. That means that train_dnn.sh, run_nnet2_common.sh, run_nnet2_multisplice.sh, Fisher data preparation scripts, etc, should remain in local/dnn
General purpose scripts that can be used by any LID example should go into lid. Scripts such as get_egs2.sh, get_lda.sh, make_multisplice_configs.py, relabel_egs2.sh, etc would go into lid/nnet2
See the contents of egs/fisher_english/s5/local for examples of what should be in local, if in doubt
Also, since we're refactoring things, please double check that all the scripts you're moving around are actually needed. For example, I think we don't need a lid/nnet2/remove_egs.sh. There's probably an identical one in steps/nnet2 that can be called instead.

s-mousmita · 2016-09-03T17:47:08Z

@david-ryan-snyder The following are correct?

get_egs2.sh, relabel_egs2.sh and get_lda.sh will go to /lid/nnet2/
train_dnn.sh, fisher_data_prep.sh, fisher_prepare_dict.sh, fisher_train_lms.sh, fisher_create_test_lang.sh, remove_dup_utts.sh, run_nnet2_multisplice.sh, run_nnet2_common.sh and train_multisplice_accel2.sh will go to local/dnn/
get_num_frames.sh, make_multisplice_configs.py, align.sh and remove_egs.sh will be used from steps/nnet2.

danpovey · 2016-09-03T18:28:52Z

train_multisplice_accel2.sh is not dataset-specific so it should go into
lid/nnet2. Otherwise that looks plausible.

Dan

On Sat, Sep 3, 2016 at 1:47 PM, Mousmita Sarma notifications@github.com
wrote:

@david-ryan-snyder https://github.com/david-ryan-snyder The following
are correct?

get_egs2.sh, relabel_egs2.sh and get_lda.sh will go to /lid/nnet2/

train_dnn.sh, fisher_data_prep.sh, fisher_prepare_dict.sh,
fisher_train_lms.sh, fisher_create_test_lang.sh, remove_dup_utts.sh,
run_nnet2_multisplice.sh, run_nnet2_common.sh and
train_multisplice_accel2.sh will go to local/dnn/

get_num_frames.sh, make_multisplice_configs.py, align.sh and
remove_egs.sh will be used from steps/nnet2.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#999 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu66_Rg_P4Tau5tSHXIZI8HVpi4X_ks5qmbKegaJpZM4JrEUQ
.

s-mousmita · 2016-09-03T19:41:01Z

I have made the changes suggested above.

danpovey · 2016-09-03T19:42:19Z

Would you mind trying to run it one more time before we commit it?
With all the name changes, there is a strong possibility of errors.

On Sat, Sep 3, 2016 at 3:41 PM, Mousmita Sarma notifications@github.com
wrote:

I have made the changes suggested above.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#999 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu1Qg_Jq5SHalGd05kD10su6i-x9uks5qmc1QgaJpZM4JrEUQ
.

david-ryan-snyder · 2016-09-04T00:14:33Z

I agree with the script locations in the current commit.

ngoel17 · 2016-09-04T04:24:51Z

Dan,
Would it make sense to run it on JHU machines so that we don't change
even the dataset paths?

Nagendra

On Sat, Sep 3, 2016, 3:42 PM Daniel Povey notifications@github.com wrote:

Would you mind trying to run it one more time before we commit it?
With all the name changes, there is a strong possibility of errors.

On Sat, Sep 3, 2016 at 3:41 PM, Mousmita Sarma notifications@github.com
wrote:

I have made the changes suggested above.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#999 (comment),
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/ADJVu1Qg_Jq5SHalGd05kD10su6i-x9uks5qmc1QgaJpZM4JrEUQ

.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#999 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIZyeB_vZIPgipJu7Tsj1ZYNF43NCbcnks5qmc2ggaJpZM4JrEUQ
.

danpovey · 2016-09-04T04:27:55Z

Yes, that sounds good.

On Sun, Sep 4, 2016 at 12:24 AM, Nagendra Goel notifications@github.com
wrote:

Dan,
Would it make sense to run it on JHU machines so that we don't change
even the dataset paths?

Nagendra

On Sat, Sep 3, 2016, 3:42 PM Daniel Povey notifications@github.com
wrote:

Would you mind trying to run it one more time before we commit it?
With all the name changes, there is a strong possibility of errors.

On Sat, Sep 3, 2016 at 3:41 PM, Mousmita Sarma <notifications@github.com

wrote:

I have made the changes suggested above.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#999 (comment),
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/ADJVu1Qg_
Jq5SHalGd05kD10su6i-x9uks5qmc1QgaJpZM4JrEUQ

.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#999 (comment),
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AIZyeB_
vZIPgipJu7Tsj1ZYNF43NCbcnks5qmc2ggaJpZM4JrEUQ>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#999 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu4WFbD8HHG2d5WteVcc_01amH29Cks5qmkgWgaJpZM4JrEUQ
.

danpovey · 2016-09-20T23:41:38Z

I assume someone is still working on this?

ngoel17 · 2016-09-21T02:47:00Z

Training is running from scratch on JHU machines.

On Tue, Sep 20, 2016, 7:41 PM Daniel Povey notifications@github.com wrote:

I assume someone is still working on this?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#999 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIZyeMZ4AW1pW_COTsTbZ4qYxVqSf3lXks5qsG83gaJpZM4JrEUQ
.

danpovey · 2016-10-17T19:44:24Z

@ngoel17, any progress on this?

s-mousmita · 2016-10-18T05:38:54Z

@danpovey @ngoel17 Its doing the last steps, I-vector extraction.

s-mousmita · 2016-10-27T13:56:36Z

I have updated the scripts.

david-ryan-snyder

I think most of it looks good. I think the main thing I'd like to see updated is that there's a huge file that is added to the commit that could instead be a symbolic link.

david-ryan-snyder · 2016-10-28T21:10:14Z

egs/lre07/v1/lid/nnet2/train_multisplice_accel2.sh

+        --transform-dir "$transform_dir" --online-ivector-dir "$online_ivector_dir" \
+        --iter $x $data $lang $dir $dir/ali_$time || exit 1
+
+      lid/nnet2/relabel_egs2.sh --cmd "$cmd" --iter $x $dir/ali_$time \


Are you sure we can't use steps/nnet2/relabel_egs2.sh here?

david-ryan-snyder · 2016-10-28T21:14:52Z

egs/lre07/v1/lid/nnet2/relabel_egs2.sh

+    $egs_in $egs_out || exit 1
+fi
+
+echo "$0: Finished relabeling training examples"


Are you sure we need this script? Can we not use the one in steps/nnet2/relabel_egs2.sh?

david-ryan-snyder · 2016-10-28T21:25:20Z

egs/lre07/v2/cmd.sh

+export decode_cmd=run.pl
+export cuda_cmd=run.pl
+export mkgraph_cmd=run.pl
+


The way the queue options are handled was changed in this commit: b3bbc03

Basically you need to update the comment and the export statements to be more similar to what you see in the link above. The default commands should include queue.pl in it.

david-ryan-snyder · 2016-10-28T21:33:36Z

egs/lre07/v2/local/lre07_eval/score_lre07.v01d.pl

+	    return -1;
+	}
+    }
+}


For files like this, that are unlikely to ever be modified, it's better to make a symbolic link to the same file in egs/lre07/v1/local.

You probably don't need to do that for any of the other scripts.

david-ryan-snyder · 2016-10-28T21:47:43Z

egs/lre07/README.txt

+ Language Evaluation.  The subdirectory v1 demonstrates the standard
+ LID system, which is an I-Vector based recipe using full covariance
+ GMM-UBM and logistic regression model. The subdirectory v2 demonstrates
+ the LID system using a Time Delay Deep Neural Network based UBM


I would consider replacing "Time Delay Deep Neural Network" with "time-delay deep neural network."

s-mousmita · 2016-10-29T16:24:18Z

@david-ryan-snyder Scripts are updated according to your recent comments.

david-ryan-snyder

I think it's ready. @danpovey, do you have any lingering doubts before accepting it?

Actually N/M since the check failed. Let's wait until we figure that out.

david-ryan-snyder · 2016-10-30T15:41:18Z

@danpovey it looks like the error is in exp-test. Have you seen this in recent PRs? I'm thinking this is probably not related to @s-mousmita's updates.

Successfully configured OpenBLAS from /home/travis/xroot/usr. /usr/bin/ld: cannot find -lgfortran collect2: ld returned 1 exit status make: *** [exp-test] Error 1 ./configure: line 222: ./exp-test: No such file or directory

danpovey · 2016-10-30T18:43:50Z

That exp-test thing was not what was causing the build to fail, it's normal
on the travis build.
The reason was:
Your test run exceeded 120 minutes.
Probably just random- there were no source changes in the PR.
I'll probably merge today.

On Sun, Oct 30, 2016 at 11:41 AM, david-ryan-snyder <
notifications@github.com> wrote:

@danpovey https://github.com/danpovey it looks like the error is in
exp-test. Have you seen this in recent PRs?

Successfully configured OpenBLAS from /home/travis/xroot/usr.
/usr/bin/ld: cannot find -lgfortran
collect2: ld returned 1 exit status
make: *** [exp-test] Error 1
./configure: line 222: ./exp-test: No such file or directory

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#999 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu1s9bmy0cJqaVEfRTKoZsXBuR2stks5q5LqhgaJpZM4JrEUQ
.

danpovey · 2016-11-01T03:44:40Z

Thanks! Merging.

s-mousmita · 2016-11-01T06:10:38Z

@danpovey @david-ryan-snyder Thank you for approving the recipe.

david-ryan-snyder reviewed Aug 23, 2016
View reviewed changes

david-ryan-snyder reviewed Aug 24, 2016
View reviewed changes

s-mousmita force-pushed the LID_DNN_update branch 4 times, most recently from dae5139 to d73b43d Compare September 3, 2016 19:37

s-mousmita force-pushed the LID_DNN_update branch from b482d15 to 76fb9da Compare October 27, 2016 12:52

david-ryan-snyder suggested changes Oct 28, 2016

View reviewed changes

Update of lre07/v2 scripts

764803b

s-mousmita force-pushed the LID_DNN_update branch from fc35eba to 764803b Compare October 29, 2016 16:30

david-ryan-snyder approved these changes Oct 30, 2016

View reviewed changes

danpovey merged commit 08869e3 into kaldi-asr:master Nov 1, 2016


		nnet_feats="ark,s,cs:apply-cmvn-sliding --center=true scp:$sdata_dnn/JOB/feats.scp ark:- \|"

		## Set up SDC features.

Update-LID using DNN #999

Update-LID using DNN #999

Uh oh!

Conversation

s-mousmita commented Aug 23, 2016

Uh oh!

david-ryan-snyder commented Aug 23, 2016

Uh oh!

david-ryan-snyder commented Aug 23, 2016

Uh oh!

s-mousmita commented Aug 23, 2016

Uh oh!

david-ryan-snyder Aug 23, 2016

Choose a reason for hiding this comment

Uh oh!

david-ryan-snyder commented Aug 23, 2016

Uh oh!

s-mousmita commented Aug 23, 2016

Uh oh!

david-ryan-snyder Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

danpovey commented Sep 3, 2016

Uh oh!

s-mousmita commented Sep 3, 2016

Uh oh!

david-ryan-snyder commented Sep 3, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s-mousmita commented Sep 3, 2016

Uh oh!

danpovey commented Sep 3, 2016

Uh oh!

s-mousmita commented Sep 3, 2016

Uh oh!

danpovey commented Sep 3, 2016

Uh oh!

david-ryan-snyder commented Sep 4, 2016

Uh oh!

ngoel17 commented Sep 4, 2016

Uh oh!

danpovey commented Sep 4, 2016

Uh oh!

danpovey commented Sep 20, 2016

Uh oh!

ngoel17 commented Sep 21, 2016

Uh oh!

danpovey commented Oct 17, 2016

Uh oh!

s-mousmita commented Oct 18, 2016

Uh oh!

s-mousmita commented Oct 27, 2016

Uh oh!

david-ryan-snyder left a comment

Choose a reason for hiding this comment

Uh oh!

david-ryan-snyder Oct 28, 2016

Choose a reason for hiding this comment

Uh oh!

david-ryan-snyder Oct 28, 2016

Choose a reason for hiding this comment

Uh oh!

david-ryan-snyder Oct 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david-ryan-snyder Oct 28, 2016

Choose a reason for hiding this comment

Uh oh!

david-ryan-snyder Oct 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david-ryan-snyder Oct 28, 2016

Choose a reason for hiding this comment

Uh oh!

s-mousmita commented Oct 29, 2016

Uh oh!

david-ryan-snyder left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

david-ryan-snyder commented Sep 3, 2016 •

edited

Loading

david-ryan-snyder Oct 28, 2016 •

edited

Loading

david-ryan-snyder Oct 28, 2016 •

edited

Loading

david-ryan-snyder left a comment •

edited

Loading

david-ryan-snyder commented Oct 30, 2016 •

edited

Loading