Xconfigs : extension #1197

vijayaditya · 2016-11-17T18:46:48Z

This is an extension of PR #1170 . I am currently testing the full pipeline. This also includes the commits from Xconfigs.

vijayaditya · 2016-11-17T22:51:01Z

@GaofengCheng Do you have time to write an xconfig class for the HLSTM. This would help us find any modifications necessary to support architectures which access internal nodes of "layers".

GaofengCheng · 2016-11-18T02:47:37Z

@vijayaditya I could do if a slight delay is tolerable

vijayaditya · 2016-11-18T16:48:29Z

@GaofengCheng no hurry do it at your convenience.

vijayaditya · 2016-11-18T21:33:18Z

I think this is ready for a round of review.
TODO: Make modifications to conform to the Google style doc. This mainly consists of adding doc_strings and conforming to 80 character/line limit.

danpovey

Looks good, some small comments.

danpovey · 2016-11-18T21:37:25Z

egs/swbd/s5c/local/chain/run_xlstm.sh

@@ -0,0 +1,233 @@
+#!/bin/bash
+
+# 6i is based on run_lstm_6h.sh, but changing the HMM context from triphone to left biphone.


is this comment out of date? Perhaps give it a different letter and run the comparison script so you can update the comment?

danpovey · 2016-11-18T21:38:14Z

egs/swbd/s5c/local/chain/run_xlstm.sh

+train_stage=-1
+get_egs_stage=-10
+speed_perturb=true
+dir=exp/chain/lstm_6i_xconf  # Note: _sp will get added to this if $speed_perturb == true.


I'd suggest 6i_xconf -> 6j or something like that.. don't want to start using _xconf in these names, we can just quietly switch over.

danpovey · 2016-11-18T21:39:17Z

egs/wsj/s5/steps/libs/nnet3/xconfig/__init__.py

@@ -0,0 +1,30 @@
+# This library has classes and methods to form neural network computation graphs,


you might want to add copyright header.

danpovey · 2016-11-18T21:39:43Z

egs/wsj/s5/steps/libs/nnet3/xconfig/__init__.py

+
+
+# 'ref.config' : which is a version of the config file used to generate
+#                a model for getting left and right context it doesn't read anything for the


danpovey · 2016-11-18T21:42:47Z

egs/wsj/s5/steps/libs/nnet3/xconfig/basic_layers.py

+
+class XconfigLayerBase(object):
+    """ A base-class for classes representing layers of xconfig files.
+        This mainly just sets self.layer_type, self.name and self.config/


I'd remove "This mainly just sets self.layer_type, self.name and self.config", as the base-class has a bunch of functions... that comment was outdated I think.

danpovey · 2016-11-18T21:45:24Z

egs/wsj/s5/steps/libs/nnet3/xconfig/basic_layers.py

+    def set_default_configs(self):
+        raise Exception("Child classes must override set_default_configs().")
+
+    # this is expected to be called after set_configs and before check_configs()


is expected to be called -> is called in the constructor.
I think this default function definition is potentially dangerous, but I guess I could be OK with it.

danpovey · 2016-11-18T21:49:06Z

egs/wsj/s5/steps/libs/nnet3/xconfig/basic_layers.py

+
+    def input_dim(self):
+        dim = 0
+        for input_name in self.get_input_descriptor_names():


I don't think this function should exist, at least not with this implementation-- in general we don't want to make any assumptions that if there are multiple input descriptors, they will be appended together. Having just 1 input is the norm, and auxiliary inputs might not be appended with the primary one [if you wanted that, you could just use Append() in the descriptor.]
Elsewhere the code sets
input_dim = self.descriptors['input']['dim']
which I think makes more sense [if there is an 'input' descriptor]. If you must have this function, I'd prefer it to use that expression.

danpovey · 2016-11-18T21:55:25Z

egs/wsj/s5/steps/libs/nnet3/xconfig/lstm.py

+
+    def set_default_configs(self):
+        self.config = {'input' : '[-1]',
+                        'cell-dim' : -1, # this is a compulsary argument


danpovey · 2016-11-18T21:57:11Z

egs/wsj/s5/steps/libs/nnet3/xconfig/lstm.py

+        configs.append("component name={0}.c1 type=ElementwiseProductComponent input-dim={1} output-dim={2}".format(name, 2 * cell_dim, cell_dim))
+        configs.append("component name={0}.c2 type=ElementwiseProductComponent input-dim={1} output-dim={2}".format(name, 2 * cell_dim, cell_dim))
+        configs.append("component name={0}.m type=ElementwiseProductComponent input-dim={1} output-dim={2}".format(name, 2 * cell_dim, cell_dim))
+        configs.append("component name={0}.c type=ClipGradientComponent dim={1} {2}".format(name, cell_dim, clipgrad_str))


Vijay, we have checked in a change to how the gradients are clipped, and that change should be applied to this PR. The component name is different and the list of options is different.

I have not included this for now as I am running comparison experiments with the 6i setup (to see if I forgot some {param,bias}-stddev initialization .

I will make the change when I am done with the experiments.

danpovey · 2016-11-18T21:59:39Z

egs/wsj/s5/steps/libs/nnet3/xconfig/parser.py

+# Apache 2.0.
+
+""" This module contains the top level xconfig parsing functions.
+It has been separated from the utils module to


you don't have to explain changes-- no-one saw the original.

vijayaditya · 2016-11-19T00:05:40Z

@freewym Could you please help check the blstm model. It is available here
/export/b01/vpeddinti/xconfig/egs/swbd/s5c/exp/chain/blstm_6j_sp/configs/ref.raw

freewym · 2016-11-19T00:06:53Z

ok will do

On Fri, Nov 18, 2016 at 7:05 PM Vijayaditya Peddinti <
notifications@github.com> wrote:

@freewym https://github.com/freewym Could you please help check the
blstm model. It is available here

/export/b01/vpeddinti/xconfig/egs/swbd/s5c/exp/chain/blstm_6j_sp/configs/ref.raw

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADWAkmBP-QNc4IfKRXhUpO1tKwQuoOZNks5q_j1YgaJpZM4K1qh5
.

Sent from my iPhone

vijayaditya · 2016-11-19T22:16:08Z

Removing pretraining seems to help a lot.

# System                  7f     7g
# WER on train_dev(tg)    14.46  13.85
# WER on train_dev(fg)    13.23  12.67
# WER on eval2000(tg)     17.0   16.5
# WER on eval2000(fg)     15.4   14.8
# Final train prob     -0.0882071 -0.0885075
# Final valid prob     -0.107545  -0.113462
# Final train prob (xent) -1.26246 -1.25788
# Final valid prob (xent) -1.35525 -1.37058

danpovey · 2016-11-19T22:27:28Z

Great! What is the improvement from, do you think?

On Sat, Nov 19, 2016 at 5:16 PM, Vijayaditya Peddinti <
notifications@github.com> wrote:

System 7f 7g# WER on train_dev(tg) 14.46 13.85# WER on train_dev(fg) 13.23 12.67# WER on eval2000(tg) 17.0 16.5# WER on eval2000(fg) 15.4 14.8# Final train prob -0.0882071 -0.0885075# Final valid prob -0.107545 -0.113462# Final train prob (xent) -1.26246 -1.25788# Final valid prob (xent) -1.35525 -1.37058

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1197 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu-q4hWVdeEdXJq-x2j83AEXWix1Eks5q_3UpgaJpZM4K1qh5
.

vijayaditya · 2016-11-19T22:30:00Z

The only thing that changed was the removal of layer-wise pretraining.

(I updated the previous message when I realized that I had forgotten to write the details of the two experiments, but github doesn't seem to send email alerts when an existing message is updated.)

freewym · 2016-11-19T22:34:58Z

@vijayaditya It seems that the max-change option is missing for all diagonal matrices in BLSTM, according to /export/b01/vpeddinti/xconfig/egs/swbd/s5c/exp/chain/blstm_6j_sp/configs/final.config

vijayaditya · 2016-11-19T22:36:31Z

OK, thanks I will modify that.

--Vijay

On Sat, Nov 19, 2016 at 5:35 PM, Yiming Wang notifications@github.com
wrote:

@vijayaditya https://github.com/vijayaditya It seems that the
max-change option is missing for all diagonal matrices in BLSTM, according
to /export/b01/vpeddinti/xconfig/egs/swbd/s5c/exp/chain/blstm_6j_sp/configs/final.config

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADtwoL1aSyWasl8Uqe7s1DixlLS9NI6rks5q_3mUgaJpZM4K1qh5
.

danpovey · 2016-11-19T22:43:59Z

I'd like to get this xconfig stuff checked in ASAP, assuming it won't break
the existing setups-- that way we can accelerate the process of converting
recipes to the new style by getting others involved. Unless you think it's
better to wait a bit..

On Sat, Nov 19, 2016 at 5:36 PM, Vijayaditya Peddinti <
notifications@github.com> wrote:

OK, thanks I will modify that.

--Vijay

On Sat, Nov 19, 2016 at 5:35 PM, Yiming Wang notifications@github.com
wrote:

@vijayaditya https://github.com/vijayaditya It seems that the
max-change option is missing for all diagonal matrices in BLSTM,
according
to /export/b01/vpeddinti/xconfig/egs/swbd/s5c/exp/chain/blstm_
6j_sp/configs/final.config

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment),
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/
ADtwoL1aSyWasl8Uqe7s1DixlLS9NI6rks5q_3mUgaJpZM4K1qh5>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1197 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu8VD6uR_GyRTG4gsZ-IpGp1U6Lv6ks5q_3nxgaJpZM4K1qh5
.

vijayaditya · 2016-11-19T22:47:02Z

The lstm training will be done in few hours. I would like to check it in
after that.

There are still some python style changes to be made, but I can do this
after the code is checked in.

--Vijay

On Sat, Nov 19, 2016 at 5:44 PM, Daniel Povey notifications@github.com
wrote:

I'd like to get this xconfig stuff checked in ASAP, assuming it won't break
the existing setups-- that way we can accelerate the process of converting
recipes to the new style by getting others involved. Unless you think it's
better to wait a bit..

On Sat, Nov 19, 2016 at 5:36 PM, Vijayaditya Peddinti <
notifications@github.com> wrote:

OK, thanks I will modify that.

--Vijay

On Sat, Nov 19, 2016 at 5:35 PM, Yiming Wang notifications@github.com
wrote:

@vijayaditya https://github.com/vijayaditya It seems that the
max-change option is missing for all diagonal matrices in BLSTM,
according
to /export/b01/vpeddinti/xconfig/egs/swbd/s5c/exp/chain/blstm_
6j_sp/configs/final.config

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment),
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/
ADtwoL1aSyWasl8Uqe7s1DixlLS9NI6rks5q_3mUgaJpZM4K1qh5>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1197 (comment),
or mute
the thread
<https://github.com/notifications/unsubscribe-
auth/ADJVu8VD6uR_GyRTG4gsZ-IpGp1U6Lv6ks5q_3nxgaJpZM4K1qh5>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1197 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADtwoNXC3N8Vyjaz7F9i3v2OvwPEqWnUks5q_3uzgaJpZM4K1qh5
.

vijayaditya · 2016-11-21T05:08:27Z

I think this is ready for merge. I will complete any style changes necessary over the next few days.
TODO: Test the architectures which use auxiliary outputs ( @GaofengCheng will test HLSTMs).

Note : I will cleanup the local/chain/run_{tdnn,lstm,blstm}.sh scripts before merge.

danpovey · 2016-11-21T05:30:23Z

It's OK with me-- merge it yourself when you think it's ready. Thanks!!

GaofengCheng · 2016-11-21T06:27:00Z

@vijayaditya I will help test after you have merged.

danpovey · 2016-11-21T07:08:03Z

trying to fix it by code change

…ontext. This prevents a crash that has been happening since pull request #1197.

2. Added ability to provide strings with '=' as values in xconfig. 3. Added swbd recipes for TDNN, LSTM and BLSTM using xconfig.

vijayaditya mentioned this pull request Nov 17, 2016

Xconfigs #1170

Closed

vijayaditya force-pushed the xconfig_vijay branch from 1f990a4 to dc26602 Compare November 17, 2016 23:32

danpovey reviewed Nov 18, 2016

View reviewed changes

vijayaditya force-pushed the xconfig_vijay branch from 75bed1c to d6fd64e Compare November 18, 2016 23:27

vijayaditya force-pushed the xconfig_vijay branch 2 times, most recently from 72e3261 to 4611e41 Compare November 21, 2016 02:04

danpovey added a commit that referenced this pull request Nov 21, 2016

Fix to nnet3-chain-copy-egs.cc to do frame-shifting before limiting c…

ac1f932

…ontext. This prevents a crash that has been happening since pull request #1197.

Xconfig code from Dan.

7130d0a

vijayaditya force-pushed the xconfig_vijay branch from 9a817a3 to 5026607 Compare November 21, 2016 17:58

xconfig : 1. Added LSTM(P,C) layers.

22c45ba

2. Added ability to provide strings with '=' as values in xconfig. 3. Added swbd recipes for TDNN, LSTM and BLSTM using xconfig.

vijayaditya force-pushed the xconfig_vijay branch from 5026607 to 22c45ba Compare November 21, 2016 18:15

vijayaditya merged commit 07a5d51 into kaldi-asr:master Nov 21, 2016

vijayaditya mentioned this pull request Nov 21, 2016

xconfig files #1124

Closed

		@@ -0,0 +1,233 @@
		#!/bin/bash

		# 6i is based on run_lstm_6h.sh, but changing the HMM context from triphone to left biphone.

		@@ -0,0 +1,30 @@
		# This library has classes and methods to form neural network computation graphs,



		# 'ref.config' : which is a version of the config file used to generate
		# a model for getting left and right context it doesn't read anything for the

Xconfigs : extension #1197

Xconfigs : extension #1197

Uh oh!

Conversation

vijayaditya commented Nov 17, 2016

Uh oh!

vijayaditya commented Nov 17, 2016

Uh oh!

GaofengCheng commented Nov 18, 2016

Uh oh!

vijayaditya commented Nov 18, 2016

Uh oh!

vijayaditya commented Nov 18, 2016

Uh oh!

danpovey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vijayaditya commented Nov 19, 2016

Uh oh!

freewym commented Nov 19, 2016

Uh oh!

vijayaditya commented Nov 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danpovey commented Nov 19, 2016

Uh oh!

vijayaditya commented Nov 19, 2016

Uh oh!

freewym commented Nov 19, 2016

Uh oh!

vijayaditya commented Nov 19, 2016

Uh oh!

danpovey commented Nov 19, 2016

Uh oh!

vijayaditya commented Nov 19, 2016

Uh oh!

vijayaditya commented Nov 21, 2016

Uh oh!

danpovey commented Nov 21, 2016

Uh oh!

GaofengCheng commented Nov 21, 2016

Uh oh!

danpovey commented Nov 21, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vijayaditya commented Nov 19, 2016 •

edited

Loading