[src,scripts,egs] CNN+TDNN-F recipes #2572

danpovey · 2018-07-25T05:25:25Z

No description provided.

Speed(gflops) size old new speedup CuVector::AddDiagMat2Shapes<double>[no-trans], (1048576, 32), 2.13 8.04 3.77x CuVector::AddDiagMat2Shapes<double>[no-trans], (524288, 64), 4.12 7.27 1.77x CuVector::AddDiagMat2Shapes<double>[no-trans], (262144, 128), 7.66 8.56 1.12x CuVector::AddDiagMat2Shapes<double>[no-trans], (131072, 256), 13.50 13.50 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (65536, 512), 22.29 22.32 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (32768, 1024), 32.26 32.35 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (16384, 2048), 32.48 32.47 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (8192, 4096), 32.54 32.57 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (4096, 8192), 32.52 32.55 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (2048, 16384), 32.46 32.49 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (1024, 32768), 32.30 32.34 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (512, 65536), 31.77 31.89 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (256, 131072), 31.74 31.71 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (128, 262144), 31.64 31.67 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (64, 524288), 32.36 32.37 1.00x CuVector::AddDiagMat2Shapes<double>[no-trans], (32, 1048576), 30.94 30.92 1.00x CuVector::AddDiagMat2Shapes<double>[trans], (1048576, 32), 1.10 8.61 7.84x CuVector::AddDiagMat2Shapes<double>[trans], (524288, 64), 2.19 8.61 3.94x CuVector::AddDiagMat2Shapes<double>[trans], (262144, 128), 4.41 8.67 1.97x CuVector::AddDiagMat2Shapes<double>[trans], (131072, 256), 8.64 8.56 0.99x CuVector::AddDiagMat2Shapes<double>[trans], (65536, 512), 15.72 8.57 0.55x CuVector::AddDiagMat2Shapes<double>[trans], (32768, 1024), 26.09 26.07 1.00x CuVector::AddDiagMat2Shapes<double>[trans], (16384, 2048), 31.51 31.26 0.99x CuVector::AddDiagMat2Shapes<double>[trans], (8192, 4096), 27.93 28.35 1.02x CuVector::AddDiagMat2Shapes<double>[trans], (4096, 8192), 31.56 31.52 1.00x CuVector::AddDiagMat2Shapes<double>[trans], (2048, 16384), 31.21 31.20 1.00x CuVector::AddDiagMat2Shapes<double>[trans], (1024, 32768), 31.40 31.36 1.00x CuVector::AddDiagMat2Shapes<double>[trans], (512, 65536), 31.52 31.55 1.00x CuVector::AddDiagMat2Shapes<double>[trans], (256, 131072), 30.96 30.95 1.00x CuVector::AddDiagMat2Shapes<double>[trans], (128, 262144), 30.00 29.99 1.00x CuVector::AddDiagMat2Shapes<double>[trans], (64, 524288), 28.43 28.78 1.01x CuVector::AddDiagMat2Shapes<double>[trans], (32, 1048576), 24.95 24.93 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (1048576, 32), 2.92 15.87 5.44x CuVector::AddDiagMat2Shapes<float>[no-trans], (524288, 64), 5.70 14.27 2.51x CuVector::AddDiagMat2Shapes<float>[no-trans], (262144, 128), 11.04 16.65 1.51x CuVector::AddDiagMat2Shapes<float>[no-trans], (131072, 256), 21.12 21.15 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (65536, 512), 38.60 38.67 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (32768, 1024), 57.21 57.29 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (16384, 2048), 63.39 63.50 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (8192, 4096), 62.63 62.71 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (4096, 8192), 63.60 63.71 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (2048, 16384), 63.07 63.09 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (1024, 32768), 62.47 62.64 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (512, 65536), 61.80 61.86 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (256, 131072), 61.03 60.99 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (128, 262144), 60.22 59.81 0.99x CuVector::AddDiagMat2Shapes<float>[no-trans], (64, 524288), 62.09 61.87 1.00x CuVector::AddDiagMat2Shapes<float>[no-trans], (32, 1048576), 52.96 53.01 1.00x CuVector::AddDiagMat2Shapes<float>[trans], (1048576, 32), 1.25 16.44 13.19x CuVector::AddDiagMat2Shapes<float>[trans], (524288, 64), 2.48 17.15 6.91x CuVector::AddDiagMat2Shapes<float>[trans], (262144, 128), 4.92 17.14 3.49x CuVector::AddDiagMat2Shapes<float>[trans], (131072, 256), 9.55 18.27 1.91x CuVector::AddDiagMat2Shapes<float>[trans], (65536, 512), 17.90 18.30 1.02x CuVector::AddDiagMat2Shapes<float>[trans], (32768, 1024), 31.49 31.48 1.00x CuVector::AddDiagMat2Shapes<float>[trans], (16384, 2048), 34.38 34.38 1.00x CuVector::AddDiagMat2Shapes<float>[trans], (8192, 4096), 51.61 51.59 1.00x CuVector::AddDiagMat2Shapes<float>[trans], (4096, 8192), 48.60 48.87 1.01x CuVector::AddDiagMat2Shapes<float>[trans], (2048, 16384), 57.47 57.52 1.00x CuVector::AddDiagMat2Shapes<float>[trans], (1024, 32768), 56.30 56.38 1.00x CuVector::AddDiagMat2Shapes<float>[trans], (512, 65536), 55.83 56.24 1.01x CuVector::AddDiagMat2Shapes<float>[trans], (256, 131072), 55.35 55.81 1.01x CuVector::AddDiagMat2Shapes<float>[trans], (128, 262144), 54.26 54.56 1.01x CuVector::AddDiagMat2Shapes<float>[trans], (64, 524288), 52.88 53.00 1.00x CuVector::AddDiagMat2Shapes<float>[trans], (32, 1048576), 47.55 47.44 1.00x

…evel.

…encies.sh failure

danpovey · 2018-07-25T05:27:13Z

@GaofengCheng, can you please investigate making this same kind of change to some larger-scale TDNN-F scripts? Our GPUs are very busy. For now you can just leave the layer sizes for the initial CNN layers the same as they are in the small setups, and tune it later.

GaofengCheng · 2018-07-25T05:33:32Z

Nice improvement
I will try this on switchboard first.
After getting good results, I can try using blocksum layer.
Gaofeng

kangshiyin · 2018-07-25T06:28:01Z

It seems this is not the newest version of AddDiagMat2. The implementation in #2560 should be 3x faster, for AddDiagMat2 with the size (1048576, 32).

danpovey · 2018-07-25T19:12:33Z

Thanks a lot, @kangshiyin; I merged your PR and merged the latest master into this PR.

danpovey · 2018-07-26T18:38:58Z

I added a WSJ example. It's a little better, but not a lot. I will try to tune it more; right now I'm trying taking out two TDNN-F layers but I don't have much hope that it will help, based on my experience in mini-librispeech. @GaofengCheng, maybe you could also try it with slightly larger l2-regularize values, since it seems to be overfitting more-- if you have a WSJ setup.

danpovey · 2018-07-26T18:49:21Z

egs/wsj/s5/local/chain/tuning/run_cnn_tdnn_1b17.sh

+
+  # the first TDNN-F layer has no bypass (since dims don't match), and a larger bottleneck so the
+  # information bottleneck doesn't become a problem.
+  tdnnf-layer name=tdnnf7 $tdnnf_first_opts dim=1024 bottleneck-dim=256 time-stride=0


The 256 here was a mistake; I intended for it to be the same as the final-layer bottleneck which it seems is 192 in the WSJ setup.

danpovey · 2018-08-20T04:05:03Z

@xiaohui-zhang, do you have time to help with this PR? I just need the scripts to be renamed to the next consecutive letter (no number), the references to the script/affixes inside the scripts themselves to be corrected, and the comparisons with non-checked-in scripts to be removed from the comments in the headers.
I'm super busy with something right now so I don't have time to do this.

I was originally hoping @GaofengCheng would be able to get CNN+TDNN-F to help (vs. TDNN-F) on larger setups,as our grid was busy, but I think he was not able to, so far. I think we can just check this in now, rather than wait.

xiaohui-zhang · 2018-08-20T04:30:19Z

no prob!

…

On Mon, Aug 20, 2018, 12:05 AM Daniel Povey ***@***.***> wrote: @xiaohui-zhang <https://github.com/xiaohui-zhang>, do you have time to help with this PR? I just need the scripts to be renamed to the next consecutive letter (no number), the references to the script/affixes inside the scripts themselves to be corrected, and the comparisons with non-checked-in scripts to be removed from the comments in the headers. I'm super busy with something right now so I don't have time to do this. I was originally hoping @GaofengCheng <https://github.com/GaofengCheng> would be able to get CNN+TDNN-F to help (vs. TDNN-F) on larger setups,as our grid was busy, but I think he was not able to, so far. I think we can just check this in now, rather than wait. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2572 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANiEEQJjpicv1A2-lAuEvahg-EljTXvRks5uSjV8gaJpZM4VfdBo> .

xiaohui-zhang · 2018-08-23T23:41:05Z

done

danpovey · 2018-08-24T20:19:30Z

Closing since merged via #2643

kangshiyin and others added 10 commits July 18, 2018 00:05

[src] Couple fixes to @kangshiyin's PR

0222773

[src] Fix bug in nnet-analyze.cc which caused crash at high verbose l…

0089943

…evel.

[src] Suppress unnecessary warning from convolution code.

6a2ffec

[src] Add dependency on sox; cosmetic changes to steps/decode.sh

8054d9b

Merge remote-tracking branch 'origin/add_diag_mat_mat_speedup'

4820bdf

Merge remote-tracking branch 'upstream/master'

741807d

[egs] Fix mix-up in mini-librispeech TDNN-F results

e86679f

[build] add sox to packages in travis.yml to stop extras/check_depend…

f9db1cd

…encies.sh failure

[src,scripts,egs] New CNN+TDNN-F scripts

8c3b1ba

danpovey added 2 commits July 25, 2018 15:09

Merge remote-tracking branch 'upstream/master'

64d5aee

Merge remote-tracking branch 'upstream/master' into cnn_tdnnf

cd4e6db

danpovey added 3 commits July 25, 2018 22:12

[egs] Update TDNN-F script for Librispeech

0fbb9a5

[egs] Fix numbering of layers in example scripts.

eb9601e

[egs] Add WSJ example with results for CNN+TDNN.

f83cbdc

danpovey commented Jul 26, 2018

View reviewed changes

xiaohui-zhang added a commit to xiaohui-zhang/kaldi that referenced this pull request Aug 23, 2018

clean up scripts in kaldi-asr#2572

5c0ac33

danpovey closed this Aug 24, 2018

xiaohui-zhang added a commit to xiaohui-zhang/kaldi that referenced this pull request Nov 6, 2018

clean up scripts in kaldi-asr#2572

4ce81ee

xiaohui-zhang added a commit to xiaohui-zhang/kaldi that referenced this pull request Nov 9, 2018

clean up scripts in kaldi-asr#2572

0e758c6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[src,scripts,egs] CNN+TDNN-F recipes #2572

[src,scripts,egs] CNN+TDNN-F recipes #2572

Uh oh!

danpovey commented Jul 25, 2018

Uh oh!

danpovey commented Jul 25, 2018

Uh oh!

GaofengCheng commented Jul 25, 2018

Uh oh!

kangshiyin commented Jul 25, 2018

Uh oh!

danpovey commented Jul 25, 2018

Uh oh!

danpovey commented Jul 26, 2018

Uh oh!

danpovey Jul 26, 2018

Uh oh!

danpovey commented Aug 20, 2018

Uh oh!

xiaohui-zhang commented Aug 20, 2018 via email

Uh oh!

xiaohui-zhang commented Aug 23, 2018

Uh oh!

danpovey commented Aug 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[src,scripts,egs] CNN+TDNN-F recipes #2572

[src,scripts,egs] CNN+TDNN-F recipes #2572

Uh oh!

Conversation

danpovey commented Jul 25, 2018

Uh oh!

danpovey commented Jul 25, 2018

Uh oh!

GaofengCheng commented Jul 25, 2018

Uh oh!

kangshiyin commented Jul 25, 2018

Uh oh!

danpovey commented Jul 25, 2018

Uh oh!

danpovey commented Jul 26, 2018

Uh oh!

danpovey Jul 26, 2018

Choose a reason for hiding this comment

Uh oh!

danpovey commented Aug 20, 2018

Uh oh!

xiaohui-zhang commented Aug 20, 2018 via email

Uh oh!

xiaohui-zhang commented Aug 23, 2018

Uh oh!

danpovey commented Aug 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants