[src] Fix divergence from orthogonal constraint. Thx: Sergey Tushev.

[danpovey]

No description provided.

[sergy5]

Would this pull request be merged?

[danpovey]

Yes, I plan to merge it, but I was relying on you to test it first.

[sergy5]

Ok, I understand, as soon as I test it I tell you about result. пятница, 06 апреля 2018г., 22:05 +03:00 от Daniel Povey notifications@github.com :

…

Yes, I plan to merge it, but I was relying on you to test it first. — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread .

[sergy5]

I have successfully trained AM with this fix. But WER of my test is 0.5% worse than previous version of swbd scenario /local/chain/run_tdnn.sh. Maybe it is ok, or it may get worse because of fixes that you have made in file src/nnet3/nnet-utils.cc?

[danpovey]

Thanks. But the problem this is intended to fix is likely quite hard to replicate, so unless you started from one of the "bad" model that you already had (after the diagnostics already started to be strange), it wouldn't test whether it fixes the divergence. Did you start from scratch or from one of your "bad" models? And did you add any debugging statements to figure out what the value of that 'ratio' usually is?

[sergy5]

Unfortunately I did not add any debugging statements, I just used your code. But the training materials, egs were the same as in 'bad' variant. So, non-patched system crashed at step 743. And patched system trained ok over all steps. Is it necessary to add debugging output and re-run experiment?

[danpovey]

These problems are nondeterministic. Unless you started it from a "bad" iteration it would not have failed, and I guess the bad models are lost now so there is no way to know whether it resolved the instability. It's OK, I'll test it locally. Unfortunately I did not add any debugging statements, I just used your

…

code. But the training materials, egs were the same as in 'bad' variant. So, non-patched system crashed at step 743. And patched system trained ok over all steps. Is it necessary to add debugging output and re-run experiment? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2336 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVuzPxVxbrUzyDLwNNCd2vjdnMaXw_ks5tnRoEgaJpZM4TF4dA> .

[danpovey]

I have tested this to at least make sure the training seems to proceed normally, and that the if-statement isn't usually taken (it's mostly taken only on the 1st iter of training)-- I think it's extremely unlikely that this is affecting the results.

The difference in your results is likely due to other causes-- it could even be random.