Skip to content

Max change per component #918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danpovey merged 13 commits into from
Oct 26, 2016
Merged

Max change per component #918

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
11e840f
add back max-change-per-component
freewym Jul 25, 2016
30420c6
update blstm results on swbd
freewym Jul 25, 2016
33351ce
apply max-change per component in trainer code
freewym Aug 4, 2016
5a39d12
changes(to be tested)
freewym Aug 15, 2016
c2d316b
fix
freewym Aug 16, 2016
10fe210
fix
freewym Sep 6, 2016
6ce191f
changes based on comments
freewym Oct 12, 2016
3c6c728
applied the same changes to chain training
freewym Oct 12, 2016
7c9eeb7
fix
freewym Oct 17, 2016
ca130ce
fix2
freewym Oct 17, 2016
21cf53f
reorder resutls
freewym Oct 17, 2016
9b31ad7
make delta_nnet_ always != NULL
freewym Oct 26, 2016
bf7ba54
added max-change-per-component options to allow them configurable
freewym Oct 26, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions egs/swbd/s5c/RESULTS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,13 +152,23 @@ exit 0
%WER 19.4 | 2628 21594 | 82.7 12.0 5.3 2.1 19.4 54.9 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_fsh_fg/score_10_0.0/eval2000_hires.ctm.callhm.filt.sys
%WER 20.8 | 2628 21594 | 81.3 13.1 5.6 2.2 20.8 56.9 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_tg/score_10_0.0/eval2000_hires.ctm.callhm.filt.sys

# bidirectional LSTM with the same configuration as the above experiment, plus self-repair of all nonliearities and clipgradient activated
%WER 10.4 | 1831 21395 | 90.5 6.2 3.3 0.9 10.4 44.2 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_fsh_fg/score_11_0.0/eval2000_hires.ctm.swbd.filt.sys
%WER 11.3 | 1831 21395 | 89.8 6.8 3.3 1.1 11.3 46.7 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_tg/score_10_0.0/eval2000_hires.ctm.swbd.filt.sys
# bidirectional LSTM with the same configuration as the above experiment, with self-repair of all nonliearities and clipgradient, and max-change-per-component activated
Copy link
Contributor

@vijayaditya vijayaditya Jul 29, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please preserve these results somewhere, may be in the actual script next to the max-change option or at the bottom of the results file.

%WER 14.9 | 4459 42989 | 86.7 9.0 4.3 1.6 14.9 50.5 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_fsh_fg/score_10_0.0/eval2000_hires.ctm.filt.sys
%WER 15.9 | 4459 42989 | 85.7 9.8 4.5 1.7 15.9 52.3 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_tg/score_10_0.0/eval2000_hires.ctm.filt.sys
%WER 10.2 | 1831 21395 | 90.8 6.1 3.2 1.0 10.2 44.4 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_fsh_fg/score_11_0.0/eval2000_hires.ctm.swbd.filt.sys
%WER 11.2 | 1831 21395 | 89.9 6.8 3.3 1.1 11.2 46.6 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_tg/score_10_0.0/eval2000_hires.ctm.swbd.filt.sys
%WER 19.4 | 2628 21594 | 82.7 11.8 5.4 2.2 19.4 54.5 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_fsh_fg/score_10_0.0/eval2000_hires.ctm.callhm.filt.sys
%WER 20.6 | 2628 21594 | 81.5 12.8 5.7 2.2 20.6 56.2 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_tg/score_10_0.5/eval2000_hires.ctm.callhm.filt.sys

(
# bidirectional LSTM with the same configuration as the above experiment, with self-repair of all nonliearities and clipgradient activated
%WER 15.0 | 4459 42989 | 86.5 9.1 4.5 1.5 15.0 50.4 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_fsh_fg/score_11_0.0/eval2000_hires.ctm.filt.sys
%WER 16.0 | 4459 42989 | 85.6 9.9 4.5 1.6 16.0 52.7 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_tg/score_10_0.0/eval2000_hires.ctm.filt.sys
%WER 10.4 | 1831 21395 | 90.5 6.2 3.3 0.9 10.4 44.2 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_fsh_fg/score_11_0.0/eval2000_hires.ctm.swbd.filt.sys
%WER 11.3 | 1831 21395 | 89.8 6.8 3.3 1.1 11.3 46.7 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_tg/score_10_0.0/eval2000_hires.ctm.swbd.filt.sys
%WER 19.6 | 2628 21594 | 82.5 12.1 5.5 2.1 19.6 54.8 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_fsh_fg/score_10_0.0/eval2000_hires.ctm.callhm.filt.sys
%WER 20.7 | 2628 21594 | 81.4 12.9 5.7 2.2 20.7 56.8 | exp/nnet3/lstm_bidirectional_sp/decode_eval2000_sw1_tg/score_10_0.0/eval2000_hires.ctm.callhm.filt.sys
)

# results with nnet3 tdnn: local/nnet3/run_tdnn.sh (11.10.2015) (2 epoch training on speed-perturbed and volume perturbed data)
%WER 12.1 | 1831 21395 | 89.1 7.1 3.8 1.3 12.1 48.1 | exp/nnet3/tdnn_sp/decode_eval2000_hires_sw1_fsh_fg/score_12_0.0/eval2000_hires.ctm.swbd.filt.sys
Expand Down
1 change: 1 addition & 0 deletions egs/swbd/s5c/local/chain/tuning/run_blstm_6i.sh
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ if [ $stage -le 12 ]; then
--recurrent-projection-dim 256 \
--non-recurrent-projection-dim 256 \
--label-delay $label_delay \
--self-repair-scale-clipgradient 1.0 \
Copy link
Contributor

@danpovey danpovey Oct 17, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the default, and how does this relate to your other changs?

Copy link
Contributor Author

@freewym freewym Oct 17, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.0 is the default value set for self-repair-scale-clipgradient in make_configs.py. Here I just explicitly tell the user there is such option.

$dir/configs || exit 1;

fi
Expand Down
Empty file modified egs/swbd/s5c/local/chain/tuning/run_tdnn_7d.sh
100644 → 100755
View file
Open in desktop
Empty file.
49 changes: 31 additions & 18 deletions egs/wsj/s5/steps/nnet3/components.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,23 +90,29 @@ def AddPermuteLayer(config_lines, name, input, column_map):
return {'descriptor': '{0}_permute'.format(name),
'dimension': input['dimension']}

def AddAffineLayer(config_lines, name, input, output_dim, ng_affine_options = ""):
def AddAffineLayer(config_lines, name, input, output_dim, ng_affine_options = "", max_change_per_component = 0.75):
components = config_lines['components']
component_nodes = config_lines['component-nodes']

components.append("component name={0}_affine type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3}".format(name, input['dimension'], output_dim, ng_affine_options))
# Per-component max-change option
max_change_options = "max-change={0:.2f}".format(max_change_per_component) if max_change_per_component is not None else ''

components.append("component name={0}_affine type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3} {4}".format(name, input['dimension'], output_dim, ng_affine_options, max_change_options))
component_nodes.append("component-node name={0}_affine component={0}_affine input={1}".format(name, input['descriptor']))

return {'descriptor': '{0}_affine'.format(name),
'dimension': output_dim}

def AddAffRelNormLayer(config_lines, name, input, output_dim, ng_affine_options = " bias-stddev=0 ", norm_target_rms = 1.0, self_repair_scale = None):
def AddAffRelNormLayer(config_lines, name, input, output_dim, ng_affine_options = " bias-stddev=0 ", norm_target_rms = 1.0, self_repair_scale = None, max_change_per_component = 0.75):
components = config_lines['components']
component_nodes = config_lines['component-nodes']

# self_repair_scale is a constant scaling the self-repair vector computed in RectifiedLinearComponent
self_repair_string = "self-repair-scale={0:.10f}".format(self_repair_scale) if self_repair_scale is not None else ''
components.append("component name={0}_affine type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3}".format(name, input['dimension'], output_dim, ng_affine_options))
# Per-component max-change option
max_change_options = "max-change={0:.2f}".format(max_change_per_component) if max_change_per_component is not None else ''

components.append("component name={0}_affine type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3} {4}".format(name, input['dimension'], output_dim, ng_affine_options, max_change_options))
components.append("component name={0}_relu type=RectifiedLinearComponent dim={1} {2}".format(name, output_dim, self_repair_string))
components.append("component name={0}_renorm type=NormalizeComponent dim={1} target-rms={2}".format(name, output_dim, norm_target_rms))

Expand Down Expand Up @@ -244,6 +250,7 @@ def AddOutputLayer(config_lines, input, label_delay = None, suffix=None, objecti

def AddFinalLayer(config_lines, input, output_dim,
ng_affine_options = " param-stddev=0 bias-stddev=0 ",
max_change_per_component = 1.5,
label_delay=None,
use_presoftmax_prior_scale = False,
prior_scale_file = None,
Expand All @@ -261,7 +268,7 @@ def AddFinalLayer(config_lines, input, output_dim,

prev_layer_output = AddAffineLayer(config_lines,
final_node_prefix , input, output_dim,
ng_affine_options)
ng_affine_options, max_change_per_component)
if include_log_softmax:
if use_presoftmax_prior_scale :
components.append('component name={0}-fixed-scale type=FixedScaleComponent scales={1}'.format(final_node_prefix, prior_scale_file))
Expand All @@ -288,7 +295,8 @@ def AddLstmLayer(config_lines,
ng_affine_options = "",
lstm_delay = -1,
self_repair_scale_nonlinearity = None,
self_repair_scale_clipgradient = None):
self_repair_scale_clipgradient = None,
max_change_per_component = 0.75):
assert(recurrent_projection_dim >= 0 and non_recurrent_projection_dim >= 0)
components = config_lines['components']
component_nodes = config_lines['component-nodes']
Expand Down Expand Up @@ -316,24 +324,26 @@ def AddLstmLayer(config_lines,
self_repair_clipgradient_string = "self-repair-scale={0:.2f}".format(self_repair_scale_clipgradient) if self_repair_scale_clipgradient is not None else ''
# Natural gradient per element scale parameters
ng_per_element_scale_options += " param-mean=0.0 param-stddev=1.0 "
# Per-component max-change option
max_change_options = "max-change={0:.2f}".format(max_change_per_component) if max_change_per_component is not None else ''
# Parameter Definitions W*(* replaced by - to have valid names)
components.append("# Input gate control : W_i* matrices")
components.append("component name={0}_W_i-xr type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3}".format(name, input_dim + recurrent_projection_dim, cell_dim, ng_affine_options))
components.append("component name={0}_W_i-xr type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3} {4}".format(name, input_dim + recurrent_projection_dim, cell_dim, ng_affine_options, max_change_options))
components.append("# note : the cell outputs pass through a diagonal matrix")
components.append("component name={0}_w_ic type=NaturalGradientPerElementScaleComponent dim={1} {2}".format(name, cell_dim, ng_per_element_scale_options))
components.append("component name={0}_w_ic type=NaturalGradientPerElementScaleComponent dim={1} {2} {3}".format(name, cell_dim, ng_per_element_scale_options, max_change_options))

components.append("# Forget gate control : W_f* matrices")
components.append("component name={0}_W_f-xr type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3}".format(name, input_dim + recurrent_projection_dim, cell_dim, ng_affine_options))
components.append("component name={0}_W_f-xr type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3} {4}".format(name, input_dim + recurrent_projection_dim, cell_dim, ng_affine_options, max_change_options))
components.append("# note : the cell outputs pass through a diagonal matrix")
components.append("component name={0}_w_fc type=NaturalGradientPerElementScaleComponent dim={1} {2}".format(name, cell_dim, ng_per_element_scale_options))
components.append("component name={0}_w_fc type=NaturalGradientPerElementScaleComponent dim={1} {2} {3}".format(name, cell_dim, ng_per_element_scale_options, max_change_options))

components.append("# Output gate control : W_o* matrices")
components.append("component name={0}_W_o-xr type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3}".format(name, input_dim + recurrent_projection_dim, cell_dim, ng_affine_options))
components.append("component name={0}_W_o-xr type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3} {4}".format(name, input_dim + recurrent_projection_dim, cell_dim, ng_affine_options, max_change_options))
components.append("# note : the cell outputs pass through a diagonal matrix")
components.append("component name={0}_w_oc type=NaturalGradientPerElementScaleComponent dim={1} {2}".format(name, cell_dim, ng_per_element_scale_options))
components.append("component name={0}_w_oc type=NaturalGradientPerElementScaleComponent dim={1} {2} {3}".format(name, cell_dim, ng_per_element_scale_options, max_change_options))

components.append("# Cell input matrices : W_c* matrices")
components.append("component name={0}_W_c-xr type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3}".format(name, input_dim + recurrent_projection_dim, cell_dim, ng_affine_options))
components.append("component name={0}_W_c-xr type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3} {4}".format(name, input_dim + recurrent_projection_dim, cell_dim, ng_affine_options, max_change_options))


components.append("# Defining the non-linearities")
Expand Down Expand Up @@ -385,7 +395,7 @@ def AddLstmLayer(config_lines,
# add the recurrent connections
if (add_recurrent_projection and add_non_recurrent_projection):
components.append("# projection matrices : Wrm and Wpm")
components.append("component name={0}_W-m type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3}".format(name, cell_dim, recurrent_projection_dim + non_recurrent_projection_dim, ng_affine_options))
components.append("component name={0}_W-m type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3} {4}".format(name, cell_dim, recurrent_projection_dim + non_recurrent_projection_dim, ng_affine_options, max_change_options))
components.append("component name={0}_r type=ClipGradientComponent dim={1} clipping-threshold={2} norm-based-clipping={3} {4}".format(name, recurrent_projection_dim, clipping_threshold, norm_based_clipping, self_repair_clipgradient_string))
component_nodes.append("# r_t and p_t")
component_nodes.append("component-node name={0}_rp_t component={0}_W-m input={0}_m_t".format(name))
Expand All @@ -396,7 +406,7 @@ def AddLstmLayer(config_lines,

elif add_recurrent_projection:
components.append("# projection matrices : Wrm")
components.append("component name={0}_Wrm type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3}".format(name, cell_dim, recurrent_projection_dim, ng_affine_options))
components.append("component name={0}_Wrm type=NaturalGradientAffineComponent input-dim={1} output-dim={2} {3} {4}".format(name, cell_dim, recurrent_projection_dim, ng_affine_options, max_change_options))
components.append("component name={0}_r type=ClipGradientComponent dim={1} clipping-threshold={2} norm-based-clipping={3} {4}".format(name, recurrent_projection_dim, clipping_threshold, norm_based_clipping, self_repair_clipgradient_string))
component_nodes.append("# r_t")
component_nodes.append("component-node name={0}_r_t_preclip component={0}_Wrm input={0}_m_t".format(name))
Expand Down Expand Up @@ -425,22 +435,25 @@ def AddBLstmLayer(config_lines,
ng_affine_options = "",
lstm_delay = [-1,1],
self_repair_scale_nonlinearity = None,
self_repair_scale_clipgradient = None):
self_repair_scale_clipgradient = None,
max_change_per_component = 0.75):
assert(len(lstm_delay) == 2 and lstm_delay[0] < 0 and lstm_delay[1] > 0)
output_forward = AddLstmLayer(config_lines, "{0}_forward".format(name), input, cell_dim,
recurrent_projection_dim, non_recurrent_projection_dim,
clipping_threshold, norm_based_clipping,
ng_per_element_scale_options, ng_affine_options,
lstm_delay = lstm_delay[0],
self_repair_scale_nonlinearity = self_repair_scale_nonlinearity,
self_repair_scale_clipgradient = self_repair_scale_clipgradient)
self_repair_scale_clipgradient = self_repair_scale_clipgradient,
max_change_per_component = max_change_per_component)
output_backward = AddLstmLayer(config_lines, "{0}_backward".format(name), input, cell_dim,
recurrent_projection_dim, non_recurrent_projection_dim,
clipping_threshold, norm_based_clipping,
ng_per_element_scale_options, ng_affine_options,
lstm_delay = lstm_delay[1],
self_repair_scale_nonlinearity = self_repair_scale_nonlinearity,
self_repair_scale_clipgradient = self_repair_scale_clipgradient)
self_repair_scale_clipgradient = self_repair_scale_clipgradient,
max_change_per_component = max_change_per_component)
output_descriptor = 'Append({0}, {1})'.format(output_forward['descriptor'], output_backward['descriptor'])
output_dim = output_forward['dimension'] + output_backward['dimension']

Expand Down
Loading