Add 'test mode' to dropout component

[freewym]

For issue #1570. Failure on derivative test need to be resolved.

[danpovey]

Minor comments but please do this right away, as I want to make some other changes that will conflict with these, and I'd rather start from your PR.

[danpovey]

Can you comment here that it only makes sense to set test-mode in the config for testing purposes?

[danpovey]

This would not work correctly for in-place propagation, if we ever change the code to make that possible, so I think it would be nicer to replace this code with:

  out->CopyFromMat(in);
  out->Scale(1.0  - dropout);
  return NULL;

[freewym]

Done

[danpovey]

@hhadian, I haven't really done any testing of this, but I'm hoping you can help with scripting and testing it in the CIFAR setup.
In your Keras-based recipes, where is the dropout applied? I.e. on which layers?

[hhadian]

Sure. Do you want me to prepare another script in local/nnet3 to test it on CIFAR?
The dropout is applied after ReLU activations (or after max-pooling which itself is applied on CNN). I.e. In the recipe I'm using it is either CNN -> ReLU -> MaxPool -> Dropout or FullyConnected -> ReLU -> Dropout

[danpovey]

Yes, make another script like that. But you'll have to add the capability for dropout to the xconfig layers for CNN and fully-connected layers, i.e. XconfigConvLayer and XconfigBasicLayer. I suggest to have it as an extra nonlinearity in addition to relu, sigmoid, etc., in XconfigBasicLayer, and an extra component in addition to 'conv', 'batchnorm' and 'relu', in XconfigConvLayer. You'll see in the code how to change it. In addition you may need to add extra options for the layer name in parser.py (don't go overboard, just add what you need). And you'll need a way to set the dropout proportion. Make it default to 0.5 (you'll modify set_default_configs in both classes to add 'dropout-proportion' : 0.5).

…

On Wed, Apr 26, 2017 at 12:17 AM, Hossein Hadian ***@***.***> wrote: Sure. Do you want me to prepare another script in local/nnet3 to test it on CIFAR? The dropout is applied after ReLU activations (or after max-pooling which itself is applied on CNN). I.e. In the recipe I'm using it is either CNN -> ReLU -> MaxPool -> Dropout or FullyConnected -> ReLU -> Dropout — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1578 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVuylkV-xYgal0UvwJkUmrN7u7bQO3ks5rzsVcgaJpZM4NIDpq> .

[hhadian]

OK. Will do.