Learning rate 'Exponential Decay' policy wrong.

[TimZaman]

Version v3.2.0-61-g0ce28b9
I often have issues with the learning rate and the decay. Sometimes it tells me it cannot process 'nan', sometimes it's correct, sometimes it jitters a bit up and down, but most often it does this. I can reproduce this only half of the time.
Base learning rate: 1.0
Policy: Exponential Decay
Gamma: 0.95

Then this happens:

[gheinrich]

Thanks for the bug report. The learning rate seems to be decreasing way too quickly indeed. Are you using Torch or Caffe? I think I can reproduce your issue with Torch.

[TimZaman]

Torch indeed. Did not try Caffe.

[gheinrich]

@TimZaman do you mind verifying that #663 fixes the issue? Thanks!

[TimZaman]

Yes. Merci beaucoup Greg!

[TimZaman]

Hmm even after the update, still something fishy here: (500 epochs used)

[gheinrich]

I just tried running 500 epochs of training. This is what the learning rate looked like:

I was using this version:

$ git describe
v3.2.0-73-gb4dd9c7

[gheinrich]

@TimZaman were you able to verify that you were using the right version of DIGITS? The exponential decay is working for me for 500 epochs.

[gheinrich]

No response and this is working for me on v3.2.0-73-gb4dd9c7 so I am closing the bug.

[TimZaman]

i was using the most up to date version for sure. There's something going on with the learning rate for sure, i will try to hunt down the cause next time instead of posting the problem.