Great work! Some thoughts on the optimization of the network

[KasperEinarson]

Dear Vahid Noroozi!
Thanks for the code - very useful! I'm interested in hearing your opinion on the optimization of the network. You say that your implementation differs from the Galen Andrews (2013) original paper

"I used RMSProp (an adaptive version of gradient descent) instead of GD with momentum. "

But he uses L-BFGS because it's a full batch optimizer because:

"
Because the correlation objective is a function of the
entire training set that does not decompose into a sum
over data points, it is not clear how to use a stochastic
optimization procedure that operates on data points
one at a time[mini batches]"

doesn't this make both RMSprop and GD with momentum invalid?
From the original paper he includes how to take the derivative of the correlation function the two outputs of the two networks - how does this special derivative of the loss come into play in your code?

Thanks again for the great work i appreciate it very much!

Kind regards
Kasper