LSTM, 'add' and 'write' produce the same results?

[Kublai-Jing]

Have been working on lstm recently. Found that 'add' and 'write' grad_reqs produce the same results

rnn_exec = rnn_sym.bind(ctx=ctx, args=arg_arrays,
                        args_grad=args_grad,
                        grad_req="write")    

which for the first two epochs, I have,

And for the 'add' version

rnn_exec = rnn_sym.bind(ctx=ctx, args=arg_arrays,
                        args_grad=args_grad,
                        grad_req="add")    

These two should not be the same ?

[antinucleon]

Yes！If you are using write, you can safely remove reset gradient code at the end of epoch

[Kublai-Jing]

I mean, if I am using 'write' for the lstm, should it produce the same (and correct) result as using 'add' ? If that's the case, where is 'add' useful? Thanks !

[Kublai-Jing]

hmm. I think I see what you mean, the 'add' only make difference when we call updater(idx, grad, weight).
Using write will overwrite the gradient inside that call, whereas using 'add' will add to that. So I guess the engine will itself add up gradients to the same parameter when we call backward, which means that using 'write' is also correct for lstm in this case.

[jermainewang]

I think 'add' means the calculated gradient should be added to rather than written to the gradient variable. This is useful for RNN since RNN will calculate 't' ('t' is the number of unrolling length) gradients of the gate parameters, and all these gradients should be accumulated to update the gate parameters. In fact, I am quite surprised that the 'write' strategy gives the same result as 'add' strategy. Let me look into it.

[pluskid]

Is it related to this? #568