Modular Interface

[ylxdzsw]

This is my implementation of the modular interface proposed by @MikeInnes.

Basically it exposes some AutoGrad logics by track and back!, track(x) creates a new tape and track(x, y) records x on the same tape as y. back! then run the backward pass and leave all gradients on the tape, which can then be retrieved by getgrad.

A model is a struct or closure which can be called. If it contains trainable parameters, it should track them in the forward pass so that them can be retrieved by params. If the input is not tracked, all parameters will also be "untracked" and the whole model is running in "prediction mode". In this case, back! is invalid.

To make code clearer for reviewing, it supports only Julia v0.6 now. We can add compat code later.

Any suggestions?

[denizyuret]

I will look at this for 0.8.6 release.

[ngphuoc]

I've tried this pull request again and it works for me. However I notice the memory usage increases rapidly, roughly 4 times more than normal. I think the following code snippet allocates new memory every minibatch. Any idea we could optimize this a bit (reuse allocated memory for example)?

"""
tracked_x = track(x)
track an input, `x` can be tuple, array or dict. If x is already tracked, it will be replaced with a new tape.
tracked_p = track(p, x)
track p on the same tape of x, return tracked p. If x is not tracked, it will also untrack p.
"""
track(x::AutoGrad.Rec) = track(x.value)
track(x, tape::AutoGrad.Tape=AutoGrad.Tape()) = AutoGrad.Rec(x, tape)
track(x, y) = x
track(x, y::AutoGrad.Rec) = AutoGrad.Rec(x, y.tapes[])
track(x::AutoGrad.Rec, y) = x.value
track(x::AutoGrad.Rec, y::AutoGrad.Rec) = x in y.tapes[] ? x : track(x.value, y)

[denizyuret]

Please check out issue #347.

[ylxdzsw]

The callable object approach in v1.1.0 is really nice. This one can be closed.