Loss computed in numpy cause performance degradation

[ThomasDelteil]

In a lot of example notebooks, you use:

curr_loss = nd.mean(loss).asscalar()

which is the equivalent of curr_loss = nd.mean(loss).asnumpy()[0]

https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.asscalar

My tests give 2-10% performance improvements when doing the .asscalar() on the moving loss at the end of the epoch rather than in every batch. (When commenting out the computation of the test and training accuracy, which we know use numpy...)

Related to this issue: apache/mxnet#9571 which you opened @zackchase 😄

To precise what I mean, calling .asnumpy(), calls a wait on read on the prediction which blocks the next batch from being loaded on the GPU.