Balance Allocation for Multiple Cards On Device KVStore

[junranhe]

我用vggnet 11layer 训练，在总的batchsize是36，3个780gpu，conv workspace 设置为 256, cuda-7.0, 但不用cudnn（觉得可以规避一些内存分配的不确定性）， 发现占用gpu：2795m，2661m，2383m，他们100多m的递减差别是怎么产生的？希望能差别小一点，这样显存的利用率高些，毕竟显卡相同，显存利用取决于占用最大的gpu了，batchsize太小我的数据有时不收敛:(

[tqchen]

This could due to the memory allocation policy used for the distributed KVStore under mode kvstore_type = 'device'. I guess this should not be the case for kvstore_type=local

When we do device type kvstore, what is needed is we allocate the reduction temporal memory on each of the device. We do it in a random assignment way.here https://github.com/dmlc/mxnet/blob/master/src/kvstore/kvstore_device.h#L36 to balance the temp weight memory on each device.

If the weight is not uniformly distributed (e.g there is a weight that is particularly big chunk of weight), then this could cause the imbalance.

[junranhe]

是的，我是使用kvstore=device，可不可以添加一个比较确定的分配策略？例如从weight从大到小贪心，总是放到当前weight最少的设备，我觉得不少人会喜欢把显存用得满满的，如果随机的话，就有偶尔爆显存的顾虑，即使是随机在显存分配上最好也有个默认的seed，这样不用担心显存每次跑起来不一样：）

[tqchen]

This seems to be a good idea. The allocation strategy code is here https://github.com/dmlc/mxnet/blob/master/src/kvstore/kvstore_device.h#L36

Maybe you can consider hack it a bit and contribute back :) ?

[junranhe]

yes，I will

[mli]

another possible way is, do random assignment if the size < bigarray_bound_, otherwise, evenly split the array into num_dev parts, and assign one part to each device

[tqchen]

This dep checking of slice might block other slices, as we do not
distinguish between non overlapping slices. So the current way is easier
On Sat, Oct 10, 2015 at 1:30 PM Mu Li [email protected] wrote:

another possible way is, do random assignment if the size <
bigarray_bound_, otherwise, evenly split the array into num_dev parts, and
assign one part to each device

—
Reply to this email directly or view it on GitHub
#252 (comment).

[tqchen]

This issue is fixed by contribution #256 Thanks to @junranhe