Memory usage for multi-label classification problem

[Gazillionth]

Hi,

I am trying to train using 1M training examples with 8600 labels per example on a g2.2xlarge (15 GB of system memory). The code crashes with and out-of-memory error (std::bad_alloc) when creating the mx.io.ImageRecordIter for the training data. I am initializing the mx.io.ImageRecordIter with both the recordio and image list files as follows:

train = mx.io.ImageRecordIter(
path_imgrec        = "/data/train.rec",
path_imglist       = "/data/train.lst",
label_width        = 8600,
...
)

Does MXNet store the training labels in memory, or if not, how do the CPU memory requirements scale with number of labels and number of training examples? (My code works ok with 100k training examples.)

Thanks,

G.

[piiswrong]

@tqchen We should tune down the default prefetching buffer size

[Gazillionth]

@piiswrong I should have noted that I have already changed the number of prefetched minibatches from 4 -> 1 on this line: https://github.com/dmlc/mxnet/blob/master/src/io/iter_image_recordio.cc#L325 and recompiled. This did not fix the out-of-memory error.

[diPDew]

@piiswrong @tqchen It seems when passing path_imglist, all of the labels will be loaded once (according to ImageLabelMap in ImageRecordIter). In this case, for OP's problem (with 8600 labels), the RAM consumption is gonna be huge, if PO is using millions of training examples.

Better to consider using a data iter for labels?

[martinbel]

I'm getting similar issues with a EC2 instance, using prefetch_buffer=1.

[diPDew]

@martinbel , my experience is that the labels are loaded at once into RAM (I've got >100GB labels to load..). I'm actually using g2.8xlarge which only has 60GB RAM.

[martinbel]

@EasonD3 You mean, your training set has 100GB of images? I'm not sure what you mean by "labels". I've converted the images to the .rec files as shown in the docs. The .rec files (compresed) are small and fit in my RAM, the raw images wouldn't for the instance I was using. I'm getting the same in my own computer, I need to kill it after 5 seconds.
The library seems promising, but I just wasn't able to make it work with my data, which doesn't make it much useful for me (as I'm not using the benchmark datasets).

[diPDew]

@martinbel , sorry about the confusion. My problem is very similar to the one stated in the original post, where each image is associated with hundreds of thousands of labels/classes (i.e., multi-label problem). If you feed path_imglist with the labels of all images in ImageRecordIter, all of the labels will be loaded for one time into RAM before the training starts. So in my case, the label file is too huge to load them directly in ImageRecordIter.

[martinbel]

@EasonD3 In my case the label's file isn't huge, but I get the same error. I guess something isn't working well with the mx.io.ImageRecordIter. I don't have so much data, but it crashes. I also don't see how a labels file can get to have 100 GB, that's insane!

[zhanghan328]

#4299 Maybe, i have the same problem.

[ysh329]

Same problem.
#5525

[yajiedesign]

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!