Memory

[WangZhouTao]

hello, I am a student.recently,I am reading the HED paper. I found that your code is very interesting,but some error is happened when I try to train the model with my own data.Error message show that Memory error.My computer in-build 32GB memory, would you give me some suggestion.Think you very much.

[CangHaiQingYue]

my GPU only has 6G memory which is fine to run the code. dont know your setting

[WangZhouTao]

Sorry,I forgot to tell you.I use 1080ti GPU and 32GB memory.I can run your code with my own data.But Memory error is happened when the iteration is about 13000.I obverse top command in ubuntu, i find Memory is insufficient（not gpu memory, is computer memory).If you could tell me some suggestion,I really appreciate what you’ve done。

[CangHaiQingYue]

Sorry, I have not had such error. This code will save a checkpoint each 100 steps, maybe 13000 is too big in your dataset( I trained at 30000 steps without memory error).

[WangZhouTao]

I already changed to save a checkpoint with each 1000 steps, but the error is still happen.Did you run this code on windows? Thank you.

[CangHaiQingYue]

I run it on Ubuntu 16.04.

[sandhawalia]

Could you please give details of your data-set ? It might be the case that your image sizes are quite large and since the VGG base model is full convolutional the intermediate representations overflow they GPU memory.

[wutachiang]

I also encountered the same problem with you, my server configuration is 8 Tesla p20, memory is 512G. Even with this configuration, memory error occurs after approximately 13,000 iterations during training. Will you solve the problem?@CangHaiQingYue

[CangHaiQingYue]

@Jasontachiangwu Well, when I cancelled the 'summary_write' operator, this problem was gone.
So I guess there maybe some bug in 'summary'. You may rebuild your own code.

[wutachiang]

@CangHaiQingYue，After I updated the tensorflow version to r1.8, no problem was found after training. It may be that there is a bug in the summary.

[yuezhilanyi]

with same code on windows 10 and ubuntu 16.04, tensorflow v1.4.0, 20000 iterations:
memory error occurs in ubuntu 16.04
while no error occurs in windows 10