Huge memory usage

[RobertCsordas]

I'm not sure if it is a bug, but the following code eats up all my memory (12 gigabytes) and I don't understand why:

local threads=require 'threads'
th=threads.Threads(8);
while true do
    th:addjob(function ()
        collectgarbage()
        print("Thread: "..(collectgarbage("count")/1024).." Mbytes")
        local d=torch.rand(10000,10000) --should be 762Mbytes
        collectgarbage()    
        return d
    end)
    print("Main: "..(collectgarbage("count")/1024).." Mbytes")
    th:dojob()
    collectgarbage()
end

The amount of memory used depends on the number of threads (about 1.6Gbyte/thread). The garbage collector shows ~1 Mbytes of memory for both thread and main block, but if I check it out with htop, I can see that it eats up all the memory and gets killed by the kernel. Shouldn't be about 800Mbytes of memory enough for this code to run? Should't the dojob() free the memory required by the thread?

This is only a toy example, I want to create a loader that fetches and prepossesses data in the background, but it uses enormous amount of memory, although the useful data fits in a much smaller space.

[alimousazy]

Hi @xdever ,

I ran the same code but I haven't had the issue. for how long should I run the code ?

Regards