Profiling memory usage

[ViralBShah]

In a couple of talks I gave recently, the question of profiling memory usage came up. That got me thinking of something that could be very useful if it existed. What if we also capture the total memory usage when the sampling profiler runs, and say also the top 5 objects by memory consumption?

This data can then be analyzed and plotted to get a very good idea of the memory behaviour of the program. I know there are a lot of ifs and buts, but I thought I would post it here to see what others things.

Cc: @timholy

[ViralBShah]

I always wanted something like this when I used Matlab. My general strategy has been to keep increasing the problem size until the program crashes and hopefully tells me where in the code it ran out of memory.

[aviks]

For useful memory profiling, possibly the simplest thing would be to sample both the number of objects, and total memory allocated for the top N objects at each sampling. This information can then be used to build further visualisation and analytical tools.

[juhljr]

@ViralBShah Just lurking around in the issue tracker. MATLAB can profile memory with profile on -memory and then do profile viewer. It's pretty neat.

[ViralBShah]

@juhljr Thanks. I did not know about the memory profiler in matlab. Fortunately, I no longer have access to matlab - but I would have loved to try it out!

[timholy]

This was closed by #7464.