smalloc cascading issue?

[pflannery]

We been investigating large memory issues for the Docpad Project and whilst looking at a generated heap snapshot cia v8-profiler I've seen that smalloc seems to do a huge cascade in the graph starting at a node 24 in distance and cascading all the way down to 58000 in distance. (see screenshot below)

The heap snapshot is here for download

I'm not an expert in analysing heap snapshots but I don't recall seeing large distances like this in V8 heap snapshots.

[pflannery]

Just adding missing info from the original post, this heap was created in windows 8.1 64bit, iojs v1.6.1 and I've updated to v1.6.2 but still got similar results.

[bnoordhuis]

@chrisdickinson Is this a streams issue?

[chrisdickinson]

Interesting. I'll take a look at the heapdump in a second – a ton of queued WriteReqs indicate a situation where lots small .write's are issued to the writable stream.

[pflannery]

@chrisdickinson thanks

I've just tested with a current master build and still getting similar results.

[Fishrock123]

ping @chrisdickinson?

[Fishrock123]

Note: smalloc will be removed in io.js 4.0 / v8 4.2: #1451

[trevnorris]

Though regardless, this will probably still be an issue even with the new implementation. You're allocations are the size of the default Buffer.poolSize. This means to me that chunks of the slab are being held onto. I'd investigate all uses of Buffer#slice() in your code.

[chrisdickinson]

This is likely due to something writing to a stream that is unable to keep up with the amount of incoming data, or is otherwise paused or corked. The WriteReq objects are a linked list containing each buffer/string that was passed to the target writable's .write function while the stream was paused, corked, or working on another chunk.

Closing this for now — streams appear to be working "as intended," though that term applies a bit loosely here!

[pflannery]

thanks for feedback all. When I get time I will try and see if this is still occurring with latest iojs version