zlib: do not leak on destroy #23734
zlib: do not leak on destroy #23734
Conversation
|
Do you know when this regressed? Or has it always done this? I am wondering if @nodejs/lts will need to backport it. I guess there is no way to unit test this, but a manual test in the PR that could be run by hand while watching sys memory use would help in verifying backports. |
|
@sam-github as far as I can see this have been an issue since Node 8 (previous to that there were no |
|
/to @nodejs/lts |
There was a problem hiding this comment.
Currently zlib streams will leak their handles if the streams are destroyed using
.destroy(), which means they'll leak when used withstream.pipelineorpumpand an error happens.
That seems somewhat surprising… if it’s true, then this is possibly not a fix for that bug. Even without the extra _close() call here, zlib handles should be weak when not currently performing write operations, so they should be garbage-collected along with the rest of the stream.
|
@addaleax oh interesting, so the _close call is not necessary in general? my leak stats my be off |
|
@mafintosh So… it is not strictly necessary in general, but it will release resources earlier than GC would, which is usually desirable I guess? So, if there are some long-lived references to the zlib stream, that could still be a memory leak – but the fix for that situation might be to not keep those references around, so that GC can do its work? (i.e. this PR is definitely improving things, but you may want to look out for deeper-lying issues.) There might also be a behavioural difference between v8.x and v10.x, at least currently. #21608 made memory allocation reporting for the native zlib handles more accurate, so GC may see lower or higher memory pressure than before. |
| @@ -592,6 +597,10 @@ function processCallback() { | |||
| assert(false, 'have should not go down'); | |||
| } | |||
|
|
|||
| if (self.destroyed) { | |||
There was a problem hiding this comment.
There is already a check for this a few line above, isn't that one sufficient?
There was a problem hiding this comment.
I guess the idea is that the push() call can trigger on('data') listeners which can destroy the stream?
Either way, it would be great to have a test for this (and the other bits in this PR)
There was a problem hiding this comment.
Ok, makes sense, I think a comment would also help.
There was a problem hiding this comment.
This is actually already tested as the existing tests fail (crash even!) without this.
|
@addaleax thanks for the explanation, let me see if I can cook up a test case based on that. |
|
(I’m adding |
|
Confirming what @mafintosh said. This can land. |
|
@mafintosh I'll land the async iterators fix as part of the other PR, can you remove it out from this one? |
PR-URL: #23734 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matteo Collina <[email protected]>
PR-URL: #23734 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matteo Collina <[email protected]>
MylesBorins
added
the
baking-for-lts
|
This lands cleanly on 10.x but requires a backport for 8.x Is it ready to land in LTS yet or does it need a bit more time to bake? |
MylesBorins
removed
the
baking-for-lts
|
This is ready, but it will need a proper backport. |
PR-URL: #23734 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matteo Collina <[email protected]>
PR-URL: #23734 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Matteo Collina <[email protected]>
Checklist
make -j4 test(UNIX), orvcbuild test(Windows) passesCurrently zlib streams will leak their handles if the streams are destroyed using
.destroy(), which means they'll leak when used withstream.pipelineorpumpand an error happens.This PR fixes that by implementing
_destroyand call_closeon the handle always.