Suspect memory leak in v6.10.1 #12033

stefanosala · 2017-03-25T08:58:25Z

Version: v6.10.1
Platform: Linux f227befc-7ce9-41a4-9b09-363eac11b499 3.13.0-112-generic Added DS_Store to gitignore #159-Ubuntu SMP Fri Mar 3 15:26:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Subsystem:

Hi there, we noticed a weird behaviour after upgrading to v6.10.1 from v6.10.0. Basically we see memory accumulating until Heroku restarts the process.

You can better notice the behaviour in this graph:

v1314 is the upgrade to v6.10.1, v1315 is the rollback to v6.10.0.

We start the process with this command: NODE_ICU_DATA=node_modules/full-icu node --max-old-space-size=900 lib/processes/oneInstance.js.

Help? :) Please let me know if and how we can provide more useful info.

Thanks a lot.
Stefano

The text was updated successfully, but these errors were encountered:

stefanosala · 2017-03-25T08:59:52Z

We had a similar behaviour when we tried node v7, I'm not sure if that helps :)

cjihrig · 2017-03-25T13:05:15Z

@stefanosala do you have any code you can share?

stefanosala · 2017-03-25T14:15:19Z

@cjihrig we haven't isolated the issue yet, I'm sorry :(

joyeecheung · 2017-03-25T14:36:07Z

You mentioned this behavior also presents in v7, can you provide the specific version of that release? Cross comparing it with #11759 we might be able to find the related commit.

lpinca · 2017-03-25T16:17:10Z

Possibly related: #12019.

stefanosala · 2017-03-25T16:43:24Z

Yep, we had the same issue with v7.6.0.

SuperTux88 · 2017-03-26T17:36:51Z

I have the same behaviour after upgrading to 6.10.1 with camo, started with this command: /usr/bin/node /srv/camo/node_modules/.bin/coffee server.coffee.

dvfeinblum · 2017-03-27T14:59:41Z

I also see similar issues. My child processes are utilizing more ram than they are provisioned (using the --max_old_space_size flag) and I'm seeing a linear increase in memory usage.

Edit: Bit more information. I am using child_process.exec() to fire off my workers and switching to 6.9 has fixed the problem. Memory is no longer increasing linearly and I'm only using 15gb as opposed to maxing out the 30 I have at my disposal. Scripts also seem to be respecting the 1.5 gb I'm giving each of them now.

kmikiy · 2017-03-28T06:23:38Z

I've also had the same issue
6.10.0 works fine

6.10.1 starts leaking

7.5.0 also works fine

7.6.0 and up starts leaking

Same code was running all 4 times, only changed the node version.

joyeecheung · 2017-03-28T07:20:58Z

Excluding benchmark, doc, meta, test, tools and build, these are the PRs both added in v6.10.1 and v7.6.0 (obtained by scripting the OP of #11185 and #11759, I might be missing something though..)

crypto: Remove expired certs from CNNIC whitelist Distrust certs issued after 00:00:00 Oct. 21, 2016 by StartCom and WoSign #9469
crypto: add cert check issued by StartCom/WoSign Distrust certs issued after 00:00:00 Oct. 21, 2016 by StartCom and WoSign #9469
dgram: remove this aliases dgram: remove this aliases #11243
lib: replace \u2019 with regular ascii quote src: support UTF-8 in compiled-in JS source files #11129
readline: update 6 comparions to strict readline: update 6 comparisons to strict #11078
src: support UTF-8 in compiled-in JS source files src: support UTF-8 in compiled-in JS source files #11129
src: fix delete operator on vm context src: fix delete operator on vm context #11266
src: unconsume stream fix in internal http impl src: unconsume stream fix #11015
stream: move legacy to lib/internal dir stream: move legacy to lib/internal dir #8197
timer: remove duplicated word in comment remove duplicated word in comment #11323

MylesBorins · 2017-03-28T09:24:15Z

@joyeecheung thank you so much for compiling that list

@nodejs/lts @nodejs/collaborators do any of the above prs stand out to you?

addaleax · 2017-03-28T11:03:55Z

src: unconsume stream fix in internal http impl #11015

Seems like the most likely candidate from the list by far … /cc @Kasher @jasnell @indutny

addaleax · 2017-03-28T11:18:03Z

Maybe strike that, I’ve just seen #12089, that looks like a good candidate for fixing the leak.

joegoldbeck · 2017-03-28T14:29:06Z

We ran into a memory leak in 4.8.1 (was not there in 4.8.0). These may be related.

shigeki · 2017-03-28T16:27:15Z

I confirmed that #12089 fixes a memory leak bug and I'm sure that it resolves this issue.

MylesBorins · 2017-03-28T17:42:07Z

As it would appear that the memory leak is also appear on v4.x I've gone ahead and made a list of commits that appears in v4.8.1, v6.10.1, and v7.6.0. It is much smaller

crypto: Remove expired certs from CNNIC whitelist (Shigeki Ohtsu) Distrust certs issued after 00:00:00 Oct. 21, 2016 by StartCom and WoSign #9469
crypto: add cert check issued by StartCom/WoSign (Shigeki Ohtsu) Distrust certs issued after 00:00:00 Oct. 21, 2016 by StartCom and WoSign #9469
~~readline: update 6 comparions to strict (Umair Ishaq) readline: update 6 comparisons to strict #11078~~ not likely to cause leak
src: unconsume stream fix in internal http impl (Roee Kasher) src: unconsume stream fix #11015

edit: @nodejs/streams do you think #11015 could have caused a leak?

shigeki · 2017-03-28T18:26:30Z

TLS and HTTP client memory leaks were fixed in a6f9494.
I'm sure it fixes this. Please test it.

I apology my bug in my commit and thanks for everyone helping and identifying the issues.

stefanosala · 2017-03-28T19:41:37Z

@shigeki thanks a lot and thanks to everybody else involved, we appreciate a lot! :)

I'll test the new release asap.

dvfeinblum · 2017-03-28T20:57:29Z

Yeah, thanks a bunch for fixing this! We'll give this patch a go next time we run our job.

The additional validity checks applied to StartCom and WoSign certificates failed to free memory before returning. Refs: #9469 Fixes: #12033 PR-URL: #12089 Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Fedor Indutny <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Myles Borins <[email protected]> Reviewed-By: Shigeki Ohtsu <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

stefanosala · 2017-03-29T10:34:49Z

Stats are looking good with v7.8.0, thanks again!

The additional validity checks applied to StartCom and WoSign certificates failed to free memory before returning. Refs: #9469 Fixes: #12033 PR-URL: #12089 Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Fedor Indutny <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Myles Borins <[email protected]> Reviewed-By: Shigeki Ohtsu <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

The additional validity checks applied to StartCom and WoSign certificates failed to free memory before returning. Refs: nodejs/node#9469 Fixes: nodejs/node#12033 PR-URL: nodejs/node#12089 Reviewed-By: Sam Roberts <[email protected]> Reviewed-By: Fedor Indutny <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Myles Borins <[email protected]> Reviewed-By: Shigeki Ohtsu <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

vsemozhetbyt added the memory Issues and PRs related to the memory management or memory footprint. label Mar 25, 2017

mscdex added the v6.x label Mar 25, 2017

joyeecheung added the v7.x label Mar 28, 2017

addaleax mentioned this issue Mar 28, 2017

crypto: fix memory leaks in cert validation #12089

Closed

2 tasks

MylesBorins added the v4.x label Mar 28, 2017

shigeki closed this as completed in a6f9494 Mar 28, 2017

MylesBorins mentioned this issue Mar 28, 2017

Regressions in v4.8.1 and v6.10.1 nodejs/Release#193

Closed

satazor mentioned this issue Mar 29, 2017

Investigate why re-metadata task terminates with out of memory npms-io/npms-analyzer#145

Open

gibfahn mentioned this issue Mar 31, 2017

Node v7.6.0+ memory leak #12019

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Suspect memory leak in v6.10.1 #12033

Suspect memory leak in v6.10.1 #12033

stefanosala commented Mar 25, 2017

stefanosala commented Mar 25, 2017

cjihrig commented Mar 25, 2017

stefanosala commented Mar 25, 2017

joyeecheung commented Mar 25, 2017 •

edited

Loading

lpinca commented Mar 25, 2017 •

edited

Loading

stefanosala commented Mar 25, 2017

SuperTux88 commented Mar 26, 2017 •

edited

Loading

dvfeinblum commented Mar 27, 2017 •

edited

Loading

kmikiy commented Mar 28, 2017

joyeecheung commented Mar 28, 2017

MylesBorins commented Mar 28, 2017

addaleax commented Mar 28, 2017

addaleax commented Mar 28, 2017

joegoldbeck commented Mar 28, 2017 •

edited

Loading

shigeki commented Mar 28, 2017

MylesBorins commented Mar 28, 2017 •

edited

Loading

shigeki commented Mar 28, 2017

stefanosala commented Mar 28, 2017 •

edited

Loading

dvfeinblum commented Mar 28, 2017

stefanosala commented Mar 29, 2017

Suspect memory leak in v6.10.1 #12033

Suspect memory leak in v6.10.1 #12033

Comments

stefanosala commented Mar 25, 2017

stefanosala commented Mar 25, 2017

cjihrig commented Mar 25, 2017

stefanosala commented Mar 25, 2017

joyeecheung commented Mar 25, 2017 • edited Loading

lpinca commented Mar 25, 2017 • edited Loading

stefanosala commented Mar 25, 2017

SuperTux88 commented Mar 26, 2017 • edited Loading

dvfeinblum commented Mar 27, 2017 • edited Loading

kmikiy commented Mar 28, 2017

joyeecheung commented Mar 28, 2017

MylesBorins commented Mar 28, 2017

addaleax commented Mar 28, 2017

addaleax commented Mar 28, 2017

joegoldbeck commented Mar 28, 2017 • edited Loading

shigeki commented Mar 28, 2017

MylesBorins commented Mar 28, 2017 • edited Loading

shigeki commented Mar 28, 2017

stefanosala commented Mar 28, 2017 • edited Loading

dvfeinblum commented Mar 28, 2017

stefanosala commented Mar 29, 2017

joyeecheung commented Mar 25, 2017 •

edited

Loading

lpinca commented Mar 25, 2017 •

edited

Loading

SuperTux88 commented Mar 26, 2017 •

edited

Loading

dvfeinblum commented Mar 27, 2017 •

edited

Loading

joegoldbeck commented Mar 28, 2017 •

edited

Loading

MylesBorins commented Mar 28, 2017 •

edited

Loading

stefanosala commented Mar 28, 2017 •

edited

Loading