Jest memory problems in node environment #6399
Comments
|
Maybe unrelated, but please make sure you have |
|
It would be super helpful if you were able to create a reproduction showing a leak (even if it doesn't leak enough to crash) |
|
Just tried with babel-jest 23 and that doesn't seem to fix it. |
|
Sure, but a reduced case would help. As is, this report isn't actionable for us |
|
I'm working on a reduced case of the problem, will get back to you when/if needed |
|
Ok, turns out the problem comes from us importing fs-extra in one of our model. We use the latest version (6.0.1). Just commenting the import fixes the memory going up between test suites. |
|
Here is a quick repro if you want: https://github.com/mrrinot/fs-extra-jest-repro |
|
Can you run with |
|
All 27 tests leak: Unfortunately this isn't Jest related |
|
Might be worth opening up an issue with (note that we use it internally, so maybe it's installed multiple times or something?) |
|
I updated the one I referenced above but the project doesn't seem to get a lot of attention. |
|
It's pretty much required for windows, unfortunately. |
|
@rickhanlonii, I'm also experiencing this bug, and I do suspect it's a bug with jest's module registry. Jest does not cache modules as node's Created a repo to illustrate this issue. |
|
Interesting find! It's something that Webpack does though - removing that import (but keeping the 2 others) keeps the memory usage stable. So I don't think it's an issue with Jest in general, but Webpack (or one of its dependents) is doing something that leaks. (note that |
|
Not sure if it matters, but doing heap snapshots between runs shows a ton of |
|
Another thing there's a lot of is a |
|
@SimenB good catch, no leak when the only dep is a huge json :) Our heap snapshot diffs also show lots of |
|
Using |
|
@SimenB yeah but
https://github.com/lev-kazakov/jest-leak/tree/ajv |
|
Jest has built in |
|
it's |
|
I think I've sanitised a test case that proves The following repo illustrates the issue: If you run Note that I'm using the |
|
@lev-kazakov @mrrinot Fixed the issue on our project by replacing const fs = require('fs');
module.exports = Object.assign(fs, { gracefulify: () => {} });We put this inside It seems that graceful is only useful when you are on Windows. Since we don't use Windows at all, we don't need it. |
|
I can confirm that @Telokis' workaround improves the situation but for me it looks like something is still leaking - but not as much as before. Have to check more thoroughly. PASS tests/articles-put.test.js (8.164s, 214 MB heap size)
PASS tests/articles-post.test.js (6.911s, 389 MB heap size)
PASS tests/articles-get-microdata.test.js (498 MB heap size)
PASS tests/articles-get.test.js (593 MB heap size)
PASS tests/articles-get-feed.test.js (688 MB heap size)
PASS tests/articles-get-locations.test.js (769 MB heap size)
PASS tests/articles-get-detail.test.js (851 MB heap size)
PASS tests/articles-delete.test.js (933 MB heap size)compared to PASS tests/articles-put.test.js (9.133s, 78 MB heap size)
PASS tests/articles-post.test.js (6.82s, 111 MB heap size)
PASS tests/articles-get-microdata.test.js (144 MB heap size)
PASS tests/articles-delete.test.js (174 MB heap size)
PASS tests/articles-get-feed.test.js (205 MB heap size)
PASS tests/articles-get.test.js (235 MB heap size)
PASS tests/articles-get-locations.test.js (266 MB heap size)
PASS tests/articles-get-detail.test.js (296 MB heap size) |
|
@pkyeck Yup, we had other leaks somewhere but nothing as bad as this one. By adding some |
|
My go to technique was to comment everything except for import/requires, duplicate the test 5-6 times, and test your files. |
|
@SimenB, If you run Memory leaks when you Libraries that do this kind of hacks:
Would you like me to open a new ticket? |
|
Interesting find! Core module as just required normally. Odd that they're not GCed, though. @cpojer @aaronabramov you probably have some more context here. Anything we can do? |
#459) Jest with webpack has known memory leaks, see jestjs/jest#6399 for details. To put a bandaid on this issue, we run only two integration tests at once in the CI so that we don't run out of memory and still can add more integration tests later (vs. increasing max_old_space_size).
|
It doesn't seem that I see a comment in |
This may fix memory leak running Jest tests. See jestjs/jest#6399 (comment) Signed-off-by: Eric Dobbertin <[email protected]>
This may fix memory leak running Jest tests. See jestjs/jest#6399 (comment) Signed-off-by: Eric Dobbertin <[email protected]>
graceful-fs and jest does not like each other, so this removes the graceful-fs implementation before running tests and adds it back afterwards. jestjs/jest#6399 jestjs/jest#6814 isaacs/node-graceful-fs#102
|
@Telokis @lev-kazakov I have tried but helped a lot. |
|
@seyfer you can run your test suite against #8331. |
|
Guys, check if you have properly setup paths where JEST is scanning files for coverage. I've just discovered it goes to "dist" folder and this is causing memory looping - after excluding it everything works like a charm again. |
|
it's just a warning, it does not break the test suite. |
|
@lev-kazakov Thanks. I removed the --verbose flag but the memory leak is still present.. |
|
I have a crash report, not sure if this help |
|
try with latest version of jest |
|
This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days. |
|
Issue with |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
💬 Questions and Help
Stack:
1. Preface
We recently switched all our API's test from Mocha to Jest.
We have around 90 tests, half of them require to run synchronously due to them using our testing database (running seeds between them), so we have to use --runInBand.
I unfortunately can't share my code as it is private.
2. The problem
Running tests one by one was fine, I then tried to run all of them at once and things went bad.
With the --logHeapUsage, it seems context memory isn't GC'd resulting in a Javascript heap out of memory.
I tried using the new option --detectOpenHandles to see what could prevent the GC to work but this is what came out:
I have around 6-8 of these, and no clue whatsoever of where to look.
I searched around and found out it was most likely the database connection so I added these as a global teardown:
This didn't change much, memory still goes up very quickly (30-40 MB per test).
In the end, I wrote a small file launching jest multiple times to avoid memory problems and stitching coverage report together but this isn't ideal.
The text was updated successfully, but these errors were encountered: