Next out of Memory issues - with a replicable example!

[royletron]

What version of Next.js are you using?

11.1.2

What version of Node.js are you using?

14.15.4

What browser are you using?

Chrome

What operating system are you using?

macOS

How are you deploying your application?

next start on Google CloudRun

Describe the Bug

When using next dev locally, we are finding across multiple machines (all macOS) that we regularly run out of memory, either through hot reloads or just lots of page loads.

Expected Behavior

It to not memout 😄

To Reproduce

Ok, so I read pretty much the entire thread over on #15855 and decided that I wouldn't bring anything up until I could reliable reproduce what I thought was the culprit. So I dug down in the inspector and found a lot of (string) info being left behind and piling up. I have now been able to recreate this outside of our private repo - albeit on a smaller scale (so no memouts - but it does highlight the issue).

I chucked it into a repo: https://github.com/royletron/next-mem-issues but really just the most basic example is required (I tried it on 11.1.3-canary.20 and the problem still persists). I'll add the instructions here too, but I also added them as a README on the repo

So the whole issue seems to stem from there being a cache of 'compiled' Javascript from the pages dir, and this cache never seeming to get invalidated/garbage collected. In this example the pages are small, so the leak is hardly noticeable - but in bigger applications this becomes a much larger issue (where a page could have 30-40mb of dev-friendly JS).

This repository is super basic, just a single page - that is handful of KBs but it does exhibit the problem

Step 1: boot

NODE_OPTIONS='--inspect' npx next dev

You'll be wanting the inspector :P

Step 2: Take an initial memory heap snap

Head to chrome://inspect and the Memory tab and take a memory Heap Snapshot at this stage nothing extraordinary. Leave this open, we will come back shortly.

Step 3: Get the homepage to recompile a few times

This one is a bit less scientific. Next has this behaviour where if it idles for a period (around 2-3 mins, though chances are you know a lot more about that than me), it decides it's going to rebuild old pages (even if those pages haven't changed). You can view this by triggering a build:

curl http://localhost:3000/

I find curl is superior as the browser has a more active connection to the server, and is harder to control. You should see:

event - build page: /
wait  - compiling...
event - compiled successfully

Now wait (don't do a thing). After a few minutes you will see the following in your log:

wait  - compiling...
event - compiled successfully

You can now run the same curl as above, and you should see it builds / again. You want to do this a few times (3-4 or more if you can stomach the wait).

Step 4: Take another memory heap snap

Head back to your inspector and get another Heap Snapshot. Open this up and change the dropdown from All Objects to Objects Allocated between Snapshot 1 and Snapshot 2. Sort the list by either allocated or retained size and open up the (string) constructor. You should find that there are several strings that are common (infact identical) - they all include ATTENTION: an 'eval-source-map'.... so they should be easy to spot. Chances are you have three or more groups of these, if you peek into the code (just hover) you will see the first group is pages/_document, the second is pages/_app, and then probably pages/index (it's v small so can sometimes fall off the list).

It's my understanding that each of the compilations that were made in Step 3 are resulting in duplicated transpiled code strings being cached on the server, and that these are causing memouts - as I say on a bigger project each of these sits at around 30-40mb each so it leaks pretty rapid until OOM errors.

_NEXT-1383

[SeanGroff]

Our team has the same specs above as @royletron. We only seem to experience this issue when using Docker.

[ijjk]

Hi, this has been updated in the latest canary of Next.js v11.1.3-canary.70, please update and give it a try!

[royletron]

@ijjk It seems to be slightly better, but still seeing a lot of duplicate strings. This screenshot is taken from our current project, and you can see some pretty meaty strings that are being duplicated.

[revskill10]

@ijjk My project got same issue with latest 12.0.1.
It's running fine with 11.1.3-canary.26
Could you please take this issue ? We can't upgrade now.

[shohinrahmonov]

Is it something that will be addressed by the team?I have found out 4 other issues similar to this one but nothing has been resolved.

[Alounv]

Each string starting with "/ * * ATTENTION... is related to a page (normal or API page).

COPY

From what I experimented, each time a change is made in the code and we reload the page, a new copy of the page is created in memory.

That would not be an issue if the page itself was small, but It can be pretty big as in your screenshot.

SIZE

In my case, I noticed that each page included all the libs packages in the repository (we use monorepo yarn workspaces for our own libs). In particular, those libs use date-fns and material-ui and together they represented 75% of the weight of each page.

It probably comes from the way the files are importing each other.

I think that because, if I take an API page and remove all imports (by copying the needed function directly inside the file), the weight of the file is drastically smaller (1000 times smaller in my case).

[remorses]

I got the same issue in production (next start in standalone mode) with Next 13, the heap size is huge (and increasing with each page navigation) and is full of strings.

The app was working fine with Next 12 and i could deploy at 512Mb of RAM, not it takes 500Mb just for the first page load

[kdath]

Each string starting with "/ * * ATTENTION... is related to a page (normal or API page).

COPY

From what I experimented, each time a change is made in the code and we reload the page, a new copy of the page is created in memory.

That would not be an issue if the page itself was small, but It can be pretty big as in your screenshot.

SIZE

In my case, I noticed that each page included all the libs packages in the repository (we use monorepo yarn workspaces for our own libs). In particular, those libs use date-fns and material-ui and together they represented 75% of the weight of each page.

It probably comes from the way the files are importing each other.

I think that because, if I take an API page and remove all imports (by copying the needed function directly inside the file), the weight of the file is drastically smaller (1000 times smaller in my case).

We're seeing exactly the same behaviour. For each change and reload, a duplicate is added to memory.

node: v14.17.3
next: v11.1.2

[timneutkens]

We're consolidating these reports into this issue, please read the issue and provide a reproduction there so that it can be investigated: #54708