[BUG] out of memory on npm install: fork bomb preparing from git repos if they have scripts

[CFT-Chris]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

npm install a package with dependencies from git.

If the package manifest from a git dependency has a relevant script for prepare (pre/post/install, prepack, prepare, or build), then npm install will be forked to install the dependency to a directory.

Because these dependencies often themselves contain similar dependencies from git, a "fork bomb" might occur where the number of concurrent npm install processes rapidly grows, usually hanging up on dependencies that may simply just take a while to download (not necessarily from git, e.g. the @angular/core library), but are dependencies used by many of the packages stored in git repos.

Anytime enough npm install processes are forked and an out of memory condition is reached, the computer generally takes much longer to finish the initial npm install as the computer is too busy swapping data between memory and swap space. CTRL+C in the command line terminal does nothing, and generally a killall to stop all npm installs is required to gain back control of the computer.

Expected Behavior

Avoid the "fork bomb", perhaps by not launching all the npm installs from git repos in parallel, so that large dependencies already extracted into directories do not get npm installed multiple times concurrently.

The propagated environment variable _PACOTE_NO_PREPARE_ in https://github.com/npm/pacote/blob/main/lib/git.js#L181 prevents circular dependencies from causing infinite "fork bombs", but practically infinite lasting "fork bombs" can still be achieved with just a handful of dependencies from git that have a reasonable number of dependencies on larger libraries.

A common scenario where this bug is encountered is having a bunch of packages for Angular components that depend on each other, all stored in and retrieved from git repos. Not only do they have devDependencies with each other if one component inherits from another, they will all share a similar set of core devDependencies such as @angular/core. All of them will likely have "build" scripts in their package.json.

Possible workaround of the problem at the moment:

Since https://github.com/npm/pacote/blob/main/lib/git.js#L164 explicitly allows us to bypass the prepare step and avoid forking npm install for packages from git repos, I have gone through all my libraries stored in and retrieved from git repos and ensured their package.json manifests do not have any build, postinstall, etc. scripts. I essentially renamed build to make, and refactored away a lot of the pre/post install scripts to avoid relying on those in the first place.

This workaround has dramatically changed the npm install times for my larger projects from hours (not an exaggeration) to a minute or two. Ideally, it would be nice to retain the build and postinstall scripts and have them participate in the normal npm lifecycles.

Steps To Reproduce

1. In package.json, create dependencies/devDependencies to packages from git (git+https://github.com/npm/pacote.git, e.g.).
2. To fully experience the "fork bomb", the git dependencies from Step 1 ideally should have dependencies on many large libraries (e.g. @angular/core, @material-icons) and perhaps dependencies other packages from git that also fit the description of this step.
3. npm install the initial package.json from Step 1.
4. View active processes using top or similar system monitoring software. Note the explosion of npm install processes and the amount of CPU and memory they hog.

Environment

• npm: 8.5.2
• Node: 16.13.1
• OS: Ubuntu 21.10

[naugtur]

The default should be not to run.

[jwbrandon]

We have this issue as well, a work-around for us was to add _PACOTE_NO_PREPARE_=true into .npmrc in the project that we are working on. Luckily we don't rely on and building during install