Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

webpack ng serve runs out of memory #1652

Closed
slubowsky opened this issue Aug 11, 2016 · 107 comments
Closed

webpack ng serve runs out of memory #1652

slubowsky opened this issue Aug 11, 2016 · 107 comments
Labels
effort2: medium (days) P3 An issue that is relevant to core functions, but does not impede progress. Important, but not urgent

Comments

@slubowsky
Copy link

slubowsky commented Aug 11, 2016

Please provide us with the following information:

  1. OS? Windows 7, 8 or 10. Linux (which distribution). Mac OSX (Yosemite? El Capitan?)
    Windows 10
  2. Versions. Please run ng --version. If there's nothing outputted, please run
    in a Terminal: node --version and paste the result here:
    angular-cli: 1.0.0-beta.11-webpack.2
    node: 6.3.1
    os: win32 x64
  3. Repro steps. Was this an app that wasn't created using the CLI? What change did you
    do on your code? etc.
    Keep ng serve running and serving a cli built app while coding for some time (a few hours?)
  4. The log given by the failure. Normally this include a stack trace and some
    more information.
...
 94% asset optimization
<--- Last few GCs --->

12936118 ms: Mark-sweep 1360.3 (1435.0) -> 1356.1 (1435.0) MB, 959.3 / 0 ms [all
ocation failure] [GC in old space requested].
12936973 ms: Mark-sweep 1356.1 (1435.0) -> 1356.0 (1435.0) MB, 854.9 / 0 ms [all
ocation failure] [GC in old space requested].
12938085 ms: Mark-sweep 1356.0 (1435.0) -> 1356.0 (1435.0) MB, 1112.2 / 0 ms [la
st resort gc].
12939012 ms: Mark-sweep 1356.0 (1435.0) -> 1355.8 (1435.0) MB, 926.7 / 0 ms [las
t resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000001F1975C9E51 <JS Object>
    1: replace [native string.js:~129] [pc=0000018EE86B74E5] (this=000000EF14282
331 <String[61]: D:/it/node_modules/@angular/common/src/pipes/common_pipes.js>,
N=000000EF140824C9 <JS RegExp>,O=00000135F03D7F21 <String[10]: !(webpack)>)
    2: shorten [D:\it\node_modules\webpack\lib\RequestShortener.js:~41] [pc=000
0018EE4290DA2] (this=0000020E88D58019 <a RequestShortener with map 00000313C3C4.
..

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memo
ry
  1. Mention any other details that might be useful.

Only happened once so far but thought it worth mentioning


Thanks! We'll be in touch soon.

@kylecordes
Copy link

I am thinking about a good way to test this. I already have a little stress test program that generates a working program with a given number of components. But unfortunately it also generates the exact same program each time. So I'm thinking about making it slightly randomize the contents of the programs it generates so that it generates changed files each time. Then run it in a loop and see how much RAM leaks.

@kylecordes
Copy link

OK, I made a way to test this. I enhanced the "angular 2 stress test" tool:

https://www.npmjs.com/package/angular2-stress-test

To stress test "ng serve":

  • install angular2-stress-test
  • ng new test-project
  • cd test-project
  • ng serve
  • leave "serve" running, and make a second command window to work in
  • cd src/app
  • for i in seq 1 1000; do angular2-stress-test 500 -r ; sleep 5 ; done
  • While that runs, keep an eye on RAM and CPU use the ng serve Node process

In my testing as above:

  • Starting up the first time, ng serve uses 463MB of RAM
  • After running for a little while, it was over 1000MB of RAM

This shows that ng serve keep using more RAM over time, as it keeps recompiling after each change. However, offhand I would guess that reducing this increase, is pretty far down the priority list of the CLI team!

@kylecordes
Copy link

I'm interested in whether anyone plays with this, with various settings, modes (-prod ?), platforms (windows linux mac) etc.

@TheLarkInn
Copy link
Member

TheLarkInn commented Aug 13, 2016

@kylecordes I believe that this could be related to webpack-dev-server but I'm not 100% sure where to look specifically. One thing that comes to mind is that in situations where webpack is running in its dev server, everything compiled is cached because it is an in-memory file system. So any convention that is using a hash in its name is going to be stored in memory until the end of time. See this explanation here.

So I think the first thing I will need to do is find instances of us in our codebase using the webpack [hash], [chunkhash], or [id] properties inside of our filenames for the ng serve and default --target='production' target.

@TheLarkInn
Copy link
Member

I think that if this is happening on just plain default ng serve this is definitely a bug, however it is essentially unavoidable in --prod target.

We separated the idea of --prod and --dev targets but made them agnostic to and unmodifiedng serve task/command so people would have a more realistic local production experience.

I think in terms of --prod we need to simply document the long term use affects.

@TheLarkInn TheLarkInn added cla: no effort1: easy (hours) P5 The team acknowledges the request but does not plan to address it, it remains open for discussion effort2: medium (days) and removed cla: no effort1: easy (hours) labels Aug 13, 2016
@TheLarkInn
Copy link
Member

@slubowsky @kylecordes I did a little hands on memory profiling and heapshot analysis. (I couldn't do it locally so @hansl helped me).

Long story short things were either pointing towards the Compilation or some TS Services retaining some objects after Garbage collection was occuring.

Luckily we just merged in a kind contribution in webpack that fixes a V8 bug causing memory leaks in the webpack compilation. I believe that this is what is causing the issue, however I think it warrents still following up after we cut a release (on the webpack team), and update it here.

https://github.com/webpack/webpack/pull/2497/files

@kylecordes
Copy link

@TheLarkInn Thanks, sounds good. I will repeat the tests I described earlier in the thread, when the next new release comes out... or for that matter anyone else could try it also.

@hansl hansl added this to the 1.0-final milestone Sep 19, 2016
@mithun-daa
Copy link
Contributor

mithun-daa commented Sep 21, 2016

Any word on this? I just migrated from beta.10 to beta.15 and I am unable to ng build. Get a similar error.

70% building modules 1345/1345 modules 0 active
<--- Last few GCs --->

  317945 ms: Mark-sweep 703.9 (837.9) -> 701.4 (811.9) MB, 331.3 / 0 ms [allocation failure] [GC in old space requested].
  318296 ms: Mark-sweep 701.4 (811.9) -> 701.4 (790.9) MB, 350.5 / 0 ms [allocation failure] [GC in old space requested].
  318730 ms: Mark-sweep 701.4 (790.9) -> 698.0 (760.9) MB, 433.7 / 0 ms [last resort gc].
  319058 ms: Mark-sweep 698.0 (760.9) -> 692.7 (751.9) MB, 328.7 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 00000298510373A9 <JS Object>
    1: /* anonymous */(aka /* anonymous */) [D:\dev\cobalt_wp\node_modules\webpack\lib\FlagDependencyExportsPlugin.js:77] [pc=0000026F721B51D6] (this=0000029851004131 <undefined>,dep=00000150FC6162C9 <a NormalModule with map 0000025741730C01>)
    2: arguments adaptor frame: 3->1
    3: InnerArrayForEach(aka InnerArrayForEach) [native array.js:~924] [pc=0000026F71EE3DCD] (this=000002985100413...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

Not sure if this has anything to do with it but I also got the following error while npm i process-nextick-args util-deprecate (#1930)

> [email protected] install D:\dev\cobalt_wp\node_modules\node-zopfli
> node-pre-gyp install --fallback-to-build

node-pre-gyp ERR! Tried to download: https://node-zopfli.s3.amazonaws.com/Release/zopfli-v1.4.0-node-v46-win32-x64.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v46 ABI) (falling back to source compile with node-gyp)
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
TRACKER : error TRK0005: Failed to locate: "CL.exe". The system cannot find the file specified. [D:\dev\cobalt_wp\node_modules\node-z opfli\build\zopfli.vcxproj]


gyp ERR! build error
gyp ERR! stack Error: `C:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (D:\dev\cobalt_wp\node_modules\node-gyp\lib\build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Windows_NT 10.0.14393
gyp ERR! command "C:\\Program Files (x86)\\Nodist\\v-x64\\nodev4.2.3\\node.exe" "D:\\dev\\cobalt_wp\\node_modules\\node-gyp\\bin\\node-gyp.js" "build" "--fallback-to-build" "--module=D:\\dev\\cobalt_wp\\node_modules\\node-zopfli\\lib\\binding\\node-v46-win32-x64\\zopfli.node" "--module_name=zopfli" "--module_path=D:\\dev\\cobalt_wp\\node_modules\\node-zopfli\\lib\\binding\\node-v46-win32-x64"
gyp ERR! cwd D:\dev\cobalt_wp\node_modules\node-zopfli
gyp ERR! node -v v4.2.3
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'C:\Program Files (x86)\Nodist\v-x64\nodev4.2.3\node.exe D:\dev\cobalt_wp\node_modules\node-gyp\bin\node-gyp.js build --fallback-to-build --module=D:\dev\cobalt_wp\node_modules\node-zopfli\lib\binding\node-v46-win32-x64\zopfli.node --module_name=zopfli --module_path=D:\dev\cobalt_wp\node_modules\node-zopfli\lib\binding\node-v46-win32-x64' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (D:\dev\cobalt_wp\node_modules\node-pre-gyp\lib\util\compile.js:83:29)
node-pre-gyp ERR! stack     at emitTwo (events.js:87:13)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:818:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
node-pre-gyp ERR! System Windows_NT 10.0.14393
node-pre-gyp ERR! command "C:\\Program Files (x86)\\Nodist\\v-x64\\nodev4.2.3\\node.exe" "D:\\dev\\cobalt_wp\\node_modules\\node-pre-gyp\\bin\\node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd D:\dev\cobalt_wp\node_modules\node-zopfli
node-pre-gyp ERR! node -v v4.2.3
node-pre-gyp ERR! node-pre-gyp -v v0.6.30
node-pre-gyp ERR! not ok
Failed to execute 'C:\Program Files (x86)\Nodist\v-x64\nodev4.2.3\node.exe D:\dev\cobalt_wp\node_modules\node-gyp\bin\node-gyp.js build --fallback-to-build --module=D:\dev\cobalt_wp\node_modules\node-zopfli\lib\binding\node-v46-win32-x64\zopfli.node --module_name=zopfli --module_path=D:\dev\cobalt_wp\node_modules\node-zopfli\lib\binding\node-v46-win32-x64' (1)
npm WARN install:[email protected] [email protected] install: `node-pre-gyp install --fallback-to-build`
npm WARN install:[email protected] Exit status 1
[email protected] D:\dev\cobalt_wp
├─┬ [email protected]
│ └─┬ [email protected]
│   └─┬ [email protected]
│     ├─┬ [email protected]
│     │ └─┬ [email protected]
│     │   └─┬ [email protected]
│     │     └── [email protected]
│     └─┬ [email protected]
│       └─┬ [email protected]
│         └─┬ [email protected]
│           └── [email protected]
├── [email protected]
└── [email protected]

I am on node 4.x and unable to proceed.

@mithun-daa
Copy link
Contributor

mithun-daa commented Sep 26, 2016

@TheLarkInn @kylecordes @hansl Any help would be appreciated.

@kylecordes
Copy link

@mithun-daa We have a team of people using CLI-webpack-edition here, and have helped numerous others get it up and running. We have noticed that it consumes quite a bit of memory which grows over time - the topic of this issue, but when we run on good development hardware it is very little practical problem. We just end up restarting CLI every couple of hours, no big deal. I think maybe the error you experience might be unrelated to the general issue here of leakage over time. A quick way to check and help provide more information for the CLI team to work from, is to simply try the exact same thing again on a machine with abundant RAM.

@mithun-daa
Copy link
Contributor

mithun-daa commented Sep 26, 2016

@kylecordes I wouldn't call my dev machine weak. These specs, I guess should be good enough:

image

@kylecordes
Copy link

Right, that is obviously more than ample RAM. So I don't think the problem you are experiencing has anything to do with the somewhat exaggerated RAM usage and leakage over time that CLI+webpack currently experiences. I think you there some other problem in your project.

I suggest making a copy of your project, then incrementally trim away nearly all of it until you have a minimum reproduction case for the failure, something which you can publish to a public repository. That will then be a solid bug report which it is likely someone could reproduce and fix. Or you might find that by trimming away pieces you eventually find the bit which triggers this problem, also a great outcome. Good luck.

@hansl
Copy link
Contributor

hansl commented Sep 26, 2016

@mithun-daa You can raise the heap size usingnode --max_old_space_size=2048 ./node_modules/.bin/ng build. We (and the webpack team) are aware of some memory problems and @TheLarkInn is the one looking into it.

@mithun-daa
Copy link
Contributor

@hansl will this work on Windows? Get the following error:

case `uname` in
^^^^

SyntaxError: Unexpected token case
    at exports.runInThisContext (vm.js:53:16)
    at Module._compile (module.js:414:25)
    at Object.Module._extensions..js (module.js:442:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:311:12)
    at Function.Module.runMain (module.js:467:10)
    at startup (node.js:136:18)
    at node.js:963:3

@hansl
Copy link
Contributor

hansl commented Sep 26, 2016

@mithun-daa Oh on windows you have to replace node_modules/.bin/ng by node_modules\.bin\ng :)

@mithun-daa
Copy link
Contributor

i did try with the \ instead of / with the same results.

@hansl
Copy link
Contributor

hansl commented Sep 26, 2016

mmmh, then I'm not sure. It seems like this is a problem with Node that using a full path doesn't work. On my VM using node node_modules\.bin\ng does not work.

@mithun-daa
Copy link
Contributor

Are there any prereqs? I am on Node 4.2.3. The global version of TS is different from the local version.

@hansl
Copy link
Contributor

hansl commented Sep 26, 2016

Not really. Node 4.2 should work. I'm seeing a different error on my side but similar reason, with Node 6 on Windows 10.

@mithun-daa
Copy link
Contributor

@hansl If i create a new project using ng new everything works fine. but my existing app is massive and would like to avoid starting from scratch. I followed the directions in the Upgrade docs (beta.10 - beta.15).

@danishin
Copy link

@xnnkmd It's about 10K LOC of .ts files. Not a big project at all. I realized that in tsconfig.json, module is set to commonjs instead of es2015 as explained in here.

Will try after making the changes. I have a feeling that what @tariknz said #1652 (comment) is the root cause of this. It seems that during the compilation, there was an error and memory leak because of it.

@brunobertechini
Copy link

Just as a heads up. It turns out when I use YARN instead of NPM to RESTORE packages ng build completes successfully.

I am using

ng build --env=prod --aot

When I restore packages using npm install -- same error from this issue (out of memory)
When I restore packages using yarn install -- no error - success

@RicTheThird
Copy link

RicTheThird commented Aug 14, 2017

Thanks @basherr!
But instead of running script from command line.
Try running build script in package json by the following script:

"scripts": {
   "build-prod": "node --max_old_space_size=5048 ./node_modules/@angular/cli/bin/ng build --prod"
}

then run "npm run build-prod"

@chronicbint
Copy link

@RicTheThird and @basherr - worked for me, thanks.

@scharlo
Copy link

scharlo commented Aug 22, 2017

@RicTheThird - thx. works for me. o/

By the way, i preferred changing the start script rather than adding a new one.

@deepakanuraags
Copy link

deepakanuraags commented Aug 25, 2017

"scripts": {
"ng-custom-serve": "node --max_old_space_size=5048 ./node_modules/@angular/cli/bin/ng serve"
}
as suggested by @RicTheThird
this is what i have done to resolve,having ng serve as well,but will this hold good?,what if the application grows?Can we hope on this and go ahead?Any help would be great?

@igorrius
Copy link

In my case problem was solved after force adding @ngtools/webpack package to dependencies. And using --max_old_space_size is was not required.

@muhammadsaaeed
Copy link

I've tried building project with latest webpack till date(i.e. @3.8.1) and it solved the similar issue for me.

@finalxcode
Copy link

npm uninstall -g @angular/cli and reinstall @angular/[email protected] solved my problem

@rejayi-experion
Copy link

rejayi-experion commented Nov 10, 2017

I had experienced the issue FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory with 'ng build --prod'. I have solved the problem by using
node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

You can add also in package.json as

"scripts": {
    "prod": "node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng  build --prod",
}

npm run prod

@MartinFreire
Copy link

Hi, in my case, none of the solutions above worked for me.
I commented here about my execution crashing when it goes above 500 mb
I leave you here the link in case that thread is useful to someone

@nhustak
Copy link

nhustak commented Nov 28, 2017

Thanks @rejayi-experion Modifying NG.cmd didn't work but just invoking it manually as you suggested did.

@ghost
Copy link

ghost commented Nov 28, 2017

I mean, is there a reasonable explanation why it takes minutes to compile HTML, TS, and (S)CSS?

@JBGoldberg
Copy link

Anyone have a more definitive solution for this? Anyone comment that @TheLarkInn is working in the issue. There is another thread open to this issue?

@isahyarima
Copy link

node --max_old_space_size=2048 ./node_modules/.bin/ng Worked for me thanks

@thecheeto
Copy link

For anyone that might be trying to debug why all of a sudden the build process started running out of memory, I was able to track the issue down to the following code in my project.

Before change (out of memory exception):

  handleResponse(response: Response) {
    if (response.status == 403) {
      this.authenticationService.expireAuthToken();
      return Observable.throw('Session Timeout');
    }

    let json = response.json();

    return Observable.of({
      success: false,
      errors: json
    });
  }

After change (no memory issues):

  handleResponse(response: Response): Observable<any> { // <--- ADDED return type
    if (response.status == 403) {
      this.authenticationService.expireAuthToken();
      return Observable.throw('Session Timeout');
    }

    let json = response.json();

    return Observable.of({
      success: false,
      errors: json
    });
  }

Using the code of the before change, every time that method was called, it caused more and more memory being used during ng serve or ng build.

Not sure why specifying the return type fixes the issue, but went from ng build using ~3gb of memory down to ~700mb.

@vitor-belim
Copy link

@thecheeto Did you add this modification to every function within your project? And even if it returns void?

@thecheeto
Copy link

@vitor-belim No, only on that function. I'm thinking it has something to do with differing return types in a function. In my case it was returning either an ErrorObservable or Observable<{success: boolean, errors: string[]}>.

@prudhvichitturi
Copy link

This has been changed in angular 7 now. I couldn't able to find the path of ng.. Please any help for finding its path in node modules..

`> node --max_old_space_size=5048 ng build --prod --aot --source-map=false

module.js:540
throw err;
^

Error: Error: Cannot find module '/Users/work/ui/node_modules/@angular/cli/bin/ng'
at Function.Module._resolveFilename (module.js:538:15)
at Function.Module._load (module.js:468:25)
at Function.Module.runMain (module.js:684:10)
at startup (bootstrap_node.js:187:16)
at bootstrap_node.js:608:3
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build-script-a7: node --max_old_space_size=5048 ng build --prod --aot --source-map=false
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build-script-a7 script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/.npm/_logs/2019-01-28T08_13_45_344Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: npm run config -- --environment=prod && npm run build-script-a7 && npm run prod-server
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/.npm/_logs/2019-01-28T08_13_45_364Z-debug.log`

@strizich
Copy link

@prudhvichitturi node --max_old_space_size=5048 ./node_modules/.bin/<ng command>

@httpete
Copy link

httpete commented Feb 15, 2019

this worked for me:

"start": "node --max_old_space_size=5048 ./node_modules/@angular/cli/bin/ng serve",

@uriel2095
Copy link

"scripts": {
"ng-custom-serve": "node --max_old_space_size=5048 ./node_modules/@angular/cli/bin/ng serve"
}
as suggested by @RicTheThird
this is what i have done to resolve,having ng serve as well,but will this hold good?,what if the application grows?Can we hope on this and go ahead?Any help would be great?

this is simple and fuction for me :3

@pauldraper
Copy link

pauldraper commented Apr 18, 2019

It's weird that I have to tell Angular CLI/Node.js how much memory it can use.

Chrome/V8 seems happy to use all my memory.

@angular-automatic-lock-bot
Copy link

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

This action has been performed automatically by a bot.

@angular-automatic-lock-bot angular-automatic-lock-bot bot locked and limited conversation to collaborators Sep 9, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
effort2: medium (days) P3 An issue that is relevant to core functions, but does not impede progress. Important, but not urgent
Projects
None yet
Development

No branches or pull requests