build-optimizer: high memory usage (node crash)

[alexeagle]

From @MrBlaise on October 26, 2017 12:8

Bug Report or Feature Request (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request

Area

- [x] devkit
- [ ] schematics

Versions

macOS 10.13 (High Sierra)
It affects Windows as well, but on mac it is higher (larger than 1.7 gigs, therefore causes crash with node by default)

Repro steps

The problem is happening with the build-optimizer version 0.0.31.

Using @angular/cli 1.5.0-rc.2 and building with
ng build --prod --aot --build-optimizer

The memory usage starts to grow radically at 95% emitting stage. On a semi-large project this reaches 1.7 gigs on mac, which in turn causes a crash. Increasing the memory limit on node solves the issue, but this seems rather high.

Running it without build-optimizer it builds fine without a crash.

The log given by the failure

$ ng build --prod --aot --build-optimizer
 95% emitting                                                                 s <--- Last few GCs --->

[10242:0x103000000]   191148 ms: Mark-sweep 1417.1 (2081.5) -> 1417.1 (2081.5) MB, 753.9 / 0.0 ms  allocation failure scavenge might not succeed
[10242:0x103000000]   191836 ms: Mark-sweep 1417.1 (2081.5) -> 1417.1 (2035.5) MB, 687.0 / 0.0 ms  last resort GC in old space requested
[10242:0x103000000]   192507 ms: Mark-sweep 1417.1 (2035.5) -> 1417.1 (2013.5) MB, 670.8 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x31fd83fa5e91 <JSObject>
    1: DoJoin(aka DoJoin) [native array.js:~95] [pc=0x112d8f5ead39](this=0x31fd64e82311 <undefined>,p=0x31fdc377f859 <JSArray[432]>,q=432,E=0x31fd64e823b1 <true>,A=0x31fd83fef239 <String[1]:  >,z=0x31fd64e82421 <false>)
    2: Join(aka Join) [native array.js:~120] [pc=0x112d8ee542a9](this=0x31fd64e82311 <undefined>,p=0x31fdc377f859 <JSArray[432]>,q=432,A=0x31fd83fef239 <String[1]:  >,z=0x31fd64e...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/balazsrostas/.nvm/versions/node/v8.8.1/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/balazsrostas/.nvm/versions/node/v8.8.1/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/balazsrostas/.nvm/versions/node/v8.8.1/bin/node]
 4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/Users/balazsrostas/.nvm/versions/node/v8.8.1/bin/node]
 5: v8::internal::Runtime_StringBuilderJoin(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/balazsrostas/.nvm/versions/node/v8.8.1/bin/node]
 6: 0x112d8ed0463d
error Command failed with signal "SIGABRT".

Desired functionality

Using the build-optimizer shouldn't exceed the default 1.7GB threshold on a semi-large project.

Copied from original issue: angular/devkit#240

[alexeagle]

From @filipesilva on November 17, 2017 16:15

Hi @MrBlaise, does this happen on a project I could look at? In the context of #5618, I expect Build Optimizer usage to always use some memory, but It's odd that it happens at the 95% stage since most of the work Build Optimizer does is before the 90% phase.

Regardless, in #5618 (comment) I also have a report of high memory usage that I can't really reproduce, and in #5618 (comment) I have a report of very long times for build optimizer.

If your project isn't available, would you be interested in following some debug steps?

[alexeagle]

From @MrBlaise on November 17, 2017 16:48

Hi @filipesilva ! Unfortunately I can't share the project because it is private but I am more than happy to help with the debugging steps If I can.

[alexeagle]

From @filipesilva on November 17, 2017 16:58

That's great to hear!

I think it can be one of two things: either a transform is taking up a lot of memory, or the purify plugin has a runaway regex.

Lets measure how much memory some runs take. Make a new npm scrip that runs the CLI with increased memory allowance:

"ng-high-mem": "node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng"

This number should be enough but increase it if you need to.

If you're on a mac the easiest way to see peak memory usage is to use

/usr/bin/time -v npm run ng-high-mem -- build --prod --build-optimizer

The number you want is Maximum resident set size (kbytes). Record this number, it will be our baseline.

Then do the same with --build-optimizer=false and save the number.

Testing purify is easiest. Go to ./node_modules/@angular/cli/models/webpack-configs-production.js and comment off extraPlugins.push(new build_optimizer_1.PurifyPlugin());. Whats the maximum memory used now?

I think one of the build optimizer transformations is using up a lot of memory on your codebase, but I can't know which ones. I can tell you how to disable each one individually though.

You can do this in ./node_modules/@angular-devkit/build-optimizer/src/build-optimizer/build-optimizer.js. Each transform is individually added to the getTransforms array. You can comment out each of the getTransforms.push( and that transform will not be added.

If you can do this for each one and time me how it impacts memory used, I might be able to figure out which of the transforms is using up a lot of memory.

[alexeagle]

From @MrBlaise on November 17, 2017 18:54

Here are my tests @filipesilva . I don't know it it is helpful, I don't see much difference with all the options :/

Testing purify enabled or disabled:

purify ENABLED build-optimizer=true (Baseline): 2728394752 bytes (2.73GB) (166s runtime)
purify DISABLED build-optimizer=true: 2676985856 bytes (2.68GB) (152s runtime)

purify ENABLED build-optimizer=false: 2166198272 bytes (2.17GB) (95s runtime)
purify DISABLED build-optimizer=false: 2002616320 bytes (2.00GB) (95s runtime)

Testing build-optimizer transformations (build-optimizer=true):

(These are wrapped in an if statement, so I am not sure which of them actually ran)

It seemed like the memory usage is increased when I disabled these, but it might be misleading
due to the lack of samples (I did each step only once)

wrap_enums_1.getWrapEnumsTransformer DISABLED: 2906730496 bytes (2.90GB) (165s runtime)
import_tslib_1.getImportTslibTransformer DISABLED: 3338690560 bytes (3.33GB) (165s runtime)
prefix_classes_1.getPrefixClassesTransformer DISABLED: 2890100736 bytes (2.89GB) (168s runtime)

Side Effect Free Branch:

prefix_functions_1.getPrefixFunctionsTransformer AND scrub_file_1.getScrubFileTransformer AND class_fold_1.getFoldFileTransformer DISABLED:
2673786880 bytes (2.67GB) (155s runtime)

NOT Side Effect Free Branch

scrub_file_1.getScrubFileTransformer AND class_fold_1.getFoldFileTransformer DISABLED:
2874585088 bytes (2.87GB) (150s runtime)

[chadbr]

For what it's worth - I'm getting the same issue after I migrated from 5 -> 7

Chad-mac:kinetix-ui chadbr$ ng build --configuration production
 92% chunk asset optimization TerserPlugin                                                 
<--- Last few GCs --->

[36424:0x104000000]   192452 ms: Scavenge 1252.3 (1432.5) -> 1238.4 (1435.0) MB, 3.5 / 0.0 ms  (average mu = 0.288, current mu = 0.254) allocation failure 
[36424:0x104000000]   192495 ms: Scavenge 1253.2 (1435.0) -> 1239.6 (1437.5) MB, 4.2 / 0.0 ms  (average mu = 0.288, current mu = 0.254) allocation failure 
[36424:0x104000000]   192540 ms: Scavenge 1254.3 (1437.5) -> 1241.0 (1439.0) MB, 4.3 / 0.0 ms  (average mu = 0.288, current mu = 0.254) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0xc97acdc01d]
    1: StubFrame [pc: 0xc97acdd5a6]
Security context: 0x37860eb1e681 <JSObject>
    2: /* anonymous */ [0x3786413e2779] [/Users/chadbr/dev/vsts/kinetix-ui/node_modules/terser/dist/bundle.js:~4000] [pc=0xc97b6d6f54](this=0x378629adf289 <AST_Scope map = 0x378686d762f1>,name=0x3786cb0e0bc1 <String[4]: dist>)
    3: /* anonymous */ [0x3786c2c6ec31] [/Users/chadbr/dev/vsts/kinetix-ui/node_modules/terser/dist...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100039dbf node::Abort() [/usr/local/bin/node]
 2: 0x100039fc9 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1001d1375 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x10059c572 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 5: 0x10059f045 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 6: 0x10059aeef v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 7: 0x1005990c4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x1005a594c v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 9: 0x1005a59cf v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x100575354 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/bin/node]
11: 0x1007fcff4 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
12: 0xc97acdc01d 
13: 0xc97acdd5a6 

[chadbr]

This one allowed me to get past the build - #12645 (comment)

but there are other problems... I'll report them separately.

[filipesilva]

@chadbr the out of memory issue when updating to 7.0 is reported and investigated in #12646. It has to do with more code being brought in.

I don't think the original issue here is present anymore. It was originally from about a year ago and since then we have addressed a lot of the performance problems with build optimizer.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}