Compile stuck at optimize chunk assets

[pherrymason]

I've started having problems to compile my build when --optimize-minimize is enabled.
I see that I've reached +/-205 build modules, and then the compilation gets stuck at 80% optimize chunk assets for a very long time (+40minutes).
If I wait enough, it ends up node failing with "fatal error: js allocation failed - process out of memory"

I guess it has to do with the uglifyjs plugin as I said this only happens when using --optimize-minime flag (When compiling through my grunt task, this also fails).

[sokra]

Could try to run uglifyjs on the output files? Maybe it's an uglifyjs bug...

[pherrymason]

I've done what you suggested, I first build with webpack with no optimization and then executed this script to compress using uglifyjs (v2) and got no issues:

#!/bin/bash
FILES=*.js
for f in $FILES
do
    echo "uglifyjs $f -o ./min/$f --source-map ./min/$f.map.js -c"
    uglifyjs $f -o ./min/$f --source-map ./min/$f.map.js -c 
done

Maybe I'm using different arguments options from those used in UglifyJsPlugin?

If I couldn't fix this, can I switch to this script to compress my build? Should I configure uglifyjs in a specific way?

[sokra]

hmm... Could you give me access to the repo or prepare a testcase so I can debug it?

[pherrymason]

I could pass you the bunch of files...

On Wed, Oct 15, 2014 at 11:25 AM, Tobias Koppers [email protected]
wrote:

hmm... Could you give me access to the repo or prepare a testcase so I can
debug it?

—
Reply to this email directly or view it on GitHub
#537 (comment).

[pherrymason]

How can I send you privately the repo/files?

[erkiesken]

I hit a similar problem. It seems that the UglifyJSPlugin can't handle a compressed return value from uglify.Compressor, specifically the ast.transform(compress) (see line 70 of UglifyJsPlugin.js) step hangs for me.

I added console logging like this around there:

if(options.compress !== false) {
    console.log("uglifying pre-compress", file);
    var compress = uglify.Compressor(options.compress);
    console.log("uglifying done-compress", file);
    ast = ast.transform(compress);
    console.log("uglifying done-transform", file);
    ast.figure_out_scope();
    if(options.mangle !== false) {
        ast.compute_char_frequency(options.mangle || {});
        ast.mangle_names(options.mangle || {});
    }
}

And in my output I can see it does indeed get stuck there:

3915ms build modules
13ms optimize
751ms build modules
21ms hashing
631ms create chunk assets
 78% additional chunk assets
uglifying pre-compress main.js
uglifying done-compress main.js
...hangs here...
^C%

Workaround for me for now was to init the uglify plugin with new webpack.optimize.UglifyJsPlugin({compress:false}) to skip that step.

The codebase in question is a quite large ReactJS/Reflux application.

[erkiesken]

If you have any ideas which Uglify compress options I should try to weed out what specific feature thet ast.transform chokes on, then let me know.

[jkimbo]

I've just come across the same problem with a large project with 26 entry points. If I disable source maps it works but hangs with source maps enabled. My current hypothesis is that because webpack keeps all the sourcemaps in memory as it is compiling it results in node hitting the 1.4Gb limit in v8. I'm not familiar with how the internals work but can the maps not be flushed to the files on disk instead of keeping them in memory?

[jkimbo]

I've managed to solve the issue by externalising some of the big libraries (React and jQuery). That seems to have reduced the memory usage considerably and I can now compress the bundles with source maps.

[jhnns]

Probably it's sufficient to flag libraries with noParse

[jkimbo]

@sokra so I've created a fairly contrived example repo that manages to reproduce the problem here: https://github.com/jkimbo/webpack-memory-issue

It has 20 entry points each of which requires fairly large libraries and then it runs it through the uglify plugin with sourcemaps. I realise it is in no way optimised but hopefully it can help debug this issue.

[pr0ton]

I am seeing a similar issue as well.

[pago]

I have the same problem without UglifyJS (continuous development build). The problem occurs after a couple of rebuilds and I believe I have an idea of where it leaks memory.

In my case, the count of ModuleNotFoundError objects created is growing at an extremely high pace. That error is thrown because of moment.js so I excluded it from parsing (noParse) and as a result the heap size stays stable (instead of growing further). If I read the dump in node-inspector correctly, it might be in the cache object of the Compilation module.

That's as much as I could find out so far.

[sokra]

You can try to disable the cache with cache: false.

[shprink]

Having the same.

[shprink]

Ended up downgrading webpack-dev-server to 1.7.0 and webpack to 1.7.3 to make it work.

[sokra]

Regarding the original issue:

new UglifyJsPlugin({
  sourceMap: false
})

instead of --optimize-minimize or -p should help...

[margielm]

Have the same issue here, without UglifyJsPlugin.
Do you have any idea when can this be fixed?

anyway. Thanks for the webpack! incredible pice of art.

[roman01la]

Same issue, sourceMap: false doesn't make any difference, removing UglifyJS plugin helps.

[jedwards1211]

I don't think it's UglifyJS or memory; my webpack config suddenly started hanging at only module 67, when it was successfully building over 400 modules before. Ironically it still works with webpack dev server and various plugins; but my config that's hanging has no plugins.

[jedwards1211]

Here's my config that hung it.

webpack.config.base.js (if I use this config by itself it doesn't hang):

var path = require('path');
var webpack = require('webpack');

module.exports = {
  devtool: 'source-map',
  entry: [
    'babel/polyfill',
    './app',
  ],
  output: {
    path: path.join(__dirname, 'dist'),
    filename: 'bundle.js',
    publicPath: '/assets/',
  },
  resolve: {
    extensions: ['', '.js', '.jsx'],
  },
  module: {
    loaders: [
      {
        test: /\.jsx?$/,
        loader: 'babel?stage=0',
        exclude: /node_modules|vendor/,
      }, 
      {
        test: /\.css$/,
        loader: 'style-loader!css-loader'
      }, 
      {
        test: /\.sass/,
        loader: 'style-loader!css-loader!sass-loader?indentedSyntax&outputStyle=expanded&' +
        "includePaths[]=" +
        (path.resolve(__dirname, "./node_modules"))
      }, 
      {
        test: /\.scss/,
        loader: 'style-loader!css-loader!sass-loader?outputStyle=expanded&' +
        "includePaths[]=" +
        (path.resolve(__dirname, "./node_modules"))
      }, 
      {
        test: /\.(png|jpg)$/,
        loader: 'url-loader?limit=8192'
      },
    ],
  },
};

webpack.config.js:

var path = require('path');
var webpack = require('webpack');
var config = require('./webpack.config.base');
var _ = require('lodash');

var getWindowReact = path.resolve(__dirname, 'getWindowReact');

module.exports = _.assign({}, config, {
  resolve: _.assign({}, config.resolve, {
    alias: _.assign({}, config.resolve.alias, {
      'react$': getWindowReact, // if I comment out only this line it doesn't hang.
      'react/addons$': getWindowReact,
    }),
  }),
});

getWindowReact.js:

// my boss wants me to use React from a CDN
export default window.React;

[jedwards1211]

@jhnns unfortunately noParse won't work for React because it uses require extensively

[jedwards1211]

@sokra I was thinking, if only I could make Webpack print out the modules one by one as it encounters them while building, then I could see which one it's hanging on, and maybe that would give me a clue what to investigate next.

[jedwards1211]

Another note, my compilation gets stuck before the optimizing chunk assets part. Who knows, there might be several unrelated issues here.

[Monokai]

I have exactly the same issue, and also at the exact same percentage (80%). Dev builds work fine.

[jedwards1211]

@Monokai As far as I've seen it always says 80% for optimizing chunk assets.

[johann-sonntagbauer]

@jedwards1211 @Monokai I faced the same problem. The main thing on my side is a bug in the libsass transpiler. Following workaround solves the problem for me:
UV_THREADPOOL_SIZE=100 npm run build
Maybe this tweek of the node internal threadpools helps.

Some context:
webpack-contrib/sass-loader#100

[dougcalobrisi]

@Tzaphkiel - Does the big dependency you added to your project have any pre-minified files? If so, that is likely the cause, as @sunlei33 recently mentioned about the UglifyJS issue.

If not, try using the workarounds for increasing the number of threads and allocated memory for the node process running webpack, as I summarized above. I have encountered the exact same issues.

[Tzaphkiel]

@dougcalobrisi - Ok, thx, that might be it, the project I'm trying to include is alloy-editor which contains ckeditor, etc. and has indeed minimized files.

Is there a plan to fix this ? I saw something about UglifyJS having a fix in master but nothing released in npm yet... the dep. would still have to be adapted in webpack dependency no ?

My workaround for the moment is simply to turn off compress in the plugin definition for UglifyJS...
This is tedious and impractical :(

[dougcalobrisi]

@Tzaphkiel - You could clone the current master repo for UglifyJS and then use npm link to create a globally installed version of the current master of UglifyJS, then delete the current UglifyJS dep in node_modules/ of your project and then install the linked version.

I haven't actually tried this, it's just off the top of my head, but should work. Something like this:

#clone the master repo and link it
mkdir ~/modules
git clone git://github.com/mishoo/UglifyJS2.git ~/modules/uglifyjs
npm link ~/modules/uglifyjs

#move to your project, remove the old uglifyjs dep, link the new one, and confirm the symlink shows
cd <path of your project>
rm -rf node_modules/uglifyjs
npm link uglifyjs
ls -l node_modules/uglifyjs

At that point your project should have the uglifyjs dependency in your project's node_modules symlinked to the globally installed version of the current master branch of UglifyJS. You'll want to unlink it once they have pushed a new release to npm.

Let me know if that works or not for you. As I said I haven't actually tested it, so no guarantees. Likely very helpful for others if it works, though.

[ConAntonakos]

Just happened to me. And I tested it again after removing devtool: inline-source-map, and the build compilation worked great and much quicker.

[hlzhang]

watch: true 

make this happens.

Disable watch solved problem for me.

[FrendEr]

@dougcalobrisi How can I know webpack's default memory and thread pool that Node provides ? And how much should I set will be best ? UV_THREADPOOL_SIZE=100 or UV_THREADPOOL_SIZE=10 ? The same to max_old_space_size

[dougcalobrisi]

@FrendEr - I believe the default heap allocation is 1.76gb for a 64bit environment. You could check it using v8.getHeapStatistics() - docs: https://nodejs.org/api/v8.html

As for the default thread pool size, I believe that is coming from libuv which has a default of 4 threads.

[FrendEr]

@dougcalobrisi thanks! I use UV_THREADPOOL_SIZE=100 node --max_old_space_size=4096 to make my webpack run 8s reduce to 6.5s at additional chunk assets, but it still hangs at 78%.

[FrendEr]

@sunlei33 [email protected] has merge the issue#1024 and publish, but it has no effect on build optimization

[FrendEr]

compilation.plugin("additional-chunk-assets", function() {
  handler(0.78, "additional chunk assets");
});

I found the code in ProgressPlugin.js, I think UglifyJS takes much time to optimize code, but the reason for webpack hang at 78% just because the code above. It won't update before it turn to next step call optimize-chunk-assets

[theatlasroom]

Had this same issue, building was taking 10mins+ and as a few people have noted, i took out my minified dependencies and it now builds in ~10s :D.

[joshwiens]

Had the same general issue as @theatlasroom and went the same direction to try and troubleshoot which oddly enough solved the issue.

[rturk]

Increasing swap worked for me!

[unformatt]

@rturk can you elaborate?

[fresheneesz]

@unformatt I also found out that I didn't have any swap space enabled on my digital ocean centos machine, which was causing the out-of-memory killer to kill my install script. Increasing swap fixed that problem for me. It still hangs for a bit at 78-80% tho.

[fresheneesz]

@sokra Could you comment on why you closed this? Has a fix from uglify been pulled in to fix the problem with already-compiled files?

[MarioGruda]

removing the --progress option worked for me

[kevyworks]

In some cases and digging, i've fixed mine by insuring that the names has a unique entries..

if (Mix.extract) {
    module.exports.plugins.push(
        new webpack.optimize.CommonsChunkPlugin({
            names: Mix.entryBuilder.extractions.concat([
                path.join(Mix.js.base, 'manifest').replace(/\\/g, '/')
            ]).filter(function (item, pos, self) {
                return self.indexOf(item) == pos;
            }),
            // minChunks: ({resource}) => /node_modules/.test(resource)
            minChunks: Infinity
        })
    );
}

Edit: Sorry, After I updated my npm libraries, I started having errors somewhere in n basic chunk optimization in laravel-mix.

Hope it helps.

[chouhan]

webpack asset optimization continuously loops on 'ng serve'. I was expecting it would do once, then start my app at localhost:4200

Any Ideas?

Did npm update, and got all latest node_modules. Here is my

// package.json

{
"name": "angular-quickstart",
"version": "1.0.0",
"description": "package.json from the documentation, supplemented with testing support",
"angular-cli": {},
"scripts": {
"start": "tsc && concurrently "tsc -w" "lite-server" ",
"e2e": "tsc && concurrently "http-server -s" "protractor protractor.config.js" --kill-others --success first",
"lint": "tslint ./app/**/*.ts -t verbose",
"lite": "lite-server",
"pree2e": "webdriver-manager update",
"test": "tsc && concurrently "tsc -w" "karma start karma.conf.js"",
"test-once": "tsc && karma start karma.conf.js --single-run",
"tsc": "tsc",
"tsc:w": "tsc -w"
},
"keywords": [],
"author": "",
"license": "MIT",
"dependencies": {
"@angular/animations": "^4.1.0",
"@angular/common": "^4.1.0",
"@angular/compiler": "^4.1.0",
"@angular/compiler-cli": "^4.1.0",
"@angular/core": "^4.1.0",
"@angular/forms": "^4.1.0",
"@angular/http": "^4.1.0",
"@angular/platform-browser": "^4.1.0",
"@angular/platform-browser-dynamic": "^4.1.0",
"@angular/platform-server": "^4.1.0",
"@angular/router": "^4.1.0",
"@ng-bootstrap/ng-bootstrap": "^1.0.0-alpha.15",
"angular-in-memory-web-api": "~0.2.1",
"angular2-jwt": "^0.1.28",
"angular2-logger": "^0.5.1",
"auth0-lock": "^10.10.2",
"bootstrap": "4.0.0-alpha.5",
"core-js": "^2.4.1",
"jquery": "1.9.1 - 3",
"reflect-metadata": "^0.1.8",
"rxjs": "5.0.0-rc.4 - 5.0.1",
"systemjs": "0.19.40",
"tether": "^1.3.7",
"typescript": "^2.3.2",
"zone.js": "^0.7.2"
},
"devDependencies": {
"@types/jasmine": "^2.5.36",
"@types/node": "^6.0.46",
"@types/selenium-webdriver": "^2.53.33",
"angular-cli": "^1.0.0-beta.22-1",
"canonical-path": "0.0.2",
"concurrently": "^3.1.0",
"http-server": "^0.9.0",
"jasmine-core": "~2.4.1",
"jasmine-spec-reporter": "^2.7.0",
"karma": "^1.3.0",
"karma-chrome-launcher": "^2.0.0",
"karma-cli": "^1.0.1",
"karma-jasmine": "^1.0.2",
"karma-jasmine-html-reporter": "^0.2.2",
"lite-server": "^2.2.2",
"lodash": "^4.16.4",
"protractor": "~4.0.13",
"rimraf": "^2.5.4",
"ts-node": "^1.7.2",
"tslint": "^3.15.1",
"typescript": "~2.0.10",
"webdriver-manager": "^11.1.0",
"@types/google-maps": "^3.2.0"
},
"repository": {}
}

Chouhan:mockii_node rakeshchouhan$ ng serve
As a forewarning, we are moving the CLI npm package to "@angular/cli" with the next release,
which will only support Node 6.9 and greater. This package will be officially deprecated
shortly after.

To disable this warning use "ng set --global warnings.packageDeprecation=false".

Your global Angular CLI version (1.0.0-beta.28.3) is greater than your local
version (1.0.0-beta.22-1). The local Angular CLI version is used.

To disable this warning use "ng set --global warnings.versionMismatch=false".
Hash: 2a5559aa5e7694288411
Time: 17424ms
chunk {0} 0.chunk.js, 0.bundle.map 54 kB {7} {6} {5} {2} {4} {3} {1} [rendered]
chunk {1} 1.chunk.js, 1.bundle.map 22.1 kB {3} {4} {2} {5} {6} {0} {7} [rendered]
chunk {2} 2.chunk.js, 2.bundle.map 17.8 kB {5} {4} {3} {1} {6} {0} {7} [rendered]
chunk {3} 3.chunk.js, 3.bundle.map 4.62 kB {1} {4} {2} {5} {6} {0} {7} [rendered]
chunk {4} 4.chunk.js, 4.bundle.map 3.77 kB {2} {3} {1} {5} {6} {0} {7} [rendered]
chunk {5} 5.chunk.js, 5.bundle.map 4.43 kB {6} {2} {4} {3} {1} {0} {7} [rendered]
chunk {6} 6.chunk.js, 6.bundle.map 4.43 kB {0} {5} {2} {4} {3} {1} {7} [rendered]
chunk {7} main.bundle.js, main.bundle.map (main) 172 kB {9} [initial] [rendered]
chunk {8} styles.bundle.js, styles.bundle.map (styles) 9.72 kB {10} [initial] [rendered]
chunk {9} vendor.bundle.js, vendor.bundle.map (vendor) 2.84 MB [initial] [rendered]
chunk {10} inline.bundle.js, inline.bundle.map (inline) 0 bytes [entry] [rendered]
webpack: Compiled successfully.
webpack: Compiling...
Hash: 2a5559aa5e7694288411
Time: 830ms
chunk {0} 0.chunk.js, 0.bundle.map 54 kB {7} {6} {5} {2} {4} {3} {1}
chunk {1} 1.chunk.js, 1.bundle.map 22.1 kB {3} {4} {2} {5} {6} {0} {7}
chunk {2} 2.chunk.js, 2.bundle.map 17.8 kB {5} {4} {3} {1} {6} {0} {7}
chunk {3} 3.chunk.js, 3.bundle.map 4.62 kB {1} {4} {2} {5} {6} {0} {7}
chunk {4} 4.chunk.js, 4.bundle.map 3.77 kB {2} {3} {1} {5} {6} {0} {7}
chunk {5} 5.chunk.js, 5.bundle.map 4.43 kB {6} {2} {4} {3} {1} {0} {7}
chunk {6} 6.chunk.js, 6.bundle.map 4.43 kB {0} {5} {2} {4} {3} {1} {7}
chunk {7} main.bundle.js, main.bundle.map (main) 172 kB {9} [initial]
chunk {8} styles.bundle.js, styles.bundle.map (styles) 9.72 kB {10} [initial]
chunk {9} vendor.bundle.js, vendor.bundle.map (vendor) 2.84 MB [initial]
chunk {10} inline.bundle.js, inline.bundle.map (inline) 0 bytes [entry]
webpack: Compiled successfully.
webpack: Compiling... Hash: 2a5559aa5e7694288411
Time: 396ms
chunk {0} 0.chunk.js, 0.bundle.map 54 kB {7} {6} {5} {2} {4} {3} {1}
chunk {1} 1.chunk.js, 1.bundle.map 22.1 kB {3} {4} {2} {5} {6} {0} {7}
chunk {2} 2.chunk.js, 2.bundle.map 17.8 kB {5} {4} {3} {1} {6} {0} {7}
chunk {3} 3.chunk.js, 3.bundle.map 4.62 kB {1} {4} {2} {5} {6} {0} {7}
chunk {4} 4.chunk.js, 4.bundle.map 3.77 kB {2} {3} {1} {5} {6} {0} {7}
chunk {5} 5.chunk.js, 5.bundle.map 4.43 kB {6} {2} {4} {3} {1} {0} {7}
chunk {6} 6.chunk.js, 6.bundle.map 4.43 kB {0} {5} {2} {4} {3} {1} {7}
chunk {7} main.bundle.js, main.bundle.map (main) 172 kB {9} [initial]
chunk {8} styles.bundle.js, styles.bundle.map (styles) 9.72 kB {10} [initial]
chunk {9} vendor.bundle.js, vendor.bundle.map (vendor) 2.84 MB [initial]
chunk {10} inline.bundle.js, inline.bundle.map (inline) 0 bytes [entry]
webpack: Compiled successfully.
webpack: Compiling... Hash: 2a5559aa5e7694288411
Time: 359ms
chunk {0} 0.chunk.js, 0.bundle.map 54 kB {7} {6} {5} {2} {4} {3} {1}
chunk {1} 1.chunk.js, 1.bundle.map 22.1 kB {3} {4} {2} {5} {6} {0} {7}
chunk {2} 2.chunk.js, 2.bundle.map 17.8 kB {5} {4} {3} {1} {6} {0} {7}
chunk {3} 3.chunk.js, 3.bundle.map 4.62 kB {1} {4} {2} {5} {6} {0} {7}
chunk {4} 4.chunk.js, 4.bundle.map 3.77 kB {2} {3} {1} {5} {6} {0} {7}
chunk {5} 5.chunk.js, 5.bundle.map 4.43 kB {6} {2} {4} {3} {1} {0} {7}
chunk {6} 6.chunk.js, 6.bundle.map 4.43 kB {0} {5} {2} {4} {3} {1} {7}
chunk {7} main.bundle.js, main.bundle.map (main) 172 kB {9} [initial]
chunk {8} styles.bundle.js, styles.bundle.map (styles) 9.72 kB {10} [initial]
chunk {9} vendor.bundle.js, vendor.bundle.map (vendor) 2.84 MB [initial]
chunk {10} inline.bundle.js, inline.bundle.map (inline) 0 bytes [entry]
webpack: Compiled successfully.

[joshwiens]

@chouhan - Questions regarding the angular-cli belong in whatever medium the have designated for support requests.

[chouhan]

@d3viant0ne - I did not get you. Are you expecting me to rewrite the question in a proper format?

I am looking for a solution to the above issue posted.

[joshwiens]

I'm not expecting you to rewrite the question, I am saying you are posting it in the wrong place.

Your question is related to the angular-cli of which webpack is a part but the right place to look for a solution related to ng serve is https://github.com/angular/angular-cli.

There is a ton of code on top of the cli's default webpack configuration that could be related to the output you are getting and considering you can't directly modify the cli's webpack configuration, this is an issue that will need to be handled by that team in whatever forum they have designated for support requests.

That said, first off I would get off of the beta build of the CLI and onto @angular/cli@latest. There were a significant number of performance improvements in the release candidates that may have a positive effect on the above which is exactly what they are going to tell you.

Secondly, don't mismatch the global cli version with the application cli version particularly with the beta builds. There are numerous resolved issues pre-final release caused by version mismatches.

Beyond those two things, you need to ask your question in the proper forum. i.e. the angular-cli repository.