OutOfMemory with 3000 docs #2165

JLKM · 2016-09-16T19:26:40Z

Having made a lot of hexo g's with 100-200 docs without problems at all, I ran into a wall yesterday when trying to run a full scale hexo generation with 3000 docs.

$ hexo g
INFO  Start processing
INFO  Files loaded in 6.02 min
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

In order to avoid a 514 MB in node.js I followed the advice from #1735 and made the follwing changes:

File: C:\Users\Name\AppData\Roaming\npm\node_modules\hexo-cli\bin\hexo

Changed: #!/usr/bin/env node
To: #!/usr/bin/env node --max_old_space_size=8192

Same result:
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Then I increased memory to 12 GB:

File: C:\Users\Name\AppData\Roaming\npm\node_modules\hexo-cli\bin\hexo

Changed: #!/usr/bin/env node --max_old_space_size=8192
To: #!/usr/bin/env node --max_old_space_size=12288

Error again - but new prompt:
FATAL ERROR: Scavenger: semi-space copy Allocation failed - process out of memory

My laptop has only 16GB of RAM, but I put my money on the swap-file and allocated 32GB RAM to node.js/Hexo:

File: C:\Users\Name\AppData\Roaming\npm\node_modules\hexo-cli\bin\hexo

Changed: #!/usr/bin/env node --max_old_space_size=12288
To: #!/usr/bin/env node --max_old_space_size=32768

Same error-prompt as in 1 and 2.
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Worth noting:

I get no other errors - like f.ex. the template-error in #1763)

Possible culprits:

Hexo
Node tranquilpeak #274
V8 - need for –max-new-space-size=XXXX
My Hexo-cli setup/version - please look below

Any advice would be most appreciated

My setup:

Name	Version
Windows	10
Node.js	6.3.1
Hexo	3.2.2
Theme	Hueman

My full hexo-cli package.json for scrutenizing:

{
  "_args": [
    [
      "hexo-cli",
      ""
    ]
  ],
  "_from": "hexo-cli@latest",
  "_id": "[email protected]",
  "_inCache": true,
  "_installable": true,
  "_location": "/hexo-cli",
  "_nodeVersion": "4.3.0",
  "_npmOperationalInternal": {
    "host": "packages-16-east.internal.npmjs.com",
    "tmp": "tmp/hexo-cli-1.0.2.tgz_1465818839377_0.7479208533186466"
  },
  "_npmUser": {
    "email": "[email protected]",
    "name": "tommy351"
  },
  "_npmVersion": "3.7.1",
  "_phantomChildren": {},
  "_requested": {
    "name": "hexo-cli",
    "raw": "hexo-cli",
    "rawSpec": "",
    "scope": null,
    "spec": "latest",
    "type": "tag"
  },
  "_requiredBy": [
    "#USER"
  ],
  "_resolved": "https://registry.npmjs.org/hexo-cli/-/hexo-cli-1.0.2.tgz",
  "_shasum": "8ebcae88cac29254f1e9ac07f8a9f07399a8a1ae",
  "_shrinkwrap": null,
  "_spec": "hexo-cli",
  "_where": "",
  "author": {
    "email": "[email protected]",
    "name": "Tommy Chen",
    "url": "http://zespia.tw"
  },
  "bin": {
    "hexo": "./bin/hexo"
  },
  "bugs": {
    "url": "https://github.com/hexojs/hexo-cli/issues"
  },
  "dependencies": {
    "abbrev": "^1.0.7",
    "bluebird": "^3.4.0",
    "chalk": "^1.1.3",
    "hexo-fs": "^0.1.5",
    "hexo-log": "^0.1.2",
    "hexo-util": "^0.6.0",
    "minimist": "^1.2.0",
    "object-assign": "^4.1.0",
    "tildify": "^1.2.0"
  },
  "description": "Command line interface for Hexo",
  "devDependencies": {
    "chai": "^3.5.0",
    "chai-as-promised": "^5.3.0",
    "eslint": "^2.12.0",
    "eslint-config-hexo": "^1.0.3",
    "hexo-renderer-marked": "^0.2.10",
    "istanbul": "^0.4.3",
    "jscs": "^3.0.4",
    "jscs-preset-hexo": "^1.0.1",
    "mocha": "^2.5.3",
    "proxyquire": "^1.7.9",
    "rewire": "^2.5.1",
    "sinon": "^1.17.4"
  },
  "directories": {
    "bin": "./bin",
    "lib": "./lib"
  },
  "dist": {
    "shasum": "8ebcae88cac29254f1e9ac07f8a9f07399a8a1ae",
    "tarball": "https://registry.npmjs.org/hexo-cli/-/hexo-cli-1.0.2.tgz"
  },
  "gitHead": "fb567e2d1e32929c4183ab4fcf1a5645fb646eb2",
  "homepage": "https://hexo.io/",
  "keywords": [
    "website",
    "blog",
    "cms",
    "framework",
    "hexo",
    "cli"
  ],
  "license": "MIT",
  "main": "lib/hexo",
  "maintainers": [
    {
      "email": "[email protected]",
      "name": "tommy351"
    }
  ],
  "name": "hexo-cli",
  "optionalDependencies": {},
  "readme": "ERROR: No README data found!",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/hexojs/hexo-cli.git"
  },
  "scripts": {
    "eslint": "eslint .",
    "jscs": "jscs .",
    "test": "mocha test/index.js",
    "test-cov": "istanbul cover --print both _mocha -- test/index.js"
  },
  "version": "1.0.2"
}

The text was updated successfully, but these errors were encountered:

soulteary · 2016-09-18T09:30:26Z

i also gotheap out of memory error when i generate more than 1000 docs.
i solve it by fatal-error-call-and-retry-last-allocation-failed-process-out-of-memory

i think this issue related to both node and hexo.

if you've dump the struct of post object, you'll see a huge object, i guess if you disable some plugin and features, you can solve it.

JLKM · 2016-09-20T14:19:07Z

Still not able to generate more than approx. 1200 posts

Thank you for your advice.

Deleted "hexo-generator-search": "^1.0.2" and "hexo-generator-json-content": "^2.2.0" from hexo package.json, but it only speeded generation up with a few seconds. It didn't make generation of more documents possible fr me.
Followed these recommendations, which mentions a number of parameters capable of expanding hexo generator. I've tested them all with not so much luck. Still wonder, how the suggestions in this doc helped you out...

I soon realised that it's much easier to put the parameters directly into the command-prompt, and the result didn't differ from putting the parameters into:

C:\Users\Username\AppData\Roaming\npm\node_modules\hexo-cli\bin\hexo.

Lessons learned from the parameters

--max_old_space_size=12288 --stack_size=12288

Did make it possible for me to generate about 1200 posts - opposed to a few hundreds with hexo g and no parameters. At this time I need to generate 3000 posts, but attempts to generate more than approx. 1200 posts resulted in various error (see below). 12288 seemed to be best fit on my 16 GB RAM laptop. Values below tend to pull down the limit, but higher values didn't push the limit up.

--optimize_for_size

Did increase the number of generations with a couple of hundreds when running hexo g together with f.ex. --max_old_space_size=8192. But the parameter was not able to move the maximum number of successfully generated posts.

--max_executable_size=12288

Also tested with 8192, 4096 and 2048. None of the different values made any observable difference.

--max_new_space_size=XXXX

No help according to your source. So I didn't test that one.

The error-collection

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

The most common error in my test.

FATAL ERROR: Scavenger: semi-space copy Allocation failed - process out of memory.

Usually turned up when fairly close to the upper limit (but on the wrong side).

FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory

Only encountered once, when the upper limit was really close.

/c/Users/Name/AppData/Roaming/npm/hexo: line 14: 3640 Segmentation fault node "$basedir/node_modules/hexo-cli/bin/hexo" "$@"

Showed up on one occation, when upper limit was even closer (<10 docs). This one is somewhat weird, since the referred file has not 14 lines but only 5 (see below).

1. #!/usr/bin/env node
2.
3. 'use strict';
4.
5. require('../lib/hexo')();

Themes

According to my tests, the themes are not to blame for the upper limit as such. Swithing themes apparently moves the upper limit of posts about 100 or 200 up or down. This is likely caused by the various amount of html needed for the different designs. In my case Icarus had the highest upper limit. Hueman ended in the middle, and the lowest amount of possible generated posts was experienced with bootstrap-blog.

Downgrade

After this (several days of nightmare) I went all in and downgraded to Hexo 3.1.1 - as suggested in #1831.

Hexo package.json was changed to:

{
  "name": "hexo-site",
  "version": "0.0.0",
  "private": true,
  "hexo": {
    "version": "3.1.1"
  },
  "dependencies": {
    "hexo": "3.1.1",

... etc. ...
  }
}

The following npm install resulted in a lot of warnings:

npm WARN deprecated [email protected]: potential XSS vulnerability in autoescape mode, and with escape filter was fixed in v2.4.3
npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated [email protected]: This package is no longer maintained
npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
npm WARN notsup Not compatible with your operating system or architecture: [email protected]
npm WARN optional Skipping failed optional dependency /nunjucks/chokidar/fsevents:
npm WARN notsup Not compatible with your operating system or architecture: [email protected]

And the following hexo g with parameters not only ended in an error message. It went totally bananas returning some of the following error-lines for each entry:

ERROR Process failed: _posts/209B3447F4562503C1257DE10041CC6A.md
Error: Category `2013` has already existed!

at D:\vscode\hexo\Odense\tredjesite\node_modules\hexo\lib\models\category.js:71:13
at tryCatcher (D:\vscode\hexo\Odense\tredjesite\node_modules\warehouse\node_modules\bluebird\js\main\util.js:26:23)
at D:\vscode\hexo\Odense\tredjesite\node_modules\warehouse\node_modules\bluebird\js\main\method.js:15:34
at D:\vscode\hexo\Odense\tredjesite\node_modules\warehouse\lib\model.js:193:12

Upgrade

Finally I took the opposite approach and upgraded my Hexo ^3.2.0 to 3.2.2 - in order to test the absolute latest version.

Unfortunately this wasn't enough to move the upper limit, either.

Please, anyone, make clear once and for all, that Hexo doesn't have an unbreakable upper limit of approx. 1000-1200 posts. And please also show exactly how to get around this limit.

soulteary · 2016-09-21T03:29:34Z

hold on, i'll check my config, it may helpful to you. i've generated approx. 1000+ post last month.

soulteary · 2016-09-21T04:15:02Z

@JLKM soulteary/hexo-blog#1

JLKM · 2016-09-25T22:40:38Z

Back to square 1.

I've tested all suggestions in Hexo-blog #1. But I have not yet been able to run a successful Hexo generate with more than 1200 docs. Not on:

Windows 10, Node 6.6.0 with 16 GB RAM with #!/usr/bin/env node --max_old_space_size=12288 --optimize_for_size --max_executable_size=12288 --stack_size=12288
Linux with 1 GB RAM with #!/usr/bin/env node --max_old_space_size=768 --optimize_for_size --max_executable_size=768 --stack_size=768.

Just can't be right, that 1200 docs eat up 12 GB of RAM...

Please step forward, anyone, who has been able to generate bigger sites with Hexo.

Or do we all really have to wait for the advertised upgrade of memory handling in Node.js 7.X.?

JLKM · 2016-09-27T06:34:28Z

Have given up generating more than 1000 docs at a time in Hexo

You can read more about Su Yang's and my resultless efforts to solve this.

Options left in my view at this time:

Hugo (but not so many well-designed themes to choose from)
Node 7.X with better memory-handling
Hexo-fix.
Insanely amounts of RAM (>16 GB)

A real showstopper for some. And a real pity. Hexo has the lead in almost every other field - in my view.

JLKM · 2016-09-27T17:39:34Z

After lots of failed tests with generation of thousands of docs in Windows with 16 GB RAM, I made the same tests on a Linux setup - also with 16 GB RAM.
With high hopes. But they were not fulfilled.

Created a user on Digitalocean and made an installation with Ubuntu 16.04, Node.js 6.6 and Hexo.

Then I ran a handful of tests with 2995 docs and then with 2007 docs.
hexo g No luck.
hexo generate "node --max_old_space_size=12288 --optimize_for_size --max_executable_size=12288 --stack_size=12288" No luck
hexo generate "node --max_old_space_size=8196 --optimize_for_size --max_executable_size=8196 --stack_size=8196" No luck

Exactly the same error each and every time (written by hand):

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [hexo]
2: 0x10dc06c [hexo]
3: v8::Utils::ReportApiFailure(char const*, char const*) [hexo]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [hexo]
5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [hexo]
6: v8::internal::Runtime_StringBuilderConcat(int, v8::internal::Object**, v8::internal::Isolate*) [hexo]
7: 0x1a78d1c092a7
Aborted (core dumped)

This is really frustrating.
Does anyone have a clue?

JLKM · 2016-09-29T06:54:20Z

Noone deserves to waste as many days, as I did, trying to tweak Hexo into generating more than 1200 docs. Nothing indicates, that it's possible at this time.
Really look forward to a solution for this issue on this otherwise great SSG.
A piece of advice, if you need this feature now: Hugo: 11.980 docs in 4 minutes and 40 seconds.

hank583746309 · 2018-07-10T05:23:28Z

hi,boys,my error:
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
i need help.

Num142857 · 2018-11-13T05:16:02Z

@hank583746309 目前我也碰到了同样的问题,文章数量在1040左右,已经开始出现同样的问题了
不知道大家是怎么解决这个问题的?

smoke-indica · 2019-07-27T17:04:10Z

Hexo generate debug outputs fine, final output 'Killed' & nothing output to public folder. 1 core, 1GB RAM VPS. hexo-server over generate? 730 posts.

anthonyrussano · 2022-06-01T18:17:46Z

I was able to generate over 3,000 files with 8gb of ram

I used the following:

node --max-old-space-size=8192 node_modules/hexo-cli/bin/hexo gen -c 1 -dw

githubbbie · 2024-09-30T20:29:06Z

Ok, trying on 33103 files and getting fail at 16 gb

#!/usr/bin/env node --max_old_space_size=16384

in node_modules/hexo-cli/bin/hexo

Error:

<--- Last few GCs --->

[137:0x7964cb02b000]    33039 ms: Scavenge (reduce) (interleaved) 3446.1 (3671.1) -> 3446.1 (3668.1) MB, pooled: 0 MB, 8.24 / 0.00 ms  (average mu = 0.274, current mu = 0.252) external memory pressure; 
[137:0x7964cb02b000]    33791 ms: Mark-Compact (reduce) 3446.3 (3668.1) -> 3444.6 (3648.6) MB, pooled: 0 MB, 96.37 / 0.00 ms  (+ 649.8 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 789 ms) (average mu = 0.262, 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

Aborted (core dumped)

AND, here is the error at 32gb

<--- Last few GCs --->

[148:0x7aed4cf6f000]    34166 ms: Scavenge (reduce) (interleaved) 3444.6 (3660.1) -> 3444.6 (3657.1) MB, pooled: 0 MB, 8.24 / 0.00 ms  (average mu = 0.262, current mu = 0.248) external memory pressure; 
[148:0x7aed4cf6f000]    34920 ms: Mark-Compact (reduce) 3444.7 (3657.1) -> 3443.1 (3638.9) MB, pooled: 0 MB, 99.86 / 0.00 ms  (+ 648.2 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 791 ms) (average mu = 0.256, 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

Aborted (core dumped)

anthonyrussano · 2024-09-30T20:37:38Z

here is the current workflow step I'm using in my github actions:

- name: Build site
  env:
    NODE_OPTIONS: --max-old-space-size=5168
  run: |
    npm install
    node_modules/hexo/bin/hexo gen -c 1

JLKM mentioned this issue Sep 25, 2016

how to use hexo generate more than 1000+ posts soulteary/hexo-blog#1

Closed

JLKM closed this as completed Sep 29, 2016

NoahDragon mentioned this issue Jun 25, 2017

hexo was killed because Out of memory #2621

Closed

githubbbie mentioned this issue Sep 30, 2024

Out of Memory on 33103 html and with image 671111 regular files #5552

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OutOfMemory with 3000 docs #2165

OutOfMemory with 3000 docs #2165

JLKM commented Sep 16, 2016

soulteary commented Sep 18, 2016

JLKM commented Sep 20, 2016 •

edited

Loading

soulteary commented Sep 21, 2016

soulteary commented Sep 21, 2016

JLKM commented Sep 25, 2016 •

edited

Loading

JLKM commented Sep 27, 2016 •

edited

Loading

JLKM commented Sep 27, 2016

JLKM commented Sep 29, 2016

hank583746309 commented Jul 10, 2018

Num142857 commented Nov 13, 2018

smoke-indica commented Jul 27, 2019 •

edited

Loading

anthonyrussano commented Jun 1, 2022

githubbbie commented Sep 30, 2024 •

edited

Loading

anthonyrussano commented Sep 30, 2024

OutOfMemory with 3000 docs #2165

OutOfMemory with 3000 docs #2165

Comments

JLKM commented Sep 16, 2016

soulteary commented Sep 18, 2016

JLKM commented Sep 20, 2016 • edited Loading

Still not able to generate more than approx. 1200 posts

Lessons learned from the parameters

The error-collection

Themes

Downgrade

Upgrade

soulteary commented Sep 21, 2016

soulteary commented Sep 21, 2016

JLKM commented Sep 25, 2016 • edited Loading

Back to square 1.

JLKM commented Sep 27, 2016 • edited Loading

Have given up generating more than 1000 docs at a time in Hexo

JLKM commented Sep 27, 2016

JLKM commented Sep 29, 2016

hank583746309 commented Jul 10, 2018

Num142857 commented Nov 13, 2018

smoke-indica commented Jul 27, 2019 • edited Loading

anthonyrussano commented Jun 1, 2022

githubbbie commented Sep 30, 2024 • edited Loading

anthonyrussano commented Sep 30, 2024

JLKM commented Sep 20, 2016 •

edited

Loading

JLKM commented Sep 25, 2016 •

edited

Loading

JLKM commented Sep 27, 2016 •

edited

Loading

smoke-indica commented Jul 27, 2019 •

edited

Loading

githubbbie commented Sep 30, 2024 •

edited

Loading