V8 serialize error on build with huge number of pages(100k+) #17233
Comments
|
@sidharthachatterjee As discussed the other day, created a repro here. Please look into it 👍 |
|
Probably some related stuffs! I still get |
|
Awesome @ganapativs thanks for creating this repro. |
gatsbot
bot
added
the
stale?
|
Hiya! This issue has gone quiet. Spooky quiet. 👻 We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open! As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing! Thanks for being a part of the Gatsby community! 💪💜 |
ganapativs
removed
the
stale?
|
Hiya! This issue has gone quiet. Spooky quiet. 👻 We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open! As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing! Thanks for being a part of the Gatsby community! 💪💜 |
gatsbot
bot
added
the
stale?
|
Hey again! It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it. Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing! Thanks again for being part of the Gatsby community! |
|
This is a serious scalability bug! Auto closing isn't helpful here. |
LekoArts
removed
the
stale?
|
cc @gatsbyjs/core |
|
Currently facing this issue with a production site running on gatsby. |
|
I have tested this issue on Updated the repro repo. |
|
Does anyone have an idea how to prevent this from happening? My site has "only" 30k pages but I run into this issue. |
|
Ok, I am looking into it now. Consider this a research post while I'm trying to dig in. I think nodejs/help#1059 is interesting because that implies that there shouldn't be a concrete difference between the v8.serialize and json.stringify, apart from a more aggressive gc schedule. This could very well be the reason. Additionally we might consider that the performance improvement of using v8.serialize over json.stringify is only perceived and the cost is ultimately still paid before the process exits. That's an interesting fact. Keep in mind, async operations may change the impact, as an async operation might give nodejs more idle time to run GC. Of course, if postponing GC leads to OOMs we need to re-evaluate that. The repro. I had to install mongo ( I had to update the script a tiny bit to get the repo working; because I was getting "invalid id" errors by mongo (added the suffix because why not, I realize the repo doesn't have that). Running it on 10k pages without bumping the memory quickly OOMs during sourcing. Running it with 12GB; Memory consumption remains fairly stable during the sourcing step (~4gb?). Here's my runtime output: The crash is not a regular OOM but a string length assertion error. Perhaps the serialized content is too much to bear. (After all, there are inherent limits to nodejs, a maximum string length is one of them.) The above ran on [email protected] in node v10.17. Bumping it to [email protected] ( This time the sourcing step improved, but there was no change to the run queries time. Debugging that it seems it doesn't use a filter at all. It seems the slowness is coming from the html handler. Will have to look into that later. After about three or four restarts (while debugging) the build now OOMs during the createPages step, which took 2s before. And I cannot get it to move forward. In this case I can see the memory grow (relatively) rapidly and after ~2 minutes the 12gb are up and it OOMs. I took a break and picked it up the next day. When I got back to it this step was fine again, not sure what is causing this... Can anyone reliably repro this problem? This makes me wonder whether there aren't two things at play here. My OOM problem certainly seems to stem from another source. I use Regenerating the db with 100 pages makes the run finish fine, in 10s. Not unexpected but good to see that still works. I checked into why the queries run so slow. Turns out they are actually under reporting their work; each query is running remark for every post on the page. By default there are 30 pages on each page so remark is called 30 times for that fact alone, but it visually counts as one step for the query. If I go into gatsby-transformer-remark and add Then I confirmed whether this problem pops up with JSON.stringify; Change the v8 bits in Now this isn't necessarily a surprise. I compared the v8.serialize to the old way of doing json.stringify (using https://github.com/stefanprobst/gatsby/blob/c043816915c0e4b632730091c1d14df08d6249d4/packages/gatsby/src/redux/persist.js as a reference point). Both ways dump about 500k of data. Before I "checked" the assert with a simple Next I checked whether I could catch the assertion error. This is more difficult because the error happens somewhere inside the serialzation call and we don't control the assert. It doesn't appear to be The way forward So after some discussion we're going to try to chunk the pages nodes. Ultimately we're hitting the arbitrary buffer limit and there's no easy way to fix that. So instead we'll first serialize the redux state without the page nodes. Then we'll try to apply some heuristics to chunk the page nodes such that they stay well below this limit. This means the redux state has to be serialized across multiple files, but that should also means that it won't fatal. We're gonna be a little busy the next week but watch this space. |
|
Thanks for picking this up and that's a great analysis. I'll try to spend some time this weekend on this and see if I can figure out something more. |
We are using `v8.serialize` to write and read the redux state. This is faster than `JSON.parse`. Unfortunately, as reported in #17233, this can lead to a fatal when the contents of the redux state is too big to be serialized to a Buffer (hard max of 2GB). Alternatively, we also hit this problem on large site like a million small md pages. The solution is to shard the `nodes` property, which holds all the page data. In this change I've added a simple heuristic to determine the max chunk size (mind you, currently that's basically `Infinity`). It will serialize about 11 individual nodes, measure their size, and based on the biggest node determine how many nodes would fit in 1.5GB. The serialization process is updated to no longer put the `nodes` in the main redux file, but rather sharded over a few specific files. When reading the state from cache, these files are all read and their contents are put together in a single Map again. If there were no nodes files this part does nothing so it's even backwards compatible.
|
For anyone bothered by this, can you confirm whether #21555 fixes your issue? I expect to merge this soon so if you're not comfortable with how to check this you can also wait for a bump. |
We are using `v8.serialize` to write and read the redux state. This is faster than `JSON.parse`. Unfortunately, as reported in #17233, this can lead to a fatal when the contents of the redux state is too big to be serialized to a Buffer (hard max of 2GB). Alternatively, we also hit this problem on large site like a million small md pages. The solution is to shard the `nodes` property, which holds all the page data. In this change I've added a simple heuristic to determine the max chunk size (mind you, currently that's basically `Infinity`). It will serialize about 11 individual nodes, measure their size, and based on the biggest node determine how many nodes would fit in 1.5GB. The serialization process is updated to no longer put the `nodes` in the main redux file, but rather sharded over a few specific files. When reading the state from cache, these files are all read and their contents are put together in a single Map again. If there were no nodes files this part does nothing so it's even backwards compatible.
We are using `v8.serialize` to write and read the redux state. This is faster than `JSON.parse`. Unfortunately, as reported in #17233, this can lead to a fatal when the contents of the redux state is too big to be serialized to a Buffer (hard max of 2GB). Alternatively, we also hit this problem on large site like a million small md pages. The solution is to shard the `nodes` property, which holds all the page data. In this change I've added a simple heuristic to determine the max chunk size (mind you, currently that's basically `Infinity`). It will serialize about 11 individual nodes, measure their size, and based on the biggest node determine how many nodes would fit in 1.5GB. The serialization process is updated to no longer put the `nodes` in the main redux file, but rather sharded over a few specific files. When reading the state from cache, these files are all read and their contents are put together in a single Map again. If there were no nodes files this part does nothing so it's even backwards compatible. Because the write is no longer atomized, the process will now write the redux cache to its own `redux` folder. When writing a new cache it will prepare the new cache in a tmp folder first, then move the existing `redux` folder to a temp location, move the new folder to `redux`, and then try to drop the old folder. This is about as transactional as you can get and should leave the cache in either a stale, empty, or updated state. But never in a partial state.
We are using `v8.serialize` to write and read the redux state. This is faster than `JSON.parse`. Unfortunately, as reported in #17233, this can lead to a fatal when the contents of the redux state is too big to be serialized to a Buffer (hard max of 2GB). Alternatively, we also hit this problem on large site like a million small md pages. The solution is to shard the `nodes` property, which holds all the page data. In this change I've added a simple heuristic to determine the max chunk size (mind you, currently that's basically `Infinity`). It will serialize about 11 individual nodes, measure their size, and based on the biggest node determine how many nodes would fit in 1.5GB. The serialization process is updated to no longer put the `nodes` in the main redux file, but rather sharded over a few specific files. When reading the state from cache, these files are all read and their contents are put together in a single Map again. If there were no nodes files this part does nothing so it's even backwards compatible.
We are using `v8.serialize` to write and read the redux state. This is faster than `JSON.parse`. Unfortunately, as reported in #17233, this can lead to a fatal when the contents of the redux state is too big to be serialized to a Buffer (hard max of 2GB). Alternatively, we also hit this problem on large site like a million small md pages. The solution is to shard the `nodes` property, which holds all the page data. In this change I've added a simple heuristic to determine the max chunk size (mind you, currently that's basically `Infinity`). It will serialize about 11 individual nodes, measure their size, and based on the biggest node determine how many nodes would fit in 1.5GB. The serialization process is updated to no longer put the `nodes` in the main redux file, but rather sharded over a few specific files. When reading the state from cache, these files are all read and their contents are put together in a single Map again. If there were no nodes files this part does nothing so it's even backwards compatible. Because the write is no longer atomized, the process will now write the redux cache to its own `redux` folder. When writing a new cache it will prepare the new cache in a tmp folder first, then move the existing `redux` folder to a temp location, move the new folder to `redux`, and then try to drop the old folder. This is about as transactional as you can get and should leave the cache in either a stale, empty, or updated state. But never in a partial state.
We are using `v8.serialize` to write and read the redux state. This is faster than `JSON.parse`. Unfortunately, as reported in #17233, this can lead to a fatal when the contents of the redux state is too big to be serialized to a Buffer (hard max of 2GB). Alternatively, we also hit this problem on large site like a million small md pages. The solution is to shard the `nodes` property, which holds all the page data. In this change I've added a simple heuristic to determine the max chunk size (mind you, currently that's basically `Infinity`). It will serialize about 11 individual nodes, measure their size, and based on the biggest node determine how many nodes would fit in 1.5GB. The serialization process is updated to no longer put the `nodes` in the main redux file, but rather sharded over a few specific files. When reading the state from cache, these files are all read and their contents are put together in a single Map again. If there were no nodes files this part does nothing so it's even backwards compatible. Because the write is no longer atomized, the process will now write the redux cache to its own `redux` folder. When writing a new cache it will prepare the new cache in a tmp folder first, then move the existing `redux` folder to a temp location, move the new folder to `redux`, and then try to drop the old folder. This is about as transactional as you can get and should leave the cache in either a stale, empty, or updated state. But never in a partial state.
We are using `v8.serialize` to write and read the redux state. This is faster than `JSON.parse`. Unfortunately, as reported in #17233, this can lead to a fatal when the contents of the redux state is too big to be serialized to a Buffer (hard max of 2GB). Alternatively, we also hit this problem on large site like a million small md pages. The solution is to shard the `nodes` property, which holds all the page data. In this change I've added a simple heuristic to determine the max chunk size (mind you, currently that's basically `Infinity`). It will serialize about 11 individual nodes, measure their size, and based on the biggest node determine how many nodes would fit in 1.5GB. The serialization process is updated to no longer put the `nodes` in the main redux file, but rather sharded over a few specific files. When reading the state from cache, these files are all read and their contents are put together in a single Map again. If there were no nodes files this part does nothing so it's even backwards compatible. Because the write is no longer atomized, the process will now write the redux cache to its own `redux` folder. When writing a new cache it will prepare the new cache in a tmp folder first, then move the existing `redux` folder to a temp location, move the new folder to `redux`, and then try to drop the old folder. This is about as transactional as you can get and should leave the cache in either a stale, empty, or updated state. But never in a partial state.
We are using `v8.serialize` to write and read the redux state. This is faster than `JSON.parse`. Unfortunately, as reported in #17233, this can lead to a fatal when the contents of the redux state is too big to be serialized to a Buffer (hard max of 2GB). Alternatively, we also hit this problem on large site like a million small md pages. The solution is to shard the `nodes` property, which holds all the page data. In this change I've added a simple heuristic to determine the max chunk size (mind you, currently that's basically `Infinity`). It will serialize about 11 individual nodes, measure their size, and based on the biggest node determine how many nodes would fit in 1.5GB. The serialization process is updated to no longer put the `nodes` in the main redux file, but rather sharded over a few specific files. When reading the state from cache, these files are all read and their contents are put together in a single Map again. If there were no nodes files this part does nothing so it's even backwards compatible. Because the write is no longer atomized, the process will now write the redux cache to its own `redux` folder. When writing a new cache it will prepare the new cache in a tmp folder first, then move the existing `redux` folder to a temp location, move the new folder to `redux`, and then try to drop the old folder. This is about as transactional as you can get and should leave the cache in either a stale, empty, or updated state. But never in a partial state.
* fix(gatsby): Chunk nodes when serializing redux to prevent OOM We are using `v8.serialize` to write and read the redux state. This is faster than `JSON.parse`. Unfortunately, as reported in #17233, this can lead to a fatal when the contents of the redux state is too big to be serialized to a Buffer (hard max of 2GB). Alternatively, we also hit this problem on large site like a million small md pages. The solution is to shard the `nodes` property, which holds all the page data. In this change I've added a simple heuristic to determine the max chunk size (mind you, currently that's basically `Infinity`). It will serialize about 11 individual nodes, measure their size, and based on the biggest node determine how many nodes would fit in 1.5GB. The serialization process is updated to no longer put the `nodes` in the main redux file, but rather sharded over a few specific files. When reading the state from cache, these files are all read and their contents are put together in a single Map again. If there were no nodes files this part does nothing so it's even backwards compatible. Because the write is no longer atomized, the process will now write the redux cache to its own `redux` folder. When writing a new cache it will prepare the new cache in a tmp folder first, then move the existing `redux` folder to a temp location, move the new folder to `redux`, and then try to drop the old folder. This is about as transactional as you can get and should leave the cache in either a stale, empty, or updated state. But never in a partial state.
|
The fix was published in [email protected] Please report if there are still problems and provide a repro if that's the case. (And hey, if your issue is now fixed, let me know too :) ) |
|
Thank you very much 👍 |
|
This, or a similar problem still occurs for me. Could you confirm this is related? #21957 |
|
Hey @pvdz my team is running into the same issue. Gatsby build breaks right after run queries with the same error log: I updated gatsby to the version mentioned and looked into gatsby-source-contentful. A site with 15k+ pages and a lot of content in rich text editor. Could this be another issue? |
|
@nadiavanleur @j218 There was a typo (see last linked PR) which kind of mooted the fix in some cases. Can you confirm whether the problem still exists in >[email protected] ? |
Description
Getting various issues(related to V8 serialize etc) when trying to build large number of pages(80k+ docs of 10kb each) with latest gatsby+remark resulting in the build failure.
Basically build crashes with below errors.
Without loki
With loki(
GATSBY_DB_NODES=loki)success run page queries - 1976.632 s — 3335/3335 1.69 queries/second Stacktrace: ptr1=0x25b9d2202321 ptr2=0x0 ptr3=0x0 ptr4=0x0 failure_message_object=0x7ffeefbed370 ==== JS stack trace ========================================= 0: ExitFrame [pc: 0x355f68c041bd] 1: StubFrame [pc: 0x355f68c85ad7] Security context: 0x25b9bbd9e6c9 <JSObject> 2: saveState [0x25b910175701] [/Users/guns/public/node_modules/gatsby/dist/db/index.js:30] [bytecode=0x25b9097edb61 offset=181](this=0x25b90cd96279 <Object map = 0x25b90f1ad969>) 3: /* anonymous */ [0x25b9850fee71](this=0x25b93d108c59 <JSGlobal Object>,0x25b9d2202321 <the_hole>) 4: StubFrame [pc: 0x355f68c42871] 5: StubFrame [pc: 0x355f68c21b9a] 6: EntryFrame [pc: 0x355f68c0ba01] ==== Details ================================================ [0]: ExitFrame [pc: 0x355f68c041bd] [1]: StubFrame [pc: 0x355f68c85ad7] [2]: saveState [0x25b910175701] [/Users/guns/public/node_modules/gatsby/dist/db/index.js:30] [bytecode=0x25b9097edb61 offset=181](this=0x25b90cd96279 <Object map = 0x25b90f1ad969>) { // stack-allocated locals var .generator_object = 0x25baaacb2ee9 <JSGenerator> var /* anonymous */ = 0x25baaacb2eb9 <Promise map = 0x25b9b4783e89> // expression stack (top to bottom) [11] : 0x25b9d2202321 <the_hole> [10] : 0x25b9097ed889 <String[24]: Error persisting state: > [09] : 0x25b921b04c89 <Object map = 0x25b983544361> [08] : 0x25b9794ede29 <JSBoundFunction (BoundTargetFunction 0x25b9794ecbf1)> [07] : 0x25b9101767b9 <FunctionContext[9]> [06] : 0x25b9850ff331 <CatchContext[5]> [05] : 0x25b9101767b9 <FunctionContext[9]> [04] : 0x25b9101767b9 <FunctionContext[9]> [03] : 0x25b90cd96279 <Object map = 0x25b90f1ad969> [02] : 0x25b910175701 <JSFunction saveState (sfi = 0x25b9996e74f9)> --------- s o u r c e c o d e --------- function saveState() {\x0a if (saveInProgress) return;\x0a saveInProgress = true;\x0a\x0a try {\x0a await Promise.all(dbs.map(db => db.saveState()));\x0a } catch (err) {\x0a report.warn(`Error persisting state: ${err && err.message || err}`);\x0a }\x0a\x0a saveInProgress = false;\x0a} ----------------------------------------- } [3]: /* anonymous */ [0x25b9850fee71](this=0x25b93d108c59 <JSGlobal Object>,0x25b9d2202321 <the_hole>) { // optimized frame --------- s o u r c e c o d e --------- <No Source> ----------------------------------------- } [4]: StubFrame [pc: 0x355f68c42871] [5]: StubFrame [pc: 0x355f68c21b9a] [6]: EntryFrame [pc: 0x355f68c0ba01] ===================== error Command failed with signal "SIGILL". info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.Interestingly the build for 200k(4.5kb each post) runs successfully on
[email protected]which usesJSON.stringifyto persist state(shows a redux persisting state warning, but everything works).Steps to reproduce
Repro repo: https://github.com/ganapativs/gatsby-v8-issue-repro (README has everything related to the issue and other observations).
Expected result
Build should be successful without
V8 serializeerror.Actual result
Build crashed with
V8 serializeerror.DANGEROUSLY_DISABLE_OOMwould have helped temporarily, but, it was removed recently 😅Environment
System:
OS: macOS 10.15
CPU: (8) x64 Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Shell: 5.7.1 - /bin/zsh
Binaries:
Node: 10.6.0 - /usr/local/bin/node
Yarn: 1.7.0 - ~/.yarn/bin/yarn
npm: 6.1.0 - /usr/local/bin/npm
Languages:
Python: 2.7.16 - /usr/bin/python
Browsers:
Chrome: 76.0.3809.132
Safari: 13.0
npmPackages:
gatsby: 2.14.0 => 2.14.0
gatsby-plugin-styled-components: 3.1.3 => 3.1.3
gatsby-remark-autolink-headers: 2.1.8 => 2.1.8
gatsby-remark-prismjs: 3.3.9 => 3.3.9
gatsby-remark-sub-sup: 1.0.0 => 1.0.0
gatsby-source-mongodb: 2.1.9 => 2.1.9
gatsby-transformer-remark: 2.6.19 => 2.6.19
npmGlobalPackages:
gatsby-cli: 2.7.40
The text was updated successfully, but these errors were encountered: