Javascript Heap out of Memory

[mansona]

Howdy 👋

Now let me prefix this with I know what I'm doing is non-standard, so if this isn't something that we really care about feel free to just close the issue.

This happens when I'm trying to pre-render roughly 2K pages during an ember-cli-deploy, I can only imagine that this is because we're rendering into memory as we go along?

My workaround solution is that I'm going to be splitting the deploy into multiple batches of roughly 300 pages using travis, but I just thought I would let you know about this issue 👍

<--- Last few GCs --->

[63121:0x102801e00]  3023388 ms: Mark-sweep 1408.1 (1547.8) -> 1408.1 (1539.8) MB, 1946.9 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1948 ms) last resort
[63121:0x102801e00]  3025254 ms: Mark-sweep 1408.1 (1539.8) -> 1408.1 (1539.8) MB, 1865.9 / 0.0 ms  last resort


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xf64a141cef1 <JSObject>
    1: _addHeaderLine [_http_incoming.js:~286] [pc=0x2f434bbc5b01](this=0x7d3a4824dd9 <IncomingMessage map = 0x1f01f81673b1>,field=0x7d3a4826879 <String[10]: x-amz-id-2>,value=0x7d3a48268a1 <String[76]: xEYHNmtXDBpKxSTsH6gbEKL8FpRLYSr7HRdnXoo5+M9vsGcHuZ9jWmTQ0+tQmXehx2kt/45Hzlw=>,dest=0x7d3a4823d49 <Object map = 0x2701ded84829>)
    3: _addHeaderLines [_http_incoming.js:131] [bytecode=0x25c435409...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::Factory::NewTransitionArray(int) [/usr/local/bin/node]
 5: v8::internal::TransitionArray::Insert(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::SimpleTransitionFlag) [/usr/local/bin/node]
 6: v8::internal::Map::CopyReplaceDescriptors(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Handle<v8::internal::LayoutDescriptor>, v8::internal::TransitionFlag, v8::internal::MaybeHandle<v8::internal::Name>, char const*, v8::internal::SimpleTransitionFlag) [/usr/local/bin/node]
 7: v8::internal::Map::CopyAddDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [/usr/local/bin/node]
 8: v8::internal::Map::CopyWithField(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::FieldType>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::TransitionFlag) [/usr/local/bin/node]
 9: v8::internal::Map::TransitionToDataProperty(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
10: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
11: v8::internal::StoreIC::LookupForWrite(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
12: v8::internal::StoreIC::UpdateCaches(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
13: v8::internal::StoreIC::Store(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node]
14: v8::internal::KeyedStoreIC::Store(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
15: v8::internal::Runtime_KeyedStoreIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
16: 0x2f434b4040dd
[1]    63121 abort      ember deploy production --activate

[ef4]

Thanks, this is a legit bug, probably a memory leak.

…

On Sun, Jan 21, 2018 at 5:50 AM Chris Manson ***@***.***> wrote: Howdy 👋 Now let me prefix this with I *know* what I'm doing is non-standard, so if this isn't something that we really care about feel free to just close the issue. This happens when I'm trying to pre-render roughly 2K pages during an ember-cli-deploy, I can only imagine that this is because we're rendering into memory as we go along? My workaround solution is that I'm going to be splitting the deploy into multiple batches of roughly 300 pages using travis, but I just thought I would let you know about this issue 👍 <--- Last few GCs ---> [63121:0x102801e00] 3023388 ms: Mark-sweep 1408.1 (1547.8) -> 1408.1 (1539.8) MB, 1946.9 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1948 ms) last resort [63121:0x102801e00] 3025254 ms: Mark-sweep 1408.1 (1539.8) -> 1408.1 (1539.8) MB, 1865.9 / 0.0 ms last resort <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0xf64a141cef1 <JSObject> 1: _addHeaderLine [_http_incoming.js:~286] [pc=0x2f434bbc5b01](this=0x7d3a4824dd9 <IncomingMessage map = 0x1f01f81673b1>,field=0x7d3a4826879 <String[10]: x-amz-id-2>,value=0x7d3a48268a1 <String[76]: xEYHNmtXDBpKxSTsH6gbEKL8FpRLYSr7HRdnXoo5+M9vsGcHuZ9jWmTQ0+tQmXehx2kt/45Hzlw=>,dest=0x7d3a4823d49 <Object map = 0x2701ded84829>) 3: _addHeaderLines [_http_incoming.js:131] [bytecode=0x25c435409... FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [/usr/local/bin/node] 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node] 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node] 4: v8::internal::Factory::NewTransitionArray(int) [/usr/local/bin/node] 5: v8::internal::TransitionArray::Insert(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::SimpleTransitionFlag) [/usr/local/bin/node] 6: v8::internal::Map::CopyReplaceDescriptors(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Handle<v8::internal::LayoutDescriptor>, v8::internal::TransitionFlag, v8::internal::MaybeHandle<v8::internal::Name>, char const*, v8::internal::SimpleTransitionFlag) [/usr/local/bin/node] 7: v8::internal::Map::CopyAddDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [/usr/local/bin/node] 8: v8::internal::Map::CopyWithField(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::FieldType>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::TransitionFlag) [/usr/local/bin/node] 9: v8::internal::Map::TransitionToDataProperty(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node] 10: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node] 11: v8::internal::StoreIC::LookupForWrite(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node] 12: v8::internal::StoreIC::UpdateCaches(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node] 13: v8::internal::StoreIC::Store(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node] 14: v8::internal::KeyedStoreIC::Store(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node] 15: v8::internal::Runtime_KeyedStoreIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node] 16: 0x2f434b4040dd [1] 63121 abort ember deploy production --activate — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATfMhxAsEEwLaXOQyQiyAbBOEMOam_Oks5tMxZ6gaJpZM4RlxcY> .

[mansona]

well it really slows down the further that you get... although one thing I have noticed is that there is an error coming out on the console so it could be in the case where the fastboot dies (for whatever reason) but I haven't replicated this issue in fastboot yet

[mansona]

So this seemed to go away once I stopped fastboot crashing on every other request (for reference see emberjs/ember.js#15662 (comment)) so I wonder if the memory leak is something to do with the error handling side of things in either prember or fastboot itself 🤔

[ianpetzer]

We are also running into this issue :(

@mansona I assume this was related to generating ember-learn/guides-app? Could you possibly confirm that you are now able to generate 2k+ pages using prember? If you are that would certainly imply that the memory leak is in my app. We are running of ember 3.5 / fastboot 2 / prember 1.0

@ef4 @mansona Do either of you know if there are any examples of running prember in batches as @mansona suggested above? Or running prember in some kind of multi-process configuration with a pool of prember workers? We're looking to generate 6k+ pages on each deploy and when data changes.

Thanks for any advise!

[mansona]

@ianpetzer yes this was related to generating the https://github.com/ember-learn/guides-app. I'm pretty sure this error was related to the error handling in fastboot and the prember process. At the time that I reported this error there were quite a few pages causing fastboot to throw an error, which we didn't actually care too much about because it was the older pages in the guides and even if they weren't pre-rendered they would still work (the Ember app still ran)

So my first question is: are you seeing any errors in your logs when you're doing the prember?

Secondly I have thought quite hard about this idea of either incremental builds or some way to only build pages that the data has changed, and in the case of the Ember Guides app we have tended towards simplicity. Even though we almost never update older versions of the Guides data (it's mostly policy to only change major bugs instead of the content) we still build all previous versions of the Guides on every build because it's just simpler to reason about.

Now it is clear that you are feeling more pain because of this but prember is very fixable from the "user space" of your application if you want to take the time to implement a solution. If you take a look how we dynamically build the urls that are ultimately reported as urlsforPrember() https://github.com/ember-learn/guides-app/blob/master/lib/content-guides-generator/index.js#L28-L42 you will see that it should be relatively straight-forward to add some sort of batch or filtering functionality as part of your build process. Especially if your urls are deterministic (i.e. the same between builds)

I don't know if this helps 😖but let me know if you would like me to go any deeper or explain anything.

[ianpetzer]

Hi @mansona,

Thanks for the comprehensive response!

With regard to your questions:

1. There are currently no errors in the logs. We did have a cannot read property 'createComment' of undefined error but I managed to resolve that.
2. Thanks for that link. We will be supplying the urls from our api as that has the list of entities and their friendly ids that we need to generate pages for and will hopefully allow us to generate partial builds when someone changes an entity in the database.
   I'm pretty confident on how to split urls.

What I wasn't sure about was how I would set up prember to only do 200 pages at a time to avoid the memory leak if I can't fix it. I can get our api to give 200 urls at a time, but I was curious if you had any solution for running a pool of prember processes each of which get 200 urls from the api and then shuts down before running out of memory

[sandstrom]

I'm doing some issue gardening 🌱🌿 🌷 and came upon this issue. Since it's quite old I just wanted to ask if this is still relevant? If it isn't, maybe we can close this issue?

By closing some old issues we reduce the list of open issues to a more manageable set.