GC segfault

[Smbc1]

Known error (nodejs/node-v0.x-archive#25648) with segfault when using --expose-gc repeats on node 5.0.0

Here is callstack:

PID 2614 received SIGSEGV for address: 0xffffffffffffffa0
/<path to modules>/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1a13)[0x7f91d067ba13]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f91d14190a0]
next engine worker(_ZN2v88internal23PointersUpdatingVisitor12VisitPointerEPPNS0_6ObjectE+0x18)[0x99f6c8]
next engine worker(_ZN2v88internal13ObjectVisitor14VisitCodeEntryEPh+0x28)[0xac3988]
next engine worker(_ZN2v88internal20MarkCompactCollector29EvacuateNewSpaceAndCandidatesEv+0xeb9)[0x9bf249]
next engine worker(_ZN2v88internal20MarkCompactCollector11SweepSpacesEv+0x2f1)[0x9e4c51]
next engine worker(_ZN2v88internal20MarkCompactCollector14CollectGarbageEv+0x2a)[0x9ec90a]
next engine worker(_ZN2v88internal4Heap11MarkCompactEv+0x9f)[0x96882f]
next engine worker(_ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorENS_15GCCallbackFlagsE+0x288)[0x97c938]
next engine worker(_ZN2v88internal4Heap14CollectGarbageENS0_16GarbageCollectorEPKcS4_NS_15GCCallbackFlagsE+0xce)[0x97cf5e]
next engine worker(_ZN2v88internal4Heap17CollectAllGarbageEiPKcNS_15GCCallbackFlagsE+0x6c)[0x97d37c]
next engine worker(_ZN2v87Isolate34RequestGarbageCollectionForTestingENS0_21GarbageCollectionTypeE+0x3d)[0x80598d]
next engine worker(_ZN2v88internal25FunctionCallbackArguments4CallEPFvRKNS_20FunctionCallbackInfoINS_5ValueEEEE+0x9c)[0x82b78c]
next engine worker[0x84ddff]
[0x36cfbc2060a2]

[evanlucas]

Is there a reproducible test case that you can share with us to help identify the issue?

[calzoneman]

Not sure if this is related, but this consistently dumps core after 5 iterations of the while loop on my machine (Arch Linux, node.js v5.0.0):

var x = [];

while (true) {
    for (var i = 0; i < 10000000; i++) x.push(Math.random());
    console.log(process.memoryUsage().rss / 1000000);
}

[cjihrig]

@calzoneman I don't think they're related. In your case, the process seems to be running out of memory by continuously pushing to x.

[calzoneman]

It's not running out of memory (it crashes after only reaching ~1.1GB rss), I think I've tracked it down to v8 having a length limit on arrays. But you're right, I think it's unrelated.

[calzoneman]

Getting this on node v4.2.2 as well, when using --expose-gc:

#0  0x0000000000aec14b in v8::internal::PointersUpdatingVisitor::VisitPointer(v8::internal::Object**) ()
#1  0x0000000000bea673 in v8::internal::ObjectVisitor::VisitCodeEntry(unsigned char*) ()
#2  0x0000000000aef867 in v8::internal::SlotsBuffer::UpdateSlots(v8::internal::Heap*) ()
#3  0x0000000000af31ba in v8::internal::MarkCompactCollector::EvacuateNewSpaceAndCandidates() ()
#4  0x0000000000af4a03 in v8::internal::MarkCompactCollector::SweepSpaces() ()
#5  0x0000000000afe778 in v8::internal::MarkCompactCollector::CollectGarbage() ()
#6  0x0000000000ab5030 in v8::internal::Heap::MarkCompact() ()
#7  0x0000000000acc948 in v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) ()
#8  0x0000000000accee9 in v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) ()
#9  0x0000000000acd1e5 in v8::internal::Heap::CollectAllGarbage(int, char const*, v8::GCCallbackFlags) ()
#10 0x00000000008f2837 in v8::Isolate::RequestGarbageCollectionForTesting(v8::Isolate::GarbageCollectionType) ()
#11 0x0000000000903fa2 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#12 0x000000000092de61 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::(anonymous namespace)::BuiltinArguments<(v8::internal::BuiltinExtraArguments)1>&) ()
#13 0x000000000092e2be in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
...

I'll see if I can dumb it down to a testcase, but it happens very intermittently so I haven't been able to track down what triggers it.

[artofspeed]

happens to me too with v4.3.0 and v4.3.2

[MylesBorins]

have you tried v4.4.0

[artofspeed]

just tried, happened on 4.4.0 as well after 1 day of running

[MylesBorins]

@artofspeed do you have a minimal test case?

[akobler]

happened here in 4.4.7 - any progress/recommandations so far?

[bnoordhuis]

@akobler More info? Any way for us to reproduce?

[akobler]

@bnoordhuis Quite seldom, but I observed occasionally a Node crash and got the following line in kern.log:
master kernel: [3644710.877233] traps: node[4114] trap invalid opcode ip:fc9369 sp:7fff8455a4e8 error:0 in node[400000+1383000]

So I just enabled core dumps, unfortunately no deeper insight yet. As you see below, the pattern matches pretty good with other observations in this issue. I deliver stack traces as soon as available.

Some general info:

• Node 4.4.7
• Ubuntu 14.04, 64bit on AWS EC2
• --enable-gc on
• after fighting some memory leaks, I started to log memory usage: every 5min global.gc() and then process.memoryUsage() is triggered

[bnoordhuis]

You could try gdb --args node followed by info symbol 0xfc9369 (the ip address) to see which function it belongs to (if any, and assuming it hasn't been mangled by ASLR.)

If you don't have gdb installed, try addr2line -e node fc9369. addr2line is less sophisticated than gdb, though; I think it only looks through the symbol sections, not DWARF debug data.

[bnoordhuis]

Closing, no real activity in > 1 year. If anyone has a reliable way of reproducing, let me know and I'll reopen.

[simonecogno]

I have a similar problem in node 4.8.0. Anyone can give me some direction on how to fix this?

I get the following crash stack trace ocasionally without any apparent pattern:

PID 24495 received SIGSEGV for address: 0x3809f3d021f8
<path_node_modules>/segfault-handler/build/Release/segfault-handler.node(+0x1a5b)[0x7f7dd565ca5b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f7dd9c20890]
/usr/bin/nodejs(_ZN2v88internal20MarkCompactCollector22ProcessWeakCollectionsEv+0xfd)[0xaec4dd]
/usr/bin/nodejs(_ZN2v88internal20MarkCompactCollector15MarkLiveObjectsEv+0x214)[0xaf3a14]
/usr/bin/nodejs(_ZN2v88internal20MarkCompactCollector14CollectGarbageEv+0x11)[0xaf47e1]
/usr/bin/nodejs(_ZN2v88internal4Heap11MarkCompactEv+0x60)[0xaaafe0]
/usr/bin/nodejs(_ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorENS_15GCCallbackFlagsE+0x4c0)[0xac2be0]
/usr/bin/nodejs(_ZN2v88internal4Heap14CollectGarbageENS0_16GarbageCollectorEPKcS4_NS_15GCCallbackFlagsE+0x238)[0xac30f8]
/usr/bin/nodejs(_ZN2v88internal4Heap15HandleGCRequestEv+0x8f)[0xac3aef]
/usr/bin/nodejs(_ZN2v88internal10StackGuard16HandleInterruptsEv+0x31c)[0xa6041c]
/usr/bin/nodejs(_ZN2v88internal18Runtime_StackGuardEiPPNS0_6ObjectEPNS0_7IsolateE+0x2b)[0xca51ab]
[0x2f2137d0963b]

[addaleax]

@simonecogno Can you let us know:

• Whether you use any native addons (find node_modules -name '*.node')
• Any code that may help in reproducing your problem
• If possible, whether this occurs with other versions of Node (v6.x, v7.x), too

and open a new issue with that information?