Segfault on Ubuntu 14.04

[jshkurti]

This is basically a duplicate of Unitech/pm2#1380

TL;DR: Node.js v0.12.4 segfaults.

kern.log:

Jun 22 15:06:02 Ubuntu-1404-trusty-64-minimal kernel: [569531.959543] PM2 v0.14.1: Go[19539]: segfault at 3fffffffa0 ip 0000000000902e38 sp 00007fff99f29290 error 4 in nodejs[400000+b60000]

Any thoughts on this ?
Thanks

@suprMax @NikitaKrasavtsev @membrive @jhansen-tt @soyuka

[max-degterev]

@wreckah

[misterdjules]

Are you able to get a core dump when node crashes? With a core dump we could at least look at a call stack and have information about native frames.

[membrive]

My node version is v0.12.6, installed on Ubuntu Server 14.04.2. This is what I see in gdb (gdb /usr/local/bin/node coredumpfile):

warning: core file may not match specified executable file.
[New LWP 1560]
[New LWP 1564]
[New LWP 1565]
[New LWP 1566]
[New LWP 1567]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: the debug information found in "/lib64/ld-2.19.so" does not match "/lib64/ld-linux-x86-64.so.2" (CRC mismatch).

Core was generated by `PM2 v0.14.3: God Daemon                                                  '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000903818 in v8::internal::PointersUpdatingVisitor::VisitPointer(v8::internal::Object**) ()
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
(gdb) bt no-filters full
#0  0x0000000000903818 in v8::internal::PointersUpdatingVisitor::VisitPointer(v8::internal::Object**) ()
No symbol table info available.
#1  0x00000000009fc1f3 in v8::internal::ObjectVisitor::VisitCodeEntry(unsigned char*) ()
No symbol table info available.
#2  0x000000000090f93c in v8::internal::UpdateSlot(v8::internal::Isolate*, v8::internal::ObjectVisitor*, v8::internal::SlotsBuffer::SlotType, unsigned char*) ()
No symbol table info available.
#3  0x000000000092640e in v8::internal::MarkCompactCollector::EvacuateNewSpaceAndCandidates() ()
No symbol table info available.
#4  0x000000000092a0e3 in v8::internal::MarkCompactCollector::SweepSpaces() ()
No symbol table info available.
#5  0x000000000092a20a in v8::internal::MarkCompactCollector::CollectGarbage() ()
No symbol table info available.
#6  0x00000000008e0fd0 in v8::internal::Heap::MarkCompact() ()
No symbol table info available.
#7  0x00000000008f039d in v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) ()
No symbol table info available.
#8  0x00000000008f07ec in v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) ()
No symbol table info available.
#9  0x00000000008f0ace in v8::internal::Heap::CollectAllGarbage(int, char const*, v8::GCCallbackFlags) ()
No symbol table info available.
#10 0x00000000007adce7 in v8::Isolate::RequestGarbageCollectionForTesting(v8::Isolate::GarbageCollectionType) ()
No symbol table info available.
#11 0x00000000007b8f42 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
No symbol table info available.
#12 0x00000000007d9bd1 in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
No symbol table info available.
#13 0x0000171905f0740e in ?? ()
No symbol table info available.
#14 0x0000171905f07361 in ?? ()
No symbol table info available.
#15 0x00007ffc5e22cc50 in ?? ()
No symbol table info available.
#16 0x00007ffc5e22cc98 in ?? ()
No symbol table info available.
#17 0x0000171907d97422 in ?? ()
No symbol table info available.
#18 0x000029b87e1afd41 in ?? ()
No symbol table info available.
#19 0x000029b87e150bf1 in ?? ()
No symbol table info available.
#20 0x000029b87e1afd41 in ?? ()
No symbol table info available.
#21 0x000013174f5ceb49 in ?? ()
No symbol table info available.
#22 0x0000347d51f91ee1 in ?? ()
No symbol table info available.
#23 0x00007ffc5e22ccd8 in ?? ()
No symbol table info available.
#24 0x0000171905f07b75 in ?? ()
No symbol table info available.
#25 0x000029b87e104121 in ?? ()
No symbol table info available.
#26 0x000029b87e104121 in ?? ()
No symbol table info available.
#27 0x0000347d51f96bd9 in ?? ()
No symbol table info available.
#28 0x0000000000000000 in ?? ()
No symbol table info available.

I don't know if it helps.

[misterdjules]

@membrive That helps a lot, thank you! From the callstack in your core dump, it seems similar to #15446, which relates to https://code.google.com/p/chromium/issues/detail?id=408380 in Chromium and https://code.google.com/p/v8/issues/detail?id=2996 in V8.

@jshkurti @suprMax Do you get a callstack (from a core dump or by any other mean) that is similar to the one posted by @membrive?

If so, we'll close this as a duplicate of #15446 and continue the discussion over there.

[misterdjules]

@membrive @jshkurti @suprMax Also, do you have any hint on how to reproduce this crash?

[membrive]

@misterdjules I don't know how to reproduce it. It just happens. I ran PM2 in three servers with Ubuntu Server 14.04 but with different kernel versions (3.10 from OVH, 3.13 from Ubuntu, and 3.16 from Ubuntu), and it crashes totally random.

I also tried with Node.js 0.12.2, 0.12.4 and 0.12.6, prebuilt from nodesource and nodejs.org, and also built from source code, without results.

Some days it crash twice or more, making impossible using it reliably.

Thank you for your help!

[soyuka]

We run with debian 7 in production with pm2 and a lot of nodejs apps (complex ones) with no segfault at all. This might be related to ubuntu. I tried to reproduce on a vm without success.

[misterdjules]

@soyuka Thank you for providing some feedback! What version of node are you running?

[soyuka]

0.12.3+

Quoting from the original issue:

I'm using node 0.12.3 with pm2 0.14.1 with a 20D+ uptime (debian 7).

On another server I've pm2 0.14.3 with node 0.12.5 1 ghost blog (8D uptime atm) and another express app without segfault (debian 8).

Those are the same versions they experienced segfaults with.

[max-degterev]

@misterdjules sorry we don't have any more dumps to help with it. As @membrive said it just crashes for no apparent reason. Sometimes it does this lovely thing too

[membrive]

If you want to generate a stack trace to post it here, follow the next steps:

• Install gdb if you hasn't got it installed already.

aptitude -y install gdb

• Activate core dumps on the operating system:

sudo ulimit -c unlimited

• Check that ulimit command had effect (if unlimited, it's ok):

membrive@my-machine ~$ ulimit
unlimited

• Run PM2 and wait for it to break again. The core file will be generated in the same directory that you executed PM2. After that:

gdb /usr/local/bin/node /path/to/core
(gdb) bt no-filters full (copy the result and paste it here)
(gdb) Quit

• Now you can deactivate the core dumps:

ulimit -c 0 (or restart the computer)

And that's all. Thank you!

[wangjia184]

do you have any native plugin in your nodejs app?

[soyuka]

I was reading the pm2 code and I stuck on those two lines that were added in the pm2 0.14 release:

  node_args.push('--expose-gc'); // Allows manual GC in the code
  node_args.push('--gc-global'); // Does full GC (smaller memory footprint)

source

We had no segfault with pm2 0.12.x so this might be the problem source (#15446). I still can't reproduce :|.

@wangjia184 regarding pm2 there are no native plugins enabled on ubuntu.

[soyuka]

So, issue seems to be related to the garbage collector and caused by global.gc called here (when gc is exposed ofc).

[jshkurti]

After further inspection it looks like node's flag --gc-global combined with --expose-gc and global.gc() was the source of this bug.
Closing this now.

[misterdjules]

@jshkurti Isn't this still a bug in Node.js' core even though you fixed what triggered it in PM2's source?

[jshkurti]

It is, indeed :/
Feel free to reopen the issue :)

[misterdjules]

@jshkurti Thanks!

[Smbc1]

Node 0.12.7, no PM2 in use, --expose-gc flag is on, no --gc-global flag. Trace:

PID 2614 received SIGSEGV for address: 0xffffffffffffffa0
/<path to modules>/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1a13)[0x7f91d067ba13]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f91d14190a0]
next engine worker(_ZN2v88internal23PointersUpdatingVisitor12VisitPointerEPPNS0_6ObjectE+0x18)[0x99f6c8]
next engine worker(_ZN2v88internal13ObjectVisitor14VisitCodeEntryEPh+0x28)[0xac3988]
next engine worker(_ZN2v88internal20MarkCompactCollector29EvacuateNewSpaceAndCandidatesEv+0xeb9)[0x9bf249]
next engine worker(_ZN2v88internal20MarkCompactCollector11SweepSpacesEv+0x2f1)[0x9e4c51]
next engine worker(_ZN2v88internal20MarkCompactCollector14CollectGarbageEv+0x2a)[0x9ec90a]
next engine worker(_ZN2v88internal4Heap11MarkCompactEv+0x9f)[0x96882f]
next engine worker(_ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorENS_15GCCallbackFlagsE+0x288)[0x97c938]
next engine worker(_ZN2v88internal4Heap14CollectGarbageENS0_16GarbageCollectorEPKcS4_NS_15GCCallbackFlagsE+0xce)[0x97cf5e]
next engine worker(_ZN2v88internal4Heap17CollectAllGarbageEiPKcNS_15GCCallbackFlagsE+0x6c)[0x97d37c]
next engine worker(_ZN2v87Isolate34RequestGarbageCollectionForTestingENS0_21GarbageCollectionTypeE+0x3d)[0x80598d]
next engine worker(_ZN2v88internal25FunctionCallbackArguments4CallEPFvRKNS_20FunctionCallbackInfoINS_5ValueEEEE+0x9c)[0x82b78c]
next engine worker[0x84ddff]
[0x36cfbc2060a2]

P.S.: "next engine worker" is just process name.

[ChALkeR]

Reported to nodejs/node as nodejs/node#3715, the discussion should continue there.

[artofspeed]

My global.gc() occasionally causes SIGSEV as well, it's very annoying.. @soyuka @jshkurti have you guys figure out how to fix this problem?