Segfault on FreeBSD

[sigmike]

I'm running this simple source:

var http = require("http");

http.createServer(function(request, response) {
  response.writeHead(200, {"Content-Type": "text/plain"});
  response.write("Hello World");
  response.end();
}).listen(5000);

It works fine until I hit Ctrl+C: it segfaults.

Here's the backtrace:

#0  0x000000080226f469 in swapcontext () from /lib/libthr.so.3
#1  0x000000080226f062 in sigaction () from /lib/libthr.so.3
#2  <signal handler called>
#3  0x00000008024f344a in _umtx_op () from /lib/libc.so.7
#4  0x00000008024e2c24 in sem_timedwait () from /lib/libc.so.7
#5  0x0000000000b53ef8 in v8::base::Semaphore::Wait ()
#6  0x00000000008c7e84 in v8::internal::SweeperThread::Run ()
#7  0x0000000000b5594d in v8::base::ThreadEntry ()
#8  0x000000080226a4f5 in pthread_create () from /lib/libthr.so.3
#9  0x0000000000000000 in ?? ()

I'm running FreeBSD 10.1-RELEASE with node installed from the ports.

If I run a script that contains only a console.log it doesn't crash when it exits.

[sigmike]

node -v returns v0.12.0

[vba-ds]

Have a same problem on FreeBSD 10.1-RELEASE-p5 with nodejs 0.12.0.

#0  0x000000080225e469 in swapcontext () from /lib/libthr.so.3
[New Thread 803407400 (LWP 100767/v8:SweeperThrea)]
[New Thread 803407000 (LWP 100374/v8:SweeperThrea)]
[New Thread 803406c00 (LWP 100331/v8:SweeperThrea)]
[New Thread 803406800 (LWP 100214/OptimizingCompi)]
[New Thread 803406400 (LWP 100081/node)]

(gdb) bt full
#0  0x000000080225e469 in swapcontext () from /lib/libthr.so.3
#1  0x000000080225e062 in sigaction () from /lib/libthr.so.3
#2  <signal handler called>
#3  0x00000008024e244a in _umtx_op () from /lib/libc.so.7
#4  0x00000008024d1c24 in sem_timedwait () from /lib/libc.so.7
#5  0x0000000000b3ce08 in v8::base::Semaphore::Wait ()
#6  0x00000000008c46c4 in v8::internal::SweeperThread::Run ()
#7  0x0000000000b3e52d in v8::base::ThreadEntry ()
#8  0x00000008022594f5 in pthread_create () from /lib/libthr.so.3
#9  0x0000000000000000 in ?? ()

(gdb) t 5
[Switching to thread 5 (Thread 803406400 (LWP 100081/node))]#0  0x000000080259904a in kevent () from /lib/libc.so.7

(gdb) bt full
#0  0x000000080259904a in kevent () from /lib/libc.so.7
#1  0x0000000000b3bad7 in uv__io_poll (loop=0x111b5c0, timeout=<value optimized out>) at ../deps/uv/src/unix/kqueue.c:142
        events = {{ident = 0, filter = 0, flags = 0, fflags = 0, data = 0, udata = 0x0}}
        events = {{ident = 6, filter = -1, flags = 1, fflags = 0, data = 0, udata = 0x0}, {ident = 9, filter = -1, flags = 1, fflags = 0, data = 0, udata = 0x0},


...stripped...

On Ubuntu 14.04 (3.13.0-45-generic) the same code works fine.

[misterdjules]

I haven't had time to reproduce/investigate, but adding it to 0.12.1 to make sure we take care of it asap.

[misterdjules]

@tjfontaine Assigning to you as discussed during the core team meeting. Please assign to @trevnorris when he can use a FreeBSD 10.1 VM somewhere. Thank you!

[trevnorris]

Was finally able to reproduce. Though I'm not sure this is a Node thing. Might be a libuv problem?

[trevnorris]

Here's a simplified test:

require('http').createServer().listen(0, function() {
  process.kill(process.pid, 'SIGINT');
});

Still narrowing down why this is happening.

[trevnorris]

I've confirmed this is also an issue in io.js. Still investigating why this could possibly happen.

[trevnorris]

More stack trace info using latest libuv:

#0  0x000000080291f469 in swapcontext () from /lib/libthr.so.3
#1  0x000000080291f062 in sigaction () from /lib/libthr.so.3
#2  <signal handler called>
#3  0x0000000802c79a1a in kill () from /lib/libc.so.7
#4  0x000000000137f149 in uv_kill (pid=41450, signum=2) at ../deps/uv/src/unix/process.c:512
#5  0x00000000012c7dca in node::Kill (args=@0x7fffffffddb0) at ../src/node.cc:1960
#6  0x0000000000b583bc in v8::internal::FunctionCallbackArguments::Call (this=0x7fffffffdeb0, f=0x12c7d30 <node::Kill(v8::FunctionCallbackInfo<v8::Value> const&)>)
    at ../deps/v8/src/arguments.cc:33
#7  0x0000000000ba6a15 in v8::internal::HandleApiCallHelper<false> (args=
        {<v8::internal::Arguments> = {<v8::internal::Embedded> = {<No data fields>}, length_ = 4, arguments_ = 0x7fffffffe0c0}, <No data fields>}, isolate=0x803853000)
    at ../deps/v8/src/builtins.cc:1139
#8  0x0000000000ba64b5 in v8::internal::Builtin_Impl_HandleApiCall (args=
        {<v8::internal::Arguments> = {<v8::internal::Embedded> = {<No data fields>}, length_ = 4, arguments_ = 0x7fffffffe0c0}, <No data fields>}, isolate=0x803853000)
    at ../deps/v8/src/builtins.cc:1156
#9  0x0000000000b9edb0 in v8::internal::Builtin_HandleApiCall (args_length=4, args_object=0x7fffffffe0c0, isolate=0x803853000) at ../deps/v8/src/builtins.cc:1155

[trevnorris]

I'm going to need more help to continue with this issue.

/cc @joyent/node-coreteam

[indutny]

Oh yeah, I seen it once. Looking...

[indutny]

The problem disappears if I'll remove SA_RESETHAND... Looking further

[indutny]

Looks like sa_handler is messed up somewhere...

[indutny]

Ha, I think it is a bug in libthr:

#include <assert.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

static void handler(int sig) {
  fprintf(stderr, "oh\n");
  fflush(stderr);

  raise(sig);
}


int main() {
  int r;
  struct sigaction sa;

  memset(&sa, 0, sizeof(sa));

  sa.sa_handler = handler;
  sa.sa_flags = SA_RESETHAND;
  sigfillset(&sa.sa_mask);
  r = sigaction(SIGINT, &sa, NULL);
  assert(r == 0);

  kill(getpid(), SIGINT);

  return 0;
}

When compiled without -pthread this program works, when compiled with it - crashes.

[indutny]

It seems that the problem happens somewhere in thr_sighandler which is supposed to wrap the normal handler, looking into it.

[indutny]

Just a short follow-up, the stack is in fact:

handle_signal # thr_sig.c:244
thr_sighandler # thr_sig.c:187

Took me awhile to disassemble and find proper chunks, the link for the crash line is:

https://github.com/freebsd/freebsd/blob/master/lib/libthr/thread/thr_sig.c#L244

It seems that thr_sighandler is called with following args: (0x2, 0x10001, 0x7fffffffe640). 0x10001 is what causing the crash. Looking more into it.

[indutny]

It is odd, because libthr proxies sigact to the libc with a following struct:

{__sigaction_u = {__sa_handler = 0x80082af30, __sa_sigaction = 0x80082af30}, sa_flags = 0x44, sa_mask = {__bits = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}}}

0x44 flags are = SA_SIGINFO | SA_RESETHAND

Going to try passing this without linking to libthr.

[indutny]

Looks like a kernel bug to me:

#include <assert.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

static void handler(int sig, siginfo_t* info, void* u) {
  fprintf(stderr, "oh %d %p %p\n", sig, info, u);
  fflush(stderr);

  raise(sig);
}


int main() {
  int r;
  struct sigaction sa;

  memset(&sa, 0, sizeof(sa));

  sa.sa_sigaction = handler;
  sa.sa_flags = SA_RESETHAND | SA_SIGINFO;
  sigfillset(&sa.sa_mask);
  r = sigaction(SIGINT, &sa, NULL);
  assert(r == 0);

  kill(getpid(), SIGINT);

  return 0;
}

Is producing:

oh 2 0x10001 0x7fffffffe660

/me is looking deeply into the kernel code now.

[indutny]

It looks like it is using:

Traditional BSD style:
                  void handler(int, int code, struct sigcontext *scp);

For the handler.

[indutny]

Looks like a combination of SA_SIGINFO and SA_RESETHAND makes kernel skip the SA_SIGINFO.

[indutny]

Related FreeBSD discussion: https://lists.freebsd.org/pipermail/freebsd-threads/2014-November/005613.html . Looks like a kernel bug for me.

[indutny]

And the relevant fix: freebsd/freebsd-src@4501dad . I'll ask on a mailing list about possibility of backporting it to 10.x FreeBSD. Looks like we'll need to work around it in io.js.

[indutny]

And node.js for sure too :)

[indutny]

Should be fixed by nodejs/node#1218

[indutny]

The fix was landed in io.js nodejs/node@b64983d , please consider backporting.

[trevnorris]

Many thanks @indutny.

/cc @tjfontaine Let's back port this.

[misterdjules]

Fixed by 61fe1fe. Thank you @indutny and @trevnorris!

[vba-ds]

Problem with building:

../src/node.cc:2793:34: error: use of undeclared identifier 'nullptr'
  CHECK_EQ(sigaction(signo, &sa, nullptr), 0);
                                 ^
../src/util.h:51:31: note: expanded from macro 'CHECK_EQ'
#define CHECK_EQ(a, b) CHECK((a) == (b))
                              ^
../src/util.h:48:37: note: expanded from macro 'CHECK'
# define CHECK(expression)   assert(expression)
                                    ^
/usr/include/assert.h:54:21: note: expanded from macro 'assert'
#define assert(e)       ((e) ? (void)0 : __assert(__func__, __FILE__, \
                          ^

I guess need to change "nullptr" to "NULL", or just make this include

#include <cstddef>

[misterdjules]

@vba-ds When integrating the change from io.js, I did not do my job correctly and did not actually test it. My apologies for that and thank you for letting us know!

Since our code base still doesn't enable C++11 features, the change should use NULL and not nullptr. Reopening the issue and adding it to the 0.12.3 milestone.

[misterdjules]

@vba-ds Fixed by #14819.

[misterdjules]

Should be fixed for good in f99eaef. Thank you and sorry for the confusion.