Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

segmentation fault at v8::internal::StoreBuffer::EnsureSpace () #11156

Closed
shyhpei opened this issue Feb 3, 2017 · 12 comments
Closed

segmentation fault at v8::internal::StoreBuffer::EnsureSpace () #11156

shyhpei opened this issue Feb 3, 2017 · 12 comments
Labels
v8 engine Issues and PRs related to the V8 dependency.

Comments

@shyhpei
Copy link

shyhpei commented Feb 3, 2017

  • Version: 4.6.0
  • Platform: Fedora Linux 64-bit
  • Subsystem:

Core file too big to attach, but back trace below:

Program terminated with signal 11, Segmentation fault.
#0  0x0000000000b13799 in v8::internal::StoreBuffer::EnsureSpace ()

Thread 9 (process 4011):
#0  0x00007f2bab1b04a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000fbddd9 in uv_cond_wait ()
No symbol table info available.
#2  0x0000000000faeca8 in uv_inet_pton ()
No symbol table info available.
#3  0x0000000000fbd939 in uv__tcp_close ()
No symbol table info available.
#4  0x00007f2bab1ac2f7 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2baaf2486d in clone () from /lib64/libc.so.6
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 8 (process 4010):
#0  0x00007f2bab1b04a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000fbddd9 in uv_cond_wait ()
No symbol table info available.
#2  0x0000000000faeca8 in uv_inet_pton ()
No symbol table info available.
#3  0x0000000000fbd939 in uv__tcp_close ()
No symbol table info available.
#4  0x00007f2bab1ac2f7 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2baaf2486d in clone () from /lib64/libc.so.6
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 7 (process 4009):
#0  0x00007f2bab1b04a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000fbddd9 in uv_cond_wait ()
No symbol table info available.
#2  0x0000000000faeca8 in uv_inet_pton ()
No symbol table info available.
#3  0x0000000000fbd939 in uv__tcp_close ()
No symbol table info available.
#4  0x00007f2bab1ac2f7 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2baaf2486d in clone () from /lib64/libc.so.6
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 6 (process 4008):
#0  0x00007f2bab1b04a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000fbddd9 in uv_cond_wait ()
No symbol table info available.
#2  0x0000000000faeca8 in uv_inet_pton ()
No symbol table info available.
#3  0x0000000000fbd939 in uv__tcp_close ()
No symbol table info available.
#4  0x00007f2bab1ac2f7 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2baaf2486d in clone () from /lib64/libc.so.6
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 5 (process 4007):
#0  0x00007f2bab1b20ed in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000fc7858 in v8::base::Semaphore::Wait ()
No symbol table info available.
#2  0x0000000000e65019 in v8::platform::TaskQueue::GetNext ()
No symbol table info available.
#3  0x0000000000e6516c in v8::platform::WorkerThread::Run ()
No symbol table info available.
#4  0x0000000000fc8810 in v8::base::Thread::~Thread$base ()
No symbol table info available.
#5  0x00007f2bab1ac2f7 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f2baaf2486d in clone () from /lib64/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 4 (process 4006):
#0  0x00007f2bab1b20ed in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000fc7858 in v8::base::Semaphore::Wait ()
No symbol table info available.
#2  0x0000000000e65019 in v8::platform::TaskQueue::GetNext ()
No symbol table info available.
#3  0x0000000000e6516c in v8::platform::WorkerThread::Run ()
No symbol table info available.
#4  0x0000000000fc8810 in v8::base::Thread::~Thread$base ()
No symbol table info available.
#5  0x00007f2bab1ac2f7 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f2baaf2486d in clone () from /lib64/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 3 (process 4005):
#0  0x00007f2bab1b20ed in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000fc7858 in v8::base::Semaphore::Wait ()
No symbol table info available.
#2  0x0000000000e65019 in v8::platform::TaskQueue::GetNext ()
No symbol table info available.
#3  0x0000000000e6516c in v8::platform::WorkerThread::Run ()
No symbol table info available.
#4  0x0000000000fc8810 in v8::base::Thread::~Thread$base ()
No symbol table info available.
#5  0x00007f2bab1ac2f7 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f2baaf2486d in clone () from /lib64/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 2 (process 4004):
#0  0x00007f2bab1b20ed in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000fc7858 in v8::base::Semaphore::Wait ()
No symbol table info available.
#2  0x0000000000e65019 in v8::platform::TaskQueue::GetNext ()
No symbol table info available.
#3  0x0000000000e6516c in v8::platform::WorkerThread::Run ()
No symbol table info available.
#4  0x0000000000fc8810 in v8::base::Thread::~Thread$base ()
No symbol table info available.
#5  0x00007f2bab1ac2f7 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f2baaf2486d in clone () from /lib64/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 1 (process 4003):
#0  0x0000000000b13799 in v8::internal::StoreBuffer::EnsureSpace ()
No symbol table info available.
#1  0x0000000000000002 in ?? ()
No symbol table info available.
#2  0x00000000019d9e50 in ?? ()
No symbol table info available.
#3  0x00000bdb8f600000 in ?? ()
No symbol table info available.
#4  0x00000000019d9e98 in ?? ()
No symbol table info available.
#5  0x00000000019da290 in ?? ()
No symbol table info available.
#6  0x00000170c2a00000 in ?? ()
No symbol table info available.
#7  0x00000000019da2d8 in ?? ()
No symbol table info available.
#8  0x0000000001130010 in vtable for v8::internal::LargeObjectIterator ()
No symbol table info available.
#9  0x00001efe91200000 in ?? ()
No symbol table info available.
#10 0x0000000000000000 in ?? ()
No symbol table info available.
@jasnell
Copy link
Member

jasnell commented Feb 4, 2017

Can you provide a use case that can be used to recreate the failure?

@shyhpei
Copy link
Author

shyhpei commented Feb 4, 2017

Sorry, this is from one of a couple core files found on a system, so we don't really know under what situation this crashed. The back trace for both core files are the same. Would the core file itself be any help? It's too big to be allowed here but I can put it on OneDrive and share it.

@mscdex mscdex added the v8 engine Issues and PRs related to the V8 dependency. label Feb 4, 2017
@mscdex
Copy link
Contributor

mscdex commented Feb 4, 2017

/cc @nodejs/v8

@bnoordhuis
Copy link
Member

@shyhpei Is testing a debug build (./configure && make -j8 -C out BUILDTYPE=Debug) an option for you? That should provide better stack traces.

For now, can you set pagination off and paste the output of disassemble and info registers? Perhaps there's a clue in there somewhere.

@shyhpei
Copy link
Author

shyhpei commented Feb 5, 2017

debug build is not doable - we used the binary build from nodejs site and never actually build the node here. And the crash is on a production system. We didn't encounter crash before in our lab. "disassemble" output will be in next comment. Here is "info registers" output:

(gdb) info register
rax 0x167846f04101 24705842036993
rbx 0x19b3438 26948664
rcx 0x19da290 27107984
rdx 0x0 0
rsi 0x19d79f0 27097584
rdi 0x7fff67089b48 140734922005320
rbp 0x7fff67089b80 0x7fff67089b80
rsp 0x7fff67089b10 0x7fff67089b10
r8 0x6 6
r9 0x254db36c3ac1 41015652924097
r10 0x40 64
r11 0x1599bf9043c1 23750088082369
r12 0x7fff67089b48 140734922005320
r13 0x4000 16384
r14 0xfa600 1025536
r15 0x19ad180 26923392
rip 0xb13799 0xb13799 v8::internal::StoreBuffer::EnsureSpace(long)+409
eflags 0x10246 [ PF ZF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
fctrl 0x37f 895
fstat 0x0 0
ftag 0xffff 65535
fiseg 0x0 0
fioff 0xa2623c 10641980
foseg 0x7fff 32767
fooff 0x67084f88 1728597896
fop 0x0 0
mxcsr 0x1fa5 [ IE ZE PE IM DM ZM OM UM PM ]

@shyhpei
Copy link
Author

shyhpei commented Feb 5, 2017

(gdb) set pagination off
(gdb) disassemble
Dump of assembler code for function _ZN2v88internal11StoreBuffer11EnsureSpaceEl:
0x0000000000b13600 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+0>: push %rbp
0x0000000000b13601 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+1>: mov %rsp,%rbp
0x0000000000b13604 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+4>: push %r13
0x0000000000b13606 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+6>: mov %rsi,%r13
0x0000000000b13609 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+9>: push %r12
0x0000000000b1360b <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+11>: push %rbx
0x0000000000b1360c <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+12>: mov %rdi,%rbx
0x0000000000b1360f <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+15>: sub $0x58,%rsp
0x0000000000b13613 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+19>: mov 0x20(%rdi),%rsi
0x0000000000b13617 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+23>: mov %rsi,%rdx
0x0000000000b1361a <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+26>: sub 0x28(%rdi),%rdx
0x0000000000b1361e <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+30>: sar $0x3,%rdx
0x0000000000b13622 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+34>: cmp %rdx,%r13
0x0000000000b13625 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+37>: jg 0xb13668 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+104>
0x0000000000b13627 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+39>: jmp 0xb13674 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+116>
0x0000000000b13629 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+41>: nopl 0x0(%rax)
0x0000000000b13630 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+48>: mov %rsi,%r12
0x0000000000b13633 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+51>: sub 0x18(%rbx),%r12
0x0000000000b13637 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+55>: mov 0x38(%rbx),%rdi
0x0000000000b1363b <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+59>: xor %ecx,%ecx
0x0000000000b1363d <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+61>: and $0xfffffffffffffff8,%r12
0x0000000000b13641 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+65>: mov %r12,%rdx
0x0000000000b13644 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+68>: callq 0xfc8080 <_ZN2v84base13VirtualMemory6CommitEPvmb>
0x0000000000b13649 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+73>: test %al,%al
0x0000000000b1364b <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+75>: je 0xb13680 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+128>
0x0000000000b1364d <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+77>: mov %r12,%rsi
0x0000000000b13650 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+80>: add 0x20(%rbx),%rsi
0x0000000000b13654 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+84>: mov %rsi,%rdx
0x0000000000b13657 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+87>: sub 0x28(%rbx),%rdx
0x0000000000b1365b <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+91>: mov %rsi,0x20(%rbx)
0x0000000000b1365f <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+95>: sar $0x3,%rdx
0x0000000000b13663 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+99>: cmp %r13,%rdx
0x0000000000b13666 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+102>: jge 0xb13674 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+116>
0x0000000000b13668 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+104>: cmp %rsi,0x30(%rbx)
0x0000000000b1366c <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+108>: ja 0xb13630 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+48>
0x0000000000b1366e <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+110>: cmpb $0x0,0x41(%rbx)
0x0000000000b13672 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+114>: je 0xb13697 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+151>
0x0000000000b13674 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+116>: add $0x58,%rsp
0x0000000000b13678 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+120>: pop %rbx
0x0000000000b13679 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+121>: pop %r12
0x0000000000b1367b <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+123>: pop %r13
0x0000000000b1367d <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+125>: pop %rbp
0x0000000000b1367e <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+126>: retq
0x0000000000b1367f <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+127>: nop
0x0000000000b13680 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+128>: mov 0x20(%rbx),%rax
0x0000000000b13684 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+132>: sub 0x28(%rbx),%rax
0x0000000000b13688 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+136>: sar $0x3,%rax
0x0000000000b1368c <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+140>: cmp %rax,%r13
0x0000000000b1368f <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+143>: jle 0xb13674 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+116>
0x0000000000b13691 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+145>: cmpb $0x0,0x41(%rbx)
0x0000000000b13695 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+149>: jne 0xb13674 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+116>
0x0000000000b13697 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+151>: mov %rbx,%rdi
0x0000000000b1369a <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+154>: callq 0xb13440 <_ZN2v88internal11StoreBuffer7CompactEv>
0x0000000000b1369f <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+159>: mov (%rbx),%rax
0x0000000000b136a2 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+162>: movb $0x1,0x41(%rbx)
0x0000000000b136a6 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+166>: movl $0x0,-0x70(%rbp)
0x0000000000b136ad <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+173>: mov 0xc08(%rax),%rdx
0x0000000000b136b4 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+180>: lea 0x48(%rdx),%rcx
0x0000000000b136b8 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+184>: mov %rdx,-0x68(%rbp)
0x0000000000b136bc <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+188>: mov 0xd0(%rdx),%rdx
0x0000000000b136c3 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+195>: mov %rcx,-0x60(%rbp)
0x0000000000b136c7 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+199>: mov %rdx,-0x58(%rbp)
0x0000000000b136cb <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+203>: mov 0xc18(%rax),%rdx
0x0000000000b136d2 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+210>: lea 0x48(%rdx),%rcx
0x0000000000b136d6 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+214>: mov %rdx,-0x50(%rbp)
0x0000000000b136da <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+218>: mov 0xd0(%rdx),%rdx
0x0000000000b136e1 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+225>: mov %rcx,-0x48(%rbp)
0x0000000000b136e5 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+229>: mov 0xc20(%rax),%rsi
0x0000000000b136ec <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+236>: lea -0x38(%rbp),%rdi
0x0000000000b136f0 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+240>: lea -0x38(%rbp),%r12
0x0000000000b136f4 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+244>: mov %rdx,-0x40(%rbp)
0x0000000000b136f8 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+248>: callq 0xb10570 <_ZN2v88internal19LargeObjectIteratorC2EPNS0_16LargeObjectSpaceE>
0x0000000000b136fd <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+253>: nopl (%rax)
0x0000000000b13700 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+256>: mov -0x70(%rbp),%eax
0x0000000000b13703 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+259>: cmp $0x1,%eax
0x0000000000b13706 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+262>: je 0xb137e0 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+480>
0x0000000000b1370c <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+268>: cmp $0x2,%eax
0x0000000000b1370f <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+271>: je 0xb13778 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+376>
0x0000000000b13711 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+273>: test %eax,%eax
0x0000000000b13713 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+275>: je 0xb13810 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+528>
0x0000000000b13719 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+281>: mov 0x20(%rbx),%rax
0x0000000000b1371d <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+285>: sub 0x28(%rbx),%rax
0x0000000000b13721 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+289>: sar $0x3,%rax
0x0000000000b13725 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+293>: cmp %rax,%r13
0x0000000000b13728 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+296>: jle 0xb13674 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+116>
0x0000000000b1372e <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+302>: mov $0x1131924,%r12d
0x0000000000b13734 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+308>: mov -0x4(%r12),%esi
0x0000000000b13739 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+313>: mov (%r12),%edx
0x0000000000b1373d <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+317>: mov %rbx,%rdi
0x0000000000b13740 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+320>: callq 0xb12bd0 <_ZN2v88internal11StoreBuffer18ExemptPopularPagesEii>
0x0000000000b13745 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+325>: mov 0x20(%rbx),%rax
0x0000000000b13749 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+329>: sub 0x28(%rbx),%rax
0x0000000000b1374d <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+333>: sar $0x3,%rax
0x0000000000b13751 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+337>: cmp %rax,%r13
0x0000000000b13754 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+340>: jle 0xb13674 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+116>
0x0000000000b1375a <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+346>: add $0x8,%r12
0x0000000000b1375e <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+350>: cmp $0x113194c,%r12
0x0000000000b13765 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+357>: jne 0xb13734 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+308>
0x0000000000b13767 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+359>: jmpq 0xb13674 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+116>
0x0000000000b1376c <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+364>: movl $0x2,-0x70(%rbp)
0x0000000000b13773 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+371>: nopl 0x0(%rax,%rax,1)
0x0000000000b13778 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+376>: mov %r12,%rdi
0x0000000000b1377b <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+379>: callq 0xb105b0 <_ZN2v88internal19LargeObjectIterator4NextEv>
0x0000000000b13780 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+384>: test %rax,%rax
0x0000000000b13783 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+387>: je 0xb13840 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+576>
0x0000000000b13789 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+393>: mov %rax,%rdx
0x0000000000b1378c <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+396>: and $0x3,%edx
0x0000000000b1378f <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+399>: cmp $0x1,%rdx
0x0000000000b13793 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+403>: jne 0xb13778 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+376>
0x0000000000b13795 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+405>: mov -0x1(%rax),%rdx
0x0000000000b13799 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+409>: cmpb $0xb3,0xb(%rdx)
0x0000000000b1379d <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+413>: jne 0xb13778 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+376>
0x0000000000b1379f <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+415>: sub $0x1,%rax
0x0000000000b137a3 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+419>: and $0xfffffffffff00000,%rax
0x0000000000b137a9 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+425>: nopl 0x0(%rax)
0x0000000000b137b0 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+432>: test %rax,%rax
0x0000000000b137b3 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+435>: je 0xb13719 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+281>
0x0000000000b137b9 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+441>: testb $0x10,0x8(%rax)
0x0000000000b137bd <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+445>: je 0xb13700 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+256>
0x0000000000b137c3 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+451>: mov $0x4,%esi
0x0000000000b137c8 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+456>: mov %rbx,%rdi
0x0000000000b137cb <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+459>: callq 0xb12a90 <_ZN2v88internal11StoreBuffer6FilterEi>
0x0000000000b137d0 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+464>: jmpq 0xb13719 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+281>
0x0000000000b137d5 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+469>: movl $0x1,-0x70(%rbp)
0x0000000000b137dc <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+476>: nopl 0x0(%rax)
0x0000000000b137e0 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+480>: mov -0x50(%rbp),%rcx
0x0000000000b137e4 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+484>: mov -0x40(%rbp),%rax
0x0000000000b137e8 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+488>: lea 0x48(%rcx),%rdx
0x0000000000b137ec <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+492>: cmp %rdx,%rax
0x0000000000b137ef <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+495>: je 0xb1376c <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+364>
0x0000000000b137f5 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+501>: mov %rax,-0x48(%rbp)
0x0000000000b137f9 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+505>: mov 0x88(%rax),%rax
0x0000000000b13800 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+512>: mov %rax,-0x40(%rbp)
0x0000000000b13804 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+516>: mov -0x48(%rbp),%rax
0x0000000000b13808 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+520>: jmp 0xb137b0 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+432>
0x0000000000b1380a <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+522>: nopw 0x0(%rax,%rax,1)
0x0000000000b13810 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+528>: mov -0x68(%rbp),%rcx
0x0000000000b13814 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+532>: mov -0x58(%rbp),%rax
0x0000000000b13818 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+536>: lea 0x48(%rcx),%rdx
0x0000000000b1381c <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+540>: cmp %rdx,%rax
0x0000000000b1381f <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+543>: je 0xb137d5 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+469>
0x0000000000b13821 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+545>: mov %rax,-0x60(%rbp)
0x0000000000b13825 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+549>: mov 0x88(%rax),%rax
0x0000000000b1382c <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+556>: mov %rax,-0x58(%rbp)
0x0000000000b13830 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+560>: mov -0x60(%rbp),%rax
0x0000000000b13834 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+564>: jmpq 0xb137b0 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+432>
0x0000000000b13839 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+569>: nopl 0x0(%rax)
0x0000000000b13840 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+576>: movl $0x3,-0x70(%rbp)
0x0000000000b13847 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+583>: jmpq 0xb13719 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+281>
End of assembler dump.
(gdb)

@bnoordhuis
Copy link
Member

Thanks. The offending instructions are these two:

0x0000000000b13795 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+405>: mov -0x1(%rax),%rdx
0x0000000000b13799 <_ZN2v88internal11StoreBuffer11EnsureSpaceEl+409>: cmpb $0xb3,0xb(%rdx)

It segfaults because rdx == 0, loaded from address 0x167846f04100 (because rax == 0x167846f04101.)

It corresponds with this block of code in PointerChunkIterator::next() in deps/v8/src/heap/spaces.h.

More specifically, it's the !heap_object->IsFixedArray() check - $0xb3 is FIXED_ARRAY_TYPE.

Even more specific, it segfaults on a HeapObject::cast(this)->map()->instance_type() == FIXED_ARRAY_TYPE check (with this=0x167846f04101) because map() returns a nullptr.

(It reads the map pointer from the word at address 0x167846f04100, the address of the heap object. Heap object pointers have their LSB set but it's masked off when accessing the object's fields; the map pointer is at offset 0; ergo, it's at address 0x167846f04100.)

The crash, in other words, happens because of a heap object in a bad state. Unfortunately, I cannot tell you why it happens.

Is there any way for us to reproduce? Did you get the node binary from nodejs.org or somewhere else?

@bnoordhuis
Copy link
Member

By the way, if nothing else, try upgrading to the latest v4.x release and see if the problem persists.

@hannespayer
Copy link
Contributor

Super detailed analysis @bnoordhuis, thanks. Unfortunately the whole remembered set implementation got replaced in more recent versions of v8. I would also give upgrading to the latest v4.x a shot.

@shyhpei
Copy link
Author

shyhpei commented Feb 6, 2017

Many thanks for looking into this.

The binary is downloaded directly from nodejs.org, and I don't know how to reproduce this either. The crash happened twice on just one of many production systems with same node binary. The core files are the reason we know it crashed.

I will give upgrading to the latest 4.x build (4.7.3 now) a try.

Thanks again!

@TimothyGu
Copy link
Member

@shyhpei how did updating work out for you? Can this issue be closed now?

@Trott
Copy link
Member

Trott commented Jul 30, 2017

This issue has been inactive for sufficiently long that it seems like perhaps it should be closed. Feel free to re-open (or leave a comment requesting that it be re-opened) if you disagree. I'm just tidying up and not acting on a super-strong opinion or anything like that.

@Trott Trott closed this as completed Jul 30, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
v8 engine Issues and PRs related to the V8 dependency.
Projects
None yet
Development

No branches or pull requests

7 participants