Running node from udev causes SEGV

[fogine]

Version: v9.6.1
Platform: Linux archlaptop 4.15.2-2-ARCH #1 SMP PREEMPT Thu Feb 8 18:54:52 UTC 2018 x86_64 GNU/Linux

I have following rule in /etc/udev/rules.d/98-monitor-hotplug.rules

KERNEL=="card0", SUBSYSTEM=="drm", ACTION=="change", ENV{DISPLAY}=":0", ENV{XAUTHORITY}="/path/to/.Xauthority", RUN+="/bin/node"

When the event is triggered the node process terminates with SEGV.

This is very similar to #16901 which has been closed with no conclusion.
It doesn't matter if I run empty script.js file or node -e "process.exit(0);" instead of executing /bin/node repl console on the udev event. It always terminates with SEGV.
I experience the same issue with node version v8.9.4 LTS.
Could anybody try to reproduce this issue, please?

Apart from providing the logs, I have currently no idea how to debug this issue, so please if you need more info, could you provide me with specific information about how I should proceed? Thank you.

gdb

             Stack trace of thread 14910:
            #0  0x000055e25cb01290 _ZN2v88internal18RegExpResultsCache5ClearEPNS0_10FixedArrayE (node)
            #1  0x000055e25c9223cd _ZN2v88internal4Heap19MarkCompactPrologueEv (node)
            #2  0x000055e25c9307f5 _ZN2v88internal4Heap11MarkCompactEv (node)
            #3  0x000055e25c93d863 _ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorENS_15GCCallbackFlagsE (node)
            #4  0x000055e25c93e023 _ZN2v88internal4Heap14CollectGarbageENS0_15AllocationSpaceENS0_23GarbageCollectionReasonENS_15GCCallbackFlagsE (node)
            #5  0x000055e25c93f50a _ZN2v88internal4Heap12ReserveSpaceEPSt6vectorINS1_5ChunkESaIS3_EEPS2_IPhSaIS7_EE (node)
            #6  0x000055e25cbe95f8 _ZN2v88internal12Deserializer12ReserveSpaceEv (node)
            #7  0x000055e25cbf4ac2 _ZN2v88internal19StartupDeserializer15DeserializeIntoEPNS0_7IsolateE (node)
            #8  0x000055e25c9deffe _ZN2v88internal7Isolate4InitEPNS0_19StartupDeserializerE (node)
            #9  0x000055e25cbf423e _ZN2v88internal8Snapshot10InitializeEPNS0_7IsolateE (node)
            #10 0x000055e25c5dad69 _ZN2v814IsolateNewImplEPNS_8internal7IsolateERKNS_7Isolate12CreateParamsE (node)
            #11 0x000055e25c465d4d _ZN4node5StartEP9uv_loop_siPKPKciS5_ (node)
            #12 0x000055e25c462a45 _ZN4node5StartEiPPc (node)
            #13 0x00007fa3c6228f4a __libc_start_main (libc.so.6)
            #14 0x000055e25c42867a _start (node)

            Stack trace of thread 14911:
            #0  0x00007fa3c65cc3bd pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1  0x00007fa3c863ccfa uv_cond_wait (libuv.so.1)
            #2  0x000055e25c4b64f4 n/a (node)
            #3  0x00007fa3c65c608c start_thread (libpthread.so.0)
            #4  0x00007fa3c62fde7f __clone (libc.so.6)

            Stack trace of thread 14914:
            #0  0x00007fa3c65cc3bd pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1  0x00007fa3c863ccfa uv_cond_wait (libuv.so.1)
            #2  0x000055e25c4b64f4 n/a (node)
            #3  0x00007fa3c65c608c start_thread (libpthread.so.0)
            #4  0x00007fa3c62fde7f __clone (libc.so.6)

            Stack trace of thread 14913:
            #0  0x00007fa3c65cc3bd pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1  0x00007fa3c863ccfa uv_cond_wait (libuv.so.1)
            #2  0x000055e25c4b64f4 n/a (node)
            #3  0x00007fa3c65c608c start_thread (libpthread.so.0)
            #4  0x00007fa3c62fde7f __clone (libc.so.6)

            Stack trace of thread 14912:
            #0  0x00007fa3c65cc3bd pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1  0x00007fa3c863ccfa uv_cond_wait (libuv.so.1)
            #2  0x000055e25c4b64f4 n/a (node)
            #3  0x00007fa3c65c608c start_thread (libpthread.so.0)
            #4  0x00007fa3c62fde7f __clone (libc.so.6)


            Reading symbols from /usr/bin/node...(no debugging symbols found)...done.
            [New LWP 14910]
            [New LWP 14911]
            [New LWP 14914]
            [New LWP 14913]
            [New LWP 14912]
            [Thread debugging using libthread_db enabled]
            Using host libthread_db library "/usr/lib/libthread_db.so.1".
            Core was generated by `/bin/node'.
            Program terminated with signal SIGSEGV, Segmentation fault.
            #0  0x000055e25cb01290 in                                                                                                 
            v8::internal::RegExpResultsCache::Clear(v8::internal::FixedArray*) ()
            [Current thread is 1 (Thread 0x7fa3c8e64b80 (LWP 14910))]

thread apply all bt full

            Thread 5 (Thread 0x7fa3c3c46700 (LWP 14912)):
            #0  0x00007fa3c65cc3bd in pthread_cond_wait@@GLIBC_2.3.2 () from                                                                                                                                 
             /usr/lib/libpthread.so.0
             No symbol table info available.                
             #1  0x00007fa3c863ccfa in uv_cond_wait () from /usr/lib/libuv.so.1
             No symbol table info available.
             #2  0x000055e25c4b64f4 in ?? ()
             No symbol table info available.
            #3  0x00007fa3c65c608c in start_thread () from /usr/lib/libpthread.so.0
            No symbol table info available.
            #4  0x00007fa3c62fde7f in clone () from /usr/lib/libc.so.6
            No symbol table info available.

            Thread 4 (Thread 0x7fa3c3445700 (LWP 14913)):
            #0  0x00007fa3c65cc3bd in pthread_cond_wait@@GLIBC_2.3.2 () from                 
             /usr/lib/libpthread.so.0
            No symbol table info available.
            #1  0x00007fa3c863ccfa in uv_cond_wait () from /usr/lib/libuv.so.1
            No symbol table info available.
            #2  0x000055e25c4b64f4 in ?? ()
            No symbol table info available.
            #3  0x00007fa3c65c608c in start_thread () from /usr/lib/libpthread.so.0
            No symbol table info available.
            #4  0x00007fa3c62fde7f in clone () from /usr/lib/libc.so.6
            No symbol table info available.

            Thread 3 (Thread 0x7fa3c2c44700 (LWP 14914)):
            #0  0x00007fa3c65cc3bd in pthread_cond_wait@@GLIBC_2.3.2 () from                 
             /usr/lib/libpthread.so.0
            No symbol table info available.
            #1  0x00007fa3c863ccfa in uv_cond_wait () from /usr/lib/libuv.so.1
            No symbol table info available.
            #2  0x000055e25c4b64f4 in ?? ()
            No symbol table info available.
            #3  0x00007fa3c65c608c in start_thread () from /usr/lib/libpthread.so.0
            No symbol table info available.
            #4  0x00007fa3c62fde7f in clone () from /usr/lib/libc.so.6
            No symbol table info available.

            Thread 2 (Thread 0x7fa3c4447700 (LWP 14911)):
            #0  0x00007fa3c65cc3bd in pthread_cond_wait@@GLIBC_2.3.2 () from                 
             /usr/lib/libpthread.so.0
            No symbol table info available.
            #1  0x00007fa3c863ccfa in uv_cond_wait () from /usr/lib/libuv.so.1
            No symbol table info available.
            #2  0x000055e25c4b64f4 in ?? ()
            No symbol table info available.
            #3  0x00007fa3c65c608c in start_thread () from /usr/lib/libpthread.so.0
            No symbol table info available.
            #4  0x00007fa3c62fde7f in clone () from /usr/lib/libc.so.6
            No symbol table info available.

            Thread 1 (Thread 0x7fa3c8e64b80 (LWP 14910)):
            #0  0x000055e25cb01290 in                 
             v8::internal::RegExpResultsCache::Clear(v8::internal::FixedArray*) ()
            No symbol table info available.
            #1  0x000055e25c9223cd in v8::internal::Heap::MarkCompactPrologue() ()
            No symbol table info available.
            #2  0x000055e25c9307f5 in v8::internal::Heap::MarkCompact() ()
            No symbol table info available.
            #3  0x000055e25c93d863 in                                 
             v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector,                 
             v8::GCCallbackFlags) ()
            No symbol table info available.
            #4  0x000055e25c93e023 in                 
             v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace,                 
             v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) ()
            No symbol table info available.
            #5  0x000055e25c93f50a in                 
             v8::internal::Heap::ReserveSpace(std::vector<v8::internal::Heap::Chunk,                 
             std::allocator<v8::internal::Heap::Chunk> >*, std::vector<unsigned char*,                 
             std::allocator<unsigned char*> >*) ()
            No symbol table info available.
            #6  0x000055e25cbe95f8 in v8::internal::Deserializer::ReserveSpace() ()
            No symbol table info available.
            #7  0x000055e25cbf4ac2 in                 
             v8::internal::StartupDeserializer::DeserializeInto(v8::internal::Isolate*) ()
            No symbol table info available.
            #8  0x000055e25c9deffe in                                 
             v8::internal::Isolate::Init(v8::internal::StartupDeserializer*) ()
            No symbol table info available.
            ---Type <return> to continue, or q <return> to quit---
            #9  0x000055e25cbf423e in v8::internal::Snapshot::Initialize(v8::internal::Isolate*) ()                
            No symbol table info available.
            #10 0x000055e25c5dad69 in v8::IsolateNewImpl(v8::internal::Isolate*,                                 
             v8::Isolate::CreateParams const&) ()
            No symbol table info available.
            #11 0x000055e25c465d4d in node::Start(uv_loop_s*, int, char const* const*, int,                 
             char const* const*) ()
            No symbol table info available.
            #12 0x000055e25c462a45 in node::Start(int, char**) ()
            No symbol table info available.
            #13 0x00007fa3c6228f4a in __libc_start_main () from /usr/lib/libc.so.6
            No symbol table info available.
            #14 0x000055e25c42867a in _start ()
            No symbol table info available.

[gireeshpunathil]

I am trying to recreate this - how do you fire the rule? (I am new to udev).
udevadm control --reload and udevadm trigger did not work.

[iczero]

@gireeshpunathil I think udevadm trigger --action=change --subsystem-match=drm should work

[gireeshpunathil]

thanks, but that did not help - I have no evidence that it triggered the action.

[bnoordhuis]

FWIW, I wasn't able to reproduce either but my guess would be it's system-specific, possibly Arch-specific. Perhaps there's a restrictive ulimit in effect (like a low ulimit -s) or a security policy that disallows JIT code.

[fogine]

@gireeshpunathil on my system it does not report that it needs root permissions when you manually retrigger the event (run udevadm with sudo).
Then you should see an error in your system's logs if the rule's RUN command failed (journalctl in my case), like:

systemd-coredump[11915]: Process 11903 (node) of user 0 dumped core.
Stack trace of thread 11903:
....                                              

The udev rule watches display outputs - hot pluggable ports like HDMI, Display port etc..
You can make sure that the rule gets included when you connect/disconnect a cable or manually trigger the event by the following command:

#path to your display output could be different
#udevadm test - does not trigger the RUN command of the rule
udevadm test /sys/class/drm/card0/card0-HDMI-A-1/

@bnoordhuis that's an idea worth exploring, thank you.
ulimit -a reports:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         unlimited
-m: resident set size (kbytes)      unlimited
-u: processes                       30983
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  16384
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 30983
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

I tried increasing stack size to 819200 with no impact. Maybe there is somewhere udev specific restriction with higher priority...

[gireeshpunathil]

I was able to recreate the crash. Earlier I was working in a container image, which does not work well with udev as I read.

1. All the mmap requests (probably coming from v8) with rwx attributes are failing when ran through udev controller:

mmap(0x3f6d68784000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 EPERM (Operation not permitted)
mmap(0x3f6d68804000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 EPERM (Operation not permitted)
mmap(0x3f6d68884000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 EPERM (Operation not permitted)
mmap(0x3f6d68904000, 503808, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 EPERM (Operation not permitted)

while I am not certain this is the root cause, this is a major deviation from the normal run.

2. The entire environ array is erased by the parent systemd-udevd before exec'ing node. This is the env of the child:

(gdb) p  environ[0]
$5 = 0x7ffc79d84e94 "ACTION=change"
(gdb) p  environ[1]
$3 = 0x7ffc79d84ea2 "DEVPATH=/devices/pci0000:00/0000:00:0f.0/drm/card0"
(gdb) p  environ[2]
$4 = 0x7ffc79d84ed5 "SUBSYSTEM=drm"
(gdb) p  environ[3]
$6 = 0x7ffc79d84ee3 "DEVNAME=/dev/dri/card0"
(gdb) p  environ[4]
$7 = 0x7ffc79d84efa "DEVTYPE=drm_minor"
(gdb) p  environ[5]
$8 = 0x7ffc79d84f0c "SEQNUM=22186"
(gdb) p  environ[6]
$9 = 0x7ffc79d84f19 "MAJOR=226"
(gdb) p  environ[7]
$10 = 0x7ffc79d84f23 "MINOR=0"
(gdb) p  environ[8]
$11 = 0x7ffc79d84f2b "USEC_INITIALIZED=3286355997479"
(gdb) p  environ[9]
$12 = 0x7ffc79d84f4a "ID_PATH=pci-0000:00:0f.0"
(gdb) p  environ[10]
$13 = 0x7ffc79d84f63 "ID_PATH_TAG=pci-0000_00_0f_0"
(gdb) p  environ[11]
$14 = 0x7ffc79d84f80 "TAGS=:uaccess:master-of-seat:seat:"
(gdb) p  environ[12]
$15 = 0x7ffc79d84fa3 "ID_FOR_SEAT=drm-pci-0000_00_0f_0"
(gdb) p  environ[13]
$16 = 0x7ffc79d84fc4 "DISPLAY=:0"
(gdb) p  environ[14]
$17 = 0x7ffc79d84fcf "XAUTHORITY=/root/.Xauthority"
(gdb) p  environ[15]
$18 = 0x0

[bnoordhuis]

Right, so there's a W^X deny policy in effect. Upstream V8 might one day support that - some work was done on that recently - but for now you'll have to either disable the policy or not run node from udev.

Thanks for clearing that up, Gireesh. I'll close this out.

[sudhakar3697]

@bnoordhuis Any update on this issue?

[bnoordhuis]

@sudhakar3697 Recent versions of Node.js have a --jitless flag but YMMV.

[JustAnotherContributor]

For the benefit of anyone coming across this issue as I did, I didn’t have the luxury of using a newer version of node with the --jitless flag because the node packages I was using hadn’t been updated in a long time and didn’t install on newer versions of node, but I found the problem referred to in this issue was caused by the presence of “MemoryDenyWriteExecute=yes” in the /lib/systemd/system/systemd-udevd.service file. However, in my case, simply changing this to no presented another problem, the process I was running from my udev rule would be terminated after a minute or so (probably 180 seconds), so I came up with a better solution and that was to create my own systemd unit file with “MemoryDenyWriteExecute=no” and modify my udev rule to simply stop and start the service represented by my unit file e.g.

/etc/systemd/system/myservice.service

[Unit]
Description=MyService
DefaultDependencies=no

[Service]
Type=simple
ExecStart=/home/user/myscript.sh
MemoryDenyWriteExecute=no
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6 AF_BLUETOOTH

/etc/udev/rules.d/120-bluetooth.rules

ACTION==“add”,\
SUBSYSTEM==“bluetooth”,\
KERNEL==“hci0”,\
RUN+=“/usr/bin/systemctl stop myservice”,\
RUN+=“/usr/bin/systemctl start myservice”

Hopefully this comment will assist someone else who encountered this issue like me.