[DEBUG] Dump GDB back traces on Segmentation Fault #153

gigony · 2021-11-17T09:33:45Z

No description provided.

ajschmidt8 · 2021-11-17T18:54:15Z

Please consider using GitHub's Draft PR feature instead of DEBUG tags in the future. Draft PRs have the benefit of preventing notifications to codeowners until PRs are marked Ready for Review. CI will still run on Draft PRs.

Some useful information about Draft PRs:

https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cucim/job/prb/job/cucim-gpu-test/CUDA=11.5,GPU_LABEL=cuda115,LINUX_VER=ubuntu18.04,PYTHON=3.7/130/consoleFull

gigony · 2021-11-19T22:39:41Z

With 85695c7

https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cucim/job/prb/job/cucim-gpu-test/CUDA=11.5,GPU_LABEL=cuda115,LINUX_VER=ubuntu18.04,PYTHON=3.7/137/consoleFull

Line 2379

13:09:56 Starting program: /opt/conda/envs/rapids/bin/python -c "import cucim.clara._cucim as cc; cc.haha(\"dd\");print(\"hoho\")"
13:09:56 warning: Error disabling address space randomization: Operation not permitted
13:09:56 [Thread debugging using libthread_db enabled]
13:09:56 Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
13:09:57 ##CuFileDriverInitializer()
13:09:57 ##load()
13:09:57 ##load(): handle_ == nullptr
13:09:57 ##load(): load_library != nullptr
13:09:57 [New Thread 0x7f1206f53700 (LWP 2120)]
13:09:58 haha
13:09:58 [New Thread 0x7f1205a9f700 (LWP 2121)]
13:09:58 Trying to import cupy...
13:09:58 Importing cupy done.
13:09:58 Trying to import clara...
13:09:58 Importing clara done.
13:09:58 hoho
13:09:58 [Thread 0x7f1205a9f700 (LWP 2121) exited]
13:09:58 ##Destructor CuFileStub for g_cufile_stub
13:09:58 ##~CuFileDriverInitializer() for s_cufile_initializer: is_available_ = true
13:09:58 CuFileDriver closed!
13:09:58 ##close_cufile_stub() start: 0x7f120efa8ca0 
13:09:58 ##unload()
13:09:58 ##unload(): handle_ != nullptr
13:09:58 ##close_cufile_stub() done
13:09:58 

13:09:58 Thread 1 "python" received signal SIGSEGV, Segmentation fault.
13:09:58 0x00007f120e8b4cb0 in ?? ()
13:09:58 #0  0x00007f120e8b4cb0 in ?? ()
13:09:58 #1  0x00007f120ee86a9d in (anonymous namespace)::run (p=<optimized out>)
13:09:58     at ../../../../libstdc++-v3/libsupc++/atexit_thread.cc:75
13:09:58 #2  0x00007f120f2a6161 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
13:09:58 #3  0x00007f120f2a625a in exit () from /lib/x86_64-linux-gnu/libc.so.6
13:09:58 #4  0x00007f120f284bfe in __libc_start_main ()
13:09:58    from /lib/x86_64-linux-gnu/libc.so.6
13:09:58 #5  0x0000561b3b36a51d in _start ()
13:09:58 process 2116
13:09:58 Mapped address spaces:
13:09:58 

13:09:58           Start Addr           End Addr       Size     Offset objfile
13:09:58          0x200000000        0x300200000 0x100200000        0x0 
13:09:58       0x561b3b1ac000     0x561b3b207000    0x5b000        0x0 /opt/conda/envs/rapids/bin/python3.7
13:09:58       0x561b3b207000     0x561b3b3f5000   0x1ee000    0x5b000 /opt/conda/envs/rapids/bin/python3.7
13:09:58       0x561b3b3f5000     0x561b3b4a5000    0xb0000   0x249000 /opt/conda/envs/rapids/bin/python3.7
13:09:58       0x561b3b4a6000     0x561b3b4a9000     0x3000   0x2f9000 /opt/conda/envs/rapids/bin/python3.7
13:09:58       0x561b3b4a9000     0x561b3b512000    0x69000   0x2fc000 /opt/conda/envs/rapids/bin/python3.7
13:09:58       0x561b3b512000     0x561b3b532000    0x20000        0x0 
13:09:58       0x561b3c96e000     0x561b3db20000  0x11b2000        0x0 [heap]
13:09:58       0x7f11e8000000     0x7f11e8021000    0x21000        0x0 
13:09:58       0x7f11e8021000     0x7f11ec000000  0x3fdf000        0x0 
13:09:58       0x7f11ec000000     0x7f11ec021000    0x21000        0x0 
13:09:58       0x7f11ec021000     0x7f11f0000000  0x3fdf000        0x0 
13:09:58       0x7f11f0000000     0x7f1200000000 0x10000000        0x0 
13:09:58       0x7f12042c5000     0x7f1204c4d000   0x988000        0x0 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.495.29.05
13:09:58       0x7f1204c4d000     0x7f1204e4d000   0x200000   0x988000 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.495.29.05
13:09:58       0x7f1204e4d000     0x7f1204f98000   0x14b000   0x988000 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.495.29.05
13:09:58       0x7f1204f98000     0x7f1204f9c000     0x4000   0xad3000 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.495.29.05
13:09:58       0x7f1204f9c000     0x7f1204fa1000     0x5000        0x0 
13:09:58       0x7f120529f000     0x7f12052a0000     0x1000        0x0 
13:09:58       0x7f12052a0000     0x7f1205aa0000   0x800000        0x0 
13:09:58       0x7f1205bea000     0x7f1205c07000    0x1d000        0x0 /lib/x86_64-linux-gnu/libudev.so.1.6.9
13:09:58       0x7f1205c07000     0x7f1205e06000   0x1ff000    0x1d000 /lib/x86_64-linux-gnu/libudev.so.1.6.9
13:09:58       0x7f1205e06000     0x7f1205e07000     0x1000    0x1c000 /lib/x86_64-linux-gnu/libudev.so.1.6.9
13:09:58       0x7f1205e07000     0x7f1205e08000     0x1000    0x1d000 /lib/x86_64-linux-gnu/libudev.so.1.6.9
13:09:58       0x7f1205e08000     0x7f1205e78000    0x70000        0x0 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
13:09:58       0x7f1205e78000     0x7f1206078000   0x200000    0x70000 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
13:09:58       0x7f1206078000     0x7f1206079000     0x1000    0x70000 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
13:09:58       0x7f1206079000     0x7f120607a000     0x1000    0x71000 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
13:09:58       0x7f120607a000     0x7f120607c000     0x2000        0x0 /opt/conda/envs/rapids/lib/libuuid.so.1.3.0
13:09:58       0x7f120607c000     0x7f1206080000     0x4000     0x2000 /opt/conda/envs/rapids/lib/libuuid.so.1.3.0
13:09:58       0x7f1206080000     0x7f1206081000     0x1000     0x6000 /opt/conda/envs/rapids/lib/libuuid.so.1.3.0
13:09:58       0x7f1206081000     0x7f1206082000     0x1000     0x6000 /opt/conda/envs/rapids/lib/libuuid.so.1.3.0
13:09:58       0x7f1206082000     0x7f1206083000     0x1000     0x7000 /opt/conda/envs/rapids/lib/libuuid.so.1.3.0
13:09:58       0x7f1206083000     0x7f12060a8000    0x25000        0x0 /lib/x86_64-linux-gnu/libselinux.so.1
13:09:58       0x7f12060a8000     0x7f12062a7000   0x1ff000    0x25000 /lib/x86_64-linux-gnu/libselinux.so.1
13:09:58       0x7f12062a7000     0x7f12062a8000     0x1000    0x24000 /lib/x86_64-linux-gnu/libselinux.so.1
13:09:58       0x7f12062a8000     0x7f12062a9000     0x1000    0x25000 /lib/x86_64-linux-gnu/libselinux.so.1
13:09:58       0x7f12062a9000     0x7f12062ab000     0x2000        0x0 
13:09:58       0x7f12062ab000     0x7f12062f2000    0x47000        0x0 /lib/x86_64-linux-gnu/libblkid.so.1.1.0
13:09:58       0x7f12062f2000     0x7f12064f2000   0x200000    0x47000 /lib/x86_64-linux-gnu/libblkid.so.1.1.0
13:09:58       0x7f12064f2000     0x7f12064f6000     0x4000    0x47000 /lib/x86_64-linux-gnu/libblkid.so.1.1.0
13:09:58       0x7f12064f6000     0x7f12064f7000     0x1000    0x4b000 /lib/x86_64-linux-gnu/libblkid.so.1.1.0
13:09:58       0x7f12064f7000     0x7f12064f8000     0x1000        0x0 
13:09:58       0x7f12064f8000     0x7f1206549000    0x51000        0x0 /lib/x86_64-linux-gnu/libmount.so.1.1.0
13:09:58       0x7f1206549000     0x7f1206748000   0x1ff000    0x51000 /lib/x86_64-linux-gnu/libmount.so.1.1.0
13:09:58       0x7f1206748000     0x7f120674a000     0x2000    0x50000 /lib/x86_64-linux-gnu/libmount.so.1.1.0
13:09:58       0x7f120674a000     0x7f120674b000     0x1000    0x52000 /lib/x86_64-linux-gnu/libmount.so.1.1.0
13:09:58       0x7f120674b000     0x7f120674c000     0x1000        0x0 
13:09:58       0x7f1206753000     0x7f1206754000     0x1000        0x0 
13:09:58       0x7f1206754000     0x7f120701f000   0x8cb000        0x0 
13:09:58       0x7f120701f000     0x7f120d01f000  0x6000000        0x0 
13:09:58       0x7f120d01f000     0x7f120d020000     0x1000        0x0 /dev/nvidia0
13:09:58       0x7f120d020000     0x7f120e3c9000  0x13a9000        0x0 /usr/lib/x86_64-linux-gnu/libcuda.so.495.29.05
13:09:58       0x7f120e3c9000     0x7f120e5c9000   0x200000  0x13a9000 /usr/lib/x86_64-linux-gnu/libcuda.so.495.29.05
13:09:58       0x7f120e5c9000     0x7f120e6bd000    0xf4000  0x13a9000 /usr/lib/x86_64-linux-gnu/libcuda.so.495.29.05
13:09:58       0x7f120e6bd000     0x7f120e7c8000   0x10b000  0x149d000 /usr/lib/x86_64-linux-gnu/libcuda.so.495.29.05
13:09:58       0x7f120e7c8000     0x7f120e7ea000    0x22000        0x0 
13:09:58       0x7f120eb21000     0x7f120ebbc000    0x9b000        0x0 /opt/conda/envs/rapids/lib/libcudart.so.11.5.50
13:09:58       0x7f120ebbc000     0x7f120edbc000   0x200000    0x9b000 /opt/conda/envs/rapids/lib/libcudart.so.11.5.50
13:09:58       0x7f120edbc000     0x7f120edc1000     0x5000    0x9b000 /opt/conda/envs/rapids/lib/libcudart.so.11.5.50
13:09:58       0x7f120edc1000     0x7f120edc2000     0x1000    0xa0000 /opt/conda/envs/rapids/lib/libcudart.so.11.5.50
13:09:58       0x7f120edc2000     0x7f120edc3000     0x1000        0x0 
13:09:58       0x7f120edc3000     0x7f120edc7000     0x4000    0xa1000 /opt/conda/envs/rapids/lib/libcudart.so.11.5.50
13:09:58       0x7f120edc7000     0x7f120edca000     0x3000        0x0 /opt/conda/envs/rapids/lib/libgcc_s.so.1
13:09:58       0x7f120edca000     0x7f120edd6000     0xc000     0x3000 /opt/conda/envs/rapids/lib/libgcc_s.so.1
13:09:58       0x7f120edd6000     0x7f120edd9000     0x3000     0xf000 /opt/conda/envs/rapids/lib/libgcc_s.so.1
13:09:58       0x7f120edd9000     0x7f120edda000     0x1000    0x11000 /opt/conda/envs/rapids/lib/libgcc_s.so.1
13:09:58       0x7f120edda000     0x7f120eddb000     0x1000    0x12000 /opt/conda/envs/rapids/lib/libgcc_s.so.1
13:09:58       0x7f120eddb000     0x7f120ee7e000    0xa3000        0x0 /opt/conda/envs/rapids/lib/libstdc++.so.6.0.28
13:09:58       0x7f120ee7e000     0x7f120eefd000    0x7f000    0xa3000 /opt/conda/envs/rapids/lib/libstdc++.so.6.0.28
13:09:58       0x7f120eefd000     0x7f120ef3e000    0x41000   0x122000 /opt/conda/envs/rapids/lib/libstdc++.so.6.0.28
13:09:58       0x7f120ef3e000     0x7f120ef49000     0xb000   0x162000 /opt/conda/envs/rapids/lib/libstdc++.so.6.0.28
13:09:58       0x7f120ef49000     0x7f120ef4d000     0x4000   0x16d000 /opt/conda/envs/rapids/lib/libstdc++.so.6.0.28
13:09:58       0x7f120ef4d000     0x7f120ef50000     0x3000        0x0 
13:09:58       0x7f120ef50000     0x7f120ef61000    0x11000        0x0 /opt/conda/envs/rapids/lib/libcucim.so.21.12.00
13:09:58       0x7f120ef61000     0x7f120ef96000    0x35000    0x11000 /opt/conda/envs/rapids/lib/libcucim.so.21.12.00
13:09:58       0x7f120ef96000     0x7f120efa6000    0x10000    0x46000 /opt/conda/envs/rapids/lib/libcucim.so.21.12.00
13:09:58       0x7f120efa6000     0x7f120efa8000     0x2000    0x55000 /opt/conda/envs/rapids/lib/libcucim.so.21.12.00
13:09:58       0x7f120efa8000     0x7f120efa9000     0x1000    0x57000 /opt/conda/envs/rapids/lib/libcucim.so.21.12.00
13:09:58       0x7f120efa9000     0x7f120efb8000     0xf000        0x0 /opt/conda/envs/rapids/lib/python3.7/site-packages/cucim/clara/_cucim.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f120efb8000     0x7f120f047000    0x8f000     0xf000 /opt/conda/envs/rapids/lib/python3.7/site-packages/cucim/clara/_cucim.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f120f047000     0x7f120f075000    0x2e000    0x9e000 /opt/conda/envs/rapids/lib/python3.7/site-packages/cucim/clara/_cucim.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f120f075000     0x7f120f076000     0x1000    0xcc000 /opt/conda/envs/rapids/lib/python3.7/site-packages/cucim/clara/_cucim.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f120f076000     0x7f120f07c000     0x6000    0xcc000 /opt/conda/envs/rapids/lib/python3.7/site-packages/cucim/clara/_cucim.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f120f07c000     0x7f120f07d000     0x1000    0xd2000 /opt/conda/envs/rapids/lib/python3.7/site-packages/cucim/clara/_cucim.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f120f07d000     0x7f120f263000   0x1e6000        0x0 
13:09:58       0x7f120f263000     0x7f120f44a000   0x1e7000        0x0 /lib/x86_64-linux-gnu/libc-2.27.so
13:09:58       0x7f120f44a000     0x7f120f64a000   0x200000   0x1e7000 /lib/x86_64-linux-gnu/libc-2.27.so
13:09:58       0x7f120f64a000     0x7f120f64e000     0x4000   0x1e7000 /lib/x86_64-linux-gnu/libc-2.27.so
13:09:58       0x7f120f64e000     0x7f120f650000     0x2000   0x1eb000 /lib/x86_64-linux-gnu/libc-2.27.so
13:09:58       0x7f120f650000     0x7f120f654000     0x4000        0x0 
13:09:58       0x7f120f654000     0x7f120f7f1000   0x19d000        0x0 /lib/x86_64-linux-gnu/libm-2.27.so
13:09:58       0x7f120f7f1000     0x7f120f9f0000   0x1ff000   0x19d000 /lib/x86_64-linux-gnu/libm-2.27.so
13:09:58       0x7f120f9f0000     0x7f120f9f1000     0x1000   0x19c000 /lib/x86_64-linux-gnu/libm-2.27.so
13:09:58       0x7f120f9f1000     0x7f120f9f2000     0x1000   0x19d000 /lib/x86_64-linux-gnu/libm-2.27.so
13:09:58       0x7f120f9f2000     0x7f120f9f9000     0x7000        0x0 /lib/x86_64-linux-gnu/librt-2.27.so
13:09:58       0x7f120f9f9000     0x7f120fbf8000   0x1ff000     0x7000 /lib/x86_64-linux-gnu/librt-2.27.so
13:09:58       0x7f120fbf8000     0x7f120fbf9000     0x1000     0x6000 /lib/x86_64-linux-gnu/librt-2.27.so
13:09:58       0x7f120fbf9000     0x7f120fbfa000     0x1000     0x7000 /lib/x86_64-linux-gnu/librt-2.27.so
13:09:58       0x7f120fbfa000     0x7f120fbfc000     0x2000        0x0 /lib/x86_64-linux-gnu/libutil-2.27.so
13:09:58       0x7f120fbfc000     0x7f120fdfb000   0x1ff000     0x2000 /lib/x86_64-linux-gnu/libutil-2.27.so
13:09:58       0x7f120fdfb000     0x7f120fdfc000     0x1000     0x1000 /lib/x86_64-linux-gnu/libutil-2.27.so
13:09:58       0x7f120fdfc000     0x7f120fdfd000     0x1000     0x2000 /lib/x86_64-linux-gnu/libutil-2.27.so
13:09:58       0x7f120fdfd000     0x7f120fe00000     0x3000        0x0 /lib/x86_64-linux-gnu/libdl-2.27.so
13:09:58       0x7f120fe00000     0x7f120ffff000   0x1ff000     0x3000 /lib/x86_64-linux-gnu/libdl-2.27.so
13:09:58       0x7f120ffff000     0x7f1210000000     0x1000     0x2000 /lib/x86_64-linux-gnu/libdl-2.27.so
13:09:58       0x7f1210000000     0x7f1210001000     0x1000     0x3000 /lib/x86_64-linux-gnu/libdl-2.27.so
13:09:58       0x7f1210001000     0x7f121001b000    0x1a000        0x0 /lib/x86_64-linux-gnu/libpthread-2.27.so
13:09:58       0x7f121001b000     0x7f121021a000   0x1ff000    0x1a000 /lib/x86_64-linux-gnu/libpthread-2.27.so
13:09:58       0x7f121021a000     0x7f121021b000     0x1000    0x19000 /lib/x86_64-linux-gnu/libpthread-2.27.so
13:09:58       0x7f121021b000     0x7f121021c000     0x1000    0x1a000 /lib/x86_64-linux-gnu/libpthread-2.27.so
13:09:58       0x7f121021c000     0x7f1210220000     0x4000        0x0 
13:09:58       0x7f1210220000     0x7f1210249000    0x29000        0x0 /lib/x86_64-linux-gnu/ld-2.27.so
13:09:58       0x7f1210249000     0x7f1210259000    0x10000        0x0 /dev/nvidia0
13:09:58       0x7f1210259000     0x7f121025a000     0x1000        0x0 /opt/conda/envs/rapids/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f121025a000     0x7f121025c000     0x2000     0x1000 /opt/conda/envs/rapids/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f121025c000     0x7f121025d000     0x1000     0x3000 /opt/conda/envs/rapids/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f121025d000     0x7f121025e000     0x1000     0x3000 /opt/conda/envs/rapids/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f121025e000     0x7f1210260000     0x2000     0x4000 /opt/conda/envs/rapids/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
13:09:58       0x7f1210260000     0x7f12102a0000    0x40000        0x0 
13:09:58       0x7f12102a0000     0x7f121043b000   0x19b000        0x0 /usr/lib/locale/locale-archive
13:09:58       0x7f121043b000     0x7f1210440000     0x5000        0x0 
13:09:58       0x7f1210440000     0x7f1210447000     0x7000        0x0 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
13:09:58       0x7f1210447000     0x7f1210449000     0x2000        0x0 
13:09:58       0x7f1210449000     0x7f121044a000     0x1000    0x29000 /lib/x86_64-linux-gnu/ld-2.27.so
13:09:58       0x7f121044a000     0x7f121044b000     0x1000    0x2a000 /lib/x86_64-linux-gnu/ld-2.27.so
13:09:58       0x7f121044b000     0x7f121044c000     0x1000        0x0 
13:09:58       0x7ffc50510000     0x7ffc50532000    0x22000        0x0 [stack]
13:09:58       0x7ffc50563000     0x7ffc50566000     0x3000        0x0 [vvar]
13:09:58       0x7ffc50566000     0x7ffc50567000     0x1000        0x0 [vdso]
13:09:58   0xffffffffff600000 0xffffffffff601000     0x1000        0x0 [vsyscall]

13:09:58 ##~CuFileDriverInitializer() for s_cufile_initializer: is_available_ = true

s_cufile_initializer: is_available_ would be true only if libcufile.so is available and cuFileDriverOpen() API succeed.

@jakirkham
Now I remember that, from CUDA 11.5, libcufile.so (GPUDirectStorage's shared library) is bundled to CUDA Toolkit 11.5.
(That's why the segfault doesn't happen in CUDA Toolkit < 11.5).

It looks like libcufile.so registers a cleaning up function with atexit() and causes the segmentation fault.
Now will look at libcufile code.

jakirkham · 2021-11-19T23:48:55Z

We were using the static library before right? Should we still be using that? Thinking about users with older CUDA versions that may not have GDS

gigony · 2021-11-20T02:24:53Z

We were using the static library before right? Should we still be using that? Thinking about users with older CUDA versions that may not have GDS

@jakirkham Thanks for the feedback! No, we don't. GDS static library had an issue (it wasn't compiled with _GLIBCXX_USE_CXX11_ABI=0 which is necessary for building manylinux2014 wheel) so we couldn't link GDS static library. cuCIM loads a shared library (libcufile.so) if it is available in user's system.

gigony · 2021-11-20T03:50:56Z

Moving stub initialization early didn't help resolve the issue.

https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cucim/job/prb/job/cucim-gpu-test/CUDA=11.5,GPU_LABEL=cuda115,LINUX_VER=ubuntu18.04,PYTHON=3.7/142/console

19:06:04 Starting program: /opt/conda/envs/rapids/bin/python -c "import cucim.clara._cucim as cc; cc.haha(\"dd\");print(\"hoho\")"
19:06:04 warning: Error disabling address space randomization: Operation not permitted
19:06:04 [Thread debugging using libthread_db enabled]
19:06:04 Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
19:06:05 ##CuFileDriverInitializer()
19:06:05 ##load()
19:06:05 ##load(): handle_ == nullptr
19:06:05 ##load(): load_library != nullptr
19:06:05 [New Thread 0x7f5d1f939700 (LWP 2089)]
19:06:06 ##impl_cuFileDriverGetProperties() called
19:06:06 haha
19:06:06 [New Thread 0x7f5d1e485700 (LWP 2090)]
19:06:06 Trying to import cupy...
19:06:06 Importing cupy done.
19:06:06 Trying to import clara...
19:06:06 Importing clara done.
19:06:06 hoho
19:06:06 [Thread 0x7f5d1e485700 (LWP 2090) exited]
19:06:06 ##~CuFileDriverInitializer() for s_cufile_initializer: is_available_ = true
19:06:06 ##impl_cuFileDriverClose() called
19:06:06 CuFileDriver closed!
19:06:06 ##unload()
19:06:06 ##Unloading libcufile.so in unload()
19:06:06 

19:06:06 Thread 1 "python" received signal SIGSEGV, Segmentation fault.
19:06:06 0x00007f5d2729acb0 in ?? ()
19:06:06 #0  0x00007f5d2729acb0 in ?? ()
19:06:06 #1  0x00007f5d2786ca9d in (anonymous namespace)::run (p=<optimized out>)
19:06:06     at ../../../../libstdc++-v3/libsupc++/atexit_thread.cc:75
19:06:06 #2  0x00007f5d27c8c161 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
19:06:06 #3  0x00007f5d27c8c25a in exit () from /lib/x86_64-linux-gnu/libc.so.6
19:06:06 #4  0x00007f5d27c6abfe in __libc_start_main ()
19:06:06    from /lib/x86_64-linux-gnu/libc.so.6
19:06:06 #5  0x0000564b21c3351d in _start ()
19:06:06 process 2085
19:06:06 Mapped address spaces:

gigony · 2021-11-24T02:09:26Z

Closing this because we root-caused the issue (see #158)

gigony added bug Something isn't working non-breaking Introduces a non-breaking change labels Nov 17, 2021

gigony self-assigned this Nov 17, 2021

gigony requested review from a team as code owners November 17, 2021 09:33

gigony force-pushed the dump_trace branch from f229c10 to 998aa59 Compare November 17, 2021 09:36

gigony changed the title ~~[TEST] Dump GDB backtraces on segfault~~ [DEBUG] Dump GDB back traces on Segmentation Fault Nov 17, 2021

gigony force-pushed the dump_trace branch 5 times, most recently from 93df698 to bee158b Compare November 17, 2021 16:47

jakirkham mentioned this pull request Nov 17, 2021

Build with NumPy 1.17 #148

Merged

gigony force-pushed the dump_trace branch from bee158b to 6cf73c8 Compare November 17, 2021 18:37

gigony force-pushed the dump_trace branch 3 times, most recently from 1d55086 to 3a80033 Compare November 18, 2021 02:27

gigony requested a review from a team as a code owner November 18, 2021 02:27

gigony force-pushed the dump_trace branch 4 times, most recently from 4e910d9 to a802658 Compare November 18, 2021 10:00

Dump GDB backtraces on segfault

20ae816

gigony force-pushed the dump_trace branch from a802658 to 20ae816 Compare November 18, 2021 11:19

gigony marked this pull request as draft November 18, 2021 16:31

gigony force-pushed the dump_trace branch 4 times, most recently from 0a37668 to 92ad2d0 Compare November 18, 2021 21:07

simplify script using external package

24e1835

gigony force-pushed the dump_trace branch from 92ad2d0 to 24e1835 Compare November 18, 2021 21:09

gigony added 2 commits November 18, 2021 15:46

remove libcucim link to _cucim

1e2aaac

Use cucim::io::Device

c1d756d

https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cucim/job/prb/job/cucim-gpu-test/CUDA=11.5,GPU_LABEL=cuda115,LINUX_VER=ubuntu18.04,PYTHON=3.7/130/consoleFull

gigony force-pushed the dump_trace branch from 78ca7ca to a8eba33 Compare November 19, 2021 05:39

Discard unnecessary code, abseil/libcuckoo/boost

ec78e3f

gigony force-pushed the dump_trace branch from a8eba33 to ec78e3f Compare November 19, 2021 06:56

gigony added 5 commits November 19, 2021 00:21

restoree abseil/libcuckoo/boost

4359ede

restore core cpp files

eb54e99

Disable part of core-cuimage/cache

28d78e5

Do not link gds module

c48090f

Pinpoint rootcause with debug info

85695c7

Do not unload GDS shared library

00bbc97

gigony mentioned this pull request Nov 20, 2021

Fix a segmentation fault due to unloaded libcufile #158

Merged

gigony added 2 commits November 19, 2021 17:54

Add more debug message

e2d8d8e

Move CuFileStub definition to header

5fdc5d1

Dump symbol table

2227415

gigony force-pushed the dump_trace branch from b13320d to 2227415 Compare November 21, 2021 16:16

Do not use thread_local

94c4de1

gigony closed this Nov 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[DEBUG] Dump GDB back traces on Segmentation Fault #153

[DEBUG] Dump GDB back traces on Segmentation Fault #153

Uh oh!

gigony commented Nov 17, 2021

Uh oh!

ajschmidt8 commented Nov 17, 2021

Uh oh!

gigony commented Nov 19, 2021

Uh oh!

jakirkham commented Nov 19, 2021

Uh oh!

gigony commented Nov 20, 2021 •

edited

Loading

Uh oh!

gigony commented Nov 20, 2021

Uh oh!

gigony commented Nov 24, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[DEBUG] Dump GDB back traces on Segmentation Fault #153

[DEBUG] Dump GDB back traces on Segmentation Fault #153

Uh oh!

Conversation

gigony commented Nov 17, 2021

Uh oh!

ajschmidt8 commented Nov 17, 2021

Uh oh!

gigony commented Nov 19, 2021

Uh oh!

jakirkham commented Nov 19, 2021

Uh oh!

gigony commented Nov 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gigony commented Nov 20, 2021

Uh oh!

gigony commented Nov 24, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gigony commented Nov 20, 2021 •

edited

Loading