Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent crash in travis regression tests #73

Closed
edyoung opened this issue Apr 5, 2020 · 6 comments
Closed

Intermittent crash in travis regression tests #73

edyoung opened this issue Apr 5, 2020 · 6 comments

Comments

@edyoung
Copy link
Member

edyoung commented Apr 5, 2020

Some jobs such as https://travis-ci.com/github/edyoung/gnofract4d/jobs/314557671 are intermittently failing with the stack below. This doesn't appear to be an error in ir.py, which is pure python code and shouldn't cause a segfault. More likely this is a refcount issue, buffer overflow, or similar.

fract4dgui/tests/test_browser.py ..Fatal Python error: Segmentation fault

Current thread 0x00007fdd1d9cc680 (most recent call first):

  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/ir.py", line 13 in __init__
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/ir.py", line 57 in __init__
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/ir.py", line 92 in __init__
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 912 in binop
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 779 in exp
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 501 in stm
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 711 in decl
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 491 in stm
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 481 in <listcomp>
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 481 in stmlist
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 155 in final
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 1288 in final
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 1311 in main
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/translate.py", line 1239 in __init__
  File "/home/travis/build/edyoung/gnofract4d/fract4d_compiler/fc.py", line 447 in get_formula
  File "/home/travis/build/edyoung/gnofract4d/fract4d/formsettings.py", line 324 in set_formula
  File "/home/travis/build/edyoung/gnofract4d/fract4d/fractal.py", line 496 in set_formula
  File "/home/travis/build/edyoung/gnofract4d/fract4d/fractal.py", line 546 in set_outer
  File "/home/travis/build/edyoung/gnofract4d/fract4d/fractal.py", line 96 in __init__
  File "/home/travis/build/edyoung/gnofract4d/fract4dgui/tests/test_browser.py", line 15 in setUp
  File "/opt/python/3.7.6/lib/python3.7/unittest/case.py", line 624 in run
  File "/opt/python/3.7.6/lib/python3.7/unittest/case.py", line 676 in __call__
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/unittest.py", line 232 in runtest
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 135 in pytest_runtest_call
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 217 in <lambda>
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 244 in from_call
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 217 in call_runtest_hook
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 186 in call_and_report
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 100 in runtestprotocol
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 85 in pytest_runtest_protocol
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 272 in pytest_runtestloop
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/lib/python3.7/site-packages/_pytest/config/__init__.py", line 125 in main
  File "/home/travis/build/edyoung/gnofract4d/.tox/py37/bin/pytest", line 8 in <module>
ERROR: InvocationError for command /home/travis/build/edyoung/gnofract4d/.tox/py37/bin/pytest fract4d fract4dgui fract4d_compiler (exited with code -11 (SIGSEGV)) (exited with code -11)
@mindhells
Copy link
Member

mindhells commented Jul 9, 2020 •
edited
Loading

It seems this is still a thing:

I also ran into the same problem while working on #147 . Thought it was related to _REENTRANT macro deletion:

@edyoung
Copy link
Member Author

edyoung commented Jul 11, 2020

Have also seen this with other stack traces. Appears there is some memory corruption issue. Have not seen this when running manually

@dragonmux
Copy link
Member

I'd be very interested to see what happens if these tests are run with the shared libraries built with -fsanitize=address and LD_PRELOAD=/usr/lib/libasan.so (or w/e libasan.so is on your system) as this feels like the kind of thing that ASAN is able to pinpoint and show you exactly where the code goes wrong..

@mindhells
Copy link
Member

I've managed to run gnofract with -fsanitize=address fract4dc module in MacOS environment.
I had to do the following:

    1. in setup.py add -fsanitize=address to the extra_link_args for the module_fract4dc Extension.
    1. then I followed the instructions described here: https://jonasdevlieghere.com/sanitizing-python-modules/ to get the actual python executable
    1. DYLD_INSERT_LIBRARIES=/usr/local/opt/llvm/lib/clang/9.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib $actual_python_exec gnofract4d

If you try to run gnofract4d with the extension linked with ASAN but without DYLD_INSERT_LIBRARIES you get this error:

==47716==ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
DYLD_INSERT_LIBRARIES=/usr/local/opt/llvm/lib/clang/9.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib
"interceptors not installed" && 0Abort trap: 6

I think with gcc you can statically link the libasan but haven't tried it.

I've (almost) never seen the errors when running the tests locally, but I sometimes get "segmenation fault" errors when running the application. This is the sanitizer output after running into that error:

==47496==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d0000c4f80 at pc 0x00011227083e bp 0x7ffee22d5130 sp 0x7ffee22d5128
READ of size 8 at 0x60d0000c4f80 thread T0
    #0 0x11227083d in calcs::pycalc(_object*, _object*, _object*) calcs.cpp:67
    #1 0x112262564 in pycalc(_object*, _object*, _object*) fract4dmodule.cpp:147
    #2 0x10e6bb4c1 in cfunction_call_varargs (Python:x86_64+0x1a4c1)
    #3 0x10e6bafb0 in _PyObject_MakeTpCall (Python:x86_64+0x19fb0)
    #4 0x10e75b7c5 in call_function (Python:x86_64+0xba7c5)
    #5 0x10e758372 in _PyEval_EvalFrameDefault (Python:x86_64+0xb7372)
    #6 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #7 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #8 0x10e7581c8 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71c8)
    #9 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #10 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #11 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #12 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #13 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #14 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #15 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #16 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #17 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #18 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #19 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #20 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #21 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #22 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #23 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #24 0x10e7581c8 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71c8)
    #25 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #26 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #27 0x10e6bd993 in method_vectorcall (Python:x86_64+0x1c993)
    #28 0x10e6bb23c in PyVectorcall_Call (Python:x86_64+0x1a23c)
    #29 0x112412a7e in pyg_closure_marshal (_gi.cpython-38-darwin.so:x86_64+0x19a7e)
    #30 0x1125ef79c in g_closure_invoke (libgobject-2.0.0.dylib:x86_64+0x779c)
    #31 0x112605c9f in signal_emit_unlocked_R (libgobject-2.0.0.dylib:x86_64+0x1dc9f)
    #32 0x112605542 in g_signal_emitv (libgobject-2.0.0.dylib:x86_64+0x1d542)
    #33 0x11240104d in pygobject_emit (_gi.cpython-38-darwin.so:x86_64+0x804d)
    #34 0x10e6bb555 in cfunction_call_varargs (Python:x86_64+0x1a555)
    #35 0x10e6bafb0 in _PyObject_MakeTpCall (Python:x86_64+0x19fb0)
    #36 0x10e75b7c5 in call_function (Python:x86_64+0xba7c5)
    #37 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #38 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #39 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #40 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #41 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #42 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #43 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #44 0x10e6bd993 in method_vectorcall (Python:x86_64+0x1c993)
    #45 0x10e6bb23c in PyVectorcall_Call (Python:x86_64+0x1a23c)
    #46 0x11241696e in pygi_signal_closure_marshal (_gi.cpython-38-darwin.so:x86_64+0x1d96e)
    #47 0x1125ef79c in g_closure_invoke (libgobject-2.0.0.dylib:x86_64+0x779c)
    #48 0x112605c9f in signal_emit_unlocked_R (libgobject-2.0.0.dylib:x86_64+0x1dc9f)
    #49 0x112606af8 in g_signal_emit_valist (libgobject-2.0.0.dylib:x86_64+0x1eaf8)
    #50 0x1126071e1 in g_signal_emit (libgobject-2.0.0.dylib:x86_64+0x1f1e1)
    #51 0x117527237 in gtk_widget_event_internal (libgtk-3.0.dylib:x86_64+0x31d237)
    #52 0x1173ab06e in gtk_propagate_event (libgtk-3.0.dylib:x86_64+0x1a106e)
    #53 0x1173aaa56 in gtk_main_do_event (libgtk-3.0.dylib:x86_64+0x1a0a56)
    #54 0x113162820 in _gdk_event_emit (libgdk-3.0.dylib:x86_64+0x15820)
    #55 0x1131906b1 in gdk_event_dispatch (libgdk-3.0.dylib:x86_64+0x436b1)
    #56 0x1124962dc in g_main_context_dispatch (libglib-2.0.0.dylib:x86_64+0x3b2dc)
    #57 0x112496658 in g_main_context_iterate (libglib-2.0.0.dylib:x86_64+0x3b658)
    #58 0x112496999 in g_main_loop_run (libglib-2.0.0.dylib:x86_64+0x3b999)
    #59 0x1173aa3e9 in gtk_main (libgtk-3.0.dylib:x86_64+0x1a03e9)
    #60 0x11264ff2c in ffi_call_unix64 (libffi.7.dylib:x86_64+0x4f2c)
    #61 0x7ffee22d791f  (<unknown module>)

0x60d0000c4f80 is located 96 bytes inside of 136-byte region [0x60d0000c4f20,0x60d0000c4fa8)
freed by thread T99 here:
    #0 0x10d97ebed in wrap__ZdlPv (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x51bed)
    #1 0x112271d60 in calculation_thread(void*) calcs.cpp:116
    #2 0x7fff6736f108 in _pthread_start (libsystem_pthread.dylib:x86_64+0x6108)
    #3 0x7fff6736ab8a in thread_start (libsystem_pthread.dylib:x86_64+0x1b8a)

previously allocated by thread T0 here:
    #0 0x10d97e7cd in wrap__Znwm (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x517cd)
    #1 0x112270df9 in parse_calc_args(_object*, _object*) calcs.cpp:123
    #2 0x112270412 in calcs::pycalc(_object*, _object*, _object*) calcs.cpp:40
    #3 0x112262564 in pycalc(_object*, _object*, _object*) fract4dmodule.cpp:147
    #4 0x10e6bb4c1 in cfunction_call_varargs (Python:x86_64+0x1a4c1)
    #5 0x10e6bafb0 in _PyObject_MakeTpCall (Python:x86_64+0x19fb0)
    #6 0x10e75b7c5 in call_function (Python:x86_64+0xba7c5)
    #7 0x10e758372 in _PyEval_EvalFrameDefault (Python:x86_64+0xb7372)
    #8 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #9 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #10 0x10e7581c8 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71c8)
    #11 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #12 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #13 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #14 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #15 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #16 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #17 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #18 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #19 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #20 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #21 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #22 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #23 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #24 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #25 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #26 0x10e7581c8 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71c8)
    #27 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #28 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #29 0x10e6bd993 in method_vectorcall (Python:x86_64+0x1c993)

Thread T99 created by T0 here:
    #0 0x10d96c53a in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x3f53a)
    #1 0x112270789 in calcs::pycalc(_object*, _object*, _object*) calcs.cpp:64
    #2 0x112262564 in pycalc(_object*, _object*, _object*) fract4dmodule.cpp:147
    #3 0x10e6bb4c1 in cfunction_call_varargs (Python:x86_64+0x1a4c1)
    #4 0x10e6bafb0 in _PyObject_MakeTpCall (Python:x86_64+0x19fb0)
    #5 0x10e75b7c5 in call_function (Python:x86_64+0xba7c5)
    #6 0x10e758372 in _PyEval_EvalFrameDefault (Python:x86_64+0xb7372)
    #7 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #8 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #9 0x10e7581c8 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71c8)
    #10 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #11 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #12 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #13 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #14 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #15 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #16 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #17 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #18 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #19 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #20 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #21 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #22 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #23 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #24 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #25 0x10e7581c8 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71c8)
    #26 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #27 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #28 0x10e6bd993 in method_vectorcall (Python:x86_64+0x1c993)
    #29 0x10e6bb23c in PyVectorcall_Call (Python:x86_64+0x1a23c)
    #30 0x112412a7e in pyg_closure_marshal (_gi.cpython-38-darwin.so:x86_64+0x19a7e)
    #31 0x1125ef79c in g_closure_invoke (libgobject-2.0.0.dylib:x86_64+0x779c)
    #32 0x112605c9f in signal_emit_unlocked_R (libgobject-2.0.0.dylib:x86_64+0x1dc9f)
    #33 0x112605542 in g_signal_emitv (libgobject-2.0.0.dylib:x86_64+0x1d542)
    #34 0x11240104d in pygobject_emit (_gi.cpython-38-darwin.so:x86_64+0x804d)
    #35 0x10e6bb555 in cfunction_call_varargs (Python:x86_64+0x1a555)
    #36 0x10e6bafb0 in _PyObject_MakeTpCall (Python:x86_64+0x19fb0)
    #37 0x10e75b7c5 in call_function (Python:x86_64+0xba7c5)
    #38 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #39 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #40 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #41 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #42 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #43 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #44 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #45 0x10e6bd993 in method_vectorcall (Python:x86_64+0x1c993)
    #46 0x10e6bb23c in PyVectorcall_Call (Python:x86_64+0x1a23c)
    #47 0x11241696e in pygi_signal_closure_marshal (_gi.cpython-38-darwin.so:x86_64+0x1d96e)
    #48 0x1125ef79c in g_closure_invoke (libgobject-2.0.0.dylib:x86_64+0x779c)
    #49 0x112605c9f in signal_emit_unlocked_R (libgobject-2.0.0.dylib:x86_64+0x1dc9f)
    #50 0x112606af8 in g_signal_emit_valist (libgobject-2.0.0.dylib:x86_64+0x1eaf8)
    #51 0x1126071e1 in g_signal_emit (libgobject-2.0.0.dylib:x86_64+0x1f1e1)
    #52 0x117527237 in gtk_widget_event_internal (libgtk-3.0.dylib:x86_64+0x31d237)
    #53 0x1173ab06e in gtk_propagate_event (libgtk-3.0.dylib:x86_64+0x1a106e)
    #54 0x1173aaa56 in gtk_main_do_event (libgtk-3.0.dylib:x86_64+0x1a0a56)
    #55 0x113162820 in _gdk_event_emit (libgdk-3.0.dylib:x86_64+0x15820)
    #56 0x1131906b1 in gdk_event_dispatch (libgdk-3.0.dylib:x86_64+0x436b1)
    #57 0x1124962dc in g_main_context_dispatch (libglib-2.0.0.dylib:x86_64+0x3b2dc)
    #58 0x112496658 in g_main_context_iterate (libglib-2.0.0.dylib:x86_64+0x3b658)
    #59 0x112496999 in g_main_loop_run (libglib-2.0.0.dylib:x86_64+0x3b999)
    #60 0x1173aa3e9 in gtk_main (libgtk-3.0.dylib:x86_64+0x1a03e9)
    #61 0x11264ff2c in ffi_call_unix64 (libffi.7.dylib:x86_64+0x4f2c)
    #62 0x7ffee22d791f  (<unknown module>)

SUMMARY: AddressSanitizer: heap-use-after-free calcs.cpp:67 in calcs::pycalc(_object*, _object*, _object*)
Shadow bytes around the buggy address:
  0x1c1a000189a0: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
  0x1c1a000189b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c1a000189c0: fa fa fa fa fa fa fa fa fa fa 00 00 00 00 00 00
  0x1c1a000189d0: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
  0x1c1a000189e0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
=>0x1c1a000189f0:[fd]fd fd fd fd fa fa fa fa fa fa fa fa fa 00 00
  0x1c1a00018a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c1a00018a10: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x1c1a00018a20: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa
  0x1c1a00018a30: fa fa 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c1a00018a40: 00 00 00 00 fa fa fa fa fa fa fa fa 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==47496==ABORTING
Abort trap: 6

@mindhells mindhells mentioned this issue Jul 13, 2020
Merged
@dragonmux
Copy link
Member

dragonmux commented Jul 13, 2020 •
edited
Loading

I've managed to run gnofract with -fsanitize=address fract4dc module in MacOS environment.
I had to do the following:

    1. in setup.py add -fsanitize=address to the extra_link_args for the module_fract4dc Extension.
    1. then I followed the instructions described here: https://jonasdevlieghere.com/sanitizing-python-modules/ to get the actual python executable
    1. DYLD_INSERT_LIBRARIES=/usr/local/opt/llvm/lib/clang/9.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib $actual_python_exec gnofract4d

Ahh.. TIL about how you inject ASAN into binaries on Mac OS.. glad it basically told you what to do though.

If you try to run gnofract4d with the extension linked with ASAN but without DYLD_INSERT_LIBRARIES you get this error:

==47716==ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
DYLD_INSERT_LIBRARIES=/usr/local/opt/llvm/lib/clang/9.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib
"interceptors not installed" && 0Abort trap: 6

This is specifically why you have to (what is on Linux) LD_PRELOAD the library as that bypasses this.

I think with gcc you can statically link the libasan but haven't tried it.

I've (almost) never seen the errors when running the tests locally, but I sometimes get "segmenation fault" errors when running the application. This is the sanitizer output after running into that error:

==47496==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d0000c4f80 at pc 0x00011227083e bp 0x7ffee22d5130 sp 0x7ffee22d5128
READ of size 8 at 0x60d0000c4f80 thread T0
    #0 0x11227083d in calcs::pycalc(_object*, _object*, _object*) calcs.cpp:67
    #1 0x112262564 in pycalc(_object*, _object*, _object*) fract4dmodule.cpp:147
    #2 0x10e6bb4c1 in cfunction_call_varargs (Python:x86_64+0x1a4c1)
    #3 0x10e6bafb0 in _PyObject_MakeTpCall (Python:x86_64+0x19fb0)
    #4 0x10e75b7c5 in call_function (Python:x86_64+0xba7c5)
    #5 0x10e758372 in _PyEval_EvalFrameDefault (Python:x86_64+0xb7372)
    #6 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #7 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #8 0x10e7581c8 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71c8)
    #9 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #10 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #11 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #12 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #13 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #14 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #15 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #16 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #17 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #18 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #19 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #20 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #21 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #22 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #23 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #24 0x10e7581c8 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71c8)
    #25 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #26 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #27 0x10e6bd993 in method_vectorcall (Python:x86_64+0x1c993)
    #28 0x10e6bb23c in PyVectorcall_Call (Python:x86_64+0x1a23c)
    #29 0x112412a7e in pyg_closure_marshal (_gi.cpython-38-darwin.so:x86_64+0x19a7e)
    #30 0x1125ef79c in g_closure_invoke (libgobject-2.0.0.dylib:x86_64+0x779c)
    #31 0x112605c9f in signal_emit_unlocked_R (libgobject-2.0.0.dylib:x86_64+0x1dc9f)
    #32 0x112605542 in g_signal_emitv (libgobject-2.0.0.dylib:x86_64+0x1d542)
    #33 0x11240104d in pygobject_emit (_gi.cpython-38-darwin.so:x86_64+0x804d)
    #34 0x10e6bb555 in cfunction_call_varargs (Python:x86_64+0x1a555)
    #35 0x10e6bafb0 in _PyObject_MakeTpCall (Python:x86_64+0x19fb0)
    #36 0x10e75b7c5 in call_function (Python:x86_64+0xba7c5)
    #37 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #38 0x10e75c12a in _PyEval_EvalCodeWithName (Python:x86_64+0xbb12a)
    #39 0x10e6bb95b in _PyFunction_Vectorcall (Python:x86_64+0x1a95b)
    #40 0x10e6bd8a6 in method_vectorcall (Python:x86_64+0x1c8a6)
    #41 0x10e75b5fb in call_function (Python:x86_64+0xba5fb)
    #42 0x10e7581e4 in _PyEval_EvalFrameDefault (Python:x86_64+0xb71e4)
    #43 0x10e6bb7f5 in function_code_fastcall (Python:x86_64+0x1a7f5)
    #44 0x10e6bd993 in method_vectorcall (Python:x86_64+0x1c993)
    #45 0x10e6bb23c in PyVectorcall_Call (Python:x86_64+0x1a23c)
    #46 0x11241696e in pygi_signal_closure_marshal (_gi.cpython-38-darwin.so:x86_64+0x1d96e)
    #47 0x1125ef79c in g_closure_invoke (libgobject-2.0.0.dylib:x86_64+0x779c)
    #48 0x112605c9f in signal_emit_unlocked_R (libgobject-2.0.0.dylib:x86_64+0x1dc9f)
    #49 0x112606af8 in g_signal_emit_valist (libgobject-2.0.0.dylib:x86_64+0x1eaf8)
    #50 0x1126071e1 in g_signal_emit (libgobject-2.0.0.dylib:x86_64+0x1f1e1)
    #51 0x117527237 in gtk_widget_event_internal (libgtk-3.0.dylib:x86_64+0x31d237)
    #52 0x1173ab06e in gtk_propagate_event (libgtk-3.0.dylib:x86_64+0x1a106e)
    #53 0x1173aaa56 in gtk_main_do_event (libgtk-3.0.dylib:x86_64+0x1a0a56)
    #54 0x113162820 in _gdk_event_emit (libgdk-3.0.dylib:x86_64+0x15820)
    #55 0x1131906b1 in gdk_event_dispatch (libgdk-3.0.dylib:x86_64+0x436b1)
    #56 0x1124962dc in g_main_context_dispatch (libglib-2.0.0.dylib:x86_64+0x3b2dc)
    #57 0x112496658 in g_main_context_iterate (libglib-2.0.0.dylib:x86_64+0x3b658)
    #58 0x112496999 in g_main_loop_run (libglib-2.0.0.dylib:x86_64+0x3b999)
    #59 0x1173aa3e9 in gtk_main (libgtk-3.0.dylib:x86_64+0x1a03e9)
    #60 0x11264ff2c in ffi_call_unix64 (libffi.7.dylib:x86_64+0x4f2c)
    #61 0x7ffee22d791f  (<unknown module>)

[...]

SUMMARY: AddressSanitizer: heap-use-after-free calcs.cpp:67 in calcs::pycalc(_object*, _object*, _object*)
Shadow bytes around the buggy address:
0x1c1a000189a0: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
0x1c1a000189b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x1c1a000189c0: fa fa fa fa fa fa fa fa fa fa 00 00 00 00 00 00
0x1c1a000189d0: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
0x1c1a000189e0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
=>0x1c1a000189f0:[fd]fd fd fd fd fa fa fa fa fa fa fa fa fa 00 00
0x1c1a00018a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1c1a00018a10: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
0x1c1a00018a20: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa
0x1c1a00018a30: fa fa 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1c1a00018a40: 00 00 00 00 fa fa fa fa fa fa fa fa 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==47496==ABORTING
Abort trap: 6

That's a nice trace.. Use after free.. so the sanitizer is saying that one of the pointers used on line 67 of calcs.cpp has already been freed, hence the occasional segfault behaviour. looking at https://github.com/fract4d/gnofract4d/blob/master/fract4d/c/fract4dc/calcs.cpp#L67, that suggests the cargs or site pointer is invalid after that pthread logic.. probably cargs if it ends up being freed in the thread that's spawned

A closer look at the compute thread says yes, cargs gets deleted as the last action of the thread.. so, for this code to work, the thread has to be started paused, otherwise the scheduler is allowed to let the thread do its thing before the rest of the function is run

@mindhells mindhells mentioned this issue Aug 17, 2020
Merged
@mindhells mindhells mentioned this issue Oct 22, 2020
Closed
@edyoung
Copy link
Member Author

edyoung commented Dec 18, 2020

Haven't seen this in a while so will assume the change above addressed it. Thanks!

@edyoung edyoung closed this as completed Dec 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@edyoung @dragonmux @mindhells