drgn segfaults on attempting to debug a userspace coredump on Fedora 34 or Debian bullseye

[rincebrain]

Core was generated by `/usr/bin/python3 /usr/local/bin/drgn -c core.ztest.442075.fedora34.1637728682'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f04fa0a20c0 in read_addrs (addrs=0x7ffc7aa89a70, n=4, read_vaddr=<synthetic pointer>, vaddr=<optimized out>, buffer_available=0x7ffc7aa899f0, buffer=0x7ffc7aa899e8, elfdata=1 '\001', elfclass=2 '\002',
    closure=<synthetic pointer>) at link_map.c:291
291               addrs[i] = LE64 (read_8ubyte_unaligned_noncvt (&(*a64)[i]));
Missing separate debuginfos, use: dnf debuginfo-install libselinux-3.2-1.fc34.x86_64 openldap-2.4.57-5.fc34.x86_64
(gdb) bt
#0  0x00007f04fa0a20c0 in read_addrs (addrs=0x7ffc7aa89a70, n=4, read_vaddr=<synthetic pointer>, vaddr=<optimized out>, buffer_available=0x7ffc7aa899f0, buffer=0x7ffc7aa899e8, elfdata=1 '\001', elfclass=2 '\002',
    closure=<synthetic pointer>) at link_map.c:291
#1  report_r_debug (memory_callback=0x7f04fa0a16f0 <integrated_memory_callback>, r_debug_info=0x7ffc7aa89b48, memory_callback_arg=0x7ffc7aa89a40, r_debug_vaddr=<optimized out>, dwfl=0x55c79e5e2d60, elfdata=1 '\001', elfclass=2 '\002')
    at link_map.c:341
#2  dwfl_link_map_report (dwfl=dwfl@entry=0x55c79e5e2d60, auxv=auxv@entry=0x7f04f94022b8, auxv_size=auxv_size@entry=336, memory_callback=memory_callback@entry=0x7f04fa0a3400 <dwfl_elf_phdr_memory_callback>,
    memory_callback_arg=memory_callback_arg@entry=0x55c79e58c030, r_debug_info=r_debug_info@entry=0x7ffc7aa89b48) at link_map.c:1054
#3  0x00007f04fa0a414e in dwfl_core_file_report (dwfl=0x55c79e5e2d60, elf=0x55c79e58c030, executable=executable@entry=0x0) at core-file.c:547
#4  0x00007f04fa10533b in userspace_report_debug_info (load=<optimized out>) at ../../libdrgn/debug_info.c:645
#5  drgn_debug_info_load (dbinfo=0x55c79e4597a0, paths=paths@entry=0x0, n=n@entry=0, load_default=load_default@entry=true, load_main=load_main@entry=true) at ../../libdrgn/debug_info.c:1118
#6  0x00007f04fa12bd3e in drgn_program_load_debug_info (prog=prog@entry=0x55c79e5c38e0, paths=paths@entry=0x0, n=n@entry=0, load_default=<optimized out>, load_main=<optimized out>) at ../../libdrgn/program.c:617
#7  0x00007f04fa0f8865 in Program_load_debug_info (self=0x55c79e5c38d0, args=<optimized out>, kwds=<optimized out>) at ../../libdrgn/python/program.c:499
#8  0x00007f04faad68f2 in cfunction_call (func=<built-in method load_debug_info of _drgn.Program object at remote 0x55c79e5c38d0>, args=<optimized out>, kwargs=<optimized out>)
    at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Objects/methodobject.c:543
#9  0x00007f04faad5778 in _PyObject_Call (tstate=0x55c79e4479e0, callable=<built-in method load_debug_info of _drgn.Program object at remote 0x55c79e5c38d0>, args=(None,), kwargs=<optimized out>)
    at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Objects/call.c:281
#10 0x00007f04faac5fc7 in do_call_core (kwdict={'default': True, 'main': True}, callargs=(None,), func=<built-in method load_debug_info of _drgn.Program object at remote 0x55c79e5c38d0>, tstate=<optimized out>)
    at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/ceval.c:5095
#11 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/ceval.c:3580
#12 0x00007f04faabf00d in _PyEval_EvalFrame (throwflag=0,
    f=Frame 0x55c79e5b4850, for file /usr/local/lib64/python3.9/site-packages/drgn-0.0.14+71.gd18be05-py3.9-linux-x86_64.egg/drgn/internal/cli.py, line 2423, in main (python_version='3.9.7', libkdumpfile='without libkdumpfile', version='drgn 0.0.14+71.gd18be05 (using Python 3.9.7, elfutils 0.185, without libkdumpfile)', parser=<ArgumentParser(description='Scriptable debugger', argument_default=None, prefix_chars='-', conflict_handler='error', _registries={'action': {None: <type at remote 0x55c79e59a9f0>, 'store': <type at remote 0x55c79e59a9f0>, 'store_const': <type at remote 0x55c79e59ada0>, 'store_true': <type at remote 0x55c79e59b150>, 'store_false': <type at remote 0x55c79e59b500>, 'append': <type at remote 0x55c79e59b8b0>, 'append_const': <type at remote 0x55c79e59bc60>, 'count': <type at remote 0x55c79e59c010>, 'help': <type at remote 0x55c79e59c3c0>, 'version': <type at remote 0x55c79e59c770>, 'parsers': <type at remote 0x55c79e59d3d0>, 'extend': <type at remote 0x55c79e59d780>}, 'type': {None: <function at ...(truncated), tstate=0x55c79e4479e0) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Include/internal/pycore_ceval.h:40
#13 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x55c79e4a2be0, kwcount=0, kwstep=1, defs=0x0,
    defcount=0, kwdefs=0x0, closure=0x0, name='main', qualname='main') at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/ceval.c:4327
#14 0x00007f04faacccee in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Objects/call.c:396
#15 0x00007f04faac038c in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55c79e4a2be0, callable=<function at remote 0x7f04f951cb80>, tstate=0x55c79e4479e0)
    at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Include/cpython/abstract.h:118
#16 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55c79e4a2be0, callable=<function at remote 0x7f04f951cb80>) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Include/cpython/abstract.h:127
#17 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55c79e4479e0) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/ceval.c:5075
#18 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/ceval.c:3518
#19 0x00007f04faabf00d in _PyEval_EvalFrame (throwflag=0, f=Frame 0x55c79e4a2a60, for file /usr/local/bin/drgn, line 33, in <module> (), tstate=0x55c79e4479e0)
    at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Include/internal/pycore_ceval.h:40
#20 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0,
    kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/ceval.c:4327
#21 0x00007f04fab3b635 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>,
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/usr/local/bin/drgn') at remote 0x7f04fa5c0c10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f04fa5ac860>, '__file__': '/usr/local/bin/drgn', '__cached__': None, 're': <module at remote 0x7f04fa48aef0>, 'sys': <module at remote 0x7f04fa5a37c0>, '__requires__': 'drgn==0.0.14+71.gd18be05', 'distribution': <function at remote 0x7f04fa430b80>, 'importlib_load_entry_point': <function at remote 0x7f04fa47faf0>, 'load_entry_point': <function at remote 0x7f04fa47faf0>}, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>,
    kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/ceval.c:4359
#22 0x00007f04fab3b5cd in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0,
    closure=0x0) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/ceval.c:4375
#23 0x00007f04fab3b57f in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/ceval.c:826
#24 0x00007f04fab68264 in run_eval_code_obj (tstate=0x55c79e4479e0, co=0x7f04fa491030,
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/usr/local/bin/drgn') at remote 0x7f04fa5c0c10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f04fa5ac860>, '__file__': '/usr/local/bin/drgn', '__cached__': None, 're': <module at remote 0x7f04fa48aef0>, 'sys': <module at remote 0x7f04fa5a37c0>, '__requires__': 'drgn==0.0.14+71.gd18be05', 'distribution': <function at remote 0x7f04fa430b80>, 'importlib_load_entry_point': <function at remote 0x7f04fa47faf0>, 'load_entry_point': <function at remote 0x7f04fa47faf0>},
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/usr/local/bin/drgn') at remote 0x7f04fa5c0c10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f04fa5ac860>, '__file__': '/usr/local/bin/drgn', '__cached__': None, 're': <module at remote 0x7f04fa48aef0>, 'sys': <module at remote 0x7f04fa5a37c0>, '__requires__': 'drgn==0.0.14+71.gd18be05', 'distribution': <function at remote 0x7f04fa430b80>, 'importlib_load_entry_point': <function at remote 0x7f04fa47faf0>, 'load_entry_point': <function at remote 0x7f04fa47faf0>}) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/pythonrun.c:1219
#25 0x00007f04fab64506 in run_mod (mod=<optimized out>, filename=<optimized out>,
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/usr/local/bin/drgn') at remote 0x7f04fa5c0c10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f04fa5ac860>, '__file__': '/usr/local/bin/drgn', '__cached__': None, 're': <module at remote 0x7f04fa48aef0>, 'sys': <module at remote 0x7f04fa5a37c0>, '__requires__': 'drgn==0.0.14+71.gd18be05', 'distribution': <function at remote 0x7f04fa430b80>, 'importlib_load_entry_point': <function at remote 0x7f04fa47faf0>, 'load_entry_point': <function at remote 0x7f04fa47faf0>},
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/usr/local/bin/drgn') at remote 0x7f04fa5c0c10>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f04fa5ac860>, '__file__': '/usr/local/bin/drgn', '__cached__': None, 're': <module at remote 0x7f04fa48aef0>, 'sys': <module at remote 0x7f04fa5a37c0>, '__requires__': 'drgn==0.0.14+71.gd18be05', 'distribution': <function at remote 0x7f04fa430b80>, 'importlib_load_entry_point': <function at remote 0x7f04fa47faf0>, 'load_entry_point': <function at remote 0x7f04fa47faf0>}, flags=<optimized out>, arena=<optimized out>)
    at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/pythonrun.c:1240
#26 0x00007f04faa3afb4 in pyrun_file (fp=<optimized out>, filename=<optimized out>, start=<optimized out>, globals=<optimized out>, locals=<optimized out>, closeit=<optimized out>, flags=0x7ffc7aa8a4a8)
    at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/pythonrun.c:1138
#27 0x00007f04fab5e563 in pyrun_simple_file (flags=0x7ffc7aa8a4a8, closeit=1, filename='/usr/local/bin/drgn', fp=0x55c79e443340) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/pythonrun.c:449
#28 PyRun_SimpleFileExFlags (fp=0x55c79e443340, filename=<optimized out>, closeit=1, flags=0x7ffc7aa8a4a8) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Python/pythonrun.c:482
#29 0x00007f04fab5b768 in pymain_run_file (cf=0x7ffc7aa8a4a8, config=0x55c79e446330) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Modules/main.c:373
#30 pymain_run_python (exitcode=0x7ffc7aa8a4a0) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Modules/main.c:598
#31 Py_RunMain () at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Modules/main.c:677
--Type <RET> for more, q to quit, c to continue without paging--
#32 0x00007f04fab2e04d in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/python3.9-3.9.7-1.fc34.x86_64/Modules/main.c:731
#33 0x00007f04fa80eb75 in __libc_start_main (main=0x55c79cf02160 <main>, argc=4, argv=0x7ffc7aa8a6b8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc7aa8a6a8) at ../csu/libc-start.c:332
#34 0x000055c79cf0209e in _start ()

(The above is from a core dump on F34; I tried both the package in Fedora and using git as of d18be05.)

If you'd like to see the core dump, here you go.

Please let me know if I can provide anything else useful.

[osandov]

This looks like a known bug in elfutils. It was introduced in elfutils 0.183 by https://sourceware.org/git/?p=elfutils.git;a=commit;h=3bf41d458fb6f78260e71ff318adf50e1afe4bb3 and fixed in 0.186 by https://sourceware.org/git/?p=elfutils.git;a=commit;h=828024afc517e266f3226b469ba33f372b401821. Can you check what version of elfutils you're using? I believe on Fedora, you can do that with rpm -qv elfutils-libs.

It's probably worth reporting this to Fedora and Debian so that they can patch the bug. Would you mind opening bug reports? I can do it if you're not able to.

In the mean time, the binary wheel in pip includes the elfutils libraries with the patch applied, so you can sudo pip3 install drgn to get something that works.

[rincebrain]

Bullseye is 0.183, Fedora 34 is 0.185, so yes, that would fit, thanks!

I can go open bugs as needed.

Curiously, doing pip3 install drgn on F34 resulted in something differently broken:

$ drgn -c core.ztest.442075.fedora34.1637728682
Traceback (most recent call last):
  File "/usr/local/bin/drgn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib64/python3.9/site-packages/drgn/internal/cli.py", line 119, in main
    prog.load_debug_info(args.symbols, **args.default_symbols)
Exception: libpcre2-8.so.0: .debug_abbrev+0x1017: unknown attribute form 0x1f20

[osandov]

That looks like a legitimate error. drgn doesn't know about attribute form 0x1f20 (DW_FORM_GNU_ref_alt). I'll look into supporting it; hopefully it's not too hard.

[osandov]

@rincebrain I pushed workarounds for both of these issues. Could you please try building from git if you get the chance? Thanks!

[rincebrain]

On bullseye, because I had it handier: $ drgn -c core drgn 0.0.14+82.ge6abfea (using Python 3.9.2, elfutils 0.183, without libkdumpfile) For help, type help(drgn).

>> import drgn >> from drgn import NULL, Object, cast, container_of, execscript, offsetof, reinterpret, sizeof >>

Seems like progress, thanks! I don't know enough to know how to poke it more rigorously though at the moment. May try when I get more time later. - Rich

…

On Wed, Dec 8, 2021 at 3:21 PM Omar Sandoval ***@***.***> wrote: @rincebrain I pushed workarounds for both of these issues. Could you please try building from git if you get the chance? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.