problem: cannot get stack trace #1965

somdoron · 2016-05-06T14:36:12Z

I'm trying to debug libzmq and cannot stack trace on assert, I'm trying the following:

void* callstack[128];
int i, frames = backtrace(callstack, 128);
char** strs = backtrace_symbols(callstack, frames);
for (i = 0; i < frames; ++i) {
  printf("%s\n", strs[i]);
}
free(strs);

But I don't get the methods names, for example:

/home/somdoron/git/libzmq/src/.libs/libzmq.so.5(+0x189c0) [0x7ff8c9ad29c0]
/home/somdoron/git/libzmq/src/.libs/libzmq.so.5(+0x11bc0) [0x7ff8c9acbbc0]
/home/somdoron/git/libzmq/src/.libs/libzmq.so.5(+0x183b2) [0x7ff8c9ad23b2]
/home/somdoron/git/libzmq/tests/.libs/lt-test_inproc_connect() [0x403231]
/home/somdoron/git/libzmq/tests/.libs/lt-test_inproc_connect() [0x401076]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff8c94f2a40]
/home/somdoron/git/libzmq/tests/.libs/lt-test_inproc_connect() [0x4010b9]

I try to configure with --enable-debug, but output is the same.

I'm actually using the libzmq tests (make check), so how do I compile the tests with symbols of libzmq?
Platform is Ubuntu and using autotools.

The text was updated successfully, but these errors were encountered:

somdoron · 2016-05-06T14:38:27Z

Also making an assert to print the full stack trace will also help

bluca · 2016-05-06T14:44:21Z

-g should be enabled by default. Try appending to ./configure: CFLAGS="-g" CXXFLAGS="-g"

bluca · 2016-05-06T14:45:07Z

Also if you run make V=1, is the -g flag passed by default?

somdoron · 2016-05-06T14:52:57Z

Try appending to ./configure: CFLAGS="-g" CXXFLAGS="-g"

still same output

Attached are the configure output and make output

I run the following commands

./configure --enable-debug CFLAGS="-g" CXXFLAGS="-g" > 1.log
make V=1

make.txt
configure.txt

bluca · 2016-05-06T14:53:46Z

Yeah just checked here, default ./configure and -g is passed when compiling and when linking both the test and the library.

somdoron · 2016-05-06T14:58:00Z

So what I'm I missing?

I'm trying to check what is causing the mutex to be recursive, so I changed the type to PTHREAD_MUTEX_ERRORCHECK. Now I'm running the tests, and tests failed as expected, however assert only give the line number and file and I need the stack trace, so I'm using this:

        inline void lock ()
        {
            int rc = pthread_mutex_lock (&mutex);

            if (rc != 0)
            {
                void* callstack[128];
                int i, frames = backtrace(callstack, 128);
                char** strs = backtrace_symbols(callstack, frames);
                for (i = 0; i < frames; ++i) {
                    printf("%s\n", strs[i]);
                }
                free(strs);
            }
            posix_assert (rc);
        }

Which doesn't print the method names.

I suspect the test binaries are missing the symbols files or debug argument.

[1] https://github.com/zeromq/libzmq/blob/master/src/mutex.hpp#L104

bluca · 2016-05-06T15:11:20Z

Yeah I can reproduce your problem:

$ LD_LIBRARY_PATH=src/.libs tests/.libs/test_thread_safe 
src/.libs/libzmq.so.5(+0x1881d) [0x7f8ae0d2481d]
src/.libs/libzmq.so.5(+0x27d58) [0x7f8ae0d33d58]
src/.libs/libzmq.so.5(+0x2e8ed) [0x7f8ae0d3a8ed]
src/.libs/libzmq.so.5(+0x431e2) [0x7f8ae0d4f1e2]
src/.libs/libzmq.so.5(+0x463d5) [0x7f8ae0d523d5]
tests/.libs/test_thread_safe() [0x400e46]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f8adfd3bb45]
tests/.libs/test_thread_safe() [0x401000]
Resource deadlock avoided (src/mutex.hpp:134)
Aborted (core dumped)

Curiously glibc symbols are there. I wonder if the backtrace functionality in glibc needs the symbols installed in /usr/lib/dbg, rather than in the binary itself?

I don't think it's a problem of our build system, since running the same test in GDB and using the backtrace command it works just fine:

$ LD_LIBRARY_PATH=src/.libs gdb tests/.libs/test_thread_safe 
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from tests/.libs/test_thread_safe...done.
(gdb) r
Starting program: /home/lboccass/git/libzmq/tests/.libs/test_thread_safe 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff6b6a700 (LWP 12756)]
[New Thread 0x7ffff6369700 (LWP 12757)]
src/.libs/libzmq.so.5(+0x1881d) [0x7ffff7b7581d]
src/.libs/libzmq.so.5(+0x27d58) [0x7ffff7b84d58]
src/.libs/libzmq.so.5(+0x2e8ed) [0x7ffff7b8b8ed]
src/.libs/libzmq.so.5(+0x431e2) [0x7ffff7ba01e2]
src/.libs/libzmq.so.5(+0x463d5) [0x7ffff7ba33d5]
/home/lboccass/git/libzmq/tests/.libs/test_thread_safe() [0x400e46]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7ffff6b8cb45]
/home/lboccass/git/libzmq/tests/.libs/test_thread_safe() [0x401000]
Resource deadlock avoided (src/mutex.hpp:134)

Program received signal SIGABRT, Aborted.
0x00007ffff6ba0067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff6ba0067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff6ba1448 in __GI_abort () at abort.c:89
#2  0x00007ffff7b7fc59 in zmq::zmq_abort (
    errmsg_=errmsg_@entry=0x7ffff6ccebbf "Resource deadlock avoided") at src/err.cpp:84
#3  0x00007ffff7b7589c in zmq::mutex_t::lock (this=<optimized out>) at src/mutex.hpp:134
#4  0x00007ffff7b84d58 in zmq::mailbox_safe_t::send (this=0x606b30, cmd_=...) at src/mailbox_safe.cpp:81
#5  0x00007ffff7b8b8ed in send_command (cmd_=..., this=0x606620) at src/object.cpp:436
#6  zmq::object_t::send_own (this=this@entry=0x606620, destination_=destination_@entry=0x606620, 
    object_=object_@entry=0x607110) at src/object.cpp:222
#7  0x00007ffff7b8dc7b in zmq::own_t::launch_child (this=this@entry=0x606620, 
    object_=object_@entry=0x607110) at src/own.cpp:90
#8  0x00007ffff7ba01e2 in zmq::socket_base_t::add_endpoint (this=this@entry=0x606620, 
    addr_=0x606c08 "tcp://127.0.0.1:5560", endpoint_=endpoint_@entry=0x607110, pipe=pipe@entry=0x0)
    at src/socket_base.cpp:981
#9  0x00007ffff7ba33d5 in zmq::socket_base_t::bind (this=0x606620, addr_=<optimized out>)
    at src/socket_base.cpp:613
#10 0x0000000000400e46 in main () at tests/test_thread_safe.cpp:52

bluca · 2016-05-06T15:36:31Z

manpage says that you need rdynamic, and with optimization the symbol names might not be available.
Tried with:
./configure --without-docs CFLAGS="-rdynamic -g -O0" CXXFLAGS="-rdynamic -g -O0" LDFLAGS="-rdynamic -g -O0"
But still no luck. I'll keep digging.

somdoron · 2016-05-06T15:37:37Z

thanks, your stack trace actually gave me a direction. But it would be nice to see the stack trace

bluca · 2016-05-06T15:44:16Z

Now that I remember, some of our stuff uses libunwind and it works, let me try and dig up an example

bluca · 2016-05-06T16:09:48Z

Ta-dan :-)

$ LD_LIBRARY_PATH=src/.libs tests/.libs/test_thread_safe 0: src/.libs/libzmq.so.5 (_ZN3zmq14mailbox_safe_t4sendERKNS_9command_tE+0x23) [0x7f77ff6e8925]
1: src/.libs/libzmq.so.5 (_ZN3zmq5ctx_t12send_commandEjRKNS_9command_tE+0x59) [0x7f77ff6cbef7]
2: src/.libs/libzmq.so.5 (_ZN3zmq8object_t12send_commandERNS_9command_tE+0x37) [0x7f77ff6f363d]
3: src/.libs/libzmq.so.5 (_ZN3zmq8object_t8send_ownEPNS_5own_tES2_+0x62) [0x7f77ff6f2bc0]
4: src/.libs/libzmq.so.5 (_ZN3zmq5own_t12launch_childEPS0_+0x52) [0x7f77ff6f6f3c]
5: src/.libs/libzmq.so.5 (_ZN3zmq13socket_base_t12add_endpointEPKcPNS_5own_tEPNS_6pipe_tE+0x2c) [0x7f77ff710b78]
6: src/.libs/libzmq.so.5 (_ZN3zmq13socket_base_t4bindEPKc+0x546) [0x7f77ff70f2e0]
7: src/.libs/libzmq.so.5 (zmq_bind+0x67) [0x7f77ff732057]
8: tests/.libs/test_thread_safe (main+0x5c) [0x402377]
9: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf5) [0x7f77fe6e8b45]
10: tests/.libs/test_thread_safe (_start+0x29) [0x4012f9]
11: ? (+0x29) [0x29]
Resource deadlock avoided (src/mutex.hpp:167)
Aborted (core dumped)

somdoron · 2016-05-06T16:13:10Z

@bluca great!!! what do I need to change?
Also do we want to make a pull request for that?

bluca · 2016-05-06T16:17:12Z

This uses libunwind. I can clear up the snippet, and add a print_backtrace() helper function, maybe in zmq_utils.
It will have to be behind a flag since it needs libunwind, which is an external library, and if available I can have zmq_assert print it. Does that sound ok?

somdoron · 2016-05-06T16:18:08Z

sounds good, thanks!!!!

bluca · 2016-05-06T16:38:24Z

Will send as soon as I clean it up. Also managed to get demangled names:

0: src/.libs/libzmq.so.5 (zmq::mailbox_safe_t::send(zmq::command_t const&)+0x23) [0x7fcebb6f19b9]
1: src/.libs/libzmq.so.5 (zmq::ctx_t::send_command(unsigned int, zmq::command_t const&)+0x59) [0x7fcebb6d4f47]
2: src/.libs/libzmq.so.5 (zmq::object_t::send_command(zmq::command_t&)+0x37) [0x7fcebb6fc6d1]
3: src/.libs/libzmq.so.5 (zmq::object_t::send_own(zmq::own_t*, zmq::own_t*)+0x62) [0x7fcebb6fbc54]
4: src/.libs/libzmq.so.5 (zmq::own_t::launch_child(zmq::own_t*)+0x52) [0x7fcebb6fffd0]
5: src/.libs/libzmq.so.5 (zmq::socket_base_t::add_endpoint(char const*, zmq::own_t*, zmq::pipe_t*)+0x2c) [0x7fcebb719c0c]
6: src/.libs/libzmq.so.5 (zmq::socket_base_t::bind(char const*)+0x546) [0x7fcebb718374]
7: src/.libs/libzmq.so.5 (zmq_bind+0x67) [0x7fcebb73b0eb]
8: tests/.libs/test_thread_safe (main+0x5c) [0x402377]
9: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf5) [0x7fceba6f1b45]
10: tests/.libs/test_thread_safe (_start+0x29) [0x4012f9]
11: ? (+0x29) [0x29]
Resource deadlock avoided (src/mutex.hpp:173)
Aborted (core dumped)

hitstergtd added the Help Request label May 6, 2016

bluca mentioned this issue May 6, 2016

Problem: no backtrace is printed on assert #1966

Merged

somdoron closed this as completed in #1966 May 6, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

problem: cannot get stack trace #1965

problem: cannot get stack trace #1965

somdoron commented May 6, 2016 •

edited

Loading

somdoron commented May 6, 2016

bluca commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016 •

edited

Loading

bluca commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016

bluca commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016

bluca commented May 6, 2016

problem: cannot get stack trace #1965

problem: cannot get stack trace #1965

Comments

somdoron commented May 6, 2016 • edited Loading

somdoron commented May 6, 2016

bluca commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016 • edited Loading

bluca commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016

bluca commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016

bluca commented May 6, 2016

somdoron commented May 6, 2016 •

edited

Loading

somdoron commented May 6, 2016 •

edited

Loading