Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 3 additions & 17 deletions bazel/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,23 +187,9 @@ Envoy can produce backtraces on demand and from assertions and other fatal
actions like segfaults. Where supported, stack traces will contain resolved
symbols, though not include line numbers. On systems where absl::Symbolization is
not supported, the stack traces written in the log or to stderr contain addresses rather
than resolved symbols. The `tools/stack_decode.py` script exists to process the output
and do symbol resolution including line numbers, to make the stack traces useful.
Any log lines not relevant to the backtrace capability
are passed through the script unchanged (it acts like a filter).

The script runs in one of two modes. If passed no arguments it anticipates
Envoy (or test) output on stdin. You can postprocess a log or pipe the output of
an Envoy process. If passed some arguments it runs the arguments as a child
process. This enables you to run a test with backtrace post processing. Bazel
sandboxing must be disabled by specifying standalone execution. Example
command line:

```
bazel test -c dbg //test/server:backtrace_test
--run_under=`pwd`/tools/stack_decode.py --strategy=TestRunner=standalone
--cache_test_results=no --test_output=all
```
than resolved symbols. If the symbols were resolved, the address is also included at
the end of the line. It can be used with a debugging tool like `addr2line` to
get a precise line number if the required debugging info is available.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a suggested bazel --run_under=.... snippet describing how we can get line numbers?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addr2line isn't a pass through by itself, although I implemented such a functionality into stack_decode.py. If you think it'll actually be useful I can modify stack_decode.py to work with the new format backtraces rather than eliminate it. Then you'll be able to use bazel --run_under= like before.

Alternatively, I can just write instructions for manually invoking addr2line with the right args and the addresses of interest (it just won't be a --run_under=).

I have a mild preference for the latter (and let stack_decode.py die), WDYT?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like run_under is easier to use unless addr2line is destructive to lines that are not part of stack traces. What's the reason you prefer to ask users to do this manually?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, let's revive stack_decode.py. Sorry about the futz here, but I love my line numbers! :D

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries! I didn't think anyone was using it! Happy to fix it up to get the best of both worlds.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, revert this doc change? The previous flow still works, yeah? So the additional detail is not strictly necessary.

@dnoe dnoe Jan 30, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flow is slightly different. Previously, stack_decode.py autodetected the object file name from the log, because backward had this information and we could log it. But we don't have it anymore and the executable file needs to be passed in all circumstances so the usage details of stack_decode.py changed slightly. I'm going to update the doc now that mac CI is confirmed to be passing.


You will need to use either a `dbg` build type or the `opt` build type to get symbol
information in the binaries.
Expand Down
15 changes: 4 additions & 11 deletions bazel/repositories.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,6 @@ def envoy_dependencies(path = "@envoy_deps//", skip_targets = []):
# semi-standard in the Bazel community, intended to avoid both duplicate
# dependencies and name conflicts.
_com_google_absl()
_com_github_bombela_backward()
_com_github_circonus_labs_libcircllhist()
_com_github_cyan4973_xxhash()
_com_github_eile_tclap()
Expand Down Expand Up @@ -321,16 +320,6 @@ def _boringssl_fips():
build_file = "@envoy//bazel/external:boringssl_fips.BUILD",
)

def _com_github_bombela_backward():
_repository_impl(
name = "com_github_bombela_backward",
build_file = "@envoy//bazel/external:backward.BUILD",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed that https://github.com/envoyproxy/envoy/blob/master/bazel/external/backward.BUILD is still in the codebase. This can be removed now I believe @dnoe

)
native.bind(
name = "backward",
actual = "@com_github_bombela_backward//:backward",
)

def _com_github_circonus_labs_libcircllhist():
_repository_impl(
name = "com_github_circonus_labs_libcircllhist",
Expand Down Expand Up @@ -529,6 +518,10 @@ def _com_google_absl():
name = "abseil_symbolize",
actual = "@com_google_absl//absl/debugging:symbolize",
)
native.bind(
name = "abseil_stacktrace",
actual = "@com_google_absl//absl/debugging:stacktrace",
)

# Require abseil_time as an indirect dependency as it is needed by the
# direct dependency jwt_verify_lib.
Expand Down
5 changes: 0 additions & 5 deletions bazel/repository_locations.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,6 @@ REPOSITORY_LOCATIONS = dict(
strip_prefix = "thrift-0.11.0",
urls = ["https://files.pythonhosted.org/packages/c6/b4/510617906f8e0c5660e7d96fbc5585113f83ad547a3989b80297ac72a74c/thrift-0.11.0.tar.gz"],
),
com_github_bombela_backward = dict(
sha256 = "ad73be31c5cfcbffbde7d34dba18158a42043a109e7f41946f0b0abd589ed55e",
strip_prefix = "backward-cpp-1.4",
urls = ["https://github.com/bombela/backward-cpp/archive/v1.4.tar.gz"],
),
com_github_circonus_labs_libcircllhist = dict(
sha256 = "9949e2864b8ad00ee5c3e9c1c3c01e51b6b68bb442a919652fc66b9776477987",
strip_prefix = "libcircllhist-fd8a14463739d247b414825cc56ca3946792a3b9",
Expand Down
27 changes: 3 additions & 24 deletions source/exe/signal_action.cc
Original file line number Diff line number Diff line change
Expand Up @@ -9,32 +9,11 @@ namespace Envoy {
constexpr int SignalAction::FATAL_SIGS[];

void SignalAction::sigHandler(int sig, siginfo_t* info, void* context) {
void* error_pc = 0;

const ucontext_t* ucontext = reinterpret_cast<const ucontext_t*>(context);
if (ucontext != nullptr) {
#ifdef REG_RIP
// x86_64
error_pc = reinterpret_cast<void*>(ucontext->uc_mcontext.gregs[REG_RIP]);
#elif defined(__APPLE__) && defined(__x86_64__)
error_pc = reinterpret_cast<void*>(ucontext->uc_mcontext->__ss.__rip);
#elif defined(__powerpc__)
error_pc = reinterpret_cast<void*>(ucontext->uc_mcontext.regs->nip);
#elif defined(__aarch64__)
error_pc = reinterpret_cast<void*>(ucontext->uc_mcontext.pc);
#elif defined(__arm__)
error_pc = reinterpret_cast<void*>(ucontext->uc_mcontext.arm_pc);
#else
#warning "Please enable and test PC retrieval code for your arch in signal_action.cc"
// x86 Classic: reinterpret_cast<void*>(ucontext->uc_mcontext.gregs[REG_EIP]);
// ARM: reinterpret_cast<void*>(ucontext->uc_mcontext.arm_pc);
#endif
}

BackwardsTrace tracer;

tracer.logFault(strsignal(sig), info->si_addr);
if (error_pc != 0) {
tracer.captureFrom(error_pc);
if (context != nullptr) {
tracer.captureFrom(context);
} else {
tracer.capture();
}
Expand Down
2 changes: 1 addition & 1 deletion source/server/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ envoy_cc_library(
name = "backtrace_lib",
hdrs = ["backtrace.h"],
external_deps = [
"backward",
"abseil_stacktrace",
"abseil_symbolize",
],
tags = ["backtrace"],
Expand Down
97 changes: 27 additions & 70 deletions source/server/backtrace.h
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
#pragma once

#include <backward.hpp>

#include "common/common/logger.h"

#include "absl/debugging/stacktrace.h"
#include "absl/debugging/symbolize.h"

namespace Envoy {
Expand All @@ -15,7 +14,7 @@ namespace Envoy {
} while (0)

/**
* Use the Backward library ( https://github.com/bombela/backward-cpp ) to log
* Use absl::Stacktrace and absl::Symbolize to log resolved symbols
* stack traces on demand. To use this just do:
*
* BackwardsTrace tracer;
Expand All @@ -29,20 +28,8 @@ namespace Envoy {
* For convenience a macro is provided BACKTRACE_LOG() which performs the
* construction, capture, and log in one shot.
*
* To resolve the addresses in the backtrace output and de-interleave
* multithreaded output use the tools/stack_decode.py command and pass the
* log/stderr output to stdin of the tool. Backtrace lines will be resolved,
* other lines will be passed through and echo'd unchanged.
*
* The stack_decode.py tool can also run envoy or a test as a child process if
* you pass the command and arguments as arguments to the tool. This enables
* you to run tests containing backtrace commands added for debugging and see
* the output like this:
*
* bazel test -c dbg //test/server:backtrace_test
* --run_under=`pwd`/tools/stack_decode.py
* --strategy=TestRunner=standalone --cache_test_results=no
* --test_output=all
* If the symbols cannot be resolved by absl::Symbolize then the raw address
* will be printed instead.
*/
class BackwardsTrace : Logger::Loggable<Logger::Id::backtrace> {
public:
Expand All @@ -53,80 +40,50 @@ class BackwardsTrace : Logger::Loggable<Logger::Id::backtrace> {
*
* The trace will begin with the call to capture().
*/
void capture() { stack_trace_.load_here(MAX_STACK_DEPTH); }
void capture() {
// Skip of one means we exclude the last call, which must be to capture().
stack_depth_ = absl::GetStackTrace(stack_trace_, MaxStackDepth, /* skip_count = */ 1);
}

/**
* Capture a stack trace from a particular address.
* Capture a stack trace from a particular context.
*
* This can be used to capture a useful stack trace from a fatal signal
* handler.
* handler. The context argument should be a pointer to the context passed
* to a signal handler registered via a sigaction struct.
*
* @param address The stack trace will begin from this address.
* @param context A pointer to ucontext_t obtained from a sigaction handler.
*/
void captureFrom(void* address) { stack_trace_.load_from(address, MAX_STACK_DEPTH); }
void captureFrom(const void* context) {
stack_depth_ =
absl::GetStackTraceWithContext(stack_trace_, MaxStackDepth, /* skip_count = */ 1, context,
/* min_dropped_frames = */ nullptr);
}

/**
* Log the stack trace.
*/
void logTrace() {
backward::TraceResolver resolver;
resolver.load_stacktrace(stack_trace_);
// If there's nothing in the captured trace we cannot do anything.
// The size must be at least two for useful info - there is a sentinel frame
// at the end that we ignore.
if (stack_trace_.size() < 2) {
ENVOY_LOG(critical, "Back trace attempt failed");
return;
}

const auto thread_id = stack_trace_.thread_id();
backward::ResolvedTrace first_frame_trace = resolver.resolve(stack_trace_[0]);
auto obj_name = first_frame_trace.object_filename;

#ifdef __APPLE__
// The stack_decode.py script uses addr2line which isn't readily available and doesn't seem to
// work when installed.
ENVOY_LOG(critical, "Backtrace thr<{}> obj<{}>:", thread_id, obj_name);
#else
char out[200];
ENVOY_LOG(critical,
"Backtrace thr<{}> obj<{}> (If unsymbolized, use tools/stack_decode.py):", thread_id,
obj_name);
#endif

// Backtrace gets tagged by ASAN when we try the object name resolution for the last
// frame on stack, so skip the last one. It has no useful info anyway.

for (unsigned int i = 0; i < stack_trace_.size() - 1; ++i) {
backward::ResolvedTrace trace = resolver.resolve(stack_trace_[i]);
if (trace.object_filename != obj_name) {
obj_name = trace.object_filename;
ENVOY_LOG(critical, "thr<{}> obj<{}>", thread_id, obj_name);
}
ENVOY_LOG(critical, "Backtrace:");

#ifdef __APPLE__
// In the absence of stack_decode.py, print the function name.
ENVOY_LOG(critical, "thr<{}> #{} {} {}", thread_id, stack_trace_[i].idx, stack_trace_[i].addr,
trace.object_function);
#else
if (absl::Symbolize(stack_trace_[i].addr, out, sizeof(out))) {
ENVOY_LOG(critical, "thr<{}> #{} {} {}", thread_id, stack_trace_[i].idx,
stack_trace_[i].addr, out);
for (int i = 0; i < stack_depth_; ++i) {
char out[1024];
const bool success = absl::Symbolize(stack_trace_[i], out, sizeof(out));
if (success) {
ENVOY_LOG(critical, "#{}: {} [{}]", i, out, stack_trace_[i]);
} else {
ENVOY_LOG(critical, "thr<{}> #{} {} (unknown)", thread_id, stack_trace_[i].idx,
stack_trace_[i].addr);
ENVOY_LOG(critical, "#{}: {}", i, stack_trace_[i]);
}
#endif
}
ENVOY_LOG(critical, "end backtrace thread {}", stack_trace_.thread_id());
}

void logFault(const char* signame, const void* addr) {
ENVOY_LOG(critical, "Caught {}, suspect faulting address {}", signame, addr);
}

private:
static const int MAX_STACK_DEPTH = 64;
backward::StackTrace stack_trace_;
static constexpr int MaxStackDepth = 64;
void* stack_trace_[MaxStackDepth];
int stack_depth_{0};
};
} // namespace Envoy
1 change: 0 additions & 1 deletion tools/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ envoy_package()

exports_files([
"gen_git_sha.sh",
"stack_decode.py",
"check_format.py",
"header_order.py",
"envoy_build_fixer.py",
Expand Down
107 changes: 0 additions & 107 deletions tools/stack_decode.py

This file was deleted.