v0.10

Major changes

This is a big release with some new major features added (though we still stay within minor version update, as there might be still some minor breaking changes). Most notable changes:

• Function arguments capture (-A argument). Retsnoop now can capture all input arguments for all traced functions and print them in human-readable form. See README for more details.
• Injected probes (-J). In addition to traced functions specified with -e and -a flags, it's now possible to also specify a single-point injected probes (kprobes, kretprobes, tracepoints, and raw tracepoints). Note that for kprobe, it's possible to specify extra offset (e.g., -J kprobe:bprm_execve+12), which allows to trace inlined functions and internals of functions (normally retsnoop only traces function entry and exit). See README for more details.
• Retsnoop can also capture extra context for injected probes, just use -A and -J together. For kprobes and kretprobes registers state is captured, for tracepoints and raw tracepoints their actual arguments are captured. See README for more details.
• (Breaking change!) It's now possible to enable only function call trace mode (-T) separately from default call stack mode. The latter now is controlled with -E flag. The important distinction and a breaking change is that with function call trace mode --success-stacks/-S option is implied, which makes most sense for function call tracing. When retsnoop -E is specified, even with -T, the original behavior of tracing and emitting only erroring call stacks (i.e., those that end up returning error from entry functions specified with -e arguments). So, in short:
  • retsnoop -T emits all function call trace, both successful and erroring;
  • retsnoop -E (or just retsnoop, as -E is the default mode) emits only erroring call stacks (no function call traces);
  • retsnoop -E -S will emit call stacks only (no function call traces), but both erroring and successful ones;
  • retsnoop -E -T will emit both call stacks and function call traces, but only erroring ones;
  • retsnoop -E -T -S will do both call stacks and function call trace for both successful and erroring cases.
• Added kernel module BTF support, improving tracing functions defined in kernel modules.
• Added advanced configuration options, specified with -C flag. See retsnoop --config-help for list of supported options and more details.
• Significant rework of --help output.
• Many smaller bug fixes and usability improvements.

What's Changed

• Fix typos in README by @Antiz96 in #57
• Makefile: use LDFLAGS for linking by @martinetd in #58
• Makefile: Update variables for package builds with external artifacts by @martinetd in #56
• Makefile: Do not rebuild the sidecar if not the default path by @martinetd in #59
• Support module BTF and lots of log improvements by @anakryiko in #60
• Lbr improvements by @anakryiko in #61
• Retsnoop session revamp by @anakryiko in #62
• Retsnoop function args capture support by @anakryiko in #63
• Retsnoop config and function args capture polish by @anakryiko in #64
• Bump gimli stack by @michel-slm in #66
• Retsnoop vararg support in printf-like functions by @anakryiko in #68
• retsnoop: make func call trace and call stack modes independent by @anakryiko in #69
• Retsnoop injection probes and other improvements by @anakryiko in #70
• retsnoop: handle idle threads properly by @anakryiko in #71
• Retsnoop improvements for LBR, stitched stacks, and interim stacks by @anakryiko in #72
• Retsnoop ARM64 improvements by @anakryiko in #75

New Contributors

• @Antiz96 made their first contribution in #57

Full Changelog: v0.9.8...v0.10

v0.9.8

What's Changed

• A few small fixes and clean ups by @anakryiko in #53
• bpftool: Pass additional compile flags as EXTRA_CFLAGS, not CFLAGS by @qmonnet in #52
• sidecar: Update addr2line dependency to 0.21 by @danielocfb in #55

New Contributors

• @danielocfb made their first contribution in #55

Full Changelog: v0.9.7...v0.9.8

retsnoop v0.9.7

What's Changed

• Add release workflow for shipping binaries and combined sources by @qmonnet in #42 and #44
• retsnoop: Fix sign extension failure logic by @anakryiko in #50
• retsnoop: Remove a limit of 4096 attachable functions by @anakryiko in #48
• retsnoop: Remove hard-coded maximum of 256 CPUs supported by @anakryiko in #49
• retsnoop: Use bpf_probe_read_kernel() instead of bpf_probe_read() by @iii-i in #46
• retsnoop: Pass the real envp to the sidecar by @erthalion in #47

New Contributors

• @qmonnet made their first contribution in #42
• @iii-i made their first contribution in #46
• @erthalion made their first contribution in #47

Full Changelog: v0.9.6...v0.9.7

retsnoop v0.9.6

What's Changed

• Fix calibration unreliability on some new kernels by @anakryiko in #41

Full Changelog: v0.9.5...v0.9.6

retsnoop v0.9.5

What's Changed

Massive improvements in how retsnoop determines whether kprobes are attachable:

• add --debug multi-kprobe mode to bisect failing multi-kprobe attachment; it quickly narrows down and logs which kprobes were attempted but failed to be attached;
• skip attaching to kernel functions that have non-unique name and some of instances are not traceable;
• resolve internal mix up of function and data ksyms;
• internal fixes to consistently take into account kernel module to which ksym/kprobe belongs to.

Overall, these fixes and improvements make retsnoop's mass-attach behavior more reliable.

Full Changelog: v0.9.4...v0.9.5

retsnoop v0.9.4

Bug fixes

• fix IP (instruction pointer) fetching on non-x86_64 architectures on older kernels;
• handle io_uring source code files better, after Linux code reorganization;
• automatically pick debugfs (/sys/kernel/debug/tracing) or tracefs (/sys/kernel/tracing), whichever is available;
• handle very old kernels that don't support BPF global data more gracefully.

retsnoop v0.9.3

What's Changed

retsnoop now supports DWARF-based symbolization (i.e.,
source code file/line info and inline functions) on
KASLR-enabled Linux kernels.

retsnoop v0.9.2

What's Changed

• -F (fentry/fexit) mode now supports tracing void-returning functions;
• few fixes for -F (fentry/fexit) mode interacting weirdly with source code globs;
• retsnoop now can be compiled across x86_64, i686, aarch64, ppc64le, s390x, riscv64 architectures by using per-architecture pre-generated minimal vmlinux.h (see gen-vmlinux-headers.sh script);
• retsnoop now builds bootstrap (lightweight) version of bpftool from submodule, which allows it to be compilable on multiple-architectures. Previously retsnoop's Makefile relied on checked in pre-built x86_64 bpftool binary;
• massive revamp of README.md;
• usage text fixes and improvements.

Full Changelog: v0.9.1...v0.9.2

retsnoop v0.9.1

Few nice improvements with no major new features:

• dropped the requirement for /proc/config.gz presence for multi-kprobe detection (just using BPF CO-RE now for detection);
• use dynamically allocated internal formatting buffers for stacks and traces, thus allowing much larger traces without dropping any information (at the expense of more memory usage, of course);
• force-flush stdout before (potentially very long) detachment to improve retsnoop usage in scripts;
• emit detected features when printing version and --verbose flag is specified:

$ sudo ./retsnoop -Vv
retsnoop v0.9.1
Feature detection:
        BPF ringbuf map supported: yes
        bpf_get_func_ip() supported: yes
        bpf_get_branch_snapshot() supported: yes
        BPF cookie supported: yes
        multi-attach kprobe supported: yes
Feature calibration:
        kretprobe IP offset: 4
        fexit sleep fix: yes
        fentry re-entry protection: yes

All just nice quality of life improvements. Enjoy!

retsnoop v0.9

--trace (-T) function calls trace mode

Add function call trace output, in addition to default stack trace and LBR output.

Example:

$ sudo ./retsnoop -e '*sys_bpf' -v -n simfail -a ':kernel/bpf/syscall.c' -a ':kernel/bpf/verifier.c' -T
...
Receiving data...
15:50:12.413878 -> 15:50:12.414193 TID/PID 1755152/1755152 (simfail/simfail):

FUNCTION CALLS TRACE                    RESULT                 DURATION
-------------------------------------   --------------------  ---------
→ __x64_sys_bpf
    → __sys_bpf
        ↔ bpf_check_uarg_tail_zero      [0]                     0.341us
        → bpf_raw_tracepoint_open
            ↔ __bpf_prog_get            [0xffffc9000c93d000]    0.255us
            → bpf_tracing_prog_attach
                ↔ bpf_link_prime        [0]                     2.530us
                ↔ bpf_link_cleanup      [void]                  3.435us
            ← bpf_tracing_prog_attach   [-ENOTSUPP]           306.161us
        ← bpf_raw_tracepoint_open       [-ENOTSUPP]           310.147us
    ← __sys_bpf                         [-ENOTSUPP]           314.846us
← __x64_sys_bpf                         [-ENOTSUPP]           315.515us

                      entry_SYSCALL_64_after_hwframe+0x44  (arch/x86/entry/entry_64.S:112:0)
                      do_syscall_64+0x2d                   (arch/x86/entry/common.c:46:12)
   315us [-ENOTSUPP]  __x64_sys_bpf+0x1c                   (kernel/bpf/syscall.c:4749:1)
   314us [-ENOTSUPP]  __sys_bpf+0x867                      (kernel/bpf/syscall.c:4689:9)
   310us [-ENOTSUPP]  bpf_raw_tracepoint_open+0x9a         (kernel/bpf/syscall.c:3063:6)
!  306us [-ENOTSUPP]  bpf_tracing_prog_attach

As you can see from the above, function calls trace mode allows to peer into exact control flow inside the kernel, but filter to according to allow/deny lists, taking into account all the filters (process name, latency, etc). In addition to call sequence, function results and duration is emitted.

Note that leaf function calls (e.g., bpf_link_prime above) are collapsed, if they don't call any other functions. This makes call trace more readable and compact. This is marked with ↔ marker, while otherwise function entry is marked with →, and function exit is marked with ←.

This mode is perfectly augments stack trace output for deeper kernel internals inspection, but also is great for discovering how kernel internals work, in general.