feat: generic probe support by manuelfelipe · Pull Request #953 · open-telemetry/opentelemetry-ebpf-profiler

manuelfelipe · 2025-11-10T21:23:01Z

Summary

Refactors probe infrastructure introduced in #651 to support kprobes, kretprobes, uprobes, and uretprobes through a unified interface with syntax inspired by bpftrace. Previously only uprobes were supported.

Motivation

We wanted to understand what causes shared memory pages to turn into private memory pages (copy-on-write) in our Ruby workers. Pitchfork workers start with everything shared post-fork but quickly grow private memory as requests come in.

Being able to attach kprobes to kernel functions like do_wp_page allows us to capture stack traces when CoW events occur, showing exactly which Ruby code is causing pages to be copied. This data could help us identify memory hotspots and optimization opportunities.

Previous uprobe-only support limited profiling to user space functions. Adding kprobe support enables profiling kernel-level events (page faults, syscalls, etc.) alongside regular CPU profiling, giving a complete picture of system behavior.

What Changed

Renamed uprobe-specific code to generic probe terminology:

--uprobe-link → --probe-link flag
UProbeLinks → ProbeLinks config
TraceOriginUProbe → TraceOriginProbe
support/ebpf/uprobe.ebpf.c → generic_probe.ebpf.c

Added kprobe support:

New kprobe__generic eBPF entry point
Unified probe attachment logic in tracer/probe.go
Support for kprobe/kretprobe/uprobe/uretprobe types

Enhanced probe-ctrl tool:

Works with all probe types
Added --list flag to show loaded eBPF programs
Better error messages

Usage


# Attach to kernel function
./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls -probe-link "kprobe:do_wp_page`

# Attach to user space function  
./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls -probe-link "uprobe:/root/ebpf-profiler:runtime.(*mheap).alloc"

Running ^ locally, with traces going to devfiler:

florianl · 2025-11-11T10:41:21Z

+	ProgName string
+}
+
+func ParseProbe(spec string) (*ProbeSpec, error) {


Should we cover USDTs here as well?

Yeah. good idea. can we do that in a follow up PR?

from a quick look, i failed trying to find any support for this in cilium/ebpf. looks like we would need to parse SDT notes, and then, in certain cases, based on https://www.sourceware.org/systemtap/wiki/UserSpaceProbeImplementation, deal with semaphores ourselves?

worth noting that i also looked into adding support for generic tracepoints but was not trivial either because we need struct pt_regs *.

my guess is that we could maybe do something like what happens here, but then that would only get us support for tracepoints on syscalls

follow up sounds good to me.

florianl · 2025-11-13T07:35:53Z

+}
+
+// uprobe__generic serves as entry point for uprobe based profiling.
+SEC("uprobe/generic")


Following feedback from #651 (comment), we should be able to skip uprobe/generic and just use kprobe/generic and attach it acccordingly.

oh yeah. good point. change it in e7dd520 to make use of kprobe/generic for all types

florianl · 2025-11-13T07:37:36Z

+}
+
+func ParseProbe(spec string) (*ProbeSpec, error) {
+	parts := strings.SplitN(spec, ":", 3)


As we handle here user input, should we handle the case where capitalized letters are input?

not sure how much further we want to get into parsing the user input.

In last rev i added https://github.com/Shopify/opentelemetry-ebpf-profiler/blob/2e9f35d274a89f8e96c1f79d07b331244e496eca/tracer/probe.go#L75 to make sure the probe type is always lower case before any comparison.

As for the other parts of the input, i feel that is a concern of the link pkg. with an invalid probe name, we get something like the error msg below which feels like the right behaviour to me:

{"level":"error","msg":"failed to attach probes: creating perf_kprobe PMU (arch-specific fallback for \"do_WP_page\"): token __x64_do_WP_page: not found: no such file or directory","time":"2025-11-14T14:07:17.902603356Z"}

no strong opinions here. strings.ToLower() is always a good start and for the rest pkg link will return an error if it does not fit.

christos68k · 2025-11-14T14:14:18Z

+
+	default:
+		return nil, fmt.Errorf("unsupported probe type: %s", probeTypeStr)
+	}


This can never happen (also maybe some linters can warn if we don't have a default here but do not exhaustively check all values for this type)

Suggested change

default:

return nil, fmt.Errorf("unsupported probe type: %s", probeTypeStr)

}

}

without this golang complains about the missing return. I could just move it out of the switch. just not sure it makes much of a difference

tracer/probe.go:115:1: missing return

christos68k · 2025-12-04T14:53:40Z

@manuelfelipe Can you resolve conflicts?

Rename uprobe-specific code to generic probe terminology and add support for kprobes alongside uprobes. Update probe-ctrl tool to handle both probe types.

* Add docs to functions and fields from probe struct

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>

manuelfelipe · 2025-12-04T15:22:52Z

done @christos68k. rebased against latest main

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>

manuelfelipe requested review from a team as code owners November 10, 2025 21:23

florianl reviewed Nov 11, 2025

View reviewed changes

florianl reviewed Nov 13, 2025

View reviewed changes

florianl approved these changes Nov 14, 2025

View reviewed changes

Comment thread tools/probe-ctrl/cmd/probe-ctrl/main.go Outdated

christos68k reviewed Nov 14, 2025

View reviewed changes

manuelfelipe and others added 7 commits December 4, 2025 10:14

feat: generic probe support

107ed38

Rename uprobe-specific code to generic probe terminology and add support for kprobes alongside uprobes. Update probe-ctrl tool to handle both probe types.

remove list functionality from probe-ctrl

2aa76fa

use generic kprobe instead of individual one per type

51816d5

* Refactor probe type handling to use enumerated types

79af01f

* Add docs to functions and fields from probe struct

fix up test

19aef5a

clean up un-used var from probe-ctrl

cb45003

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>

probe-ctrl: apply suggestions from code review

b0807bd

manuelfelipe force-pushed the generic-probe-support branch from ba86928 to b0807bd Compare December 4, 2025 15:15

christos68k approved these changes Dec 4, 2025

View reviewed changes

christos68k merged commit 9d4a7fd into open-telemetry:main Dec 4, 2025
29 checks passed

dalehamel pushed a commit to Shopify/opentelemetry-ebpf-profiler that referenced this pull request Dec 5, 2025

feat: generic probe support (open-telemetry#953)

a6904af

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>

bobrik mentioned this pull request Feb 3, 2026

The first firing of a probe is missed #1150

Closed

gnurizen mentioned this pull request Mar 9, 2026

Optimize distro QEMU tests to be more efficient parca-dev/opentelemetry-ebpf-profiler#228

Closed

Conversation

manuelfelipe commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

What Changed

Usage

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christos68k commented Dec 4, 2025

Uh oh!

manuelfelipe commented Dec 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

manuelfelipe commented Nov 10, 2025 •

edited

Loading