Skip to content

add support for uprobes#651

Merged
fabled merged 8 commits intomainfrom
draft-dynamic-uprobe
Aug 21, 2025
Merged

add support for uprobes#651
fabled merged 8 commits intomainfrom
draft-dynamic-uprobe

Conversation

@florianl
Copy link
Copy Markdown
Member

@florianl florianl commented Jul 23, 2025 •
edited
Loading

Add support to dynamically attach to symbols of executables.

E.g. to inspect memory allocations in Go one could attach a uprobe to runtime.mallocgc:

sudo ./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls -off-cpu-threshold 0.1 -uprobe /path/to/my/go/executable:runtime.mallocgc

Another interesting uprobe in Go executables is runtime.gopark, that puts a Go routine into a waiting state.

To identify potential uprobes, one could use something like this:

bpftrace -l 'uprobe:/path/to/my/executable:*'

Visualize uprobes with https://github.com/elastic/devfiler

fabled
fabled reviewed Jul 23, 2025
View reviewed changes
Comment thread support/ebpf/tracemgmt.h Outdated
@florianl florianl mentioned this pull request Jul 25, 2025
Closed
@florianl

This comment was marked as outdated.

Sign in to view
@florianl florianl force-pushed the draft-dynamic-uprobe branch from 877b141 to 89940ca Compare July 25, 2025 14:39
florianl added a commit that referenced this pull request Jul 30, 2025
@florianl
tools: add strobelight-ctrl
0bac425 
strobelight-ctrl is a supplementary tool demoing the use of -load-probe from #651.

This tool is inspired by the blog post [1] around always-on profiling and enabling users to dynamically attach probes to various points.

[1]: https://engineering.fb.com/2025/01/21/production-engineering/strobelight-a-profiling-service-built-on-open-source-technology/

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
florianl added a commit that referenced this pull request Jul 30, 2025
@florianl
tools: add strobelight-ctrl
fbce312 
strobelight-ctrl is a supplementary tool demoing the use of -load-probe from #651.

This tool is inspired by the blog post [1] around always-on profiling and enabling users to dynamically attach probes to various points.

[1]: https://engineering.fb.com/2025/01/21/production-engineering/strobelight-a-profiling-service-built-on-open-source-technology/

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
@florianl florianl mentioned this pull request Jul 30, 2025
Merged
@florianl florianl changed the title [poc] add support for uprobes add support for uprobes Jul 30, 2025
@florianl florianl marked this pull request as ready for review July 30, 2025 13:01
@florianl florianl requested review from a team as code owners July 30, 2025 13:01
florianl added a commit that referenced this pull request Jul 30, 2025
@florianl
tools: add strobelight-ctrl
afddb38 
strobelight-ctrl is a supplementary tool demoing the use of -load-probe from #651.

This tool is inspired by the blog post [1] around always-on profiling and enabling users to dynamically attach probes to various points.

[1]: https://engineering.fb.com/2025/01/21/production-engineering/strobelight-a-profiling-service-built-on-open-source-technology/

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
florianl added 3 commits August 5, 2025 09:47
@florianl
[poc] add support for uprobes
28f3a95 
Add support to dynamically attach to symbols of executables.

E.g. to inspect memory allocations in Go one could attach a uprobe to runtime.mallocgc:

```
sudo ./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls -off-cpu-threshold 0.1 -uprobe /path/to/my/go/executable:runtime.mallocgc
```

Another interesting uproble in Go executables is runtime.gopark, that puts a Go routine into a waiting state.

While ebpf-profiler allows attaching only to a single uprobe, using this project as OTel collector allows to attach to multiple uprobes at ones.

Visualize uprobes with elastic/devfiler#38

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
@florianl
fixup: reuse kprobes as uprobes
1663bfb 
Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
@florianl
fixup: allow to load generic eBPF program without attaching it
1b1ea8d 
Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
@florianl florianl force-pushed the draft-dynamic-uprobe branch from b15ce93 to 1b1ea8d Compare August 5, 2025 07:48
fabled
fabled reviewed Aug 13, 2025
View reviewed changes
Comment thread tracer/tracer.go Outdated
fabled
fabled reviewed Aug 13, 2025
View reviewed changes
Comment thread cli_flags.go Outdated
@florianl
fixup: use fs.Func()
6a6a726 
Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
@florianl
fixup: reuse kprobes
9e3841e 
Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
fabled
fabled approved these changes Aug 14, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@fabled fabled left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One nit about not needing to copy/prepend the helpers but load them first in a separate block. Pre-approving though.

Comment thread tracer/tracer.go Outdated
florianl and others added 2 commits August 14, 2025 16:56
@florianl
fixup: preload tailcall destinations
ecd510e 
Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
@florianl
Merge branch 'main' into draft-dynamic-uprobe
8e1b710
@fabled fabled merged commit fa0cef5 into main Aug 21, 2025
28 checks passed
@fabled fabled deleted the draft-dynamic-uprobe branch August 21, 2025 07:31
This was referenced Sep 22, 2025
Open
Closed
simonswine added a commit to simonswine/opentelemetry-ebpf-profiler that referenced this pull request Sep 23, 2025
@simonswine
feat: Propate the uprobe link into sample type
aeb3d34 
With open-telemetry#651 it is possible to attach uprobes, but they currently are
merged together. This PR introduces the use of cookies (available from kernel 5.15+) to keep
track of the links that lead to the events being generated.

It now uses the symbol part of the uprobe link and propages it into the sample type. That is something that we currently don't do well enough.

So let's look at this example:

```
--uprobe-link /nix/store/r7pnxs3cfl3qxwacj38iakpm5v1ch6lz-glibc-2.40-66/lib/libc.so.6:malloc --uprobe-link /nix/store/r7pnxs3cfl3qxwacj38iakpm5v1ch6lz-glibc-2.40-66/lib/libc.so.6:open --uprobe-link /proc/4501/root/usr/lib/aarch64-linux-gnu/libc.so.6:open --uprobe-link /proc/4501/root/usr/lib/aarch64-linux-gnu/libc.so.6:malloc
```

There would be resulting two sample types

- `uprobe_malloc_events:count`
- `uprobe_open_events:count`
simonswine added a commit to simonswine/opentelemetry-ebpf-profiler that referenced this pull request Sep 23, 2025
@simonswine
feat: Propate the uprobe link into sample type
49c080c 
With open-telemetry#651 it is possible to attach uprobes, but they currently are
merged together. This PR introduces the use of cookies (available from kernel 5.15+) to keep
track of the links that lead to the events being generated.

It now uses the symbol part of the uprobe link and propages it into the sample type. That is something that we currently don't do well enough.

So let's look at this example:

```
--uprobe-link /nix/store/r7pnxs3cfl3qxwacj38iakpm5v1ch6lz-glibc-2.40-66/lib/libc.so.6:malloc --uprobe-link /nix/store/r7pnxs3cfl3qxwacj38iakpm5v1ch6lz-glibc-2.40-66/lib/libc.so.6:open --uprobe-link /proc/4501/root/usr/lib/aarch64-linux-gnu/libc.so.6:open --uprobe-link /proc/4501/root/usr/lib/aarch64-linux-gnu/libc.so.6:malloc
```

There would be resulting two sample types

- `uprobe_malloc_events:count`
- `uprobe_open_events:count`
gnurizen pushed a commit to parca-dev/opentelemetry-ebpf-profiler that referenced this pull request Sep 30, 2025
@florianl @gnurizen
add support for uprobes (open-telemetry#651)
cf15ffb 
Signed-off-by: Florian Lehner <florian.lehner@elastic.co>
@gnurizen gnurizen mentioned this pull request Sep 30, 2025
Merged
@luweglarz luweglarz mentioned this pull request Oct 23, 2025
Closed
This was referenced Nov 10, 2025
Merged
Merged
@gnurizen gnurizen mentioned this pull request Mar 9, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fabled fabled fabled approved these changes

@christos68k christos68k christos68k approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@florianl @fabled @christos68k