pyroscope.ebpf: otel profiler implementation by korniltsev · Pull Request #2920 · grafana/alloy

korniltsev · 2025-03-06T12:15:03Z

PR Description

This PR changes the implementation of pyroscope.ebpf compoenent from grafana/pyroscope/ebpf to opentelemetry-ebpf-profiler - a fork of opentelemetry-ebpf-profiler.
The notable changes of the fork: native frame symbolization, dynamic profiling policy, target discovery labels.

The fork uses a rust library for symbol extraction. See https://github.com/open-telemetry/opentelemetry-ebpf-profiler/tree/main/rust-crates.

In this PR I put all pyroscope.ebpf code behind a pyroscope_ebpf go tag.

Some component arguments became no-op as there are no corresponding knobs in the new implementations. These args a deprecated

// deprecated
	PidCacheSize int `alloy:"pid_cache_size,attr,optional"`
	// deprecated
	BuildIDCacheSize int `alloy:"build_id_cache_size,attr,optional"`
	// deprecated
	SameFileCacheSize int `alloy:"same_file_cache_size,attr,optional"`
	// deprecated
	CacheRounds int `alloy:"cache_rounds,attr,optional"`
	// deprecated
	GoTableFallback bool `alloy:"go_table_fallback,attr,optional"`
	// deprecated
	SymbolsMapSize int `alloy:"symbols_map_size,attr,optional"`

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

CHANGELOG.md updated
Documentation added
Tests updated
Config converters updated

github-actions · 2025-03-10T05:21:04Z

💻 Deploy preview deleted.

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

…orniltsev/otel-profiler

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

marcsanmi

LGTM 👏 , just left some nits!

marcsanmi · 2025-06-26T09:54:42Z

+| `python_enabled`          | `bool`                   | A flag to enable or disable python profiling.                                                                        | `true`   | no       |
+| `ruby_enabled`            | `bool`                   | A flag to enable or disable Ruby profiling.                                                                          | `true`   | no       |
+| `same_file_cache_size`    | `int`                    | Deprecated (no-op), previously controlled the size of the elf file -> symbols table LRU cache.                       | `8`      | no       |
+| `sample_rate`             | `int`                    | How many times per second to collect profile samples.                                                                | `97`     | no       |


I think the new default is 19?

yep, updating

marcsanmi · 2025-06-26T10:18:30Z

+| `go_table_fallback`       | `bool`                   | Deprecated (no-op), previously enabled symbol lookup in `.sym` / `.dynsym` sections when `.gopclntab` lookup failed. | `false`  | no       |
+| `hotspot_enabled`         | `bool`                   | A flag to enable ordisable hotspot profiling.                                                                        | `true`   | no       |
+| `perl_enabled`            | `bool`                   | A flag to enable or disable Perl profiling.                                                                          | `true`   | no       |
+| `php_enabled`             | `bool`                   | A flag to enable or disable PHP profiling.                                                                           | `true`   | no       |


Should go_enabled be also documented?

The current revision of the go consumes shit ton of memory and cpu and it works much better without it (i.e. using our symbolizer). We may want to add go here after we update our fork to include open-telemetry/opentelemetry-ebpf-profiler#456 which is not straightforward because it removes the rust lib and we rely on it. tldr - yes, but later.

marcsanmi · 2025-06-26T10:21:35Z

+| `perl_enabled`            | `bool`                   | A flag to enable or disable Perl profiling.                                                                          | `true`   | no       |
+| `php_enabled`             | `bool`                   | A flag to enable or disable PHP profiling.                                                                           | `true`   | no       |
+| `pid_cache_size`          | `int`                    | Deprecated (no-op), previously controlled the size of the PID -> proc symbols table LRU cache.                       | `32`     | no       |
+| `pid_map_size`            | `int`                    | The size of eBPF PID map.                                                                                            | `2048`   | no       |


This is no longer used? Do you keep it for backwards compatibility? I think we should either update it to deprecated or just remove it.

Nice catch. deprecated. We may want to expose tracer.MapScaleFactor in the future.

marcsanmi · 2025-06-26T10:50:28Z

+			time.Sleep(c.cfg.ReporterInterval)
+			continue
 		}
+		defer ctlr.Shutdown()


Do we need to cleanup FileObserver to avoid resource leaking?

defer func() { ctlr.Shutdown() if c.cfg.FileObserver != nil { c.cfg.FileObserver.Cleanup() } }()

thanks, nice catch

marcsanmi

LGTM! 🚀

korniltsev force-pushed the korniltsev/otel-profiler branch from d9663da to 3779c4e Compare March 6, 2025 12:21

korniltsev commented Mar 6, 2025

View reviewed changes

Comment thread go.mod Outdated

korniltsev commented Mar 6, 2025

View reviewed changes

Comment thread go.mod

korniltsev commented Mar 6, 2025

View reviewed changes

Comment thread tools/build-image/Dockerfile Outdated

korniltsev force-pushed the korniltsev/otel-profiler branch from 3779c4e to 7c540d2 Compare March 6, 2025 12:32

korniltsev added 2 commits March 6, 2025 19:52

build-image: add rust toolchains

4846a04

pyroscope.ebpf: otel implementation

ca84882

korniltsev force-pushed the korniltsev/otel-profiler branch from 7c540d2 to ca84882 Compare March 6, 2025 12:53

korniltsev mentioned this pull request Mar 6, 2025

build-image: add rust toolchains #2921

Merged

4 tasks

go mod tidy

f30538b

korniltsev commented Mar 6, 2025

View reviewed changes

Comment thread .gitignore Outdated

korniltsev added 6 commits March 6, 2025 20:00

revert .gitignore changes

afb4561

revert .dockerignore changes

8d31d67

temp: install rust toolchain

0b5a7af

fix pprof reporter creation

de05df5

fix

d0b5167

docs

79ac6f1

korniltsev added 11 commits March 10, 2025 16:42

docs

f4704c4

use new build-image

17cfb0f

Merge remote-tracking branch 'origin/main' into korniltsev/otel-profiler

6f04247

revert go.mod

e17b5d6

moar tags

17d013e

test go 1.23 version

84e266a

Merge remote-tracking branch 'origin/main' into korniltsev/otel-profiler

1449494

nl

b205f65

update dependency

79b4df6

fix build

c2bc065

docs

38cf014

korniltsev marked this pull request as ready for review March 11, 2025 08:21

update ebpf-profiler dependency

c2a10ad

korniltsev mentioned this pull request Jun 18, 2025

chore: bump otel to 128 #3843

Merged

4 tasks

korniltsev added 6 commits June 24, 2025 10:34

Merge remote-tracking branch 'origin/main' into korniltsev/otel-profiler

625a50b

fix build

31eb2f8

cleanup

091cf26

cleanup

9dbe9fe

lint

c5cf7b8

revert gitignore

21cdd37

korniltsev marked this pull request as ready for review June 24, 2025 06:41

clayton-cornell reviewed Jun 24, 2025

View reviewed changes

Comment thread docs/sources/reference/components/pyroscope/pyroscope.ebpf.md Outdated

Update docs/sources/reference/components/pyroscope/pyroscope.ebpf.md

f92c399

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

korniltsev requested review from marcsanmi and simonswine June 25, 2025 08:18

korniltsev mentioned this pull request Jun 25, 2025

Service Discovery on k8s grafana/pyroscope#4270

Closed

korniltsev added 2 commits June 25, 2025 23:08

Merge remote-tracking branch 'origin/main' into korniltsev/otel-profiler

d6778fe

Merge remote-tracking branch 'origin/korniltsev/otel-profiler' into k…

b80a7b4

…orniltsev/otel-profiler

clayton-cornell reviewed Jun 25, 2025

View reviewed changes

Comment thread docs/sources/reference/components/pyroscope/pyroscope.ebpf.md Outdated

Update docs/sources/reference/components/pyroscope/pyroscope.ebpf.md

4343497

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

marcsanmi reviewed Jun 26, 2025

View reviewed changes

korniltsev added 3 commits June 26, 2025 18:50

review fixes

cd10b20

deprecate more

9892ce7

FileObserver cleanup

d3163b9

korniltsev requested a review from marcsanmi June 26, 2025 12:00

marcsanmi approved these changes Jun 26, 2025

View reviewed changes

korniltsev mentioned this pull request Jun 27, 2025

chore(modulestore): add public coredump modulestore API open-telemetry/opentelemetry-ebpf-profiler#543

Closed

filenames

b4d2a96

korniltsev merged commit bd763f8 into main Jul 14, 2025
45 checks passed

korniltsev deleted the korniltsev/otel-profiler branch July 14, 2025 03:49

github-actions Bot added the frozen-due-to-age label Aug 14, 2025

github-actions Bot locked as resolved and limited conversation to collaborators Aug 14, 2025

Conversation

korniltsev commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marcsanmi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcsanmi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

korniltsev commented Mar 6, 2025 •

edited

Loading

github-actions Bot commented Mar 10, 2025 •

edited

Loading