pyroscope.ebpf: otel profiler implementation #2920
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,19 +15,25 @@ title: pyroscope.ebpf | |
| `pyroscope.ebpf` configures an eBPF profiling job for the current host. | ||
| The collected performance profiles are forwarded to the list of receivers passed in `forward_to`. | ||
|
|
||
| The `pyroscope.ebpf` component embeds the [`grafana/opentelemetry-ebpf-profiler`][] which is a fork of [`open-telemetry/opentelemetry-ebpf-profiler`][]. | ||
|
|
||
| [`grafana/opentelemetry-ebpf-profiler`]: https://github.com/grafana/opentelemetry-ebpf-profiler | ||
| [`open-telemetry/opentelemetry-ebpf-profiler`]: https://github.com/open-telemetry/opentelemetry-ebpf-profiler | ||
|
|
||
| {{< admonition type="note" >}} | ||
| To use the `pyroscope.ebpf` component you must run {{< param "PRODUCT_NAME" >}} as root and inside the host PID namespace. | ||
| {{< /admonition >}} | ||
|
|
||
|
Contributor
There was a problem hiding this comment. Does it make sense to add a note pointing to the underlying profiler being used?
Contributor
There was a problem hiding this comment. We've done that in other component docs - example: https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.windows/ |
||
| {{< admonition type="note" >}} | ||
| The profiler requires file system storage at `/tmp/symb-cache` to store symbol cache data. Ensure this directory is accessible and has sufficient storage space. | ||
| {{< /admonition >}} | ||
|
|
||
| You can specify multiple `pyroscope.ebpf` components by giving them different labels, however it's not recommended as it can lead to additional memory and CPU usage. | ||
|
|
||
| ## Supported languages | ||
|
|
||
| - Native code (C/C++, Rust, Zig, Go, etc. without debug symbols on host) | ||
| - Broad set of HLLs (Hotspot JVM, Python, Ruby, PHP, Node.JS, V8, Perl). | ||
|
|
||
| ## Usage | ||
|
|
||
|
|
@@ -44,27 +50,36 @@ The component configures and starts a new eBPF profiling job to collect performa | |
|
|
||
| You can use the following arguments with `pyroscope.ebpf`: | ||
|
|
||
| | Name | Type | Description | Default | Required | | ||
| |---------------------------|--------------------------|----------------------------------------------------------------------------------------------------------------------|----------|----------| | ||
| | `forward_to` | `list(ProfilesReceiver)` | List of receivers to send collected profiles to. | | yes | | ||
| | `targets` | `list(map(string))` | List of targets to group profiles by container ID. | | yes | | ||
| | `build_id_cache_size` | `int` | Deprecated (no-op), previously controlled the size of the elf file build id -> symbols table LRU cache. | `64` | no | | ||
| | `cache_rounds` | `int` | Deprecated (no-op), previously controlled the number of cache rounds. | | no | | ||
| | `collect_interval` | `duration` | How frequently to collect profiles. | `"15s"` | no | | ||
| | `collect_kernel_profile` | `bool` | Deprecated (no-op), previously enabled collection of kernelspace profiles. | `true` | no | | ||
| | `collect_user_profile` | `bool` | Deprecated (no-op), previously enabled collection of userspace profiles. | `true` | no | | ||
| | `container_id_cache_size` | `int` | The size of the PID -> container ID table LRU cache. | `1024` | no | | ||
| | `demangle` | `string` | C++ demangle mode. Available options are: `none`, `simplified`, `templates`, or `full`. | `"none"` | no | | ||
| | `dotnet_enabled` | `bool` | A flag to enable or disable .NET profiling. | `true` | no | | ||
| | `go_table_fallback` | `bool` | Deprecated (no-op), previously enabled symbol lookup in `.sym` / `.dynsym` sections when `.gopclntab` lookup failed. | `false` | no | | ||
| | `hotspot_enabled` | `bool` | A flag to enable ordisable hotspot profiling. | `true` | no | | ||
| | `perl_enabled` | `bool` | A flag to enable or disable Perl profiling. | `true` | no | | ||
| | `php_enabled` | `bool` | A flag to enable or disable PHP profiling. | `true` | no | | ||
|
Contributor
There was a problem hiding this comment. Should
Contributor
Author
There was a problem hiding this comment. The current revision of the go consumes shit ton of memory and cpu and it works much better without it (i.e. using our symbolizer). We may want to add go here after we update our fork to include open-telemetry/opentelemetry-ebpf-profiler#456 which is not straightforward because it removes the rust lib and we rely on it. tldr - yes, but later. |
||
| | `pid_cache_size` | `int` | Deprecated (no-op), previously controlled the size of the PID -> proc symbols table LRU cache. | `32` | no | | ||
| | `pid_map_size` | `int` | Deprecated (no-op), previously controlled the size of eBPF PID map. | `2048` | no | | ||
| | `python_enabled` | `bool` | A flag to enable or disable python profiling. | `true` | no | | ||
| | `ruby_enabled` | `bool` | A flag to enable or disable Ruby profiling. | `true` | no | | ||
| | `same_file_cache_size` | `int` | Deprecated (no-op), previously controlled the size of the elf file -> symbols table LRU cache. | `8` | no | | ||
| | `sample_rate` | `int` | How many times per second to collect profile samples. | `19` | no | | ||
| | `symbols_map_size` | `int` | Deprecated (no-op), previously controlled the size of eBPF symbols map . | `16384` | no | | ||
| | `v8_enabled` | `bool` | A flag to enable/disable V8 profiling. | `true` | no | | ||
|
|
||
|
Contributor
There was a problem hiding this comment. It would be good to also pass through
Contributor
Author
There was a problem hiding this comment. On the one hand I like this idea, but on the other I want to introduce as little arguments as possible. It is currently possible to enable this flag through env variable. I'd prefer to keep it an undocumented debugging-facility feature. WDYT? |
||
| Only the `forward_to` and `targets` fields are required. | ||
| Omitted fields take their default values. | ||
|
|
||
| Several arguments are marked as "Deprecated (no-op)". These arguments were previously used for configuring various cache sizes and behaviors, but they no longer have any effect. They are kept for backward compatibility but will be removed in a future release. It is recommended to remove these arguments from your configuration. | ||
|
|
||
| ## Blocks | ||
|
|
||
| The `pyroscope.ebpf` component doesn't support any blocks. You can configure this component with arguments. | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Deploying that using the helm chart, I am getting this error:
This fixes it:
I think as this change might be quite disruptive, we maybe also should investigate, if we do that when the folder is not mounted:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can try mounting it ourselves. We're superuser afterall. I remember I did this in a branch for convenience.
https://github.com/grafana/pyroscope/pull/3184/files#diff-ed660caba4be49d0b3854003a14e854719b063ad0141ec736c8e1119a3c7a9a7R30
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added tracefs automount 5bf3ab9#diff-8899fb2a871c6c203d47853b2398f1818fd5fa5817aabd462fd37a4d0554c495R281-R301
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello, I currently run alloy using pyroscope.ebpf with only few capabilities (BFP,PERF,PTRACE), and I believe that the mount syscall at least requires SYS_ADMIN, would there be any workaround or it is mandatory to mount this fs ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can mount the fs yourself and the component will not issue any mount syscalls