From aabc1b50e1821c36f2844976929fed2868e2df49 Mon Sep 17 00:00:00 2001 From: Christos Kalkanis Date: Thu, 5 Mar 2026 07:24:40 -0500 Subject: [PATCH 1/8] Add profiles concepts page This is a high-level introduction to OpenTelemetry profiles, setting the context and linking to the OTel eBPF profiling agent and the specification. --- content/en/docs/concepts/signals/profiles.md | 133 +++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 content/en/docs/concepts/signals/profiles.md diff --git a/content/en/docs/concepts/signals/profiles.md b/content/en/docs/concepts/signals/profiles.md new file mode 100644 index 000000000000..b1bda82f7c50 --- /dev/null +++ b/content/en/docs/concepts/signals/profiles.md @@ -0,0 +1,133 @@ +--- +title: Profiles +weight: 5 +description: A recording of resource usage at the code level. +--- + +**Status**: [Development](/docs/specs/otel/document-status/) + +> [!NOTE] +> +> The profiles signal is still experimental and under active development. +> Breaking changes may be introduced in future versions. + +A **profile** is a **collection** of samples and associated metadata that +shows where applications consume resources during execution. A sample records +values encountered in some program context (typically a stack trace), +optionally augmented with auxiliary information like the trace id +corresponding to a higher-level request. + +The moment of capturing a sample is known as a **sample event** and consists +not only of the observation data point, but also the time at which it was captured. + +For example, an On-CPU profile contains samples (aggregated stack traces) for code +that was running on the CPU when samples were captured, together with the timestamps +and number of times each stack trace was observed. + +## Profiles overview + +Profiles are emerging as the fourth essential signal of observability, alongside +logs, metrics, and traces. They offer unparalleled insights into system and +application behavior, often uncovering performance bottlenecks overlooked by +other signals. + +Profiles provide granular, time-based views of resource consumption and +code execution, encompassing: + +* **Application-level profiling**: Reveals how software functions consume CPU, +memory, and other resources, highlighting slow or inefficient code. + +* **System-level profiling**: Offers a holistic view of the infrastructure, +pinpointing issues in operating system calls, kernel operations, and I/O. + +This performance picture can lead to: + +* **Faster Root Cause Analysis**: Quickly identifies the exact cause of +performance degradation. +* **Proactive Optimization**: Identifies potential issues before user impact. +* **Improved Resource Utilization**: Optimizes infrastructure for cost savings +and efficiency. +* **Enhanced Developer Productivity**: Helps developers validate code performance +and prevent regressions. + +## How profiles complement other signals + +Each OpenTelemetry signal answers a different question: + +| Signal | Question | +| --- | --- | +| **Logs** | What discrete events occurred? (insights into system behavior) | +| **Metrics** | What is happening at the system level? (e.g. CPU usage is 90%) | +| **Traces** | How does a request travel through a distributed system? | +| **Profiles** | Which code is responsible for consuming resources? | + +OpenTelemetry profiles support bi-directional links with other signals. These +correlations work across two dimensions: + +* **Request context correlation**: Linking profiling data to a specific trace or + span so you can see what code was running during a particular request. + +* **Resource context correlation**: Linking profiling data to the same + [resource](/docs/concepts/resources/) that emitted the associated metrics, + logs or traces (e.g. the same service instance). + +Profiles become especially powerful when correlated with other signals: + +* **Logs to profiles**: From an out of memory log entry find the code paths + responsible for the memory pressure. +* **Metrics to profiles**: From a spike in CPU or memory usage jump directly to + the functions consuming those resources. +* **Traces to profiles**: From a slow span in a trace see the corresponding + profile to identify the code responsible for the latency. + +## Profile types + +Profiling can capture many different kinds of resource usage. Some common +profile types include: + +* **On-CPU**: Which functions are consuming processor time? +* **Off-CPU**: Where are threads blocked or waiting (e.g. locks, I/O) instead of running? +* **Heap (memory)**: Which functions have allocated memory that is still in use? +* **Allocations (memory)**: Which code paths are responsible for the most memory + allocations (regardless of whether that memory has been freed)? + +The OpenTelemetry profiles signal is flexible enough to accommodate all of these. +However, the specific profile types available depend on the language runtime and +profiler being used. + +## How profiling works + +There are multiple approaches to collecting profiles and we designed OpenTelemetry +profiles to support all of them: + +* **Sampling-based profiling**: A profiler periodically interrupts the program + (e.g. using timer-based interrupts) and records the current stack trace. This is + the most common approach for CPU profiling. On Linux, profilers can use eBPF + to capture stack traces from the kernel without modifying userspace applications + at all. This approach enables zero-instrumentation, whole-system profiling + (including code produced by compiled languages without runtime support) and is + designed for continuous, low-overhead production use. +* **Instrumentation-based profiling**: Runtime hooks or bytecode instrumentation + report events like memory allocations, lock acquisitions or garbage collections + along with their associated stack traces. + +Regardless of the collection method, the resulting data is serialized into +OpenTelemetry's common profile data model and exported via OTLP. + +## Collecting profiles + +OpenTelemetry provides an [eBPF-based profiling agent](https://github.com/open-telemetry/opentelemetry-ebpf-profiler) +for Linux, capable of profiling most languages with zero code changes. + +Additional language-specific profiling integrations that tap into built-in runtime +profiling frameworks (e.g. JFR for Java, pprof for Go) will also become available as +the signal matures. + +Profiles can be exported via OTLP to the OpenTelemetry Collector or directly to any +compatible backend. + +## Specification + +To learn more about profiles in OpenTelemetry, see the +[profiles specification](/docs/specs/otel/profiles/). + From 62cfba52dd12f9e88213b995df166f1b2a1da08c Mon Sep 17 00:00:00 2001 From: Christos Kalkanis Date: Thu, 5 Mar 2026 07:40:36 -0500 Subject: [PATCH 2/8] Update link to Profiles Now points to Profiles concepts page. --- content/en/docs/concepts/signals/_index.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/content/en/docs/concepts/signals/_index.md b/content/en/docs/concepts/signals/_index.md index f4c5a469adf7..50eb1a8fe267 100644 --- a/content/en/docs/concepts/signals/_index.md +++ b/content/en/docs/concepts/signals/_index.md @@ -24,11 +24,9 @@ OpenTelemetry currently supports: Also under development or at the [proposal][] stage: - [Events][], a specific type of [log](logs) -- [Profiles][] are being worked on by the Profiling Working Group. +- [Profiles](profiles) [Events]: /docs/specs/otel/logs/data-model/#events -[Profiles]: - https://github.com/open-telemetry/opentelemetry-specification/blob/main/oteps/profiles/0212-profiling-vision.md [proposal]: https://github.com/open-telemetry/opentelemetry-specification/tree/main/oteps/#readme [signals]: /docs/specs/otel/glossary/#signals From 2be092bcc04c299c6fb2ccad4a67617328a3eb0c Mon Sep 17 00:00:00 2001 From: Christos Kalkanis Date: Thu, 5 Mar 2026 08:05:40 -0500 Subject: [PATCH 3/8] Apply test-and-fix --- content/en/docs/concepts/signals/profiles.md | 121 ++++++++++--------- 1 file changed, 61 insertions(+), 60 deletions(-) diff --git a/content/en/docs/concepts/signals/profiles.md b/content/en/docs/concepts/signals/profiles.md index b1bda82f7c50..f64a8e4b6d13 100644 --- a/content/en/docs/concepts/signals/profiles.md +++ b/content/en/docs/concepts/signals/profiles.md @@ -11,18 +11,18 @@ description: A recording of resource usage at the code level. > The profiles signal is still experimental and under active development. > Breaking changes may be introduced in future versions. -A **profile** is a **collection** of samples and associated metadata that -shows where applications consume resources during execution. A sample records -values encountered in some program context (typically a stack trace), -optionally augmented with auxiliary information like the trace id -corresponding to a higher-level request. +A **profile** is a **collection** of samples and associated metadata that shows +where applications consume resources during execution. A sample records values +encountered in some program context (typically a stack trace), optionally +augmented with auxiliary information like the trace ID corresponding to a +higher-level request. -The moment of capturing a sample is known as a **sample event** and consists -not only of the observation data point, but also the time at which it was captured. +The moment of capturing a sample is known as a **sample event** and consists not +only of the observation data point, but also the time at which it was captured. -For example, an On-CPU profile contains samples (aggregated stack traces) for code -that was running on the CPU when samples were captured, together with the timestamps -and number of times each stack trace was observed. +For example, an On-CPU profile contains samples (aggregated stack traces) for +code that was running on the CPU when samples were captured, together with the +timestamps and number of times each stack trace was observed. ## Profiles overview @@ -31,53 +31,53 @@ logs, metrics, and traces. They offer unparalleled insights into system and application behavior, often uncovering performance bottlenecks overlooked by other signals. -Profiles provide granular, time-based views of resource consumption and -code execution, encompassing: +Profiles provide granular, time-based views of resource consumption and code +execution, encompassing: -* **Application-level profiling**: Reveals how software functions consume CPU, -memory, and other resources, highlighting slow or inefficient code. +- **Application-level profiling**: Reveals how software functions consume CPU, + memory, and other resources, highlighting slow or inefficient code. -* **System-level profiling**: Offers a holistic view of the infrastructure, -pinpointing issues in operating system calls, kernel operations, and I/O. +- **System-level profiling**: Offers a holistic view of the infrastructure, + pinpointing issues in operating system calls, kernel operations, and I/O. This performance picture can lead to: -* **Faster Root Cause Analysis**: Quickly identifies the exact cause of -performance degradation. -* **Proactive Optimization**: Identifies potential issues before user impact. -* **Improved Resource Utilization**: Optimizes infrastructure for cost savings -and efficiency. -* **Enhanced Developer Productivity**: Helps developers validate code performance -and prevent regressions. +- **Faster Root Cause Analysis**: Quickly identifies the exact cause of + performance degradation. +- **Proactive Optimization**: Identifies potential issues before user impact. +- **Improved Resource Utilization**: Optimizes infrastructure for cost savings + and efficiency. +- **Enhanced Developer Productivity**: Helps developers validate code + performance and prevent regressions. ## How profiles complement other signals Each OpenTelemetry signal answers a different question: -| Signal | Question | -| --- | --- | -| **Logs** | What discrete events occurred? (insights into system behavior) | -| **Metrics** | What is happening at the system level? (e.g. CPU usage is 90%) | -| **Traces** | How does a request travel through a distributed system? | -| **Profiles** | Which code is responsible for consuming resources? | +| Signal | Question | +| ------------ | -------------------------------------------------------------- | +| **Logs** | What discrete events occurred? (insights into system behavior) | +| **Metrics** | What is happening at the system level? (e.g. CPU usage is 90%) | +| **Traces** | How does a request travel through a distributed system? | +| **Profiles** | Which code is responsible for consuming resources? | OpenTelemetry profiles support bi-directional links with other signals. These correlations work across two dimensions: -* **Request context correlation**: Linking profiling data to a specific trace or +- **Request context correlation**: Linking profiling data to a specific trace or span so you can see what code was running during a particular request. -* **Resource context correlation**: Linking profiling data to the same +- **Resource context correlation**: Linking profiling data to the same [resource](/docs/concepts/resources/) that emitted the associated metrics, logs or traces (e.g. the same service instance). Profiles become especially powerful when correlated with other signals: -* **Logs to profiles**: From an out of memory log entry find the code paths +- **Logs to profiles**: From an out of memory log entry find the code paths responsible for the memory pressure. -* **Metrics to profiles**: From a spike in CPU or memory usage jump directly to +- **Metrics to profiles**: From a spike in CPU or memory usage jump directly to the functions consuming those resources. -* **Traces to profiles**: From a slow span in a trace see the corresponding +- **Traces to profiles**: From a slow span in a trace see the corresponding profile to identify the code responsible for the latency. ## Profile types @@ -85,49 +85,50 @@ Profiles become especially powerful when correlated with other signals: Profiling can capture many different kinds of resource usage. Some common profile types include: -* **On-CPU**: Which functions are consuming processor time? -* **Off-CPU**: Where are threads blocked or waiting (e.g. locks, I/O) instead of running? -* **Heap (memory)**: Which functions have allocated memory that is still in use? -* **Allocations (memory)**: Which code paths are responsible for the most memory +- **On-CPU**: Which functions are consuming processor time? +- **Off-CPU**: Where are threads blocked or waiting (e.g. locks, I/O) instead of + running? +- **Heap (memory)**: Which functions have allocated memory that is still in use? +- **Allocations (memory)**: Which code paths are responsible for the most memory allocations (regardless of whether that memory has been freed)? -The OpenTelemetry profiles signal is flexible enough to accommodate all of these. -However, the specific profile types available depend on the language runtime and -profiler being used. +The OpenTelemetry profiles signal is flexible enough to accommodate all of +these. However, the specific profile types available depend on the language +runtime and profiler being used. ## How profiling works -There are multiple approaches to collecting profiles and we designed OpenTelemetry -profiles to support all of them: +There are multiple approaches to collecting profiles and we designed +OpenTelemetry profiles to support all of them: -* **Sampling-based profiling**: A profiler periodically interrupts the program - (e.g. using timer-based interrupts) and records the current stack trace. This is - the most common approach for CPU profiling. On Linux, profilers can use eBPF - to capture stack traces from the kernel without modifying userspace applications - at all. This approach enables zero-instrumentation, whole-system profiling - (including code produced by compiled languages without runtime support) and is - designed for continuous, low-overhead production use. -* **Instrumentation-based profiling**: Runtime hooks or bytecode instrumentation - report events like memory allocations, lock acquisitions or garbage collections - along with their associated stack traces. +- **Sampling-based profiling**: A profiler periodically interrupts the program + (e.g. using timer-based interrupts) and records the current stack trace. This + is the most common approach for CPU profiling. On Linux, profilers can use + eBPF to capture stack traces from the kernel without modifying userspace + applications at all. This approach enables zero-instrumentation, whole-system + profiling (including code produced by compiled languages without runtime + support) and is designed for continuous, low-overhead production use. +- **Instrumentation-based profiling**: Runtime hooks or bytecode instrumentation + report events like memory allocations, lock acquisitions or garbage + collections along with their associated stack traces. Regardless of the collection method, the resulting data is serialized into OpenTelemetry's common profile data model and exported via OTLP. ## Collecting profiles -OpenTelemetry provides an [eBPF-based profiling agent](https://github.com/open-telemetry/opentelemetry-ebpf-profiler) +OpenTelemetry provides an +[eBPF-based profiling agent](https://github.com/open-telemetry/opentelemetry-ebpf-profiler) for Linux, capable of profiling most languages with zero code changes. -Additional language-specific profiling integrations that tap into built-in runtime -profiling frameworks (e.g. JFR for Java, pprof for Go) will also become available as -the signal matures. +Additional language-specific profiling integrations that tap into built-in +runtime profiling frameworks (e.g. JFR for Java, pprof for Go) will also become +available as the signal matures. -Profiles can be exported via OTLP to the OpenTelemetry Collector or directly to any -compatible backend. +Profiles can be exported via OTLP to the OpenTelemetry Collector or directly to +any compatible backend. ## Specification To learn more about profiles in OpenTelemetry, see the [profiles specification](/docs/specs/otel/profiles/). - From a1118fae23021bb325bc82a5a1a13b5a43da6120 Mon Sep 17 00:00:00 2001 From: Christos Kalkanis Date: Fri, 6 Mar 2026 15:32:41 +0100 Subject: [PATCH 4/8] Update content/en/docs/concepts/signals/profiles.md Review feedback Co-authored-by: Fabrizio Ferri-Benedetti --- content/en/docs/concepts/signals/profiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/signals/profiles.md b/content/en/docs/concepts/signals/profiles.md index f64a8e4b6d13..d02274838a5d 100644 --- a/content/en/docs/concepts/signals/profiles.md +++ b/content/en/docs/concepts/signals/profiles.md @@ -122,7 +122,7 @@ OpenTelemetry provides an for Linux, capable of profiling most languages with zero code changes. Additional language-specific profiling integrations that tap into built-in -runtime profiling frameworks (e.g. JFR for Java, pprof for Go) will also become +runtime profiling frameworks, such as JFR for Java or pprof for Go, will also become available as the signal matures. Profiles can be exported via OTLP to the OpenTelemetry Collector or directly to From 20ccfce4e02e121a5043cea704ba6ad4fabe4660 Mon Sep 17 00:00:00 2001 From: Christos Kalkanis Date: Fri, 6 Mar 2026 15:34:57 +0100 Subject: [PATCH 5/8] Update content/en/docs/concepts/signals/profiles.md Review feedback Co-authored-by: Fabrizio Ferri-Benedetti --- content/en/docs/concepts/signals/profiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/signals/profiles.md b/content/en/docs/concepts/signals/profiles.md index d02274838a5d..5a6e55a18ac5 100644 --- a/content/en/docs/concepts/signals/profiles.md +++ b/content/en/docs/concepts/signals/profiles.md @@ -102,7 +102,7 @@ There are multiple approaches to collecting profiles and we designed OpenTelemetry profiles to support all of them: - **Sampling-based profiling**: A profiler periodically interrupts the program - (e.g. using timer-based interrupts) and records the current stack trace. This + for example using timer-based interrupts, and records the current stack trace. This is the most common approach for CPU profiling. On Linux, profilers can use eBPF to capture stack traces from the kernel without modifying userspace applications at all. This approach enables zero-instrumentation, whole-system From 6c21a79d5403a37a442dee248f9245e7abe80dcc Mon Sep 17 00:00:00 2001 From: Christos Kalkanis Date: Fri, 6 Mar 2026 15:35:34 +0100 Subject: [PATCH 6/8] Update content/en/docs/concepts/signals/profiles.md Review feedback Co-authored-by: Fabrizio Ferri-Benedetti --- content/en/docs/concepts/signals/profiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/signals/profiles.md b/content/en/docs/concepts/signals/profiles.md index 5a6e55a18ac5..874ac09b7943 100644 --- a/content/en/docs/concepts/signals/profiles.md +++ b/content/en/docs/concepts/signals/profiles.md @@ -69,7 +69,7 @@ correlations work across two dimensions: - **Resource context correlation**: Linking profiling data to the same [resource](/docs/concepts/resources/) that emitted the associated metrics, - logs or traces (e.g. the same service instance). + logs or traces, such as the same service instance. Profiles become especially powerful when correlated with other signals: From fa5a813b0df81d84d574599475daf1b6689ed5bf Mon Sep 17 00:00:00 2001 From: Christos Kalkanis Date: Fri, 6 Mar 2026 09:43:24 -0500 Subject: [PATCH 7/8] Review feedback --- content/en/docs/concepts/signals/profiles.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/concepts/signals/profiles.md b/content/en/docs/concepts/signals/profiles.md index 874ac09b7943..f44bb1bbeee8 100644 --- a/content/en/docs/concepts/signals/profiles.md +++ b/content/en/docs/concepts/signals/profiles.md @@ -119,13 +119,13 @@ OpenTelemetry's common profile data model and exported via OTLP. OpenTelemetry provides an [eBPF-based profiling agent](https://github.com/open-telemetry/opentelemetry-ebpf-profiler) -for Linux, capable of profiling most languages with zero code changes. +for Linux, capable of profiling most languages without any code changes. Additional language-specific profiling integrations that tap into built-in runtime profiling frameworks, such as JFR for Java or pprof for Go, will also become available as the signal matures. -Profiles can be exported via OTLP to the OpenTelemetry Collector or directly to +You can export Profiles through OTLP to the OpenTelemetry Collector or directly to any compatible backend. ## Specification From 98f324bac639bd0c13312be04b024b08aab02d7a Mon Sep 17 00:00:00 2001 From: otelbot <197425009+otelbot@users.noreply.github.com> Date: Tue, 10 Mar 2026 07:52:06 +0000 Subject: [PATCH 8/8] Results from /fix directive --- content/en/docs/concepts/signals/profiles.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/content/en/docs/concepts/signals/profiles.md b/content/en/docs/concepts/signals/profiles.md index f44bb1bbeee8..3bcf3abec03e 100644 --- a/content/en/docs/concepts/signals/profiles.md +++ b/content/en/docs/concepts/signals/profiles.md @@ -102,9 +102,9 @@ There are multiple approaches to collecting profiles and we designed OpenTelemetry profiles to support all of them: - **Sampling-based profiling**: A profiler periodically interrupts the program - for example using timer-based interrupts, and records the current stack trace. This - is the most common approach for CPU profiling. On Linux, profilers can use - eBPF to capture stack traces from the kernel without modifying userspace + for example using timer-based interrupts, and records the current stack trace. + This is the most common approach for CPU profiling. On Linux, profilers can + use eBPF to capture stack traces from the kernel without modifying userspace applications at all. This approach enables zero-instrumentation, whole-system profiling (including code produced by compiled languages without runtime support) and is designed for continuous, low-overhead production use. @@ -122,11 +122,11 @@ OpenTelemetry provides an for Linux, capable of profiling most languages without any code changes. Additional language-specific profiling integrations that tap into built-in -runtime profiling frameworks, such as JFR for Java or pprof for Go, will also become -available as the signal matures. +runtime profiling frameworks, such as JFR for Java or pprof for Go, will also +become available as the signal matures. -You can export Profiles through OTLP to the OpenTelemetry Collector or directly to -any compatible backend. +You can export Profiles through OTLP to the OpenTelemetry Collector or directly +to any compatible backend. ## Specification