Skip to content
Merged
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
bcdae54
Opening new PR for file.
carolxob Mar 6, 2023
796e90b
Add otel metrics raw processor doc.
carolxob Mar 6, 2023
4c776cb
Minor title change.
carolxob Mar 8, 2023
db3b38a
Delete mutate-event.md
carolxob Mar 8, 2023
b17024d
Minor edits to format data as table.
carolxob Mar 13, 2023
82660fc
Minor edits.
carolxob Mar 13, 2023
ddf1626
Updates based on technical feedback.
carolxob Mar 21, 2023
f6ea41a
Minor updates to common metrics section.
carolxob Mar 21, 2023
f66f303
Incorporated technical feedback.
carolxob Mar 23, 2023
4a9c051
Minor updates.
carolxob Mar 24, 2023
3e106d1
Minor rephrasing.
carolxob Apr 14, 2023
b014f6c
Update otel-metrics-raw.md
carolxob Apr 14, 2023
93bfc5b
Updated phrasing.
carolxob Apr 18, 2023
59a12ec
Update _data-prepper/pipelines/configuration/processors/otel-metrics-…
carolxob Apr 19, 2023
52cc8bd
Update _data-prepper/pipelines/configuration/processors/otel-metrics-…
carolxob Apr 19, 2023
554e90c
Update _data-prepper/pipelines/configuration/processors/otel-metrics-…
carolxob Apr 19, 2023
e44bf92
Update _data-prepper/pipelines/configuration/processors/otel-metrics-…
carolxob Apr 19, 2023
1db2f84
Update _data-prepper/pipelines/configuration/processors/otel-metrics-…
carolxob Apr 19, 2023
5e85db6
Update _data-prepper/pipelines/configuration/processors/otel-metrics-…
carolxob Apr 19, 2023
ccd00b9
Update _data-prepper/pipelines/configuration/processors/otel-metrics-…
carolxob Apr 19, 2023
3d99539
Update _data-prepper/pipelines/configuration/processors/otel-metrics-…
carolxob Apr 19, 2023
3c0f3c8
Minor edits.
carolxob Apr 20, 2023
c64f366
Minor updates.
carolxob Apr 24, 2023
a74ca07
Minor updates.
carolxob Apr 24, 2023
4e14952
Minor updates.
carolxob Apr 24, 2023
ca27844
Minor updated
carolxob Apr 24, 2023
fdd33af
Minor updates.
carolxob Apr 24, 2023
e1243b0
Minor updates to metadata and first heading to ensure page displays c…
carolxob Apr 24, 2023
ae04275
Updated title.
carolxob Apr 24, 2023
6b29a7c
Renamed file for consistency with processor title.
carolxob Apr 24, 2023
c18bf05
Update _data-prepper/pipelines/configuration/processors/otel-metrics.md
carolxob Apr 25, 2023
41ffa8f
Update _data-prepper/pipelines/configuration/processors/otel-metrics.md
carolxob Apr 25, 2023
d923a8d
Update _data-prepper/pipelines/configuration/processors/otel-metrics.md
carolxob Apr 25, 2023
402af35
Update _data-prepper/pipelines/configuration/processors/otel-metrics.md
carolxob Apr 25, 2023
9f7dab0
Update _data-prepper/pipelines/configuration/processors/otel-metrics.md
carolxob Apr 25, 2023
acfcd8f
Minor update.
carolxob Apr 25, 2023
b710bd7
Minor changes.
carolxob Apr 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 150 additions & 0 deletions _data-prepper/pipelines/configuration/processors/otel-metrics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
---
layout: default
title: otel_metrics processor
parent: Processors
grand_parent: Pipelines
nav_order: 72
---

# otel_metrics processor

The `otel_metrics` processor serializes a collection of `ExportMetricsServiceRequest` records sent from the [OTel metrics source]({{site.url}}{{site.baseurl}}//data-prepper/pipelines/configuration/sources/otel-metrics-source/) into a collection of string records.

## Usage

To get started, add the following processor to your `pipeline.yaml` configuration file:

``` yaml
processor:
- otel_metrics_raw_processor:
```
{% include copy.html %}

## Configuration

You can use the following optional parameters to configure histogram buckets and their default values. A histogram displays numerical data by grouping data into buckets. You can use histogram buckets to view sets of events that are organized by the total event count and aggregate sum for all events. For more detailed information, see [OpenTelemetry Histograms](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#histogram).

| Parameter | Default value | Description |
| :--- | :--- | :--- |
| `calculate_histogram_buckets` | `True` | Whether or not to calculate histogram buckets. |
| `calculate_exponential_histogram_buckets` | `True` | Whether or not to calculate exponential histogram buckets. |
| `exponential_histogram_max_allowed_scale` | `10` | Maximum allowed scale in exponential histogram calculation. |
| `flatten_attributes` | `False` | Whether or not to flatten the `attributes` field in the JSON data. |

### calculate_histogram_buckets

If `calculate_histogram_buckets` is not set to `false`, then the following `JSON` file will be added to every histogram JSON. If `flatten_attributes` is set to `false`, the `JSON` string format of the metrics does not change the attributes field. If `flatten_attributes` is set to `true`, the values in the attributes field are placed in the parent `JSON` object. The default value is `true`. See the following `JSON` example:

```json
"buckets": [
{
"min": 0.0,
"max": 5.0,
"count": 2
},
{
"min": 5.0,
"max": 10.0,
"count": 5
}
]
```

You can create detailed representations of histogram buckets and their boundaries. You can control this feature by using the following parameters in your `pipeline.yaml` file:

```yaml
processor:
- otel_metrics_raw_processor:
calculate_histogram_buckets: true
calculate_exponential_histogram_buckets: true
exponential_histogram_max_allowed_scale: 10
flatten_attributes: false
```
{% include copy.html %}

Each array element describes one bucket. Each bucket contains the lower boundary, upper boundary, and its value count. This is a specific form of more detailed OpenTelemetry representation that is a part of the `JSON` output created by the `otel_metrics` processor. See the following `JSON` file, which is added to each `JSON` histogram by the `otel_metrics` processor:

```json
"explicitBounds": [
5.0,
10.0
],
"bucketCountsList": [
2,
5
]
```



### calculate_exponential_histogram_buckets

If `calculate_exponential_histogram_buckets` is set to `true` (the default setting), the following `JSON` values are added to each `JSON` histogram:

```json

"negativeBuckets": [
{
"min": 0.0,
"max": 5.0,
"count": 2
},
{
"min": 5.0,
"max": 10.0,
"count": 5
}
],
...
"positiveBuckets": [
{
"min": 0.0,
"max": 5.0,
"count": 2
},
{
"min": 5.0,
"max": 10.0,
"count": 5
}
],
```

The following `JSON` file is a more detailed form of OpenTelemetry representation that consists of negative and positive buckets, a scale parameter, an offset, and a list of bucket counts:


```json
"negative": [
1,
2,
3
],
"positive": [
1,
2,
3
],
"scale" : -3,
"negativeOffset" : 0,
"positiveOffset" : 1
```


### exponential_histogram_max_allowed_scale

The `exponential_histogram_max_allowed_scale` parameter defines the maximum allowed scale for an exponential histogram. If you increase this parameter, you will increase potential memory consumption. See the [OpenTelemetry specifications](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto) for more information on exponential histograms and their computational complexity.

All exponential histograms that have a scale that is above the configured parameter (by default, a value of `10`) are discarded and logged with an error level. You can check the log that Data Prepper creates to see the `ERROR` log message.

The absolute scale value is used for comparison, so a scale of `-11` that is treated equally to `11` exceeds the configured value of `10` and can be discarded.
{: .note}

## Metrics

The following table describes metrics that are common to all processors.

| Metric name | Type | Description |
| ------------- | ---- | -----------|
| `recordsIn` | Counter | Metric representing the number of ingress records. |
| `recordsOut` | Counter | Metric representing the number of egress records. |
| `timeElapsed` | Timer | Metric representing the time elapsed during execution of records. |