diff --git a/CHANGELOG.md b/CHANGELOG.md index a4e91d11f5a..368dfb94d1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,8 @@ release. ([#4966](https://github.com/open-telemetry/opentelemetry-specification/pull/4966)) - Stabilize Prometheus SDK Exporter host configuration. ([#4984](https://github.com/open-telemetry/opentelemetry-specification/issues/4984)) +- Stabilize OpenTelemetry Exemplar to Prometheus Exemplar transformation. + ([#4964](https://github.com/open-telemetry/opentelemetry-specification/pull/4964)) ### SDK Configuration diff --git a/spec-compliance-matrix.md b/spec-compliance-matrix.md index 1ac0812b85e..2982b41e821 100644 --- a/spec-compliance-matrix.md +++ b/spec-compliance-matrix.md @@ -372,7 +372,7 @@ Disclaimer: Declarative configuration is currently in Development status - work | [Delta Histograms become Cumulative Prometheus Histograms](specification/compatibility/prometheus_and_openmetrics.md#histograms-1) | X | - | - | - | - | - | - | - | - | - | - | - | - | | [Attributes Keys are Sanitized](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes) | | + | + | + | + | - | - | - | + | + | + | + | - | | [Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes) | | + | + | - | - | - | - | - | + | - | - | - | - | -| [Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1) | X | + | - | - | - | - | - | - | - | - | - | - | - | +| [Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion) | X | + | - | - | - | - | - | - | - | - | - | - | - | | [`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1) | X | + | + | + | + | - | - | - | + | - | - | - | - | ## OpenCensus Compatibility diff --git a/spec-compliance-matrix/cpp.yaml b/spec-compliance-matrix/cpp.yaml index a468cb9c87c..20808ee4576 100644 --- a/spec-compliance-matrix/cpp.yaml +++ b/spec-compliance-matrix/cpp.yaml @@ -655,7 +655,7 @@ sections: status: '+' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '-' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '-' diff --git a/spec-compliance-matrix/dotnet.yaml b/spec-compliance-matrix/dotnet.yaml index 1da19bc2c17..69c691a1369 100644 --- a/spec-compliance-matrix/dotnet.yaml +++ b/spec-compliance-matrix/dotnet.yaml @@ -655,7 +655,7 @@ sections: status: '+' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '-' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '-' diff --git a/spec-compliance-matrix/erlang.yaml b/spec-compliance-matrix/erlang.yaml index 44ea8d19c42..05345966b7c 100644 --- a/spec-compliance-matrix/erlang.yaml +++ b/spec-compliance-matrix/erlang.yaml @@ -655,7 +655,7 @@ sections: status: '-' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '-' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '-' diff --git a/spec-compliance-matrix/go.yaml b/spec-compliance-matrix/go.yaml index 5b2ba48f343..fbc414a43c9 100644 --- a/spec-compliance-matrix/go.yaml +++ b/spec-compliance-matrix/go.yaml @@ -655,7 +655,7 @@ sections: status: '+' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '+' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '+' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '+' diff --git a/spec-compliance-matrix/java.yaml b/spec-compliance-matrix/java.yaml index d82904aa7ba..90016ef9020 100644 --- a/spec-compliance-matrix/java.yaml +++ b/spec-compliance-matrix/java.yaml @@ -655,7 +655,7 @@ sections: status: '+' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '+' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '+' diff --git a/spec-compliance-matrix/js.yaml b/spec-compliance-matrix/js.yaml index 70850b3f12b..ced63d7f181 100644 --- a/spec-compliance-matrix/js.yaml +++ b/spec-compliance-matrix/js.yaml @@ -655,7 +655,7 @@ sections: status: '+' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '-' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '+' diff --git a/spec-compliance-matrix/kotlin.yaml b/spec-compliance-matrix/kotlin.yaml index 6ba4ebec682..3bc72be8bde 100644 --- a/spec-compliance-matrix/kotlin.yaml +++ b/spec-compliance-matrix/kotlin.yaml @@ -655,7 +655,7 @@ sections: status: '-' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '-' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '-' diff --git a/spec-compliance-matrix/php.yaml b/spec-compliance-matrix/php.yaml index 31df84ceb71..ecb87bcb088 100644 --- a/spec-compliance-matrix/php.yaml +++ b/spec-compliance-matrix/php.yaml @@ -655,7 +655,7 @@ sections: status: '-' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '-' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '-' diff --git a/spec-compliance-matrix/python.yaml b/spec-compliance-matrix/python.yaml index 78b1348a83d..443e480dc56 100644 --- a/spec-compliance-matrix/python.yaml +++ b/spec-compliance-matrix/python.yaml @@ -655,7 +655,7 @@ sections: status: '+' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '-' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '+' diff --git a/spec-compliance-matrix/ruby.yaml b/spec-compliance-matrix/ruby.yaml index 03e84cfcc04..05a28c40d63 100644 --- a/spec-compliance-matrix/ruby.yaml +++ b/spec-compliance-matrix/ruby.yaml @@ -655,7 +655,7 @@ sections: status: '-' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '-' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '-' diff --git a/spec-compliance-matrix/rust.yaml b/spec-compliance-matrix/rust.yaml index f987489ce78..d493f22d96e 100644 --- a/spec-compliance-matrix/rust.yaml +++ b/spec-compliance-matrix/rust.yaml @@ -655,7 +655,7 @@ sections: status: '+' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '+' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '+' diff --git a/spec-compliance-matrix/swift.yaml b/spec-compliance-matrix/swift.yaml index e1a0596503a..bdd8ac6d62d 100644 --- a/spec-compliance-matrix/swift.yaml +++ b/spec-compliance-matrix/swift.yaml @@ -655,7 +655,7 @@ sections: status: '+' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' status: '-' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' status: '-' - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' status: '-' diff --git a/spec-compliance-matrix/template.yaml b/spec-compliance-matrix/template.yaml index 18aa739c52d..9c5cd9fb97d 100644 --- a/spec-compliance-matrix/template.yaml +++ b/spec-compliance-matrix/template.yaml @@ -427,7 +427,7 @@ sections: optional: true - name: '[Attributes Keys are Sanitized](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' - name: '[Colliding sanitized attribute keys are merged](specification/compatibility/prometheus_and_openmetrics.md#metric-attributes)' - - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplars-1)' + - name: '[Exemplars for Histograms and Monotonic sums](specification/compatibility/prometheus_and_openmetrics.md#exemplar-conversion)' optional: true - name: '[`target_info` metric from Resource](specification/compatibility/prometheus_and_openmetrics.md#resource-attributes-1)' optional: true diff --git a/specification/compatibility/prometheus_and_openmetrics.md b/specification/compatibility/prometheus_and_openmetrics.md index 8bbf4072a5d..4ef57debd27 100644 --- a/specification/compatibility/prometheus_and_openmetrics.md +++ b/specification/compatibility/prometheus_and_openmetrics.md @@ -39,7 +39,7 @@ aliases: * [Exponential Histograms](#exponential-histograms) * [Summaries](#summaries-1) * [Metric Attributes](#metric-attributes) - * [Exemplars](#exemplars-1) + * [Exemplar Conversion](#exemplar-conversion) * [Resource Attributes](#resource-attributes-1) @@ -473,6 +473,8 @@ a Prometheus Unknown-typed metric if the `prometheus.type` key of [metric.metadata][metricMetadata] is `unknown`. Otherwise, it MUST be converted to a Prometheus Gauge. +Exemplars on OpenTelemetry Gauges SHOULD be dropped. + ### Sums **Status**: [Development](../document-status.md) @@ -493,6 +495,13 @@ to a Prometheus Gauge. If the metric name for monotonic Sum metric points does not end in a suffix of `_total` a suffix of `_total` SHOULD be added by default, otherwise the name MUST remain unchanged. Exporters SHOULD provide a configuration option to disable the addition of `_total` suffixes. Monotonic Sum metric points with `StartTimeUnixNano` should export the `{name}_created` metric as well. +If Sum is converted to a Prometheus Counter, then `Exemplars` MUST be converted +as described in the [Exemplar Conversion](#exemplar-conversion) section. +Otherwise, `Exemplars` SHOULD be dropped. If the Prometheus protocol only +supports a single exemplar on the Counter sample, the latest exemplar SHOULD be +converted. This matches the behavior of Prometheus client libraries, which is to keep the +latest exemplar for counter instruments. + ### Histograms **Status**: [Development](../document-status.md) @@ -501,9 +510,13 @@ An [OpenTelemetry Histogram](../metrics/data-model.md#histogram) with a cumulati - A single `{name}_count` metric denoting the count field of the histogram. All attributes of the histogram point are converted to Prometheus labels. - `{name}_sum` metric denoting the sum field of the histogram, reported only if the sum is positive and monotonic. The sum is positive and monotonic when all buckets are positive. All attributes of the histogram point are converted to Prometheus labels. -- A series of `{name}_bucket` metric points that contain all attributes of the histogram point recorded as labels. Additionally, a label, denoted as `le` is added denoting the bucket boundary. The label's value is the stringified floating point value of bucket boundaries, ordered from lowest to highest. The value of each point is the sum of the count of all histogram buckets up to the boundary reported in the `le` label. These points will include a single exemplar that falls within `le` label and no other `le` labelled point. The final bucket metric MUST have an `+Inf` threshold. +- A series of `{name}_bucket` metric points that contain all attributes of the histogram point recorded as labels. Additionally, a label, denoted as `le` is added denoting the bucket boundary. The label's value is the stringified floating point value of bucket boundaries, ordered from lowest to highest. The value of each point is the sum of the count of all histogram buckets up to the boundary reported in the `le` label. The final bucket metric MUST have an `+Inf` threshold. - Histograms with `StartTimeUnixNano` set should export the `{name}_created` metric as well. +`Exemplars` are converted as described in the [Exemplar Conversion](#exemplar-conversion) section. +If the Prometheus protocol only supports a single exemplar per-bucket, the latest +exemplar that falls into each bucket SHOULD be converted. + OpenTelemetry Histograms with Delta aggregation temporality SHOULD be aggregated into a Cumulative aggregation temporality and follow the logic above, or MUST be dropped. ### Exponential Histograms @@ -539,6 +552,7 @@ Histogram as follows: result being that the Offset fields are different-by-one. - `Min` and `Max` are not used. - `StartTimeUnixNano` is not used. +- `Exemplars` are converted as described in the [Exemplar Conversion](#exemplar-conversion) section. [OpenTelemetry Exponential Histogram](../metrics/data-model.md#exponentialhistogram) metrics with the delta aggregation temporality are dropped. @@ -562,6 +576,8 @@ An [OpenTelemetry Summary](../metrics/data-model.md#summary-legacy) MUST be conv each point is the computed value of the quantile point. - Summaries with `StartTimeUnixNano` set should export the `{name}_created` metric as well. +Exemplars on OpenTelemetry Summaries SHOULD be dropped. + ### Metric Attributes **Status**: [Stable](../document-status.md) @@ -579,23 +595,24 @@ added by this specification, may cause different OpenTelemetry keys to map to the same Prometheus key. In such cases, the values MUST be concatenated together, separated by `;`, and ordered by the lexicographical order of the original keys. -### Exemplars +### Exemplar Conversion -**Status**: [Development](../document-status.md) +**Status**: [Stable](../document-status.md) -[Exemplars](../metrics/data-model.md#exemplars) on OpenTelemetry Histograms and Monotonic Sums SHOULD -be converted to Prometheus exemplars. Exemplars on other OpenTelemetry data -points MUST be dropped. For Prometheus Remote Write exporters, multiple exemplars are -able to be added to each bucket, so all exemplars SHOULD be converted. For -Prometheus pull endpoints, only a single exemplar is able to be added to each -bucket, so the largest exemplar from each bucket MUST be used, if attaching -exemplars. If no exemplars exist on a bucket, the highest exemplar from a lower -bucket MUST be used, even though it is a duplicate of another bucket's exemplar. -Prometheus Exemplars MUST use the `trace_id` and `span_id` keys for the trace -and span IDs, respectively. Timestamps MUST be added as timestamps on the -Prometheus exemplar, and `filtered_attributes` MUST be added as labels on the -Prometheus exemplar unless they would exceed the -[limit on characters](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#exemplars). +When an exemplar is converted per the metric-type-specific sections above, +the [OpenTelemetry Exemplar](../metrics/data-model.md#exemplars) MUST be converted +to a Prometheus exemplar if the Prometheus (push or pull) protocol being used +supports them, as follows: + +* If present, the OpenTelemetry Exemplar's Trace ID and Span ID MUST be added as + Exemplar labels using the `trace_id` and `span_id` keys, respectively. These + labels MUST take precedence over labels from `filtered_attributes` in cases + where there is a key collision. +* Timestamps MUST be added as timestamps on the Prometheus exemplar. +* `filtered_attributes` MUST be added as labels on the Prometheus exemplar, + unless they would exceed the Prometheus protocol's exemplar limits. For + example, OpenMetrics 1.0 imposes a + [128 character limit](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#exemplars). ### Resource Attributes