From bb3d0a0d14f41ccd3f78852316c77705e722edfa Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Fri, 1 Dec 2023 16:33:44 -0500 Subject: [PATCH] Clarifications and "flexibility" fixes in Exemplar Specification (#3760) Fixes #2205 Fixes #3674 Fixes #3669 Partially fixes #2421 ## Changes - Update example exemplar algorithm to account for initial reservoir fill - Update fixed-size defaults to account for memory contention / optimization in Java impl - Set a default for exponential histogram aggregation - Clarify that ExemplarFilter should be configured on MeterProvider - Make it clear that ONE reservoir is create PER timeseries datapoint (not one reservoir per view or metric name). - Allow flexibility in Reservoir `offer` definition based on feedback from Go impl. * Related issues #3756 --------- Co-authored-by: David Ashpole Co-authored-by: Joshua MacDonald --- CHANGELOG.md | 2 ++ specification/metrics/sdk.md | 52 ++++++++++++++++++++++++++++-------- 2 files changed, 43 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e369eea4e90..51606bf41f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ release. - Add optional configuration for Prometheus exporters to promote resource attributes to metric attributes ([#3761](https://github.com/open-telemetry/opentelemetry-specification/pull/3761)) +- Clarifications and flexibility in Exemplar speicification. + ([#3760](https://github.com/open-telemetry/opentelemetry-specification/pull/3760)) ### Logs diff --git a/specification/metrics/sdk.md b/specification/metrics/sdk.md index 0ac9ad9f29e..ee62c5a4557 100644 --- a/specification/metrics/sdk.md +++ b/specification/metrics/sdk.md @@ -54,6 +54,7 @@ linkTitle: SDK + [AlwaysOn](#alwayson) + [AlwaysOff](#alwaysoff) + [TraceBased](#tracebased) + + [Configuration](#configuration-2) * [ExemplarReservoir](#exemplarreservoir) * [Exemplar defaults](#exemplar-defaults) + [SimpleFixedSizeExemplarReservoir](#simplefixedsizeexemplarreservoir) @@ -972,11 +973,21 @@ Using this ExemplarFilter is as good as disabling Exemplar feature. An ExemplarFilter which makes those measurements eligible for being an Exemplar, which are recorded in the context of a sampled parent span. +#### Configuration + +The ExemplarFilter SHOULD be a configuration parameter of a `MeterProvider` for +an SDK. The default value SHOULD be `TraceBased`. The filter configuration +SHOULD follow the [environment variable specification](../configuration/sdk-environment-variables.md#exemplar). + ### ExemplarReservoir The `ExemplarReservoir` interface MUST provide a method to offer measurements to the reservoir and another to collect accumulated Exemplars. +A new `ExemplarReservoir` MUST be created for every known timeseries data point, +as determined by aggregation and view configuration. This data point, and its +set of defining attributes, are referred to as the associated timeseries point. + The "offer" method SHOULD accept measurements, including: - The `value` of the measurement. @@ -993,18 +1004,26 @@ span context and baggage can be inspected at this point. The "offer" method does not need to store all measurements it is given and MAY further sample beyond the `ExemplarFilter`. +The "offer" method MAY accept a filtered subset of `Attributes` which diverge +from the timeseries the reservoir is associated with. This MUST be clearly +documented in the API interface and the reservoir MUST be given the `Attributes` +associated with its timeseries point either at construction so that additional +sampling performed by the reservoir has access to all attributes from a +measurement in the "offer" method. SDK authors are encouraged to benchmark +whether this option works best for their implementation. + The "collect" method MUST return accumulated `Exemplar`s. Exemplars are expected to abide by the `AggregationTemporality` of any metric point they are recorded with. In other words, Exemplars reported against a metric data point SHOULD have -occurred within the start/stop timestamps of that point. SDKs are free to +occurred within the start/stop timestamps of that point. SDKs are free to decide whether "collect" should also reset internal storage for delta temporal aggregation collection, or use a more optimal implementation. `Exemplar`s MUST retain any attributes available in the measurement that -are not preserved by aggregation or view configuration. Specifically, at a -minimum, joining together attributes on an `Exemplar` with those available -on its associated metric data point should result in the full set of attributes -from the original sample measurement. +are not preserved by aggregation or view configuration for the associated +timeseries. Joining together attributes on an `Exemplar` with +those available on its associated metric data point should result in the +full set of attributes from the original sample measurement. The `ExemplarReservoir` SHOULD avoid allocations when sampling exemplars. @@ -1015,9 +1034,15 @@ The SDK SHOULD include two types of built-in exemplar reservoirs: 1. `SimpleFixedSizeExemplarReservoir` 2. `AlignedHistogramBucketExemplarReservoir` -By default, explicit bucket histogram aggregation with more than 1 bucket will -use `AlignedHistogramBucketExemplarReservoir`. All other aggregations will use -`SimpleFixedSizeExemplarReservoir`. +By default: + +- Explicit bucket histogram aggregation with more than 1 bucket will +use `AlignedHistogramBucketExemplarReservoir`. +- Base2 Exponential Histogram Aggregation SHOULD use a + `SimpleFixedSizeExemplarReservoir` with a reservoir equal to the + smaller of the maximum number of buckets configured on the aggregation or + twenty (e.g. `min(20, max_buckets)`). +- All other aggregations will use `SimpleFixedSizeExemplarReservoir`. #### SimpleFixedSizeExemplarReservoir @@ -1027,7 +1052,11 @@ measurements should be sampled. For example, the [simple reservoir sampling algorithm](https://en.wikipedia.org/wiki/Reservoir_sampling) can be used: ``` - bucket = random_integer(0, num_measurements_seen) + if num_measurements_seen < num_buckets then + bucket = num_measurements_seen + else + bucket = random_integer(0, num_measurements_seen) + end if bucket < num_buckets then reservoir[bucket] = measurement end @@ -1038,8 +1067,9 @@ cycle. For the above example, that would mean that the `num_measurements_seen` count is reset every time the reservoir is collected. This Exemplar reservoir MAY take a configuration parameter for the size of the -reservoir pool. If no size configuration is provided, the default size of `1` -SHOULD be used. +reservoir. If no size configuration is provided, the default size MAY be +the number of possible concurrent threads (e.g. numer of CPUs) to help reduce +contention. Otherwise, a default size of `1` SHOULD be used. #### AlignedHistogramBucketExemplarReservoir