[processor/servicegraph] Fixes the number of bucket counts#3
Merged
Conversation
Signed-off-by: Jared Tan <jian.tan@daocloud.io>
JaredTan95
changed the title
JaredTan95
pushed a commit
that referenced
this pull request
Oct 4, 2024
… Histo --> Histogram (open-telemetry#33824) ## Description This PR adds a custom metric function to the transformprocessor to convert exponential histograms to explicit histograms. Link to tracking issue: Resolves open-telemetry#33827 **Function Name** ``` convert_exponential_histogram_to_explicit_histogram ``` **Arguments:** - `distribution` (_upper, midpoint, uniform, random_) - `ExplicitBoundaries: []float64` **Usage example:** ```yaml processors: transform: error_mode: propagate metric_statements: - context: metric statements: - convert_exponential_histogram_to_explicit_histogram("random", [10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0]) ``` **Converts:** ``` Resource SchemaURL: ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: response_time -> Description: -> Unit: -> DataType: ExponentialHistogram -> AggregationTemporality: Delta ExponentialHistogramDataPoints #0 Data point attributes: -> metric_type: Str(timing) StartTimestamp: 1970-01-01 00:00:00 +0000 UTC Timestamp: 2024-07-31 09:35:25.212037 +0000 UTC Count: 44 Sum: 999.000000 Min: 40.000000 Max: 245.000000 Bucket (32.000000, 64.000000], Count: 10 Bucket (64.000000, 128.000000], Count: 22 Bucket (128.000000, 256.000000], Count: 12 {"kind": "exporter", "data_type": "metrics", "name": "debug"} ``` **To:** ``` Resource SchemaURL: ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: response_time -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Delta HistogramDataPoints #0 Data point attributes: -> metric_type: Str(timing) StartTimestamp: 1970-01-01 00:00:00 +0000 UTC Timestamp: 2024-07-30 21:37:07.830902 +0000 UTC Count: 44 Sum: 999.000000 Min: 40.000000 Max: 245.000000 ExplicitBounds #0: 10.000000 ExplicitBounds #1: 20.000000 ExplicitBounds #2: 30.000000 ExplicitBounds #3: 40.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 60.000000 ExplicitBounds #6: 70.000000 ExplicitBounds #7: 80.000000 ExplicitBounds #8: 90.000000 ExplicitBounds #9: 100.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 2 Buckets #4, Count: 5 Buckets #5, Count: 0 Buckets #6, Count: 3 Buckets #7, Count: 7 Buckets #8, Count: 2 Buckets #9, Count: 4 Buckets #10, Count: 21 {"kind": "exporter", "data_type": "metrics", "name": "debug"} ``` ### Testing - Several unit tests have been created. We have also tested by ingesting and converting exponential histograms from the `statsdreceiver` as well as directly via the `otlpreceiver` over grpc over several hours with a large amount of data. - We have clients that have been running this solution in production for a number of weeks. ### Readme description: ### convert_exponential_hist_to_explicit_hist `convert_exponential_hist_to_explicit_hist([ExplicitBounds])` the `convert_exponential_hist_to_explicit_hist` function converts an ExponentialHistogram to an Explicit (_normal_) Histogram. `ExplicitBounds` is represents the list of bucket boundaries for the new histogram. This argument is __required__ and __cannot be empty__. __WARNING:__ The process of converting an ExponentialHistogram to an Explicit Histogram is not perfect and may result in a loss of precision. It is important to define an appropriate set of bucket boundaries to minimize this loss. For example, selecting Boundaries that are too high or too low may result histogram buckets that are too wide or too narrow, respectively. --------- Co-authored-by: Kent Quirk <kentquirk@gmail.com> Co-authored-by: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com>
JaredTan95
pushed a commit
that referenced
this pull request
Jan 22, 2025
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Generates simple histograms using telemetrygen <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes <!--Describe what testing was performed and which tests were added.--> #### Testing Test with a local otel collector with debug output ``` bin/telemetrygen metrics --metrics 5 --otlp-http --otlp-endpoint "localhost:4318" --metric-type Histogram --otlp-insecure ``` Output from debug Exporter: ``` Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.633365 +0000 UTC Timestamp: 2024-11-13 16:22:51.633367 +0000 UTC Count: 0 Sum: 0.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 0 Buckets #4, Count: 0 ResourceMetrics #1 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.639942 +0000 UTC Timestamp: 2024-11-13 16:22:51.639942 +0000 UTC Count: 1 Sum: 1.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 1 Buckets #2, Count: 0 Buckets #3, Count: 0 Buckets #4, Count: 0 ResourceMetrics #2 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.6404 +0000 UTC Timestamp: 2024-11-13 16:22:51.640401 +0000 UTC Count: 2 Sum: 4.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 1 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 ResourceMetrics #3 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.640729 +0000 UTC Timestamp: 2024-11-13 16:22:51.640729 +0000 UTC Count: 3 Sum: 3.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 1 Buckets #1, Count: 1 Buckets #2, Count: 1 Buckets #3, Count: 0 Buckets #4, Count: 0 ResourceMetrics #4 Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 StartTimestamp: 2024-11-13 16:22:50.641073 +0000 UTC Timestamp: 2024-11-13 16:22:51.641073 +0000 UTC Count: 4 Sum: 12.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 1.000000 ExplicitBounds #2: 2.000000 ExplicitBounds #3: 3.000000 ExplicitBounds #4: 4.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 1 Buckets #3, Count: 2 Buckets #4, Count: 1 {"kind": "exporter", "data_type": "metrics", "name": "debug"} ``` <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.--> --------- Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
JaredTan95
pushed a commit
that referenced
this pull request
Apr 11, 2025
…ry#38146) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This PR adds support for configuring metric temporality (delta or cumulative) in the telemetrygen tool. This allows users to generate metrics with different temporality types, which is particularly useful for testing different metric collection scenarios. ##### Changes - Added `temporalityType` flag for metrics pipeline that accepts values: `delta` or `cumulative` <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#38073 <!--Describe what testing was performed and which tests were added.--> #### Testing Testing was performed by setting up as simple collector with `otlp `receiver and `debug` exporter ##### Sum Datapoint ``` Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0 Resource attributes: -> service.name: Str(telemetrygen) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: gen -> Description: -> Unit: -> DataType: Histogram -> AggregationTemporality: Delta ``` ##### Histogram Datapoint ``` Count: 0 Sum: 3940.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds #7: 250.000000 ExplicitBounds #8: 500.000000 ExplicitBounds #9: 750.000000 ExplicitBounds #10: 1000.000000 ExplicitBounds #11: 2500.000000 ExplicitBounds #12: 5000.000000 ExplicitBounds #13: 7500.000000 ExplicitBounds #14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 1 Buckets #3, Count: 0 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 3 Buckets #7, Count: 4 Buckets #8, Count: 1 Buckets #9, Count: 1 Buckets #10, Count: 0 Buckets #11, Count: 0 Buckets #12, Count: 0 Buckets #13, Count: 0 Buckets #14, Count: 0 ``` - [x] Need to add tests for the config --------- Signed-off-by: Nikos Angelopoulos <nikolaos.angelopoulos@grafana.com>
Frapschen
pushed a commit
that referenced
this pull request
Feb 25, 2026
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Added configuration schemas to following components: * activedirectorydsreceiver * aerospikereceiver * apachereceiver * apachesparkreceiver * azureblobreceiver * azureeventhubreceiver * azuremonitorreceiver * bigipreceiver Schemas were generated with [schemagen](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/schemagen/README.md) script. Link to tracking issue open-telemetry#42214
Frapschen
pushed a commit
that referenced
this pull request
Feb 25, 2026
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Add `Base64Encode` OTTL converter function to encode strings into base64 format with support for multiple variants (`base64`, `base64-raw`, `base64-url`, `base64-raw-url`). This function addresses the need to handle characters not allowed by certain exporters like NATS Core. <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#46071 <!--Describe what testing was performed and which tests were added.--> #### Testing Tested manually and added unit and e2e tests. Test config: ```yaml processors: transform: log_statements: - context: log statements: - set(attributes["encoded_default"], Base64Encode(attributes["plain_text"])) - set(attributes["encoded_base64_raw"], Base64Encode(attributes["plain_text"], "base64-raw")) - set(attributes["encoded_base64_url"], Base64Encode(attributes["plain_text"], "base64-url")) - set(attributes["encoded_base64_raw_url"], Base64Encode(attributes["plain_text"], "base64-raw-url")) ``` Test input: ```json {"plain_text": "test string"} {"plain_text": "hello world"} {"plain_text": "special chars: @#$%"} {"plain_text": "URL encoding test: https://example.com?param=value&other=123"} ``` Test result: ``` LogRecord #0 Body: Str({"plain_text": "test string"}) Attributes: -> log.file.name: Str(test_data.log) -> plain_text: Str(test string) -> encoded_default: Str(dGVzdCBzdHJpbmc=) -> encoded_base64_raw: Str(dGVzdCBzdHJpbmc) -> encoded_base64_url: Str(dGVzdCBzdHJpbmc=) -> encoded_base64_raw_url: Str(dGVzdCBzdHJpbmc) LogRecord #1 Body: Str({"plain_text": "hello world"}) Attributes: -> log.file.name: Str(test_data.log) -> plain_text: Str(hello world) -> encoded_default: Str(aGVsbG8gd29ybGQ=) -> encoded_base64_raw: Str(aGVsbG8gd29ybGQ) -> encoded_base64_url: Str(aGVsbG8gd29ybGQ=) -> encoded_base64_raw_url: Str(aGVsbG8gd29ybGQ) LogRecord #2 Body: Str({"plain_text": "special chars: @#$%"}) Attributes: -> log.file.name: Str(test_data.log) -> plain_text: Str(special chars: @#$%) -> encoded_default: Str(c3BlY2lhbCBjaGFyczogQCMkJQ==) -> encoded_base64_raw: Str(c3BlY2lhbCBjaGFyczogQCMkJQ) -> encoded_base64_url: Str(c3BlY2lhbCBjaGFyczogQCMkJQ==) -> encoded_base64_raw_url: Str(c3BlY2lhbCBjaGFyczogQCMkJQ) LogRecord #3 Body: Str({"plain_text": "URL encoding test: https://example.com?param=value&other=123"}) Attributes: -> log.file.name: Str(test_data.log) -> plain_text: Str(URL encoding test: https://example.com?param=value&other=123) -> encoded_default: Str(VVJMIGVuY29kaW5nIHRlc3Q6IGh0dHBzOi8vZXhhbXBsZS5jb20/cGFyYW09dmFsdWUmb3RoZXI9MTIz) -> encoded_base64_raw: Str(VVJMIGVuY29kaW5nIHRlc3Q6IGh0dHBzOi8vZXhhbXBsZS5jb20/cGFyYW09dmFsdWUmb3RoZXI9MTIz) -> encoded_base64_url: Str(VVJMIGVuY29kaW5nIHRlc3Q6IGh0dHBzOi8vZXhhbXBsZS5jb20_cGFyYW09dmFsdWUmb3RoZXI9MTIz) -> encoded_base64_raw_url: Str(VVJMIGVuY29kaW5nIHRlc3Q6IGh0dHBzOi8vZXhhbXBsZS5jb20_cGFyYW09dmFsdWUmb3RoZXI9MTIz) ``` <!--Describe the documentation added.--> #### Documentation Updated `README.md` with function documentation, usage examples, and supported variants. <!--Please delete paragraphs that you did not use before submitting.--> --------- Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…-contrib/pull/16025
Signed-off-by: Jared Tan jian.tan@daocloud.io
Description:
Link to tracking Issue:
Testing:
Documentation: