Receive OTLP HTTP metrics, forward to Prometheus, collision on metric names #21471

marcalff · 2023-04-17T19:29:31Z

Setup:

C++ application (test app, opentelemetry-cpp 1.9.0, OTLP HTTP exporter)
opentelemetry-collector 0.75.0
prometheus 2.42.0

In a test application, define meters and metrics as follows:

Meter my_meter_1
- Metric my_metric_a
- Metric my_metric_b
Meter my_meter_2
- Metric my_metric_a
- Metric my_metric_b
- Metric my_metric_c

As far as I understand, this is valid, because what identifies a metric is the instrumentation scope + the metric name, so that my_meter_1.my_metric_a and my_meter_2.my_metric_a are two different metrics.

Export this using OTLP HTTP, to an opentelemetry-collector.

From the collector logs, the data received is:

2023-04-17T20:40:58.805+0200	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 5}
2023-04-17T20:40:58.805+0200	info	ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.version: Str(1.9.0)
     -> telemetry.sdk.language: Str(cpp)
     -> service.version: Str(xxx)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> service.instance.id: Str(xxx)
     -> service.name: Str(xxx)
     -> service.namespace: Str(xxx)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope my_meter_1 1.0
Metric #0
Descriptor:
     -> Name: my_metric_b
     -> Description: description b
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-04-17 18:40:37.784020265 +0000 UTC
Timestamp: 2023-04-17 18:40:58.697529476 +0000 UTC
Value: 110
Metric open-telemetry/opentelemetry-collector#1
Descriptor:
     -> Name: my_metric_a
     -> Description: description a
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-04-17 18:40:37.784020265 +0000 UTC
Timestamp: 2023-04-17 18:40:58.697529476 +0000 UTC
Value: 108
ScopeMetrics open-telemetry/opentelemetry-collector#1
ScopeMetrics SchemaURL: 
InstrumentationScope my_meter_2 1.0
Metric #0
Descriptor:
     -> Name: my_metric_b
     -> Description: description b
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-04-17 18:40:37.784020265 +0000 UTC
Timestamp: 2023-04-17 18:40:58.697764025 +0000 UTC
Value: 116
Metric open-telemetry/opentelemetry-collector#1
Descriptor:
     -> Name: my_metric_a
     -> Description: description a
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-04-17 18:40:37.784020265 +0000 UTC
Timestamp: 2023-04-17 18:40:58.697764025 +0000 UTC
Value: 114
Metric open-telemetry/opentelemetry-collector#2
Descriptor:
     -> Name: my_metric_c
     -> Description: description c
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-04-17 18:40:37.784020265 +0000 UTC
Timestamp: 2023-04-17 18:40:58.697764025 +0000 UTC
Value: 118
	{"kind": "exporter", "data_type": "metrics", "name": "logging"}

So far, this works as expected: the opentelemetry collector receives data that describe:

a measurement for instrumentation scope my_meter_1 and metric my_metric_a
another measurement for instrumentation scope my_meter_2 and metric my_metric_a

Now, in the opentelemetry collector, forward the metric data to prometheus:

receivers:
  otlp:
    protocols:
      http:

processors:
  batch:
  memory_limiter:
    # 75% of maximum memory up to 4G
    limit_mib: 1536
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s

exporters:
  logging:
    loglevel: debug

  # Forward metrics to Prometheus
  prometheusremotewrite:
    endpoint: "http://localhost:9090/api/v1/write"
    # For official Prometheus (e.g. running via Docker)
    # endpoint: 'http://prometheus:9090/api/v1/write'
    # tls:
    #   insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging, prometheusremotewrite]

Data is sent using prometheusremotewrite, and is visible in prometheus.

The issue is that prometheus only displays:

my_metric_a
my_metric_b
my_metric_c

The fully qualified name (my_meter_1.my_metric_a) is lost in the chain, so that data from different meters (my_meter_1 and my_meter_2) is mixed when there is a name collision on the metric name (my_meter_a).

Expected result:

my_meter_1_my_metric_a visible in prometheus (or something similar)
my_meter_2_my_metric_a visible in prometheus

In other words, there should be some way to preserve the fully qualified name of a metric when exporting it from OTLP to prometheus, so that metrics from different meters can be separated.

The text was updated successfully, but these errors were encountered:

bogdandrutu · 2023-05-03T04:50:52Z

Move to contrib, since this is an issue in translating otlp to prometheus remote write.

github-actions · 2023-05-03T16:20:12Z

Pinging code owners for exporter/prometheusremotewrite: @Aneurysm9 @rapphil. See Adding Labels via Comments if you do not have permissions to add labels yourself.

Aneurysm9 · 2023-05-03T16:49:00Z

The PRW translation logic needs to be updated to conform to the spec for handling instrumentation scope in Prometheus exporters. The PRW exporter should also be given a configuration option to disable adding these labels.

marcalff · 2023-05-15T13:55:30Z

Familiar neither with the code nor with go, so I can't provide a patch.

From code reading, it seems the naming translation happens at:

File pkg/translator/prometheus/normalize_name.go
Function BuildPromCompliantName()
Function normalizeName()

github-actions · 2023-07-17T03:35:27Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil
pkg/translator/prometheusremotewrite: @Aneurysm9 @kovrus

See Adding Labels via Comments if you do not have permissions to add labels yourself.

marcalff · 2023-07-17T07:28:09Z

@Aneurysm9 @rapphil @kovrus

Ping to avoid automatically closing the issue, the problem is still relevant.

If this is tracked by another issue number, please link it.

github-actions · 2023-09-18T03:30:50Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil
pkg/translator/prometheusremotewrite: @Aneurysm9

See Adding Labels via Comments if you do not have permissions to add labels yourself.

marcalff · 2023-09-18T08:06:07Z

Again:

@Aneurysm9 @rapphil @kovrus

Ping to avoid automatically closing the issue, the problem is still relevant.

If this is tracked by another issue number, please link it.

In light of:

Add "/" to valid characters for instrument names opentelemetry-specification#3684
Increase metric name maximum length from 63 to 255 characters opentelemetry-specification#3648

I would like to know how an application is expected to instrument metrics here:

Should the metric name be globally unique, avoiding the collision ?
Is it ok to have the same metric name provided by different meters, and if so how is the prometheus exporter handling the collision ?

github-actions · 2023-11-20T03:29:47Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil
pkg/translator/prometheusremotewrite: @Aneurysm9

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions · 2024-01-19T05:19:14Z

This issue has been closed as inactive because it has been stale for 120 days with no activity.

marcalff · 2024-06-20T13:16:33Z

Ping @Aneurysm9 @rapphil @kovrus @bogdandrutu
CC @svrnm

This issue is still relevant, and is still blocking to deploy an application using:
opentelemetry-cpp --> opentelemetry-collector --> prometheus

Per previous analysis from @Aneurysm9:

The PRW translation logic needs to be updated to conform to the spec for handling instrumentation scope in Prometheus exporters. The PRW exporter should also be given a configuration option to disable adding these labels.

This is a confirmed bug in the prometheus write exporter.

Why are verified bugs automatically closed by github actions, which purpose does that serve ?

marcalff added the bug Something isn't working label Apr 17, 2023

bogdandrutu transferred this issue from open-telemetry/opentelemetry-collector May 3, 2023

bogdandrutu added the exporter/prometheusremotewrite label May 3, 2023

Aneurysm9 added the pkg/translator/prometheusremotewrite label May 3, 2023

Aneurysm9 assigned rapphil May 3, 2023

marcalff mentioned this issue May 15, 2023

Prometheus Exporter should add InstrumentationScope labels #21091

Closed

github-actions bot added the Stale label Jul 17, 2023

github-actions bot removed the Stale label Jul 18, 2023

github-actions bot added the Stale label Sep 18, 2023

github-actions bot removed the Stale label Sep 19, 2023

github-actions bot added the Stale label Nov 20, 2023

github-actions bot added the closed as inactive label Jan 19, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 19, 2024

marcalff mentioned this issue Jun 20, 2024

[TRACKING] List of upstream issues impacting opentelemetry-cpp open-telemetry/opentelemetry-cpp#2708

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Receive OTLP HTTP metrics, forward to Prometheus, collision on metric names #21471

Receive OTLP HTTP metrics, forward to Prometheus, collision on metric names #21471

marcalff commented Apr 17, 2023 •

edited

Loading

bogdandrutu commented May 3, 2023

github-actions bot commented May 3, 2023

Aneurysm9 commented May 3, 2023

marcalff commented May 15, 2023

github-actions bot commented Jul 17, 2023

marcalff commented Jul 17, 2023

github-actions bot commented Sep 18, 2023

marcalff commented Sep 18, 2023

github-actions bot commented Nov 20, 2023

github-actions bot commented Jan 19, 2024

marcalff commented Jun 20, 2024

Receive OTLP HTTP metrics, forward to Prometheus, collision on metric names #21471

Receive OTLP HTTP metrics, forward to Prometheus, collision on metric names #21471

Comments

marcalff commented Apr 17, 2023 • edited Loading

bogdandrutu commented May 3, 2023

github-actions bot commented May 3, 2023

Aneurysm9 commented May 3, 2023

marcalff commented May 15, 2023

github-actions bot commented Jul 17, 2023

marcalff commented Jul 17, 2023

github-actions bot commented Sep 18, 2023

marcalff commented Sep 18, 2023

github-actions bot commented Nov 20, 2023

github-actions bot commented Jan 19, 2024

marcalff commented Jun 20, 2024

marcalff commented Apr 17, 2023 •

edited

Loading