Skip to content

fix(telemetry): export properly resources configured on prometheus#7394

Merged
bnjjj merged 23 commits intodevfrom
bnjjj/fix_telemetry_prom_resource
Jun 6, 2025
Merged

fix(telemetry): export properly resources configured on prometheus#7394
bnjjj merged 23 commits intodevfrom
bnjjj/fix_telemetry_prom_resource

Conversation

@bnjjj
Copy link
Contributor

@bnjjj bnjjj commented May 5, 2025

When configuring resource to globally add labels on metrics like this:

telemetry:
  apollo:
    client_name_header: name_header
    client_version_header: version_header
  exporters:
    metrics:
      common:
        resource:
          "test-resource": "test"
      prometheus:
        enabled: true
        resource_selector: all # This will add resources on every metrics

test-resource label was never exported to prometheus, this bug only occurs with prometheus and not otlp.
This PR fixes this behavior and will no longer filter resources.


Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

  • Changes are compatible1
  • Documentation2 completed
  • Performance impact assessed and acceptable
  • Tests added and passing3
    • Unit Tests
    • Integration Tests
    • Manual Tests

Exceptions

Note any exceptions here

Notes

Footnotes

  1. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this.

  2. Configuration is an important part of many changes. Where applicable please try to document configuration examples.

  3. Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions.

Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@bnjjj bnjjj requested a review from a team May 5, 2025 12:43
@github-actions

This comment has been minimized.

@svc-apollo-docs
Copy link
Collaborator

svc-apollo-docs commented May 5, 2025

✅ Docs preview ready

The preview is ready to be viewed. View the preview

File Changes

0 new, 1 changed, 0 removed
* graphos/routing/(latest)/observability/telemetry/metrics-exporters/prometheus.mdx

Build ID: 2a5812e2b6ed7af8c5659ab8

URL: https://www.apollographql.com/docs/deploy-preview/2a5812e2b6ed7af8c5659ab8

@bnjjj bnjjj added the backport-1.x Backport this PR to 1.x label May 5, 2025
@bnjjj bnjjj requested review from BrynCooke and carodewig May 5, 2025 13:04
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@bnjjj bnjjj requested a review from a team as a code owner May 5, 2025 13:10
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
bnjjj added 3 commits May 6, 2025 17:09
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@bnjjj bnjjj requested review from carodewig and goto-bus-stop May 7, 2025 08:49
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Copy link
Contributor

@BrynCooke BrynCooke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not clear this is the best way to go, usually target_info has the resource info:

# HELP target_info Target metadata
# TYPE target_info gauge
target_info{process_executable_name="router",service_name="unknown_service:router",service_version="2.2.1",test="test"} 1

https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#resource-attributes-1

bnjjj added 3 commits May 23, 2025 11:14
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@bnjjj bnjjj requested review from BrynCooke and goto-bus-stop May 23, 2025 09:27
bnjjj and others added 4 commits May 23, 2025 14:32
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@BrynCooke
Copy link
Contributor

I think all that's needed is that this is a config option in the prometheus config. Here's the relevant part of the spec:

In Prometheus exporters, an OpenTelemetry Resource SHOULD be converted to a target info metric if the resource is not empty. The Resource attributes MAY be copied to labels of exported metric families if required by the exporter configuration, or MUST be dropped. The target info metric MUST be an info-typed metric whose labels MUST include the resource attributes, and MUST NOT include any other labels.

So to merge the PR is to move away from the "SHOULD" behaviour.

We can add an option on the prometheus exporter for this to fulfil: The Resource attributes MAY be copied to labels of exported metric families if required by the exporter configuration

@BrynCooke
Copy link
Contributor

Suggest:

telemetry:
  exporters:
    metrics:
      prometheus:
        enabled: true
        resource_selector: all

@bnjjj
Copy link
Contributor Author

bnjjj commented Jun 4, 2025

@BrynCooke Still unsure to understand because if I set this configuration to false I have no way to export the resources mentioned in the config. I think I'll just get rid of ResourceSelector::All and use RessourceSelector::KeyAllowList (https://docs.rs/opentelemetry-prometheus/latest/opentelemetry_prometheus/enum.ResourceSelector.html#variant.KeyAllowList) it would cause less issues

bnjjj added 6 commits June 4, 2025 11:44
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@goto-bus-stop goto-bus-stop removed their request for review June 6, 2025 09:09
Copy link
Contributor

@BrynCooke BrynCooke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You just need to add docs for the new option

Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
@bnjjj bnjjj requested review from BrynCooke and goto-bus-stop June 6, 2025 11:39
@bnjjj bnjjj enabled auto-merge (squash) June 6, 2025 11:58
@bnjjj bnjjj disabled auto-merge June 6, 2025 11:58
@bnjjj bnjjj enabled auto-merge (squash) June 6, 2025 11:58
@bnjjj bnjjj merged commit 731fd23 into dev Jun 6, 2025
15 checks passed
@bnjjj bnjjj deleted the bnjjj/fix_telemetry_prom_resource branch June 6, 2025 12:04
bnjjj added a commit that referenced this pull request Jun 6, 2025
…7394)

Signed-off-by: Benjamin <5719034+bnjjj@users.noreply.github.com>
(cherry picked from commit 731fd23)

# Conflicts:
#	apollo-router/src/plugins/telemetry/metrics/prometheus.rs
#	apollo-router/src/plugins/telemetry/mod.rs
#	apollo-router/src/plugins/telemetry/snapshots/apollo_router__plugins__telemetry__tests__it_test_prometheus_metrics.snap
#	apollo-router/src/plugins/telemetry/snapshots/apollo_router__plugins__telemetry__tests__it_test_prometheus_metrics_units_are_included.snap
#	apollo-router/tests/integration/telemetry/metrics.rs
#	docs/source/routing/observability/telemetry/metrics-exporters/prometheus.mdx
@abernix abernix mentioned this pull request Jul 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-1.x Backport this PR to 1.x

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants