Skip to content

[Exporter.Prometheus] Fix scope metadata#7237

Merged
martincostello merged 25 commits into
open-telemetry:mainfrom
martincostello:fix-scope-metadata
Jun 3, 2026
Merged

[Exporter.Prometheus] Fix scope metadata#7237
martincostello merged 25 commits into
open-telemetry:mainfrom
martincostello:fix-scope-metadata

Conversation

@martincostello
Copy link
Copy Markdown
Member

@martincostello martincostello commented May 1, 2026

Changes

Scope-related fixes for OpenMetrics and Prometheus specification compliance.

  • Include instrumentation scope metadata on samples using otel_scope_* labels, including scope version, schema URL, and prefixed scope attributes.
  • OpenMetrics output no longer emits a separate otel_scope_info scope metadata metric.
  • Drop conflicting scope attributes named name, version, and schema_url to avoid collisions with generated scope labels.

Merge requirement checklist

  • CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
  • Unit tests added/updated
  • Appropriate CHANGELOG.md files updated for non-trivial changes
  • Changes in public API reviewed (if applicable)

- Emit OpenMetrics scope metadata as a single `otel_scope` metric family with `otel_scope_info` samples instead of repeating metadata for every scope.
- Include instrumentation scope metadata on samples using `otel_scope_*` labels, including scope version, schema URL, and prefixed scope attributes.
- Drop conflicting scope attributes named `name`, `version`, and `schema_url` to avoid collisions with generated scope labels.
@github-actions github-actions Bot added pkg:OpenTelemetry.Exporter.Prometheus.HttpListener Issues related to OpenTelemetry.Exporter.Prometheus.HttpListener NuGet package pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package labels May 1, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented May 1, 2026

Codecov Report

❌ Patch coverage is 93.75000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.03%. Comparing base (78f6d59) to head (06d421f).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...heus.HttpListener/Internal/PrometheusSerializer.cs 93.44% 4 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #7237      +/-   ##
==========================================
- Coverage   90.04%   90.03%   -0.02%     
==========================================
  Files         276      276              
  Lines       14271    14283      +12     
==========================================
+ Hits        12850    12859       +9     
- Misses       1421     1424       +3     
Flag Coverage Δ
unittests-Project-Experimental 89.92% <93.75%> (-0.02%) ⬇️
unittests-Project-Stable 89.84% <93.75%> (-0.09%) ⬇️
unittests-UnstableCoreLibraries-Experimental 48.62% <93.75%> (+0.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...tpListener/Internal/PrometheusCollectionManager.cs 93.52% <100.00%> (+1.21%) ⬆️
...heus.HttpListener/Internal/PrometheusSerializer.cs 97.65% <93.44%> (+0.13%) ⬆️

... and 3 files with indirect coverage changes

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Prometheus/OpenMetrics exporter serialization to be compliant with OpenMetrics/Prometheus scope metadata expectations by emitting scope metadata once per scrape and attaching instrumentation scope details to metric samples.

Changes:

  • Emit OpenMetrics scope metadata as a single otel_scope metric family (with otel_scope_info samples) instead of repeating TYPE/HELP per scope.
  • Add instrumentation scope metadata onto metric samples using otel_scope_* labels (version, schema URL, and prefixed scope attributes).
  • Update/extend unit + integration tests and exporter changelogs to reflect the new output.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusSerializerTests.cs Updates serializer expectations and adds tests for scope-label prefixing and reserved-name dropping.
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusHttpListenerTests.cs Updates integration expected output for new scope metric family + prefixed scope attributes.
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusCollectionManagerTests.cs Adds coverage ensuring OpenMetrics scope metadata is written once as a single metric family.
test/OpenTelemetry.Exporter.Prometheus.AspNetCore.Tests/PrometheusExporterMiddlewareTests.cs Updates middleware integration expectations for new scope metric family + prefixed scope attributes.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializerExt.cs Adjusts histogram bucket serialization to account for WriteTags no longer leaving a trailing comma.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs Implements scope-metadata family split (metadata vs samples) and writes scope labels (version/schema/tags) with otel_scope_ prefix.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs Emits scope metadata once and deduplicates scope identities across metrics during OpenMetrics scrapes.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md Documents the scope-metadata and scope-label behavior changes.
src/OpenTelemetry.Exporter.Prometheus.AspNetCore/CHANGELOG.md Documents the scope-metadata and scope-label behavior changes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@martincostello martincostello marked this pull request as ready for review May 1, 2026 21:37
@martincostello martincostello requested a review from a team as a code owner May 1, 2026 21:37
Add more test coverage for patch.
@martincostello martincostello enabled auto-merge May 4, 2026 09:29
@martincostello martincostello added the keep-open Prevents issues and pull requests being closed as stale label May 8, 2026
@Kielek
Copy link
Copy Markdown
Member

Kielek commented May 19, 2026

From Codex:

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs:664
Scope attribute label collisions are not handled. TryCreateScopeLabel normalizes each scope attribute independently, so tags like library.mascot and library-mascot both serialize as otel_scope_library_mascot, producing duplicate label names. The OTel Prometheus compatibility spec says colliding converted labels must be concatenated with ;, ordered by original key.

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs:264
The synthetic otel_scope metric family bypasses existing metadata conflict handling. If user code exports a metric whose OpenMetrics metadata name is otel_scope, the scrape can contain both # TYPE otel_scope info and the user metric’s # TYPE otel_scope ..., violating the existing “no duplicate/conflicting TYPE comments” rule. This should be reserved/seeded into the same conflict logic that handles real metrics.

- Fix missing handling for post-normalization collisions.
- Fix metadata conflict handling being bypassed in some cases.
@martincostello
Copy link
Copy Markdown
Member Author

Both comments addressed.

Comment thread src/OpenTelemetry.Exporter.Prometheus.AspNetCore/CHANGELOG.md Outdated
Comment thread src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md Outdated
@Kielek
Copy link
Copy Markdown
Member

Kielek commented May 26, 2026

2 new codex findings:
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs:422
The new reservation only drops user metrics whose OpenMetrics metadata name is exactly otel_scope, but the synthetic info family also emits the sample name otel_scope_info. A user metric such as otel.scope.info still sanitizes to a separate otel_scope_info family and emits an otel_scope_info{...} sample, clashing with the synthetic info sample. OpenMetrics explicitly calls out suffix-based sample-name clashes for info families, so this should reserve/drop otel_scope_info as well when scope metadata is emitted.

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs:441
Scope labels are built and written before metric point tags, but they are not merged with the point labels. A scope attribute like library.mascot produces otel_scope_library_mascot; a metric point attribute already named otel_scope_library_mascot will then produce duplicate label names in the same label set. The OTel compatibility spec says scope attributes follow the Metric Attributes rules, and collisions with labels added by the spec must concatenate values with ;.

Use canonical values for labels that are `float` or `double` to resolve TODO.
- Ensure `otel_scope_info` is sanitized.
- Fix missing collision handling.
[Theory]
[InlineData(false)]
[InlineData(true)]
public void WriteMetricConcatenatesPointTagsThatCollideWithScopeLabels(bool useOpenMetrics)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more findings against spec complianace:

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs:725
CreateMetricPointLabels now concatenates point-tag collisions with existing scope labels, but it always appends the point value after the existing value. The OTel Prometheus compatibility spec requires colliding values to be ordered by the lexicographical order of the original keys. This works for the current test case, but fails for cases like scope attribute z="scope" plus point attribute otel_scope_z="point": both serialize to otel_scope_z, and the required order is point;scope, while this code emits scope;point.

This test could be changed to something like

[Theory]
[InlineData(false, "library.mascot", "dotnetbot", "otel_scope_library_mascot", "otter", "otel_scope_library_mascot", "dotnetbot;otter")]
[InlineData(true, "library.mascot", "dotnetbot", "otel_scope_library_mascot", "otter", "otel_scope_library_mascot", "dotnetbot;otter")]
[InlineData(false, "z", "scope", "otel_scope_z", "point", "otel_scope_z", "point;scope")]
[InlineData(true, "z", "scope", "otel_scope_z", "point", "otel_scope_z", "point;scope")]
public void WriteMetricConcatenatesPointTagsThatCollideWithScopeLabels(
    bool useOpenMetrics,
    string scopeTagKey,
    string scopeTagValue,
    string pointTagKey,
    string pointTagValue,
    string expectedLabelKey,
    string expectedLabelValue)
{

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be addressed now.

# Conflicts:
#	src/OpenTelemetry.Exporter.Prometheus.AspNetCore/CHANGELOG.md
#	src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md
#	src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs
Update PrometheusSerializer so scope-tag collisions with point tags now preserve the scope tag’s raw original key through the final merge, which fixes lexicographic value ordering for cases like `z` vs. `otel_scope_z`.
Remove synthetic OpenMetrics `otel_scope`/`otel_scope_info` metadata family.

#if !NET
#if NET
private static readonly ImmutableHashSet<string> ReservedScopeLabelNames = ["otel_scope_name", "otel_scope_schema_url", "otel_scope_version"];
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FrozenSet should be more performant

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

edit: alternatively, all these 3 items could be made a cosnt, and the comparision could be done by swithc/case/==. Depends on the level of optimization you need

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added where available.

@Kielek Kielek disabled auto-merge June 3, 2026 09:37
Copy link
Copy Markdown
Member

@Kielek Kielek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with some minor comment. Auto merge disabled dur to this potential change.

Use `FrozenSet<T>` for .NET 9+.
@martincostello martincostello enabled auto-merge June 3, 2026 09:39
@martincostello martincostello added this pull request to the merge queue Jun 3, 2026
@martincostello martincostello removed the keep-open Prevents issues and pull requests being closed as stale label Jun 3, 2026
Merged via the queue into open-telemetry:main with commit a3e996c Jun 3, 2026
75 checks passed
@martincostello martincostello deleted the fix-scope-metadata branch June 3, 2026 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package pkg:OpenTelemetry.Exporter.Prometheus.HttpListener Issues related to OpenTelemetry.Exporter.Prometheus.HttpListener NuGet package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants