Skip to content

[Exporter.Prometheus] Reduce allocations#7465

Merged
martincostello merged 31 commits into
open-telemetry:mainfrom
martincostello:cache-label-collections
Jul 3, 2026
Merged

[Exporter.Prometheus] Reduce allocations#7465
martincostello merged 31 commits into
open-telemetry:mainfrom
martincostello:cache-label-collections

Conversation

@martincostello

@martincostello martincostello commented Jun 25, 2026

Copy link
Copy Markdown
Member

Changes

Reduce allocations by caching scope labels and their data for each metric.

Discovered while benchmarking v1.15.2 against v1.16.0.

Builds on top of #7440.

See #7465 (comment) for benchmark results

Old benchmark results

Indicative improvements:

Prometheus serializer: allocated bytes per call (net10.0, Default job)

Benchmark 1.15.2 1.16.0 PR #7440 #7440 + this this vs 1.16.0 this vs #7440
WriteMetric (4 metrics) 104 4,856 3,360 736 −84.8% −78.1%
WriteHistogramMetric 0 1,448 1,128 472 −67.4% −58.2%
WriteMetricWithTypedLabels 104 1,344 744 88 −93.5% −88.2%

Prometheus serializer: time per call in ns (net10.0, Default job, @1000 scale)

Benchmark 1.15.2 1.16.0 PR #7440 #7440 + this this vs 1.16.0 this vs 1.15.2
WriteMetric 3,364 3,285 3,165 2,536 −22.8% −24.6%
WriteHistogramMetric 2,804 1,667 1,670 1,527 −8.4% −45.5%
WriteMetricWithTypedLabels 278 565 515 469 −17.0% +68.7%

Merge requirement checklist

  • CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
  • Unit tests added/updated
  • Appropriate CHANGELOG.md files updated for non-trivial changes
  • Changes in public API reviewed (if applicable)

Add missing `in` modifiers on methods accepting parameters of type `PrometheusProtocol`.
Add support for the `dots` and `values` escaping schemes.

Contributes to open-telemetry#7246.
Fix incorrectly sorted `[InlineData]` attributes.
- Update comments.
- Add `Debug.Assert()`.
- Add PR number to CHANGELOGs.
- Add missing coverage.
- Remove unused code.
Add support for the `allow-utf-8` escaping scheme.

Fixes open-telemetry#7246.
Add a new benchmark that compares the different text formats and escaping schemes.
Avoid allocations when a label does not need to be sanitized.
Make lines less long.
Further avoid allocations when writing labels.
Move entries to the right place.
Reduce allocations by caching scope labels and their data for each metric.
@github-actions github-actions Bot added pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package pkg:OpenTelemetry.Exporter.Prometheus.HttpListener Issues related to OpenTelemetry.Exporter.Prometheus.HttpListener NuGet package perf Performance related labels Jun 25, 2026
@codecov

codecov Bot commented Jun 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.28%. Comparing base (68c29d8) to head (d55091b).
✅ All tests successful. No failed tests found.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #7465      +/-   ##
==========================================
+ Coverage   90.18%   90.28%   +0.09%     
==========================================
  Files         287      281       -6     
  Lines       15553    15031     -522     
==========================================
- Hits        14027    13571     -456     
+ Misses       1526     1460      -66     
Flag Coverage Δ
unittests-PowerShellScripts ?
unittests-Project-Experimental 90.13% <100.00%> (-0.08%) ⬇️
unittests-Project-Stable 90.13% <100.00%> (-0.11%) ⬇️
unittests-UnstableCoreLibraries-Experimental 50.58% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ernal/Shared/Serialization/TextFormatSerializer.cs 99.49% <100.00%> (+<0.01%) ⬆️

... and 13 files with indirect coverage changes

Add missing `this`.
- Handle label vs. metric name escaping correctly.
- Fix colliding exemplars with different escapings.
- Fix scope labels not being escaped.
@martincostello martincostello added the keep-open Prevents issues and pull requests being closed as stale label Jul 2, 2026
@github-actions github-actions Bot removed the pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package label Jul 3, 2026
@martincostello

Copy link
Copy Markdown
Member Author

Just need to re-run the benchmarks after merging latest changes.

@martincostello

Copy link
Copy Markdown
Member Author

Allocations (deterministic — the metric this change targets)

PrometheusSerializerBenchmarks (per serialize call):

Method main branch Δ
WriteMetric 3,360 B 736 B −78%
WriteHistogramMetric 1,128 B 472 B −58%
WriteMetricWithTypedLabels 744 B 88 B −88%

PrometheusSerializerEscapingBenchmarks (allocation per WriteMetrics call, summing 4 metrics), all 16 format × escaping combos:

Format main branch Δ
PrometheusText V0 (all schemes) 4.72 KB 1.52 KB −68%
PrometheusText V1 4.72–4.89 KB 1.52–1.69 KB −65% to −68%
OpenMetrics V0 6.93 KB 1.80 KB −74%
OpenMetrics V1 6.99–7.16 KB 1.80–1.98 KB −72% to −74%

A consistent ~3.2 KB drop per call across every format and escaping scheme — confirming the caching removes the per-call scope-label collection rebuilds regardless of the negotiated escaping, exactly as intended.

@martincostello
martincostello marked this pull request as ready for review July 3, 2026 10:46
@martincostello
martincostello requested a review from a team as a code owner July 3, 2026 10:46
Copilot AI review requested due to automatic review settings July 3, 2026 10:46
@martincostello martincostello removed the keep-open Prevents issues and pull requests being closed as stale label Jul 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces Prometheus text serialization allocations by caching the computed otel_scope_* label collections per Metric, so they can be reused across repeated WriteMetric calls rather than rebuilt every time.

Changes:

  • Added ConditionalWeakTable caches for scope label LabelData and the merged scope label key/value pairs, keyed by Metric.
  • Switched serialization paths to use cached scope labels via GetScopeLabelData / GetScopeLabels.
  • Introduced small helper methods to encapsulate cache access while keeping existing scope-label construction logic intact.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

private static readonly ConditionalWeakTable<Metric, List<KeyValuePair<string, string>>> ScopeLabelsCache = [];
private static readonly ConditionalWeakTable<Metric, List<LabelData>> ScopeLabelDataCache = [];
#else
private static readonly ConditionalWeakTable<Metric, List<KeyValuePair<string, string>>> ScopeLabelsCache = new();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note for myself: [] is not working on .NET Fx for this case.

@martincostello
martincostello added this pull request to the merge queue Jul 3, 2026
Merged via the queue into open-telemetry:main with commit fc89dbe Jul 3, 2026
77 checks passed
@martincostello
martincostello deleted the cache-label-collections branch July 3, 2026 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

perf Performance related pkg:OpenTelemetry.Exporter.Prometheus.HttpListener Issues related to OpenTelemetry.Exporter.Prometheus.HttpListener NuGet package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants