[Exporter.Prometheus] Improve performance#7170
Conversation
- Improve the performance of `PrometheusSerializer` for .NET 8+ by using `SearchValues<T>`. - Add fuzz tests for `PrometheusSerializer`.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #7170 +/- ##
==========================================
- Coverage 89.69% 89.59% -0.11%
==========================================
Files 272 272
Lines 13366 13590 +224
==========================================
+ Hits 11989 12176 +187
- Misses 1377 1414 +37
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Handle buffer resize.
Handle different exception type between .NET and .NET Framework.
There was a problem hiding this comment.
Pull request overview
Improves the runtime performance of the Prometheus HttpListener exporter’s serialization path on .NET 8+ by using SearchValues<T>-based scanning and span-based UTF-8 writes, and adds new fuzz/property tests to validate serializer behavior.
Changes:
- Optimize
PrometheusSerializerstring/number writing on .NET 8+ usingSearchValues<char>andEncoding.UTF8.GetBytes(ReadOnlySpan<char>, Span<byte>). - Update collection retry logic to also treat
ArgumentExceptionas a “buffer too small” signal (in addition toIndexOutOfRangeException). - Add new unit tests and a new FsCheck-based fuzz test project for
PrometheusSerializer.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusSerializerTests.cs | Adds focused tests for ASCII writing, escaping behavior, and UTF-16 code-unit preservation. |
| test/OpenTelemetry.Exporter.Prometheus.HttpListener.FuzzTests/PrometheusSerializerFuzzTests.cs | Adds property-based tests comparing serializer outputs to reference implementations. |
| test/OpenTelemetry.Exporter.Prometheus.HttpListener.FuzzTests/OpenTelemetry.Exporter.Prometheus.HttpListener.FuzzTests.csproj | Introduces new fuzz test project referencing the HttpListener exporter and FsCheck.Xunit. |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/OpenTelemetry.Exporter.Prometheus.HttpListener.csproj | Adds InternalsVisibleTo for the new fuzz test assembly. |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs | Implements .NET 8+ optimized escaping/writing using SearchValues<char> and span-based UTF-8 encoding. |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs | Broadens retryable exceptions to include ArgumentException (buffer too small from span-based UTF-8 encoding). |
| OpenTelemetry.slnx | Adds the new fuzz test project to the solution. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Update comment.
Improve `PrometheusSerializer` performance further by: - Writing common label value types directly instead of relying on `ToString()` - Formatting numeric values directly as UTF-8 on .NET 8+ - Caching serialized metric names, metadata names, units, and static tag prefixes - Reusing serialized tags across histogram bucket/sum/count output - Use `stackalloc` for small temporary tag buffers on .NET 8+
This comment was marked as outdated.
This comment was marked as outdated.
Add missing patch coverage.
Encode the strings.
Add missing coverage for static labels.
Add remaining (easy) coverage and remove unused code.
Remove now-unused parameters from benchmarks.
This comment was marked as resolved.
This comment was marked as resolved.
Use helper for ASCII checking.
Fix some test changes that were removed during merge.
Fix broken UTF-8 tests.
Factor away some duplicated code paths.
Tidy up some of the formatting.
Add polyfills for `char`'s `IsAsciiDigit`, `IsAsciiLetterOrDigit` and `IsAsciiLetterLower` methods and remove helpers.
Latest BenchmarksTL;DRThis PR compared to main serializes metrics:
Percentages vary by scenario. Copilot SummaryBenchmark summary:
|
| Runtime | Method | Calls | Duration (main -> perf) |
Duration change | Duration ratio | Allocations (main -> perf) |
Allocation change | Allocation ratio |
|---|---|---|---|---|---|---|---|---|
| .NET 10.0 | WriteMetric | 1 | 4046.1 ns -> 1464.5 ns | -63.8% | 0.36x | 104 B -> 0 B | -100.0% | 0.00x |
| .NET 10.0 | WriteMetric | 1000 | 4010806.8 ns -> 1456272.4 ns | -63.7% | 0.36x | 104000 B -> 0B | -100.0% | 0.00x |
| .NET 10.0 | WriteMetric | 10000 | 40214670.5 ns -> 15353037.5 ns | -61.8% | 0.38x | 1040000 B -> 13 B | -99.9988% | 0.00001x |
| .NET 10.0 | WriteHistogramMetric | 1 | 3411.9 ns -> 1011.9 ns | -70.3% | 0.30x | 0 B -> 0 B | 0.0% | 1.00x |
| .NET 10.0 | WriteHistogramMetric | 1000 | 3238743.2 ns -> 1024652.2 ns | -68.4% | 0.32x | 0 B -> 0 B | 0.0% | 1.00x |
| .NET 10.0 | WriteHistogramMetric | 10000 | 32792324.8 ns -> 10747484.1 ns | -67.2% | 0.33x | 0 B -> 0 B | 0.0% | 1.00x |
| .NET 10.0 | WriteMetricWithTypedLabels | 1 | 331.3 ns -> 178.4 ns | -46.2% | 0.54x | 104 B -> 0 B | -100.0% | 0.00x |
| .NET 10.0 | WriteMetricWithTypedLabels | 1000 | 334419.5 ns -> 182936.2 ns | -45.3% | 0.55x | 104000 B -> 0 B | -100.0% | 0.00x |
| .NET 10.0 | WriteMetricWithTypedLabels | 10000 | 3279539.7 ns -> 1940527.0 ns | -40.8% | 0.59x | 1040000 B -> 0 B | -100.0% | 0.00x |
| .NET Framework 4.6.2 | WriteMetric | 1 | 9941.7 ns -> 5508.3 ns | -44.6% | 0.55x | 1492 B -> 1492 B | 0.0% | 1.00x |
| .NET Framework 4.6.2 | WriteMetric | 1000 | 9849356.6 ns -> 6332700.4 ns | -35.7% | 0.64x | 1492365 B -> 1492366 B | +0.00007% | 1.00x |
| .NET Framework 4.6.2 | WriteMetric | 10000 | 96584934.4 ns -> 62290357.0 ns | -35.5% | 0.64x | 14923224 B -> 14923223 B | -0.000007% | 1.00x |
| .NET Framework 4.6.2 | WriteHistogramMetric | 1 | 7650.7 ns -> 3948.3 ns | -48.4% | 0.52x | 1147 B -> 1147 B | 0.0% | 1.00x |
| .NET Framework 4.6.2 | WriteHistogramMetric | 1000 | 7491491.4 ns -> 4271289.0 ns | -43.0% | 0.57x | 1147338 B -> 1147338 B | 0.0% | 1.00x |
| .NET Framework 4.6.2 | WriteHistogramMetric | 10000 | 76410881.0 ns -> 43874466.7 ns | -42.6% | 0.57x | 11473577 B -> 11473677 B | +0.00087% | 1.00x |
| .NET Framework 4.6.2 | WriteMetricWithTypedLabels | 1 | 731.9 ns -> 624.0 ns | -14.7% | 0.85x | 185 B -> 185 B | 0.0% | 1.00x |
| .NET Framework 4.6.2 | WriteMetricWithTypedLabels | 1000 | 741720.2 ns -> 659912.7 ns | -11.0% | 0.89x | 184538 B -> 184538 B | 0.0% | 1.00x |
| .NET Framework 4.6.2 | WriteMetricWithTypedLabels | 10000 | 7470659.1 ns -> 6701965.1 ns | -10.3% | 0.90x | 1845397 B -> 1845397 B | 0.0% | 1.00x |
Benchmark Results
Expand to view
main
BenchmarkDotNet v0.15.8, Windows 11 (10.0.26200.8246/25H2/2025Update/HudsonValley2)
13th Gen Intel Core i7-13700H 2.90GHz, 1 CPU, 20 logical and 14 physical cores
.NET SDK 10.0.203
[Host] : .NET 10.0.7 (10.0.7, 10.0.726.21808), X64 RyuJIT x86-64-v3
.NET 10.0 : .NET 10.0.7 (10.0.7, 10.0.726.21808), X64 RyuJIT x86-64-v3
.NET Framework 4.6.2 : .NET Framework 4.8.1 (4.8.9325.0), X64 RyuJIT VectorSize=256
| Method | Job | Runtime | NumberOfSerializeCalls | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Allocated | Alloc Ratio |
|---|---|---|---|---|---|---|---|---|---|---|---|
| WriteMetric | .NET 10.0 | .NET 10.0 | 1 | 4,046.1 ns | 80.81 ns | 141.54 ns | 1.00 | 0.05 | 0.0076 | 104 B | 1.00 |
| WriteMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1 | 9,941.7 ns | 88.45 ns | 82.74 ns | 2.46 | 0.09 | 0.2289 | 1492 B | 14.35 |
| WriteHistogramMetric | .NET 10.0 | .NET 10.0 | 1 | 3,411.9 ns | 5.02 ns | 3.92 ns | 1.00 | 0.00 | - | - | NA |
| WriteHistogramMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1 | 7,650.7 ns | 78.25 ns | 73.19 ns | 2.24 | 0.02 | 0.1678 | 1147 B | NA |
| WriteMetricWithTypedLabels | .NET 10.0 | .NET 10.0 | 1 | 331.3 ns | 3.90 ns | 3.65 ns | 1.00 | 0.02 | 0.0081 | 104 B | 1.00 |
| WriteMetricWithTypedLabels | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1 | 731.9 ns | 9.84 ns | 9.21 ns | 2.21 | 0.04 | 0.0286 | 185 B | 1.78 |
| WriteMetric | .NET 10.0 | .NET 10.0 | 1000 | 4,010,806.8 ns | 78,737.36 ns | 133,702.02 ns | 1.00 | 0.05 | 7.8125 | 104000 B | 1.00 |
| WriteMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1000 | 9,849,356.6 ns | 36,155.93 ns | 30,191.84 ns | 2.46 | 0.08 | 234.3750 | 1492365 B | 14.35 |
| WriteHistogramMetric | .NET 10.0 | .NET 10.0 | 1000 | 3,238,743.2 ns | 9,025.53 ns | 8,442.49 ns | 1.00 | 0.00 | - | - | NA |
| WriteHistogramMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1000 | 7,491,491.4 ns | 84,946.24 ns | 79,458.77 ns | 2.31 | 0.02 | 179.6875 | 1147338 B | NA |
| WriteMetricWithTypedLabels | .NET 10.0 | .NET 10.0 | 1000 | 334,419.5 ns | 3,451.25 ns | 3,228.30 ns | 1.00 | 0.01 | 7.8125 | 104000 B | 1.00 |
| WriteMetricWithTypedLabels | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1000 | 741,720.2 ns | 4,232.27 ns | 3,751.80 ns | 2.22 | 0.02 | 29.2969 | 184538 B | 1.77 |
| WriteMetric | .NET 10.0 | .NET 10.0 | 10000 | 40,214,670.5 ns | 791,163.59 ns | 1,056,180.76 ns | 1.00 | 0.04 | 76.9231 | 1040000 B | 1.00 |
| WriteMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 10000 | 96,584,934.4 ns | 246,936.38 ns | 230,984.44 ns | 2.40 | 0.06 | 2333.3333 | 14923224 B | 14.35 |
| WriteHistogramMetric | .NET 10.0 | .NET 10.0 | 10000 | 32,792,324.8 ns | 642,673.18 ns | 921,702.74 ns | 1.00 | 0.04 | - | - | NA |
| WriteHistogramMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 10000 | 76,410,881.0 ns | 210,803.87 ns | 197,186.07 ns | 2.33 | 0.07 | 1714.2857 | 11473577 B | NA |
| WriteMetricWithTypedLabels | .NET 10.0 | .NET 10.0 | 10000 | 3,279,539.7 ns | 25,928.04 ns | 22,984.53 ns | 1.00 | 0.01 | 82.0313 | 1040000 B | 1.00 |
| WriteMetricWithTypedLabels | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 10000 | 7,470,659.1 ns | 49,956.53 ns | 41,715.97 ns | 2.28 | 0.02 | 289.0625 | 1845397 B | 1.77 |
This PR
BenchmarkDotNet v0.15.8, Windows 11 (10.0.26200.8246/25H2/2025Update/HudsonValley2)
13th Gen Intel Core i7-13700H 2.90GHz, 1 CPU, 20 logical and 14 physical cores
.NET SDK 10.0.203
[Host] : .NET 10.0.7 (10.0.7, 10.0.726.21808), X64 RyuJIT x86-64-v3
.NET 10.0 : .NET 10.0.7 (10.0.7, 10.0.726.21808), X64 RyuJIT x86-64-v3
.NET Framework 4.6.2 : .NET Framework 4.8.1 (4.8.9325.0), X64 RyuJIT VectorSize=256
| Method | Job | Runtime | NumberOfSerializeCalls | Mean | Error | StdDev | Median | Ratio | RatioSD | Gen0 | Allocated | Alloc Ratio |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| WriteMetric | .NET 10.0 | .NET 10.0 | 1 | 1,464.5 ns | 28.59 ns | 50.82 ns | 1,455.9 ns | 1.00 | 0.05 | - | - | NA |
| WriteMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1 | 5,508.3 ns | 109.06 ns | 176.11 ns | 5,424.7 ns | 3.77 | 0.17 | 0.2365 | 1492 B | NA |
| WriteHistogramMetric | .NET 10.0 | .NET 10.0 | 1 | 1,011.9 ns | 19.88 ns | 33.22 ns | 991.0 ns | 1.00 | 0.05 | - | - | NA |
| WriteHistogramMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1 | 3,948.3 ns | 78.58 ns | 143.69 ns | 3,902.1 ns | 3.91 | 0.19 | 0.1755 | 1147 B | NA |
| WriteMetricWithTypedLabels | .NET 10.0 | .NET 10.0 | 1 | 178.4 ns | 3.54 ns | 6.10 ns | 180.2 ns | 1.00 | 0.05 | - | - | NA |
| WriteMetricWithTypedLabels | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1 | 624.0 ns | 12.49 ns | 20.51 ns | 634.8 ns | 3.50 | 0.16 | 0.0286 | 185 B | NA |
| WriteMetric | .NET 10.0 | .NET 10.0 | 1000 | 1,456,272.4 ns | 28,990.07 ns | 47,631.52 ns | 1,428,828.1 ns | 1.00 | 0.05 | - | - | NA |
| WriteMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1000 | 6,332,700.4 ns | 71,098.90 ns | 66,505.95 ns | 6,296,635.2 ns | 4.35 | 0.14 | 234.3750 | 1492366 B | NA |
| WriteHistogramMetric | .NET 10.0 | .NET 10.0 | 1000 | 1,024,652.2 ns | 20,310.62 ns | 33,934.48 ns | 1,003,637.6 ns | 1.00 | 0.05 | - | - | NA |
| WriteHistogramMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1000 | 4,271,289.0 ns | 82,968.91 ns | 81,486.57 ns | 4,223,455.1 ns | 4.17 | 0.16 | 179.6875 | 1147338 B | NA |
| WriteMetricWithTypedLabels | .NET 10.0 | .NET 10.0 | 1000 | 182,936.2 ns | 3,652.25 ns | 5,577.37 ns | 184,753.9 ns | 1.00 | 0.04 | - | - | NA |
| WriteMetricWithTypedLabels | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 1000 | 659,912.7 ns | 6,925.20 ns | 6,477.84 ns | 663,846.5 ns | 3.61 | 0.11 | 29.2969 | 184538 B | NA |
| WriteMetric | .NET 10.0 | .NET 10.0 | 10000 | 15,353,037.5 ns | 304,241.36 ns | 711,154.41 ns | 15,865,493.8 ns | 1.00 | 0.07 | - | 13 B | 1.00 |
| WriteMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 10000 | 62,290,357.0 ns | 175,519.99 ns | 164,181.51 ns | 62,289,211.1 ns | 4.07 | 0.19 | 2333.3333 | 14923223 B | 1,147,940.23 |
| WriteHistogramMetric | .NET 10.0 | .NET 10.0 | 10000 | 10,747,484.1 ns | 44,403.20 ns | 34,667.10 ns | 10,741,144.5 ns | 1.00 | 0.00 | - | - | NA |
| WriteHistogramMetric | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 10000 | 43,874,466.7 ns | 59,774.99 ns | 49,914.83 ns | 43,880,050.0 ns | 4.08 | 0.01 | 1750.0000 | 11473677 B | NA |
| WriteMetricWithTypedLabels | .NET 10.0 | .NET 10.0 | 10000 | 1,940,527.0 ns | 26,452.40 ns | 24,743.59 ns | 1,924,872.9 ns | 1.00 | 0.02 | - | - | NA |
| WriteMetricWithTypedLabels | .NET Framework 4.6.2 | .NET Framework 4.6.2 | 10000 | 6,701,965.1 ns | 84,667.47 ns | 79,198.00 ns | 6,655,671.1 ns | 3.45 | 0.06 | 289.0625 | 1845397 B | NA |
|
Will rebase on top of #7194. |
There was a problem hiding this comment.
Pull request overview
This PR focuses on improving the runtime performance of the Prometheus HttpListener exporter’s serialization path (especially PrometheusSerializer) by reducing per-write overhead and enabling more reuse/caching, and it adds fuzz testing to validate serializer behavior.
Changes:
- Optimizes serialization by using UTF-8 formatting APIs on .NET, caching pre-serialized metric/name/unit/static-tag bytes, and reusing serialized tags across histogram outputs.
- Adds object label-value formatting paths (common primitives + invariant-culture formatting) and corresponding unit tests.
- Adds fuzz tests that compare serializer output against a reference implementation.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusSerializerTests.cs | Adds/adjusts unit tests for object label-value formatting and invariant-culture behavior. |
| test/OpenTelemetry.Exporter.Prometheus.HttpListener.FuzzTests/PrometheusSerializerFuzzTests.cs | Adds fuzz coverage for object label-value serialization against a reference implementation. |
| src/Shared/CharExtensions.cs | Introduces a shared shim for char.IsAscii* APIs on non-NET TFMs. |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/OpenTelemetry.Exporter.Prometheus.HttpListener.csproj | Links the new shared CharExtensions shim into the HttpListener exporter project. |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializerExt.cs | Refactors metric writing (notably histogram) to reuse/copy pre-serialized tag bytes and rent buffers when needed. |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs | Major serializer changes: UTF-8 direct formatting, escaped-string fast paths, static-tag caching, and new object label-value overload. |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusMetric.cs | Caches ASCII byte representations of metric names/metadata/unit and stores serialized static tags. |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs | Expands buffer-growth exception handling to include ArgumentException as well as IndexOutOfRangeException. |
| src/OpenTelemetry.Exporter.Prometheus.AspNetCore/OpenTelemetry.Exporter.Prometheus.AspNetCore.csproj | Links the new shared CharExtensions shim into the ASP.NET Core exporter project. |
| src/OpenTelemetry.Api/OpenTelemetry.Api.csproj | Links the new shared CharExtensions shim into the API project (for char.IsAscii* usage). |
| src/OpenTelemetry.Api/Context/Propagation/TraceStateUtils.cs | Switches ASCII validation logic to char.IsAscii* APIs. |
| src/OpenTelemetry.Api/Context/Propagation/TraceContextPropagator.cs | Switches some ASCII validation logic to char.IsAscii* APIs. |
| OpenTelemetry.slnx | Adds the new shared CharExtensions.cs file to the solution. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Throw if buffer is too small. - Fix inconsistent handling of formatting and Unicode.
Update TODO comments related to JSON.
If this is only helpful for above, then I dont think it's worth it, given it is a non prod component. If this is in shared code and benefits aspnetcore prometheus, then definitely worth it. |
|
It applies to both. |
@cijothomas, even if it's not supported, it's definitely used in production. Thanks, @martincostello! |
Use `Task.WaitAsync()`.
|
Superseded by #7279. |
Changes
PrometheusSerializerby:SearchValues<T>on .NET 8+ToString()stackallocfor small temporary tag buffers on .NET 8+Benchmark Results
See #7170 (comment) for latest results.
Details
Copilot Summary
PrometheusSerializer-perfis 0.833x geometric mean versusmain.Per-config summary
NumberOfSerializeCalls=1WriteMetricNumberOfSerializeCalls=1000WriteMetricNumberOfSerializeCalls=10000WriteMetricTakeaways
PrometheusSerializer-perf shows consistent runtime improvement.
Detailed Results
Expand to view
main
This PR
Merge requirement checklist
AppropriateCHANGELOG.mdfiles updated for non-trivial changesChanges in public API reviewed (if applicable)