[Exporter.Prometheus] Make maximum buffer size configurable

[martincostello]

Changes

• Allow the maximum Prometheus buffer size to be configured to allow for scrape responses beyond 100,000 metrics.
• Return an HTTP 500 if the buffer is not large enough to contain the scrape response.

---

I was doing some testing and found that there was a practical and unavoidable limit of ~100,000 metric series being handled by the exporter before it hit the limits of the internal buffer length and would then just return empty scrape responses, regardless of the configured cardinality limits.

opentelemetry-dotnet/src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/Shared/PrometheusCollectionManager.cs

Line 788 in 8592991

private const int MaxBufferSize = 100 * 1024 * 1024; // 100 MB

With these changes from some testing, the exporter can now handle up to somewhere in the region of ~2.4 million series for PrometheusText format and somewhere in the region of ~1.8 million for OpenMetrics format.

The buffer length limit appears to have been an arbitrary cap added in #2610:

opentelemetry-dotnet/src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusCollectionManager.cs

Lines 188 to 197 in f94bc5f

	// there are two cases we might run into the following condition:
	// 1. we have many metrics to be exported - in this case we probably want
	// to put some upper limit and allow the user to configure it.
	// 2. we got an IndexOutOfRangeException which was triggered by some other
	// code instead of the buffer[cursor++] - in this case we should give up
	// at certain point rather than allocating like crazy.
	if (bufferSize > 100 * 1024 * 1024)
	{
	throw;
	}

Merge requirement checklist

• CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
• Unit tests added/updated
• Appropriate CHANGELOG.md files updated for non-trivial changes
• Changes in public API reviewed (if applicable)

[codecov]

Codecov Report

❌ Patch coverage is 95.91837% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.22%. Comparing base (0fd1382) to head (354dc37).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...ometheus.AspNetCore/PrometheusAspNetCoreOptions.cs	50.00%	1 Missing ⚠️
...ner/Internal/Shared/PrometheusCollectionManager.cs	92.30%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7487      +/-   ##
==========================================
+ Coverage   90.19%   90.22%   +0.02%     
==========================================
  Files         287      287              
  Lines       15584    15616      +32     
==========================================
+ Hits        14056    14089      +33     
+ Misses       1528     1527       -1     

Flag	Coverage Δ
unittests-Project-Experimental	90.20% <95.91%> (-0.01%)	⬇️
unittests-Project-Stable	90.21% <95.91%> (-0.03%)	⬇️
unittests-UnstableCoreLibraries-Experimental	50.96% <95.91%> (+0.27%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...metheus.AspNetCore/PrometheusExporterMiddleware.cs	100.00% <100.00%> (ø)
...HttpListener/Internal/Shared/PrometheusExporter.cs	100.00% <100.00%> (ø)
...r/Internal/Shared/PrometheusExporterEventSource.cs	50.00% <100.00%> (+1.85%)	⬆️
...tener/Internal/Shared/PrometheusExporterOptions.cs	100.00% <100.00%> (ø)
....Prometheus.HttpListener/PrometheusHttpListener.cs	88.81% <100.00%> (+0.28%)	⬆️
...theusHttpListenerMeterProviderBuilderExtensions.cs	83.33% <100.00%> (+0.47%)	⬆️
...heus.HttpListener/PrometheusHttpListenerOptions.cs	100.00% <100.00%> (ø)
...ometheus.AspNetCore/PrometheusAspNetCoreOptions.cs	50.00% <50.00%> (ø)
...ner/Internal/Shared/PrometheusCollectionManager.cs	92.37% <92.30%> (+1.64%)	⬆️

... and 5 files with indirect coverage changes

[Copilot]

Pull request overview

Adds a configurable upper bound for Prometheus scrape response buffering in both the HttpListener and ASP.NET Core exporters, and ensures oversized scrapes fail loudly (HTTP 500) instead of returning misleading empty 200 responses.

Changes:

• Introduces MaxScrapeResponseSizeBytes on PrometheusHttpListenerOptions and PrometheusAspNetCoreOptions (default ~166 MiB) and wires it through to the shared exporter implementation.
• Updates the collection/buffering logic to cap buffer growth at the configured maximum and to distinguish “successful but empty” vs “failed” collections.
• Returns HTTP 500 on collection failure due to response size limits, and adds/updates unit tests, changelogs, and public API tracking.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusHttpListenerTests.cs	Adds regression coverage ensuring HttpListener returns 500 when the configured max scrape response size is exceeded.
test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusCollectionManagerTests.cs	Adds coverage validating that MaxScrapeResponseSizeBytes bounds buffer growth and affects collection success/failure.
test/OpenTelemetry.Exporter.Prometheus.AspNetCore.Tests/PrometheusExporterMiddlewareTests.cs	Adds regression coverage ensuring ASP.NET Core middleware returns 500 when the configured max is exceeded.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListenerOptions.cs	Adds MaxScrapeResponseSizeBytes option (with validation) and sets a new default.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListenerMeterProviderBuilderExtensions.cs	Plumbs the new max-size option into the shared exporter options.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListener.cs	Returns HTTP 500 for failed collections and logs scrape failure.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/Shared/PrometheusExporterOptions.cs	Defines initial buffer size and default max-size constants; adds shared MaxScrapeResponseSizeBytes option.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/Shared/PrometheusExporterEventSource.cs	Adds a dedicated event for scrape collection failures.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/Shared/PrometheusExporter.cs	Carries the configured max-size option into the collection manager.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/Shared/PrometheusCollectionManager.cs	Implements max-bounded buffer growth and returns a success/failure flag for collections.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md	Documents the new option and the HTTP 500 behavior change.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/.publicApi/PublicAPI.Unshipped.txt	Tracks the newly added public API surface for HttpListener options.
src/OpenTelemetry.Exporter.Prometheus.AspNetCore/PrometheusExporterMiddleware.cs	Returns HTTP 500 for failed collections and logs scrape failure.
src/OpenTelemetry.Exporter.Prometheus.AspNetCore/PrometheusAspNetCoreOptions.cs	Adds MaxScrapeResponseSizeBytes to the ASP.NET Core options wrapper.
src/OpenTelemetry.Exporter.Prometheus.AspNetCore/CHANGELOG.md	Documents the new option and the HTTP 500 behavior change.
src/OpenTelemetry.Exporter.Prometheus.AspNetCore/.publicApi/PublicAPI.Unshipped.txt	Tracks the newly added public API surface for ASP.NET Core options.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Kielek]

LGTM with direct feedback