[Prometheus.HttpListener] Fix shutdown hang - #7587
Conversation
Avoid disposal race in `PrometheusHttpListener` that can cause shutdown to hang indefinitely.
Pull request dashboard statusMerged · refreshed 2026-08-04 21:27 UTC Status above doesn't look right?
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7587 +/- ##
==========================================
- Coverage 90.51% 90.49% -0.02%
==========================================
Files 290 290
Lines 16029 16039 +10
==========================================
+ Hits 14509 14515 +6
- Misses 1520 1524 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
6 runs with no Prometheus related failures. There was one failure here on attempt 5, but that has no specific cause logged but appears to be within OpenTelemetry.Tests. |
There was a problem hiding this comment.
Pull request overview
This PR addresses a shutdown hang in the Prometheus HttpListener exporter by bounding shutdown waits and ensuring in-flight scrapes fail fast (HTTP 503) once disposal begins, with accompanying test hardening to avoid race-driven flakiness.
Changes:
- Add bounded timeouts to listener shutdown/drain logic and log when the processing loop fails to stop promptly.
- Ensure scrapes that are still collecting when disposal starts return HTTP 503 instead of completing with a successful response.
- Harden integration/unit tests by adding explicit scrape/promtool timeouts and avoiding thread-pool scheduling races during disposal.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PromToolFixture.cs | Adds scrape and overall promtool execution timeouts to prevent hangs in container-based validation. |
| test/OpenTelemetry.Exporter.Prometheus.HttpListener.Tests/PrometheusHttpListenerTests.cs | Makes the disposal-vs-scrape race test deterministic (thread-pool availability + dedicated disposal thread). |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/PrometheusHttpListener.cs | Implements bounded shutdown waits and returns 503 when disposal begins mid-collection. |
| src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md | Adds Unreleased entries describing the behavioral changes (needs PR link fixups). |
Suppressed comments (1)
src/OpenTelemetry.Exporter.Prometheus.HttpListener/CHANGELOG.md:24
- This CHANGELOG entry still uses a placeholder PR reference (#TODO). Please update it to the real PR link before merging.
([#TODO](https://github.com/open-telemetry/opentelemetry-dotnet/pull/TODO))
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add PR number.
Changes
Avoid disposal race in
PrometheusHttpListenerthat can cause shutdown to hang indefinitely and flaky tests (1, 2, 3).Merge requirement checklist
CHANGELOG.mdfiles updated for non-trivial changesChanges in public API reviewed (if applicable)