fix: metrics flush timeout canceled shutdown - #2990
Merged
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe PR centralizes metric flush handling in graph server shutdown, removes synchronous flush paths from metric stores, and adds bounded concurrent export with buffer pooling updates and regression tests. ChangesCentralized metrics flush refactor
Bounded exporter concurrency
Estimated code review effort: 4 (Complex) | ~45 minutes Related PRs: #2838 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
Router-nonroot image scan passed✅ No security vulnerabilities found in image: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
router/pkg/metric/oltp_connection_metric_store.go (1)
99-102:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winBug:
errors.Joindiscards previously accumulated errors.
errors.Join(regErr)ignores the existingerrvalue, so only the last error survives. This should join the new error with the accumulated error.🐛 Proposed fix
for _, reg := range h.instrumentRegistrations { if regErr := reg.Unregister(); regErr != nil { - err = errors.Join(regErr) + err = errors.Join(err, regErr) } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@router/pkg/metric/oltp_connection_metric_store.go` around lines 99 - 102, In the for loop iterating over h.instrumentRegistrations where Unregister() is called, the errors.Join function is being called with only regErr, which discards the previously accumulated err value. Fix this by passing both err and regErr to errors.Join so that all errors encountered during unregistration are properly accumulated and returned together.router/pkg/metric/prom_connection_metric_store.go (1)
98-102:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winBug:
errors.Joindiscards previously accumulated errors.Same issue as in
oltp_connection_metric_store.go— only the last unregister error will be returned.🐛 Proposed fix
for _, reg := range h.instrumentRegistrations { if regErr := reg.Unregister(); regErr != nil { - err = errors.Join(regErr) + err = errors.Join(err, regErr) } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@router/pkg/metric/prom_connection_metric_store.go` around lines 98 - 102, The error handling in the loop that unregisters h.instrumentRegistrations is not accumulating all errors properly. Currently, errors.Join is being called with only the current regErr, which overwrites the err variable and discards any previously accumulated errors from earlier loop iterations. Fix this by passing both the accumulated err variable and the current regErr to the errors.Join function so that all errors encountered during the unregistration loop are collected together, not just the last one.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@router/pkg/metric/oltp_connection_metric_store.go`:
- Around line 99-102: In the for loop iterating over h.instrumentRegistrations
where Unregister() is called, the errors.Join function is being called with only
regErr, which discards the previously accumulated err value. Fix this by passing
both err and regErr to errors.Join so that all errors encountered during
unregistration are properly accumulated and returned together.
In `@router/pkg/metric/prom_connection_metric_store.go`:
- Around line 98-102: The error handling in the loop that unregisters
h.instrumentRegistrations is not accumulating all errors properly. Currently,
errors.Join is being called with only the current regErr, which overwrites the
err variable and discards any previously accumulated errors from earlier loop
iterations. Fix this by passing both the accumulated err variable and the
current regErr to the errors.Join function so that all errors encountered during
the unregistration loop are collected together, not just the last one.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: fd3ab86b-5005-4a3c-bddd-ee2bbceaa4c0
📒 Files selected for processing (8)
router/core/graph_server.gorouter/core/graphql_prehandler.gorouter/pkg/metric/connection_metric_store.gorouter/pkg/metric/metric_store.gorouter/pkg/metric/oltp_connection_metric_store.gorouter/pkg/metric/oltp_stream_metric_store.gorouter/pkg/metric/prom_connection_metric_store.gorouter/pkg/metric/stream_metric_store.go
💤 Files with no reviewable changes (2)
- router/pkg/metric/oltp_stream_metric_store.go
- router/pkg/metric/stream_metric_store.go
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2990 +/- ##
==========================================
- Coverage 65.49% 61.39% -4.10%
==========================================
Files 335 259 -76
Lines 48737 30053 -18684
Branches 5424 0 -5424
==========================================
- Hits 31918 18451 -13467
+ Misses 16793 10116 -6677
- Partials 26 1486 +1460
🚀 New features to boost your workflow:
|
alepane21
approved these changes
Jun 19, 2026
Noroth
marked this pull request as ready for review
June 22, 2026 07:31
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@router/internal/exporter/exporter_test.go`:
- Around line 61-69: The test for putBatchBuffer is re-reading from sync.Pool
via getBatchBuffer, which can return a different buffer and hide failures;
update the assertion to inspect the original backing array from buf after
putBatchBuffer(buf) instead. Use the existing putBatchBuffer and getBatchBuffer
symbols to locate the test, but make the check deterministic by verifying
buf[:cap(buf)] was cleared directly, not a newly fetched pooled buffer.
- Around line 117-154: The concurrency test in NewExporter leaves export
goroutines blocked on sink.release if an assertion fails before the manual
close, so add cleanup immediately after exporter creation and reuse it on the
success path. Update the test around NewExporter, sink.release, and e.Shutdown
to register a deferred cleanup/close guard right after e is created, then remove
the late one-off close so the release channel is always freed even on failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ad3ef704-6abd-485c-b141-4143d641a365
📒 Files selected for processing (2)
router/internal/exporter/exporter.gorouter/internal/exporter/exporter_test.go
endigma
self-requested a review
June 30, 2026 09:04
endigma
requested changes
Jul 1, 2026
endigma
left a comment
endigma
left a comment
Member
There was a problem hiding this comment.
looks like tests are not really using testify otherwise looks good
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@router/core/graph_server.go`:
- Around line 1049-1057: The deferred cleanup in graphServer’s graph mux build
path can panic because gm.Shutdown is called even when buildOperationCaches has
not yet initialized the cache fields. Update graphMux.Shutdown to be nil-safe
when closing cache resources, or split the defer in graph_server.go so it only
cleans up resources that have actually been initialized before the build error.
Refer to graphMux.Shutdown and gm.buildOperationCaches(s) to keep the cleanup
path safe on early failures.
- Around line 972-978: The shutdown guard in Shutdown is not atomic because
finalization currently uses a separate Load() and Store(true), which can still
allow concurrent callers to run cleanup twice. Update the finalized guard to use
an atomic CompareAndSwap(false, true) check at the start of Shutdown, and only
proceed with mux shutdown and cleanup when the swap succeeds; otherwise return
immediately. Keep the existing shutdown flow in graph_server.go unchanged apart
from replacing the guard logic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 8eaa11b8-ea41-432d-ac78-d50164a8d1b7
📒 Files selected for processing (6)
router/core/graph_server.gorouter/pkg/metric/cache_metrics.gorouter/pkg/metric/engine_metrics.gorouter/pkg/metric/otlp_metric_store.gorouter/pkg/metric/prom_connection_metric_store.gorouter/pkg/metric/router_runtime_metrics.go
🚧 Files skipped from review as they are similar to previous changes (1)
- router/pkg/metric/prom_connection_metric_store.go
Address review feedback: convert manual t.Fatal / nil checks to require assertions and replace the waitFor helper with require.Eventually. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
endigma
approved these changes
Jul 1, 2026
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
MaxConcurrentExports, with a default).Checklist
Open Source AI Manifesto
This project follows the principles of the Open Source AI Manifesto. Please ensure your contribution aligns with its principles.