Latch receivers on ApplicationStopping for immediate graceful shutdown#2288
Merged
jeremydmiller merged 1 commit intomainfrom Mar 11, 2026
Merged
Latch receivers on ApplicationStopping for immediate graceful shutdown#2288jeremydmiller merged 1 commit intomainfrom
jeremydmiller merged 1 commit intomainfrom
Conversation
…ing during shutdown Hook into IHostApplicationLifetime.ApplicationStopping so all message receivers are latched the moment SIGTERM fires, rather than waiting for IHostedService.StopAsync which may be delayed by other hosted services stopping first. This prevents messages already in internal queues from being picked up after the shutdown signal. Also reorders StopAsync to drain endpoints before releasing ownership, adds bounded WaitForCompletionAsync calls in DurableReceiver and BufferedReceiver with configurable DrainTimeout (default 30s), and includes GracefulShutdown/RollingRestart chaos tests. Closes #2282 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Mar 12, 2026
jeremydmiller
added a commit
that referenced
this pull request
Mar 12, 2026
When a rate-limited message triggers PauseListenerContinuation, the pause calls StopAndDrainAsync → DrainAsync from within the receiver block's execute function. The WaitForCompletionAsync added in #2288 waits for in-flight items to finish, but the current message IS an in-flight item — creating a deadlock that times out after DrainTimeout (30s), causing the Redis rate limiting test to fail. Fix: only wait for completion when Latch() was previously called (indicating shutdown via OnApplicationStopping), not when DrainAsync is the first to set _latched (indicating a pipeline-triggered pause). Closes #2291 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3 tasks
This was referenced Mar 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IHostApplicationLifetime.ApplicationStoppingto immediately latch all message receivers the moment SIGTERM fires, preventing queued messages from being processed after the shutdown signalStopAsyncto drain endpoints before releasing ownership and tearing down agents, so in-flight handlers complete before messages are released back to the inboxWaitForCompletionAsyncinDurableReceiverandBufferedReceiverwith configurableDrainTimeout(default 30s) to prevent indefinite hangs during shutdownLatch()methods toBufferedReceiver,ListeningAgent, andDurableLocalQueuefor immediate latching without full drainGracefulShutdownandRollingRestartchaos test scripts with 6 test methodsContext
Addresses the scenario from #2282 where messages already in Wolverine's internal processing queues continue to be executed between the SIGTERM signal and Wolverine's own
IHostedService.StopAsyncbeing called. Since .NET stops hosted services in reverse registration order, other services may stop first, during which time Wolverine keeps processing. TheApplicationStoppinghook fires immediately on SIGTERM, closing this timing gap.Test plan
Closes #2282
🤖 Generated with Claude Code