Fix stale messages during rollout and improve graceful shutdown#2287
Merged
jeremydmiller merged 1 commit intomainfrom Mar 11, 2026
Merged
Fix stale messages during rollout and improve graceful shutdown#2287jeremydmiller merged 1 commit intomainfrom
jeremydmiller merged 1 commit intomainfrom
Conversation
During rolling deployments, a race condition can strand messages with owner_id pointing to a node that no longer exists. The DurabilityAgent now periodically releases these orphaned messages by resetting owner_id to 0 for any message whose owner is not in the active nodes list. For main databases, uses a subquery against the co-located wolverine_nodes table. For ancillary/tenant databases, fetches active node numbers from the main store and builds an explicit NOT IN list. Closes #2279 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jeremydmiller
changed the title
This was referenced Mar 12, 2026
|
Weird, I don't see changes related to Graceful shutdown improvements (#2282) about WaitForCompletionAsync |
This was referenced Mar 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Orphaned message recovery (#2279)
ReleaseOrphanedMessagesOperation(main databases): usesNOT IN (SELECT node_number FROM wolverine_nodes)subqueryReleaseOrphanedMessagesForAncillaryOperation(ancillary/tenant databases): fetches active node numbers from the main store, builds explicitNOT IN (1, 2, 3)listDurabilityAgent.buildOperationBatch()Graceful shutdown improvements (#2282)
WaitForCompletionAsyncinDurableReceiver.DrainAsync()andBufferedReceiver.DrainAsync()so in-flight message handlers finish before ownership is releasedDrainTimeoutsetting (default 30 seconds) toDurabilitySettingsGracefulShutdownandRollingRestartchaos test scripts exercising shutdown scenariosTest plan
Closes #2279
Closes #2282
🤖 Generated with Claude Code