Repository navigation
Durable receiver hardening, part 3: drain release vs. a new receiver's first claim; heartbeat only while holding (F18) - #5158
Merged
Conversation
… heartbeat runs only while something is held (F18) Release race. A drained DurableReceiver checked "is any other receiver at this address busy?" and then issued its blanket ReleaseIncomingAsync as two separate steps. A new receiver whose first claim at the address landed between them (its first inbox write, or a row its listener had already dequeued) had that row handed to any node underneath it. DurableReceiverRegistry now makes the two exclusive per address: TryBeginBlanketRelease checks and registers the release under one lock and holds it until the release statements finish, and DurableReceiver.ActivateAsync marks a receiver as working under the same lock and then waits out any blanket release of its addresses already in flight. ListeningAgent activates a durable receiver before it builds the listener, and a resumed DurableLocalQueue activates its new receiver before installing it, so rows claimed before the receiver sees them (database-backed dequeue, the queue's own store, scheduled poller, recovery) are covered. Receipt paths activate lazily on first use; the hot path pays one volatile read. The startup ClearInFlightIncomingAsync sweep goes through the same gate. F18. The InboxStaleTime heartbeat timer now starts with the first envelope taken into custody and stops on the first tick that finds custody empty. Dispose() and DisposeAsync() no longer drop held batch members: the heartbeat keeps running for them until their batch settles, then the timer is disposed. It is also disposed once the host is stopping. A post or requeue that never reaches the block leaves custody instead of leaking an entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…ed receiver schema after each test It left the inbox keyed by (id, received_at) with rows sharing an id, so any default-identity class in the same 'receiver' schema that ran after it failed all 75 tests migrating back to PRIMARY KEY (id) (23505). Order-dependent, so it surfaced only on some runs. MessageStoreCompliance gains a protected afterHostDisposedAsync hook for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
This was referenced Oct 23, 2026
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third durable receiver hardening pass, closing the two items #5157 left open.
A drain's blanket release can no longer catch a new receiver's first claim
#5157 made a drained
DurableReceivercheck "is any other receiver at this address busy?" before its blanketReleaseIncomingAsync. Those were two steps, and the release is a database round trip: a new receiver whose first claim at the address landed in between (its first inbox write, or a row its listener had already dequeued) had that row handed back to any node underneath it, a possible concurrent duplicate.DurableReceiverRegistrynow makes the two exclusive per address:TryBeginBlanketReleasechecks and registers the release under one lock, and holds it until the release statements finish.DurableReceiver.ActivateAsyncmarks a receiver as working under the same lock, then waits out any blanket release of its addresses already in flight.ListeningAgentactivates a durable receiver before it builds the listener, and a resumedDurableLocalQueueactivates its new receiver before installing it, because a row can be claimed before the receiver ever sees it (database-backed dequeue, the queue's own store, the scheduled poller, recovery). Receipt paths still activate lazily as a fallback; the hot path pays one volatile read.ClearInFlightIncomingAsyncsweep goes through the same gate.The InboxStaleTime heartbeat runs only while something is held (F18)
Dispose()andDisposeAsync()keep the heartbeat for batch members still in custody until their batch settles, then dispose the timer. It is also disposed once the host is stopping.Test isolation fix in PostgresqlTests
PostgresqlMessageStore_with_IdAndDestination_Identityleft the sharedreceiverschema's inbox keyed by(id, received_at)with rows sharing an id. Any default-identity class in that schema that ran after it failed all 75 of its tests migrating back toPRIMARY KEY (id)(23505), so the failure depended on test order. The class now drops the schema after each test, through a newprotected virtual afterHostDisposedAsync()hook onMessageStoreCompliance.Tests
a_drain_release_is_exclusive_with_a_new_receivers_first_claim.csandthe_inbox_heartbeat_runs_only_while_something_is_held.cs: 7 of the 8 tests fail on the unfixed code (the 8th is a guard), all 8 pass after.Local verification
wolverine.slnxbuild (-f net9.0): clean.external_message_tables.pull_in_message_that_goes_to_dead_letter_queue_and_replay_it, the known full-suite timeout (PostgresqlTests: pull_in_message_that_goes_to_dead_letter_queue_and_replay_it times out in a full-suite run but passes in isolation #4682) that also reproduces onmain. It passed 3 of 3 alone on this branch and onmain.Left open
🤖 Generated with Claude Code
https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U