Skip to content

Durable receiver hardening, part 3: drain release vs. a new receiver's first claim; heartbeat only while holding (F18) - #5158

Merged
jeremydmiller merged 2 commits into
mainfrom
durable-hardening-3
Oct 23, 2026
Merged

jeremydmiller merged 2 commits into
mainfrom
durable-hardening-3

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Third durable receiver hardening pass, closing the two items #5157 left open.

A drain's blanket release can no longer catch a new receiver's first claim

#5157 made a drained DurableReceiver check "is any other receiver at this address busy?" before its blanket ReleaseIncomingAsync. Those were two steps, and the release is a database round trip: a new receiver whose first claim at the address landed in between (its first inbox write, or a row its listener had already dequeued) had that row handed back to any node underneath it, a possible concurrent duplicate.

DurableReceiverRegistry now makes the two exclusive per address:

  • TryBeginBlanketRelease checks and registers the release under one lock, and holds it until the release statements finish.
  • DurableReceiver.ActivateAsync marks a receiver as working under the same lock, then waits out any blanket release of its addresses already in flight.
  • ListeningAgent activates a durable receiver before it builds the listener, and a resumed DurableLocalQueue activates its new receiver before installing it, because a row can be claimed before the receiver ever sees it (database-backed dequeue, the queue's own store, the scheduled poller, recovery). Receipt paths still activate lazily as a fallback; the hot path pays one volatile read.
  • The startup ClearInFlightIncomingAsync sweep goes through the same gate.

The InboxStaleTime heartbeat runs only while something is held (F18)

  • The timer starts with the first envelope taken into custody and stops on the first tick that finds custody empty.
  • Dispose() and DisposeAsync() keep the heartbeat for batch members still in custody until their batch settles, then dispose the timer. It is also disposed once the host is stopping.
  • A post or requeue that never reaches the block leaves custody instead of leaking an entry.

Test isolation fix in PostgresqlTests

PostgresqlMessageStore_with_IdAndDestination_Identity left the shared receiver schema's inbox keyed by (id, received_at) with rows sharing an id. Any default-identity class in that schema that ran after it failed all 75 of its tests migrating back to PRIMARY KEY (id) (23505), so the failure depended on test order. The class now drops the schema after each test, through a new protected virtual afterHostDisposedAsync() hook on MessageStoreCompliance.

Tests

  • a_drain_release_is_exclusive_with_a_new_receivers_first_claim.cs and the_inbox_heartbeat_runs_only_while_something_is_held.cs: 7 of the 8 tests fail on the unfixed code (the 8th is a guard), all 8 pass after.

Local verification

Left open

  • A durable local queue that pauses while one of its own inbox writes is still in flight holds that envelope until it resumes. If the paused receiver's blanket release runs after that write commits, the row goes back to any node. That's a narrower, different window from the one fixed here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U

jeremydmiller and others added 2 commits October 22, 2026 19:08
… heartbeat runs only while something is held (F18)

Release race. A drained DurableReceiver checked "is any other receiver at this
address busy?" and then issued its blanket ReleaseIncomingAsync as two separate
steps. A new receiver whose first claim at the address landed between them (its
first inbox write, or a row its listener had already dequeued) had that row
handed to any node underneath it. DurableReceiverRegistry now makes the two
exclusive per address: TryBeginBlanketRelease checks and registers the release
under one lock and holds it until the release statements finish, and
DurableReceiver.ActivateAsync marks a receiver as working under the same lock
and then waits out any blanket release of its addresses already in flight.
ListeningAgent activates a durable receiver before it builds the listener, and
a resumed DurableLocalQueue activates its new receiver before installing it, so
rows claimed before the receiver sees them (database-backed dequeue, the queue's
own store, scheduled poller, recovery) are covered. Receipt paths activate
lazily on first use; the hot path pays one volatile read. The startup
ClearInFlightIncomingAsync sweep goes through the same gate.

F18. The InboxStaleTime heartbeat timer now starts with the first envelope taken
into custody and stops on the first tick that finds custody empty. Dispose() and
DisposeAsync() no longer drop held batch members: the heartbeat keeps running
for them until their batch settles, then the timer is disposed. It is also
disposed once the host is stopping. A post or requeue that never reaches the
block leaves custody instead of leaking an entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…ed receiver schema after each test

It left the inbox keyed by (id, received_at) with rows sharing an id, so any default-identity class in the
same 'receiver' schema that ran after it failed all 75 tests migrating back to PRIMARY KEY (id) (23505).
Order-dependent, so it surfaced only on some runs. MessageStoreCompliance gains a protected
afterHostDisposedAsync hook for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
@jeremydmiller
jeremydmiller merged commit 9c950fc into main Oct 23, 2026
48 checks passed
This was referenced Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant