Recover dormant inbox rows for durable local queues (GH-3856)

[jeremydmiller]

Fixes #3856. Reported and diagnosed by @erdtsieck, who traced the exact mechanism — this PR follows their direction, applied one level up so both halves of the guard stay in agreement.

The bug

PartitionedMessageTopology forces ListenerScope.Exclusive onto every slot it builds, local queues included:

endpoint.UsedInShardedTopology = true;
endpoint.ListenerScope = ListenerScope.Exclusive;

The GH-3590 carve-out then hands inbox recovery for those endpoints to a ListenerInboxRecoveryLoop owned by the listening node — but a local queue never gets a ListeningAgent at all, for three independent reasons:

• LocalQueue.BuildListenerAsync() throws NotSupportedException
• StartListenersAsync() filters x is not LocalQueue
• ExclusiveListeners() filters and not LocalQueue

So the durability agent declined and nothing else picked them up. Envelopes sat at status='Incoming', owner_id=0 indefinitely, surviving rolling deploys because the replacement node had no recovery path either.

Why the obvious one-line fix isn't enough

There are two independent guards implementing the GH-3590 hand-off, both added by 629d8b6, and each re-derived the predicate on its own:

1. EndpointCollection.IsSingleNodeListener(Uri)
2. RecoverIncomingMessagesCommand.DeterminePageSize() — on the raw ListenerScope

Fixing only the first is worse than the current bug. Measured against a real Postgres host, the durability agent then issues a recovery command every single polling pass and recovers nothing:

Issuing a command to recover 5 incoming messages … to destination local://activiteiten3/
Unable to recover inbox messages to destination local://activiteiten3/. Listener has status Accepting …

…on an endless loop. The silent stall becomes log spam, and the messages still never move.

The fix

One predicate instead of three re-derivations — Endpoint.IsSingleNodeListener, overridden by LocalQueue to false, consumed by both guards and by ListeningAgent.startInboxRecoveryIfNecessary().

A local queue exists on every node, so the per-database durability agent is a perfectly good owner for its inbox recovery — which is exactly the asymmetry the GH-3590 comment relies on for external listeners, and exactly why it does not apply here. FindListenerCircuit() already falls back to the durable local queue for local:// addresses.

Two guards that must agree should never each re-derive the rule; that is what caused this.

Verification

Evidence	Without fix	With fix
New PG end-to-end test (the reported topology, dormant owner_id=0 rows)	fails — 0 of 5 recovered	passes — agent drains all 5
3 new unit tests (predicate + both guards)	fail	pass
dotnet build wolverine.slnx -c Release -f net9.0	—	clean, 0 warnings
CoreTests full suite	—	2282 / 2285

The one CoreTests failure (remembered_application_assembly_reuse_warning.a_normal_single_assembly_host_does_not_warn) is pre-existing on clean main — baselined by stashing this branch and re-running.

The existing single_node_listener_recovery_exclusion suite used new LocalQueue(...) as the stand-in endpoint for a rabbitmq:// exclusive listener; the fix correctly breaks that, so it now uses SingleNodeListenerEndpoint.

Note for a follow-up

ListenWithStrictOrdering(), ExclusiveNodeWithParallelism(), and ExclusiveNodeWithSessionOrdering() all throw NotSupportedException for a LocalQueue, yet PartitionedMessageTopology assigns ListenerScope.Exclusive directly — bypassing that guard, and the only route into this state. This PR makes the assignment harmless rather than removing it, so the fix also covers any future path that sets the scope. Removing it from the topology may still be worth doing as a cleanup.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JKfy5EzLX1i149gjUb3Tfg