Dormant inbox rows for a durable LocalQueue with ListenerScope.Exclusive are never recovered: the durability agent skips it and it never gets a ListeningAgent

[erdtsieck]

On 6.24.6, so both GH-3590 (#3598) and GH-3680 (#3695) are in. 1,350 envelopes have been sitting at status = 'Incoming', owner_id = 0, attempts = 0 for hours and nothing picks them up.

The gap

PartitionedMessageTopology forces every partitioned endpoint to ListenerScope.Exclusive, local queues included:

// PartitionedMessageTopology, ctor
var endpoint = buildEndpoint(options, name);
endpoint.UsedInShardedTopology = true;
endpoint.ListenerScope = ListenerScope.Exclusive;

So a PublishToPartitionedLocalMessaging("activiteiten", 16, …) topology produces sixteen durable local://activiteitenN/ endpoints with ListenerScope.Exclusive.

Now both recovery paths decline:

1. The durability agent skips it. CheckRecoverableIncomingMessagesOperation.PostProcessingCommands():

// GH-3590: … Recovery for those endpoints is owned by the listening node itself (ListenerInboxRecovery).
if (_endpoints.IsSingleNodeListener(incoming.Destination))
{
    continue;
}

and

bool IsSingleNodeListener(Uri address)
    => EndpointFor(address) is { ListenerScope: not ListenerScope.CompetingConsumers };

No LocalQueue carve-out, so this returns true and recovery is skipped.

2. But the listening node never owns it either, because a local queue never gets a ListeningAgent. startInboxRecoveryIfNecessary() is called at the end of ListeningAgent.StartAsync(), and the only loop that creates those agents excludes local queues:

public async Task StartListenersAsync()
{
    …
    var listeningEndpoints = _options.Transports.SelectMany(x => x.Endpoints())
        .Where(x => x is not LocalQueue)
        .Where(x => x.ShouldAutoStartAsListener(_options.Durability));

ExclusiveListeners() excludes them too (… and not LocalQueue), which is consistent with what we see in wolverine_node_assignments: of 10,878 assignments, zero have a local:// URI — only event-subscriptions://, wolverinedb:// and the two leader agents.

So the GH-3590 carve-out hands ownership to a ListenerInboxRecoveryLoop that, for a local queue, is never constructed.

Observed

A sharded deployment (512 tenant message databases, 5 nodes, Wolverine 6.24.6):

dormant rows in one tenant database	1,350
endpoints	all sixteen local://activiteiten1..16/
state	status='Incoming', owner_id=0, attempts=0
unchanged for	hours, across several node replacements
recovered … log lines in 20 minutes	0, while the same window has 1,237 lines of ordinary traffic on those queues

They get into that state the normal ways: envelopes marked replayable out of the dead letter table, and in-flight rows released to owner_id = 0 when a node is replaced. A rolling deploy does not clear them, because the node that comes back has no recovery path for these endpoints either.

Suggested fix

The GH-3590 reasoning is that an exclusive external listener runs on exactly one node while the durability agent is assigned per database and may be elsewhere. A local queue does not have that problem: it exists on every node, so the durability agent is a perfectly good owner for its inbox recovery. The carve-out over-reaches.

bool IsSingleNodeListener(Uri address)
    => EndpointFor(address) is { ListenerScope: not ListenerScope.CompetingConsumers } and not LocalQueue;

That puts local partitioned queues back under the durability agent, which already has FindListenerCircuit() fall back to the durable local queue — the comment above the skip even mentions that fallback as the reason the check has to come first.

Happy to send that as a PR with a regression test if you agree with the direction. Reproduces continuously here.

[erdtsieck]

Control measurement, since "maybe the rows are simply in a database nobody looks at" and "maybe there is no listener for those queues any more" are the obvious competing explanations. Both are ruled out by the same application, which happens to run two kinds of durable local queue side by side:

• sixteen partitioned ones from PublishToPartitionedLocalMessaging("activiteiten", 16, …), so ListenerScope.Exclusive
• ordinary per-message-type local queues, local://application.<namespace>.<messagetype>/, left at the default ListenerScope.CompetingConsumers

Both live in the same tenant databases and are fed by the same nodes. In one tenant database:

queue kind	status	owner_id	rows
partitioned, Exclusive	Incoming	0	1,350
ordinary, CompetingConsumers	Scheduled	0	2

The two ordinary rows are Scheduled, i.e. simply waiting for their execution time — nothing dormant. The same database holds zero dead letters, which is consistent with where the 1,350 came from: they were marked replayable out of the dead letter table, left it, and stalled in the inbox.

So:

• not the database — the CompetingConsumers queues in that very database are clean, and the durability agent visibly recovers those. (ListenerInboxRecovery also sweeps every tenant database by design, per its own doc comment.)
• not a missing listener — all sixteen activiteitenN queues are configured in the running build and are actively processing new traffic: 1,237 log lines mentioning them in a twenty-minute window, while recovered / globally owned / latched appear zero times in the same window.

The only difference between the two groups is ListenerScope, which is exactly the branch IsSingleNodeListener splits on. That is the whole bug: the scope sends recovery to the listening node, and a LocalQueue never becomes a ListeningAgent.

Earlier in this investigation I had two other theories and both were wrong, for the record: that recovery only queried the main store (it does not — it goes through MessageStoreCollection deliberately), and that GH-3590/GH-3680 were simply not in our version (they are, since 6.24.5).

[erdtsieck]

Three more things from digging into this today, all of which make the suggested carve-out look safer rather than riskier.

1. Still there on 6.24.7, and it is not one topology

Same behaviour after moving to 6.24.7. And it is not specific to the one topology I quoted: the application has six PublishToPartitionedLocalMessaging(...) topologies — 8 + 16 + 8 + 8 + 8 + 8 = 56 durable local queues, every one of them ListenerScope.Exclusive and every one of them with no recovery path. A dead-lettered message on any of them can never be replayed, and any in-flight envelope released to owner_id = 0 when a pod is replaced is stranded permanently.

The one that made this urgent is a message whose handler performs an external, non-idempotent submission. It is designed to be replayable — the handler verifies with the remote party before it pushes — but the replay never reaches the handler, so the design does not help.

2. On a LocalQueue, ListenerScope.Exclusive has no upside to lose

This is the part I had not checked when I filed it. The flag is not doing anything useful for a local queue anywhere else in the codebase:

• ExclusiveListeners() already excludes them (… and not LocalQueue), so they are never handed out as distributed agents — every node runs all 56 queues regardless of the scope.
• A local send always resolves in-process, so a partitioned local topology only ever ordered messages per group within one node. Never across the cluster.
• ListenerConfiguration is explicit about that: ListenWithStrictOrdering() and ExclusiveNodeWithParallelism() both throw NotSupportedException for a LocalQueue, with a message that says to use an external queue if you want cluster-wide ordering.

So on a LocalQueue the only observable effect of ListenerScope.Exclusive is the one in this issue: it switches inbox recovery off. Nothing else reads it for that endpoint kind. That makes the carve-out a pure bug fix rather than a trade-off.

3. Two implementation notes for whoever picks this up

There are two copies of IsSingleNodeListener. A default interface implementation on IEndpointCollection:

// IEndpointCollection
bool IsSingleNodeListener(Uri address)
    => EndpointFor(address) is { ListenerScope: not ListenerScope.CompetingConsumers };

and a caching override on EndpointCollection (the one the runtime actually calls):

public bool IsSingleNodeListener(Uri address)
{
    if (_singleNodeListeners.TryFind(address, out var isSingleNode)) return isSingleNode;
    isSingleNode = EndpointFor(address) is { ListenerScope: not ListenerScope.CompetingConsumers };
    …
}

A fix applied to only the default method compiles, reads correctly, and changes nothing at runtime.

Fixing it there covers every store. The predicate has exactly three call sites, all of them the same "skip inbox recovery for this destination" branch: CheckRecoverableIncomingMessagesOperation (RDBMS), CosmosDbDurabilityAgent.Incoming and RavenDbDurabilityAgent.Incoming. One carve-out, all three fixed.

The alternative — give a durable LocalQueue its own ListenerInboxRecoveryLoop — also works, but every node would then sweep the same rows for the same queue, whereas the durability agent is already the single per-database owner. That seems strictly worse for the local-queue case.

Workaround in the meantime

For anyone else hitting this before a fix ships, resetting the scope in the configure lambda is enough, because the topology constructor has already run by then:

.PublishToPartitionedLocalMessaging("facturen", 8, topology =>
{
    foreach (var slot in topology.Slots)
    {
        slot.ListenerScope = ListenerScope.CompetingConsumers;
    }

    topology.Message<…>();
});

Given point 2 that gives up nothing that was actually in effect, and it puts the queues back under the durability agent.

Still happy to send the carve-out as a PR with a regression test (dormant row on a partitioned local:// endpoint gets recovered) if you agree with the direction.

[jeremydmiller]

Thanks — this is an excellent report. Every claim in it checks out against source, and your reading of why the GH-3590 carve-out over-reaches for local queues is exactly right. #3857 is up.

I did take the fix one level up from the line you proposed, because there's a second guard that the issue doesn't mention, and it bites hard.

Both guards were added by the same commit (629d8b6, #3598), and each re-derives the predicate independently:

1. EndpointCollection.IsSingleNodeListener(Uri) — the one you found
2. RecoverIncomingMessagesCommand.DeterminePageSize() — on the raw ListenerScope:

// GH-3590 defense in depth. …
if (listener.Endpoint.ListenerScope != ListenerScope.CompetingConsumers)
{
    return 0;
}

So with only the one-liner applied, the destination gets past the skip in PostProcessingCommands(), a RecoverIncomingMessagesCommand is issued, and then DeterminePageSize() returns 0. I ran exactly that against a Postgres host with your topology and dormant owner_id = 0 rows — it loops forever, every polling pass:

Issuing a command to recover 5 incoming messages … to destination local://activiteiten3/
Unable to recover inbox messages to destination local://activiteiten3/. Listener has status Accepting, queued count 0 …

Your 1,350 rows would have stayed exactly where they are, with the silence replaced by log spam. Worth knowing if you'd already patched this locally — that symptom would look like a different bug.

The fix in #3857 is a single Endpoint.IsSingleNodeListener predicate that LocalQueue overrides to false, consumed by both guards and by ListeningAgent.startInboxRecoveryIfNecessary(). Same conclusion as yours, just derived in one place so the two sides can't disagree again.

Two things your report turned up that are worth recording:

• A local queue is never a single-node listener for three independent reasons, not one: BuildListenerAsync() throws, StartListenersAsync() filters local queues out, and ExclusiveListeners() excludes them. Your wolverine_node_assignments observation — zero local:// URIs out of 10,878 — is that showing up in the data.
• ListenWithStrictOrdering() / ExclusiveNodeWithParallelism() / ExclusiveNodeWithSessionOrdering() all throw NotSupportedException for a LocalQueue, yet PartitionedMessageTopology sets ListenerScope.Exclusive directly and bypasses that. It's the only route into this state. Recover dormant inbox rows for durable local queues (GH-3856) #3857 makes the assignment harmless rather than removing it, so the predicate also covers any future path that sets the scope.

The PR ships a regression test built on your exact topology — PublishToPartitionedLocalMessaging("activiteiten", 4, …) with durable queues, seeded dormant rows — which fails on main with 0 of 5 recovered and passes with the fix.