Bound the cached node-number release by a high-water mark (GH-3850)

[jeremydmiller]

Closes #3850. Follow-up to #3847 (merged), which cached the active node numbers per node instead of fetching them per database.

That trade is right — the failure #3847 broke was self-amplifying (pool exhaustion → heartbeat timeouts → the leader declaring healthy nodes stale → reassignment churning the table being read). This PR handles the window it widens.

The staleness is not symmetric

ReleaseOrphanedMessagesForAncillaryOperation resets owner_id to 0 for every owner missing from the cached list:

stale list…	consequence	severity
still names a dead node	its messages are not released this tick	benign — one interval of delay
is missing a live node	that node's in-flight rows are reset to owner_id = 0	another node claims work it is already doing

A cached list cannot describe a node that registered after it was taken, and MessageRoute.cs:197 stamps OwnerId on outbox persist — so a newcomer owns rows within milliseconds of registering. Registration does strictly precede ownership (PersistAsync returns the number before Options.Durability.AssignedNodeNumber is set), which is what keeps this narrow, but it does not close it.

Worth noting the Main store was never exposed: ReleaseOrphanedMessagesOperation evaluates not in (select node_number from wolverine_nodes) in-transaction. Only ancillary databases take the cached list.

Why not max(active)

Node numbers are database-generated and monotonic — SERIAL on Postgres (PostgresqlMessageStore.cs:695), AutoNumber() on SQL Server (SqlServerMessageStore.cs:570) — so the obvious guard is and owner_id <= max(cached list).

That is wrong, and the reason is worth recording. If the highest-numbered node dies, its number leaves the active list, the max drops below it, and its orphaned messages become permanently unreclaimable. Trading a rare duplicate for a permanent leak is a bad trade.

ActiveNodeNumberCache therefore keeps a high-water mark — the highest number it has ever observed, raised across fetches and never lowered:

• node 7 was active at some point → observed → still reclaimable after it departs
• node 8 registers after the last fetch → never observed → protected

Remaining gap, stated in the code: a node that both registers and departs between two fetches is never observed, so its rows stay owned until the mark rises. That parks messages rather than double-processing them — the safe direction — and the next registration clears it.

Also

Documents the gate's cost, which #3847 left implicit: one SemaphoreSlim means a hung LoadAllNodesAsync parks every database's timer callback rather than each failing on its own timeout. That is the intended trade (one slow query beats one per database), but someone diagnosing a stall should not have to derive it from the code.

Tests

Four on the cache — the mark is the highest seen, it does not drop when the highest node departs, it rises for a joiner, and a node joining mid-interval sits above the mark. Four on the generated SQL, including that both the inbox and outbox statements carry the bound.

Verified non-vacuous: disabling the ceiling turns 3 of the 4 SQL tests red. The fourth stays green by design — it asserts the guard is absent when no mark is supplied, so 0 restores the un-bounded behaviour rather than releasing nothing.

• CoreTests — 2264/0
• SqlServerTests orphan-release suites — 7/7 (includes the pre-existing Solo tests that call the changed buildOperationBatch signature)
• dotnet build wolverine.slnx -c Release -f net9.0 — clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m