Stop the high water mark stalling forever behind idle advisory-lock sessions (#5108, #5125)

[jeremydmiller]

Fixes #5108. Fixes #5125.

The root cause is shared

Both reports are the same failure at default settings: the high water mark pins under a dead sequence gap and never advances again, with AdvanceHighWaterMarkToLatestAsync() as the only recovery.

The gap-liveness gate (#4953/#4977) will not skip a gap while any transaction that could still fill it looks alive, and its open-transaction clause counts every client backend whose transaction predates the gap. A session parked idle in transaction on an advisory lock trips that clause forever, because it never commits.

#5057 already introduced the sound answer — the allocation fence: the latest server time at which the sequence had handed out nothing above the mark. Everything in the gap was allocated later, so any session that has provably executed nothing since cannot be holding it. That works. The problem is that the fence lived only in the detector instance's memory (_allocationHistory), so a detector that started after the gap formed had none, silently degraded to the pre-#5057 behavior, and never advanced.

• Dead sequence gaps + idle-in-transaction sessions: defaults stall the high-water mark permanently; SkipStaleGapsDespiteLiveTransactionsAfter has two weaknesses #5108 §1 — deploy, crash or shard rebalance while a rollback gap exists. Fresh detector, no history, and the fleet's ~57 idle Wolverine listener sessions read as live reservers.
• Daemon's own transaction-scoped advisory-lock session stalls gap detection at mark 0 — is_called fence (#5091) doesn't cover a used-then-cleaned sequence #5125 — no restart needed. The store was gapped before any daemon ever ran, so no poll can ever observe the sequence at or below the mark and the fence is unobtainable by construction. The session tripping the clause is the daemon's own HotCold leadership lock, transaction-scoped by default and therefore idle-in-transaction for its whole tenure.

The fix

1. The fence is durable (ecda70074)

No history is needed in the database. The lookup wants the newest observation whose allocated high is at or below the mark, and both values only move forward, so a single row maintained incrementally is exactly equivalent. Promote it while the store is caught up; the instant a gap opens, allocation passes the mark, promotion stops, and the row freezes at the last moment nothing above the mark had been handed out.

It's a reserved mt_event_progression row rather than a table of its own, deliberately: this repairs a default-configuration stall, so it has to engage on every existing database the moment Marten is upgraded, with no DDL migration standing between the user and the fix.

The write re-reads the sequence inside its own statement rather than trusting the poll's reading — a timestamp stamped against a stale allocation reading can sit later than the moment it claims, and a fence that is too late wrongly excludes a genuine reserver.

Two guards the reserved row requires:

• TryCorrectProgressInDatabaseAsync updates the table with no WHERE clause and would restamp the fence into a claim that was never observed.
• ProjectionProgressStatement returns every row, which would surface high-water bookkeeping as a projection shard in AllProjectionProgress().

2. The daemon's own advisory-lock sessions are ruled out structurally (ecda70074)

This turned out to be essential rather than a niche add-on, and the test is what proved it: with only the durable fence, the restart case still stalled, because the replacement daemon's own leadership lock is a new session that postdates the fence and is therefore never quiescent. So #5125's mechanism dominates #5108 §1 too.

Those connections exist only to hold the lock and never append, so they can never be a gap's reserver. Identified server-side from pg_locks against Marten's own deterministic lock ids — ProjectionLockIds.Compute over this store's schema, shards and DaemonLockId — so nothing has to be plumbed out of the running coordinator. Further restricted to backends with no xid, so anything that has actually written is never excluded. This one needs no fence, so it also applies to the per-tenant path, where no sound fence can ever exist.

A blanket "holds an advisory lock" exclusion would not be sound: RichEventAppender reserves sequence numbers through EventSequenceFetcher as a separate statement before its inserts, so a genuine Rich-mode reserver can be sitting there with no xid and no relation lock. That window is exactly why the open-transaction clause exists, and it is also why the tempting "no backend_xid ⇒ not a reserver" shortcut does not hold on its own.

3. The whole dead span clears in one cycle (893c13811)

Addresses #5108 §2. The skip used to advance to the next gap edge, so N gaps cost N detection cycles and each re-paid StaleSequenceThreshold from a fresh observation — reported as a mark grinding 38 → 426 over 21 seconds against ~50 gaps.

One liveness verdict already covers the entire span: every sequence number at or below the ceiling recorded at observation was handed out at or before that moment (nextval runs inside the reserving transaction, so its xact_start cannot postdate it), and the probe just established that no transaction from before the observation is still alive. Same argument that already licensed skipping the first gap, applied to all of them at once. The ReservedCeiling clamp is unchanged.

Tests

End-to-end proof (src/DaemonTests.ManualOnly/Coordination/high_water_advances_past_dead_gaps_with_slow_transactions.cs) — real HotCold daemons, real advisory locks, deliberately slow idle-in-transaction sessions, driven through actual daemon polling. Both scenarios stall for the full 60s timeout on master and pass in ~8s here. ManualOnly because they are slow by construction.

CI-level regression coverage (Bug_5108_durable_allocation_fence, Bug_5108_dead_gap_span_clears_in_one_cycle) since CI does not run ManualOnly. Checked against master's production code: the four behavioral tests fail, and the one that passes is the safety test — a genuinely live reserver must still hold the gap — which is the signature it should have.

Suite	Result
DaemonTests	273/273, net10.0 and net9.0
EventSourcingTests	1626/1626
TenantPartitionedEventsTests	238/238
ManualOnly proof tests	2/2

Reviewer notes

• Two behavior changes are deliberate. detecting_the_high_water_mark's multi-gap assertion encoded the old one-gap-per-cycle behavior and now pins the new one. Dropping the long-standing Daemon enters unrecoverable state if event sequence is higher than high water mark #2681 empty-tail fallback regressed six tests, so that path is preserved explicitly.
• CoreTests fails 5 tests, pre-existing. Baselined against unmodified master: identical 5 failures (Unable to attain a global lock). Not from this change.
• Docs badge says 9.23; current version is 9.22.2. Adjust if that is not the target release.
• The UseAdvisoryLockTransaction = false guidance is now historical — leaked and live leadership lock sessions are both recognized, so the docs note it as a workaround for older versions only.
• Unrelated but adjacent: now() at time zone 'utc' written into timestamptz columns skews mt_last_modified and mt_events.timestamp by the server's UTC offset #5136 (now() at time zone 'utc' written into timestamptz columns) was found while reviewing SkipStaleGapsDespiteLiveTransactionsAfter survives detector restarts (evidence-based stall onset) #5109. This PR deliberately does no arithmetic on mt_events.timestamp, so the two are independent.

Relationship to #5109

This supersedes #5109. That PR left the liveness verdict alone and instead made the SkipStaleGapsDespiteLiveTransactionsAfter cap's clock durable — but the cap is opt-in and lossy by design, so it cannot fix either report's default configuration, and its evidence query is inert in #5125's exact shape (nothing committed above the mark). Once the default path recovers on its own, the cap-clock defect stops mattering.

Credit where due: @uniquelau's detector-churn test in #5109 is the best executable statement of the problem, and their diagnosis of the churn reset is correct. Worth salvaging that test.

🤖 Generated with Claude Code