fix(#4953): high water detection never crosses outstanding event sequences

[jeremydmiller]

Closes the remaining silent-event-skip mechanisms from discussion #4953 (follow-up to #4964/#4972): the async daemon's high water detection could advance across outstanding sequence numbers — reserved by transactions still in flight — permanently skipping those events once they committed. Four mechanisms, each reproduced deterministically against 9.16.0 before fixing:

1. GapDetector per-statement snapshot skew (Normal path, silent, threshold-free). The detector batched three statements in one command; under READ COMMITTED each takes its own snapshot, so commits landing mid-command defeated both gap checks (a null leading-gap probe bypasses the Async daemon can silently skip a committed event when a partial page's ceiling advances over a visibility gap (from #4953) #4964 hold entirely) and the max(seq_id) fallback crossed in-flight sequences. This is the mechanism that ignores StaleSequenceThreshold completely and the best explanation for skips persisting with a 10-minute threshold.
2. Threshold-free safe-zone skips from rebuild/catch-up. CheckNowAsync looped DetectInSafeZone (which skips by design) to the reserved last_value — any rebuild or forced catch-up during an import mowed through in-flight gaps.
3. The Daemon enters unrecoverable state if event sequence is higher than high water mark #2681 -32 fallback teleported into pure reservations, gated on mt_event_progression.last_updated — ancient on an idle store, so the gate passed instantly on the first crush of traffic (idle store → sudden load with slow first commits = thousands of events skipped).
4. The wall-clock stale skip cannot tell "slow" from "rolled back" — any append slower than the threshold at the leading edge lost its events.

The fix: prove the gap dead instead of guessing with wall clocks

• Single-snapshot detection: GapDetector and HighWaterStatisticsDetector are now exactly one SQL statement each — the leading-gap probe, interior-gap query, and max fallback are mutually consistent by construction. Closes mechanism 1 outright.
• Transaction-evidence gating (GapLivenessProbe, on by default via ProjectionOptions.UseTransactionEvidenceForGapSkipping): before any safe-zone skip, detection checks whether a transaction that could still fill the gap is alive — granted RowExclusiveLocks on the mt_events lineage held by transactions from before the gap was first observed (pg_locks, visible to every role), open transactions from before the observation (pg_stat_activity), and in-progress write xids below the snapshot xmax recorded at observation (pg_snapshot_xip, pure MVCC data, no privileges needed). Evidence ⇒ hold: a slow append is waited out. All three signals are time-fenced to the first observation of the specific gap, so they drain naturally — steady new load never wedges skipping.
• Per-gap stale clock: the threshold is measured from when this gap was first observed (mirroring Per-tenant high water freezes after the first persisted mark — second batch of events per tenant never projects (UseTenantPartitionedEvents) #4867's per-tenant clock), never from last_updated.
• Proven dead ⇒ bounded skip: the skip advances at most to the reserved ceiling recorded when the gap was first observed (later reservations belong to newer, unproven transactions). This replaces the magic -32 — a dead tail is skipped fully and promptly instead of pinning the daemon Stale forever when the tail was shorter than 32. Every skip logs its exact range at Warning.
• CheckNowAsync (JasperFx.Events 2.29.1, [Documentation] PostgreSQL Options #530) targets the new IHighWaterDetector.FetchCommittedHighWaterCeilingAsync (overridden here with max(seq_id)), time-bounded — rebuild/catch-up now waits for in-flight appends instead of skipping them. AdvanceHighWaterMarkToLatest likewise marks committed height only.
• Per-tenant parity: trySkipStaleTenantGapAsync applies the same evidence rule under UseTenantPartitionedEvents.
• Escape hatches: UseTransactionEvidenceForGapSkipping = false restores the pre-Async daemon can advance projection progress past committed events during concurrent appends #4953 behavior verbatim; SkipStaleGapsDespiteLiveTransactionsAfter (default null = never knowingly skip a live appender) trades safety for guaranteed progress against leaked idle-in-transaction sessions — PostgreSQL's idle_in_transaction_session_timeout is the recommended backstop.

Verification

• 8 new Bug_4953_* regressions (7 red before the fixes), including the reporter's pg_sleep-trigger technique end-to-end: a rebuild during an in-flight append waits and projects everything; the running daemon holds for a slow append instead of stale-skipping it.
• pg_locks/pg_stat_activity/pg_current_snapshot visibility and cost verified empirically (PG 17): all monitoring-grade proc-array scans, probed only while a stale gap exists; cross-role redaction degrades the probe conservatively.
• Full DaemonTests suite green (230/230, net10.0) against published JasperFx 2.29.1; the two legacy HighWaterDetectorTests theories are updated to the new semantics (hold on first sighting; dead tails advance to the observed ceiling).

🤖 Generated with Claude Code