Make the stale-gap cap clock survive detector restarts (#5109)

[jeremydmiller]

Carries forward the defect and the tests from #5109 by @uniquelau, with a different mechanism.

The defect (their diagnosis, confirmed)

SkipStaleGapsDespiteLiveTransactionsAfter is documented as the bounded override for the transaction-evidence hold — once a gap has been stuck past the cap, the skip proceeds even though a candidate reserver still looks alive. But it measured from _stuckGap.Since, an in-memory observation that dies with the detector instance. Under managed-distribution agent churn, or restart cycles shorter than the cap, the clock reset every cycle and the documented bound never fired.

#5163 did not address this — it repaired the allocation fence, which is a different mechanism. Confirmed empirically: applying their test file to current master, 4 of 6 pass (#5163 handles those scenarios) and the 2 detector-churn cases fail. This fixes those two.

The mechanism

Record the first sighting durably: a reserved mt_event_progression row holding the pinned mark and the moment it was first seen pinned. The mark only ever advances, so while it has not moved that timestamp stays a sound lower bound on how long the gap has been stuck — the direction a cap needs. Keying the row on the mark makes it self-invalidating: a different mark is a different gap, and the clock restarts.

The stale threshold deliberately stays on the in-memory clock, so a fresh detector still gives a just-appeared gap a full settle window. A null cap (the default) is untouched.

Why not the original approach

#5109 dated the stall from the timestamp of the earliest committed event above the mark. That reads mt_events.timestamp, which is client-written in Rich append mode (the default) and skewed by the server's UTC offset in Quick mode — see #5136, which was found while reviewing that PR. A behind-clock client or a non-UTC server could age a gap past the cap and drop a live append's events. Using the mark's own pinning time keeps everything in server time and needs no trust in event timestamps at all.

The original PR's last_updated-only rejection still stands and its guard test (idle_store_first_live_append_is_not_cap_skipped) is carried over intact.

Tests

All six tests are @uniquelau's from #5109, unchanged in substance, with co-authorship on the commit:

Test	master	here
cap_skips_a_pre_existing_dead_gap_despite_an_idle_listener_and_no_allocation_history	pass	pass
idle_store_first_live_append_is_not_cap_skipped (guard)	pass	pass
live_reserver_is_not_cap_skipped_before_the_cap_genuinely_elapses (guard)	pass	pass
running_daemon_resumed_after_the_gap_formed_skips_within_the_cap	pass	pass
detector_churn_does_not_postpone_the_cap_forever	fail	pass
conjoined_tenancy_routes_store_global_and_the_cap_still_bounds_detector_churn	fail	pass

DaemonTests 280/280 net10.0. Further suites running locally; will report.

Housekeeping

The new reserved row is excluded from ProjectionProgressStatement and from the blanket UPDATE in TryCorrectProgressInDatabaseAsync, same as the allocation fence. #5164's isProgressRow allowlist already ignores unrecognised bookkeeping names, so WaitForNonStaleProjectionDataAsync needed no change — which is that hardening paying for itself immediately.

🤖 Generated with Claude Code

[jeremydmiller]

Local sweep complete on net10.0: DaemonTests 280/280, EventSourcingTests 1627/1627, TenantPartitionedEventsTests 238/240 (2 pre-existing skips).