High-water allocation fence: read is_called so mark 0 is fenceable (#5091) - #5092
Merged
Conversation
…sequence The #4953/#5057 allocation fence was built from the sequence's RESERVED ceiling (mt_events_sequence.last_value), and Postgres reports last_value = 1 for a sequence nothing has drawn from yet. No poll could ever report a value at or below a stuck mark of 0, so findAllocationFence(0) always returned null: a leading gap — the mark pinned at 0 under a hole at the very start of the sequence — could never be fenced, and any permanently idle open transaction held it forever. That is the shape reported in #5090 ("holding before the sequence gap above 0"), where the quiescent-session exclusion was inert on 9.21.0. The statistics reading now also carries is_called, which distinguishes "1 has been handed out" from "1 is the next value to hand out", giving the highest ALLOCATED sequence number — 0 for a pristine sequence. The fence compares against that instead, which makes mark 0 fenceable without weakening the proof: everything above the allocated high was handed out strictly after the poll that observed it. The reserved ceiling still bounds how far a proven-dead skip may advance (StuckGapObservation.ReservedCeiling is unchanged), and per-tenant partitioned stores report no allocation reading at all, so they keep today's conservative no-fence behaviour. Also documents that a process starting more than one daemon-hosting IHost over its lifetime should prefer Events.UseAdvisoryLockTransaction = false, since a session-scoped leadership lock holds no open transaction that gap detection has to treat as a possible in-flight append. Tests cover the dead leading gap skipping past an idle advisory-lock session, a live reserver of the leading gap still holding the mark, and a detector that never saw the pristine sequence still holding conservatively. The first two are verified RED before the fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JasperFx 2.36.3 (#592/#593): ProjectionCoordinatorBase.StartAsync cancels AND drains any existing leadership loop instead of disposing the old CancellationTokenSource without cancelling it, so a bare ResumeAsync can no longer orphan a live loop that outlives StopAsync. Weasel 9.20.2 (weasel#396/#397): AdvisoryLock can no longer strand a handle acquired after DisposeAsync drained — the actual cause of the permanently 'idle in transaction' advisory-lock session in #5090. Weasel 9.20.2 also carries weasel#399/#400, found while making this bump: 9.18.0's computed-column delta detection made TableColumn.MatchesForDelta compare through the non-virtual Equals(TableColumn), bypassing subclass overrides of Equals(object). That is the seam RevisionColumn uses to tolerate an existing bigint mt_version (#4614/#4742), so every Weasel from 9.18.0 through 9.20.1 fails Bug_4614's assert-check test with an empty change set. 9.20.2 is the first version this repo can move to. Verified on the published packages: CoreTests 494, DaemonTests 260, EventSourcingTests 1478 — all green on net10.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5091. Third and last piece of the #5090 chain; the other two shipped upstream and are adopted here.
The fence could never fire at mark 0
#5057 added the allocation fence so a provably-quiescent
idle in transactionsession can be ruled out as a candidate reserver of a stale sequence gap instead of pinning the high water mark forever.findAllocationFence(mark)looks for a recorded poll where the sequence was at or below the stuck mark — but it compared against the reserved ceiling (mt_events_sequence.last_value), and Postgres reportslast_value = 1for a sequence nothing has drawn from yet. No poll can ever report a value at or below a mark of 0, so a leading gap — the mark pinned at 0 under a hole at the very start of the sequence — was never fenceable, and any permanently idle open transaction held it forever. That is the reporter's shape in #5090:Daemon high water detection is holding before the sequence gap above 0, with the #5057 exclusion inert on 9.21.0.is_calleddistinguishes "1 has been handed out" from "1 is the next value to hand out", which yields the highest allocated sequence number — 0 for a pristine sequence. The fence now compares against that.This does not weaken the proof: everything above the allocated high was handed out strictly after the poll that observed it, which is the same monotonicity argument #5057 relies on. Unchanged in both directions:
StuckGapObservation.ReservedCeilingstill bounds how far a proven-dead skip may advance, and still comes fromlast_value.AllocatedSequenceHighis null there and they keep today's conservative no-fence behaviour.Tests
DaemonTests/Bugs/Bug_5091_allocation_fence_at_mark_zero.cs:a_dead_leading_gap_skips_despite_an_idle_advisory_lock_session— the fix.a_live_reserver_of_the_leading_gap_still_holds_the_mark— fencing mark 0 must not become a licence to skip live appends.a_detector_that_never_saw_the_pristine_sequence_still_holds_conservatively— the documented degradation is intact.The first two are verified RED before the fix (mark stuck at 0 in both); the third passes either way, as intended.
Dependencies — this is what actually fixes #5090
ProjectionCoordinatorBase.StartAsynccancels and drains any existing leadership loop rather than disposing the oldCancellationTokenSourcewithout cancelling it. A bareResumeAsyncused to orphan a live loop that outlivedStopAsync, re-took leadership afterReleaseAllLocks, and won a lock handle after that lock had been disposed.AdvisoryLockcan no longer strand a handle acquired afterDisposeAsyncdrained — the actual cause of Stopped host can leak the HotCold coordinator's advisory-xact-lock session ("idle in transaction"), which 9.16.1 gap detection treats as an eternal in-flight append — stalling every later daemon in the same process #5090's permanentlyidle in transactionsession onpg_try_advisory_xact_lock.Weasel 9.20.2 also carries weasel#399/#400, a regression found while making this bump and worth flagging: 9.18.0's computed-column delta detection routed every column through
TableColumn.MatchesForDelta, where a bareEquals(actual)binds to the protected non-virtualEquals(TableColumn)overload and bypasses subclass overrides ofEquals(object). That override is exactly howRevisionColumntolerates an existingbigintmt_version(#4614/#4742), soBug_4614's assert-check test fails with an empty change set on every Weasel from 9.18.0 through 9.20.1. 9.20.2 is the first version this repo can move to.Docs
Notes that a process starting more than one daemon-hosting
IHostover its lifetime — the usual xUnit integration-suite shape — should preferEvents.UseAdvisoryLockTransaction = false, since a session-scoped leadership lock holds no open transaction for gap detection to treat as a possible in-flight append. markdownlint and cspell clean.Verification
On the published packages, net10.0: CoreTests 494/494, DaemonTests 260/260, EventSourcingTests 1478/1478 (7 skipped).
🤖 Generated with Claude Code