SkipStaleGapsDespiteLiveTransactionsAfter survives detector restarts (evidence-based stall onset) - #6
Closed
uniquelau wants to merge 2 commits into
Closed
SkipStaleGapsDespiteLiveTransactionsAfter survives detector restarts (evidence-based stall onset)#6uniquelau wants to merge 2 commits into
uniquelau wants to merge 2 commits into
Conversation
…s detector restarts The JasperFx#5057 cap is documented as the bounded override for the transaction- evidence hold: once a gap has been stuck past the cap, the skip proceeds even though a candidate reserver still appears alive. But the clock it was measured from — the detector-scoped _stuckGap observation — resets with every fresh HighWaterDetector, and a daemon resumed after the gap formed also has no allocation history to fence the probe with, so long-lived idle-in-transaction sessions (Wolverine's advisory-lock listeners) hold the gate on every cycle. Under daemon restarts or managed-distribution agent churn shorter than the cap, the override never fires and a dead gap pins the high water mark indefinitely — the exact unbounded stall the cap exists to rule out. Fix: floor the cap comparison with durable evidence of the stall's onset. mt_event_progression.last_updated alone is NOT that evidence — it records the last ADVANCE and ages just as much on a caught-up idle store, where it would fire the cap against a seconds-old live append. The onset is instead the timestamp of the earliest committed event ABOVE the pinned mark (its sequence number was allocated strictly after the gap's, so its append postdates the gap's birth), clamped no earlier than the mark's last advance, computed in one server-side statement. When nothing is committed above the mark there is no durable evidence and the in-memory clock governs alone, which correctly holds for a store whose first append after an idle stretch is still in flight. The stale threshold stays on the in-memory clock, so a fresh detector still gives a just-appeared gap a full settle window, and a null cap (the default) is untouched — no change to the never-knowingly-skip posture of JasperFx#4977. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bal path Conjoined tenancy shares one event sequence and one HighWaterMark row — the vectorized per-tenant detection path is gated on UseTenantPartitionedEvents, not on TenancyStyle — so a conjoined store's daemon takes the same store-global DetectInSafeZone path as a single-tenant store, durable cap floor included. The new test pins that routing (SupportsTenantPartitioning false, the per-tenant API collapsing to the store-global reading) and re-runs the detector-churn scenario on a conjoined store: it fails without the durable floor and passes with it. The per-tenant cap check keeps its detector-scoped clock; the comment there records why the durable floor is inert on that path (the TenantedHighWaterCoordinator re-marks each tenant's progression row on every vectorized poll, so its last_updated clamps any evidence onset to roughly now). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Promoted upstream: issue JasperFx#5108, draft PR JasperFx#5109 (same branch). Closing the staging copy. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #5, section 3 (staged on this fork for review; intended for JasperFx/marten once approved).
What
The cap's stuck-gap clock (
_stuckGap.Since) is detector-scoped, so daemon churn — managed-distribution rebalancing, pause/resume cycles, restarts — resets it each time; with detector lifetimes shorter than threshold+cap, the documented "bounded override" never fires and a genuinely dead gap (rolled-back optimistic-concurrency loser) holds the high-water mark indefinitely, withAdvanceHighWaterMarkToLatestAsync()as the only recovery.Fix: floor the cap comparison with a durable, evidence-based stall onset computed server-side in one statement:
An event committed above the mark drew its sequence strictly after the gap's numbers — proof the gap has existed at least that long. When nothing is committed above the mark (an idle store's first live append), there is no durable floor and the in-memory clock governs, which holds.
Deliberately rejected design, pinned by a guard test: flooring with
last_updatedalone. It measures time-since-advance, which conflates idle time with stall time — an overnight-idle store's first live append would have its in-flight gap cap-skipped instantly, committing events behind the mark (permanently unprojected). The adversarial repro for that shape is nowidle_store_first_live_append_is_not_cap_skippedand must keep holding.Scope guarantees: null-cap default path byte-identical (JasperFx#4977's evidence gating untouched); the liveness probe is consulted unchanged; the evidence query runs only when a cap is configured and the probe reports live; the
UseTenantPartitionedEventspath is deliberately not floored (its progressionlast_updatedis a per-poll heartbeat — any onset clamps to ~now; why-comment in code, follow-up conversation welcome). Rich-append caveat documented at the handler:mt_events.timestampis client-written, so evidence age is bounded by client skew (QuickAppend writestransaction_timestamp()).Tests (first direct coverage of the cap)
Runs (net10.0, dedicated database): new suite 6/6 (timing-sensitive trio stable across 3 consecutive runs); targeted store-global set 32/32; per-tenant suites 16/16;
advanced_async_tracking7/7. net9.0 TFM not run locally (no runtime installed) — CI covers it.The conjoined-tenancy test also pins routing: a
TenancyStyle.Conjoinedstore reportsSupportsTenantPartitioning == falseand takes the store-global detection path — the reported incident's configuration.🤖 Generated with Claude Code