ExtendedProgressionWriter: don't lose the first status transition, and say when a duplicate writer attaches - #631
Merged
Conversation
A second ExtendedProgressionWriter on one database used to announce itself as lock contention -- two writers issuing multi-row UPDATEs over the same rows in plan-dependent order is a deadlock hazard. #630 made those writes one row per transaction in shard-name order, which makes a duplicate writer harmless to correctness and therefore SILENT: it just quietly does the same work twice on a second connection. That is worth knowing about, because the condition it indicates is real. The tracker is shared per database and building a daemon does not go through a cache, so a lifecycle bug can leave two STARTED daemons on one database, each arming its own writer. #621 gated arming so a daemon built only to READ state never subscribes one, which removed the common cause -- it did not remove the possibility. ShardStateTracker now logs a warning when an IExclusiveTrackerObserver attaches to a tracker that already has one of its role. Deliberately reported and not refused: the duplicate observer is the symptom, not the bug, and swallowing the subscription would hide the lifecycle bug rather than surface it. Unsubscribing removes the listener, so an ordinary daemon restart does not trip it -- pinned by its own test, since a warning that fires on every restart would be worth nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CTtw2kVRSZKp1p5RTxgAy
…n again (#631) Every store's extended-progression write is update-only -- "shards without a progression row yet are skipped silently" is the documented contract, and the row is not created until a shard's first batch commits. SubscriptionAgent.StartAsync publishes its Started state at floor 0, before that, so on a fresh shard the write matches zero rows and lands nowhere. That was harmless while the 5s periodic beat existed: it wrote again a moment later, once the row was there. #622 turned the beat off by default, which made Started the ONLY telemetry write -- so agent_status, heartbeat and running_on_node stayed NULL for the entire life of a healthy agent. Precisely the case those columns exist for: a consumer polling the database because the publishing node is down and there is no in-memory ShardState to read. Reproduced end to end against PostgreSQL through Marten: a Balanced two-node cluster ran its projection to sequence 30 and reported running_on_node=NULL, agent_status=NULL, heartbeat=NULL. Bouncing the agent so the same Started transition lands on a row that now exists produced agent_status=Running with a heartbeat, which isolates the cause to row absence rather than the transition write itself. A transition published at sequence 0 now marks the shard, and the first later publication carrying a committed sequence -- proof the row exists -- writes it once and clears the mark. Later heartbeats are dropped again exactly as #622 intends. Lock cost, since marten#5167 is why this file is careful: one extra single-row UPDATE per shard per agent start, one-shot, on the same one-row-per-transaction shard-name- ordered path -- one brief row lock, no convoy. The sequence-0 write it compensates for takes no lock at all, because it matches no rows. The ordering is safe by construction: the store commits the batch that creates the row BEFORE the agent calls MarkSuccessAsync, so the publication that triggers the replay always follows the row write. Also corrects two now-stale doc comments: ProjectionProgressRow.AgentStatus still claimed no store writes it (untrue since #537), and neither it nor ShardState.LastHeartbeat warned that the persisted heartbeat freezes at the last transition once #622's beat is off -- so a monitor thresholding now - LastHeartbeat off the persisted column reports every shard as dead. #5180 restored reading that column in the same unreleased batch, which makes saying so load-bearing. Both new tests fail against the neutralized guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016v2Aijyo8MX2AdPUZL5VtG
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two
ExtendedProgressionWriterlifecycle items. The second commit is a release blocker for the 2.39.3 line — it was caught proving the unreleased Marten/JasperFx packs against CritterWatch, where it turned a green regression guard red.fix(#631): a status transition with no progression row to land on is written again
Every store's extended-progression write is update-only — "shards without a progression row yet are skipped silently" is the documented contract — and the row is not created until a shard's first batch commits.
SubscriptionAgent.StartAsyncpublishes itsStartedstate at floor 0, before that, so on a fresh shard the write matches zero rows.That was harmless while the 5s periodic beat existed: it wrote again a moment later. #622 turned the beat off by default, which made
Startedthe only telemetry write — soagent_status,heartbeatandrunning_on_nodestay NULL for the entire life of a healthy agent. Precisely the case those columns exist for: a consumer polling the database because the publishing node is down and there is no in-memoryShardStateto read.Reproduced end to end against PostgreSQL through Marten. A Balanced two-node cluster ran its projection to sequence 30:
Bouncing the agent so the same
Startedtransition lands on a row that now exists:which isolates the cause to row absence, not the transition write itself.
The fix. A transition published at sequence 0 marks the shard; the first later publication carrying a committed sequence — proof the row exists — writes it once and clears the mark. Later heartbeats are dropped again exactly as #622 intends.
Lock cost, since marten#5167 is why this file is careful:
It rides marten#5186's one-row-per-transaction, shard-name-ordered path, so one brief row lock and no convoy. The sequence-0 write it compensates for takes no lock at all — it matches no rows. And the ordering is safe by construction:
ProjectionExecutioncommits the batch that creates the row before callingMarkSuccessAsync, and MarkSuccess is what publishes the state that triggers the replay.Also corrects two stale doc comments:
ProjectionProgressRow.AgentStatusstill claimed no store writes it (untrue since #537), and neither it norShardState.LastHeartbeatwarned that the persisted heartbeat freezes at the last transition once #622's beat is off. marten#5180 restored reading that column in the same unreleased batch, which makes saying so load-bearing.feat: a duplicate ExtendedProgressionWriter says so
(Pre-existing commit on this branch.) #630 made the writes one row per transaction, which makes a duplicate writer harmless to correctness and therefore silent — it just does the same work twice on a second connection.
ShardStateTrackernow logs a warning when anIExclusiveTrackerObserverattaches to a tracker that already has one of its role. Reported, not refused: the duplicate observer is the symptom, not the bug. Unsubscribing removes the listener, so an ordinary daemon restart does not trip it.Verification
EventTests730/730 on net9.0 and net10.0.RunningOnNodeMultiNodeTests, which was 42/42 on released Marten 9.22.3 and went 41/42 on the unreleased pack.🤖 Generated with Claude Code
https://claude.ai/code/session_016v2Aijyo8MX2AdPUZL5VtG