Skip to content

ExtendedProgressionWriter: don't lose the first status transition, and say when a duplicate writer attaches - #631

Merged
jeremydmiller merged 2 commits into
mainfrom
feat/duplicate-progression-writer-warning
Aug 4, 2026
Merged

ExtendedProgressionWriter: don't lose the first status transition, and say when a duplicate writer attaches#631
jeremydmiller merged 2 commits into
mainfrom
feat/duplicate-progression-writer-warning

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Two ExtendedProgressionWriter lifecycle items. The second commit is a release blocker for the 2.39.3 line — it was caught proving the unreleased Marten/JasperFx packs against CritterWatch, where it turned a green regression guard red.

fix(#631): a status transition with no progression row to land on is written again

Every store's extended-progression write is update-only — "shards without a progression row yet are skipped silently" is the documented contract — and the row is not created until a shard's first batch commits. SubscriptionAgent.StartAsync publishes its Started state at floor 0, before that, so on a fresh shard the write matches zero rows.

That was harmless while the 5s periodic beat existed: it wrote again a moment later. #622 turned the beat off by default, which made Started the only telemetry write — so agent_status, heartbeat and running_on_node stay NULL for the entire life of a healthy agent. Precisely the case those columns exist for: a consumer polling the database because the publishing node is down and there is no in-memory ShardState to read.

Reproduced end to end against PostgreSQL through Marten. A Balanced two-node cluster ran its projection to sequence 30:

Trip agent runs on Wolverine node 2. mt_event_progression:
  running_on_node=NULL, agent_status=NULL, heartbeat=NULL, last_seq_id=30

Bouncing the agent so the same Started transition lands on a row that now exists:

after first start (row created by progress AFTER Started): agent_status=NULL,    heartbeat=NULL
after restart      (row already existed):                  agent_status=Running, heartbeat=2026-08-04T17:15:13Z

which isolates the cause to row absence, not the transition write itself.

The fix. A transition published at sequence 0 marks the shard; the first later publication carrying a committed sequence — proof the row exists — writes it once and clears the mark. Later heartbeats are dropped again exactly as #622 intends.

Lock cost, since marten#5167 is why this file is careful:

pre-#622 #622 alone with this fix
writes per shard 12/min, forever 1 at start, lost 1 at start (lands nowhere) + 1 replay, one-shot
row locks taken 12/min 0 1, briefly

It rides marten#5186's one-row-per-transaction, shard-name-ordered path, so one brief row lock and no convoy. The sequence-0 write it compensates for takes no lock at all — it matches no rows. And the ordering is safe by construction: ProjectionExecution commits the batch that creates the row before calling MarkSuccessAsync, and MarkSuccess is what publishes the state that triggers the replay.

Also corrects two stale doc comments: ProjectionProgressRow.AgentStatus still claimed no store writes it (untrue since #537), and neither it nor ShardState.LastHeartbeat warned that the persisted heartbeat freezes at the last transition once #622's beat is off. marten#5180 restored reading that column in the same unreleased batch, which makes saying so load-bearing.

feat: a duplicate ExtendedProgressionWriter says so

(Pre-existing commit on this branch.) #630 made the writes one row per transaction, which makes a duplicate writer harmless to correctness and therefore silent — it just does the same work twice on a second connection. ShardStateTracker now logs a warning when an IExclusiveTrackerObserver attaches to a tracker that already has one of its role. Reported, not refused: the duplicate observer is the symptom, not the bug. Unsubscribing removes the listener, so an ordinary daemon restart does not trip it.

Verification

  • EventTests 730/730 on net9.0 and net10.0.
  • Both new tests verified red against the neutralized guard.
  • Downstream: the Marten-level reproduction above is CritterWatch's RunningOnNodeMultiNodeTests, which was 42/42 on released Marten 9.22.3 and went 41/42 on the unreleased pack.

🤖 Generated with Claude Code

https://claude.ai/code/session_016v2Aijyo8MX2AdPUZL5VtG

jeremydmiller and others added 2 commits August 4, 2026 11:28
A second ExtendedProgressionWriter on one database used to announce itself as lock
contention -- two writers issuing multi-row UPDATEs over the same rows in plan-dependent
order is a deadlock hazard. #630 made those writes one row per transaction in
shard-name order, which makes a duplicate writer harmless to correctness and therefore
SILENT: it just quietly does the same work twice on a second connection.

That is worth knowing about, because the condition it indicates is real. The tracker is
shared per database and building a daemon does not go through a cache, so a lifecycle bug
can leave two STARTED daemons on one database, each arming its own writer. #621
gated arming so a daemon built only to READ state never subscribes one, which removed the
common cause -- it did not remove the possibility.

ShardStateTracker now logs a warning when an IExclusiveTrackerObserver attaches to a
tracker that already has one of its role. Deliberately reported and not refused: the
duplicate observer is the symptom, not the bug, and swallowing the subscription would hide
the lifecycle bug rather than surface it. Unsubscribing removes the listener, so an
ordinary daemon restart does not trip it -- pinned by its own test, since a warning that
fires on every restart would be worth nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CTtw2kVRSZKp1p5RTxgAy
…n again (#631)

Every store's extended-progression write is update-only -- "shards without a
progression row yet are skipped silently" is the documented contract, and the row is
not created until a shard's first batch commits. SubscriptionAgent.StartAsync publishes
its Started state at floor 0, before that, so on a fresh shard the write matches zero
rows and lands nowhere.

That was harmless while the 5s periodic beat existed: it wrote again a moment later,
once the row was there. #622 turned the beat off by default, which made Started the ONLY
telemetry write -- so agent_status, heartbeat and running_on_node stayed NULL for the
entire life of a healthy agent. Precisely the case those columns exist for: a consumer
polling the database because the publishing node is down and there is no in-memory
ShardState to read.

Reproduced end to end against PostgreSQL through Marten: a Balanced two-node cluster ran
its projection to sequence 30 and reported running_on_node=NULL, agent_status=NULL,
heartbeat=NULL. Bouncing the agent so the same Started transition lands on a row that
now exists produced agent_status=Running with a heartbeat, which isolates the cause to
row absence rather than the transition write itself.

A transition published at sequence 0 now marks the shard, and the first later
publication carrying a committed sequence -- proof the row exists -- writes it once and
clears the mark. Later heartbeats are dropped again exactly as #622 intends.

Lock cost, since marten#5167 is why this file is careful: one extra single-row UPDATE
per shard per agent start, one-shot, on the same one-row-per-transaction shard-name-
ordered path -- one brief row lock, no convoy. The sequence-0 write it compensates for
takes no lock at all, because it matches no rows. The ordering is safe by construction:
the store commits the batch that creates the row BEFORE the agent calls MarkSuccessAsync,
so the publication that triggers the replay always follows the row write.

Also corrects two now-stale doc comments: ProjectionProgressRow.AgentStatus still claimed
no store writes it (untrue since #537), and neither it nor ShardState.LastHeartbeat warned
that the persisted heartbeat freezes at the last transition once #622's beat is off -- so
a monitor thresholding now - LastHeartbeat off the persisted column reports every shard as
dead. #5180 restored reading that column in the same unreleased batch, which makes saying
so load-bearing.

Both new tests fail against the neutralized guard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016v2Aijyo8MX2AdPUZL5VtG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant