fix(#572): close the ShardStatusWatcher lost-wakeup race rather than narrowing it

[jeremydmiller]

Fixes #572.

The remaining hole after #568

#568 gave ShardStatusWatcher a snapshot re-check so a state published before the wait was set up could still be found. It narrowed the window but could not close it, because the snapshot it re-read was written by the very publication walk it was racing.

ShardStateTracker subscribed itself as the first listener and updated _states from its own OnNext, on the block's consumer thread. So:

1. publish captures the listener list and begins the walk;
2. a watcher calls Subscribe — too late for that walk, no OnNext;
3. the watcher re-checks CurrentStates() — but the tracker's own OnNext hasn't run yet, so the state isn't in the snapshot either.

Both paths miss. If nothing further is ever published for that shard — exactly a high water agent that has reached the head and has nothing left to detect — the wait can only end in a timeout, however generous. That is why raising Marten's per-step budget from 2s to 10s didn't help: the wakeup was lost, not late.

The fix

PublishAsync records the state synchronously, before posting to the block. Delivery to listeners stays asynchronous, but "what the tracker knows" is now synchronous with the caller, so a snapshot read can never lag a publication that has already happened. This also removes the second writer of the state map — the consumer thread replaying an older state could momentarily walk CurrentState backwards.

Subscribe-then-snapshot is now one atomic step (SubscribeAndCaptureCurrentStates) taken under the same lock that records the state and that captures the listener list for a walk. That totally orders the two sides: either the watcher wins the lock and the following walk sees it as a listener, or the publication wins and the state is in the snapshot handed back. There is no longer an interleaving where neither path sees it.

Observers are still notified outside the lock, so user code in OnNext cannot deadlock the tracker by publishing or subscribing from a callback.

Incidental, and a real bug on its own: _listeners and _states were both read-modify-write on plain fields. Two concurrent Subscribe calls could drop one subscriber outright — a permanently lost wakeup rather than a delayed one. All mutations now take the lock, and HighWaterMark is read/written volatile.

Tests

Five new tests in ShardStateTrackerTests. Three of them fail against main as it stands:

• watcher_sees_a_state_that_is_published_but_not_yet_delivered — parks the delivery thread inside an observer so the next publication is provably queued and un-walked, then sets up the wait. Nothing can arrive at OnNext while the gate is shut, so the watcher must find it in the snapshot or burn its whole timeout on a mark already reached. This is the deterministic form of ShardStatusWatcher can still miss an already-published state after #568 (lost wakeup, not a late one) #572.
• wait_for_shard_condition_sees_a_state_that_is_published_but_not_yet_delivered — same, through WaitForShardCondition.
• concurrent_subscribers_are_never_silently_dropped — 100 subscribers registering in parallel must all be notified.

Plus two that document the invariant the fix rests on: the state map is never behind a publication the caller has already made, and never regresses to an older state still in flight.

EventTests (642) and EventStoreTests (72) are green.

@JasperFx/marten#5054 — happy to have this validated against the Marten suite before it ships.

🤖 Generated with Claude Code