[jeremydmiller]

Fixes #1027. Two commits, so the fix can be reviewed (or taken) without the new health check.

1. The #1027 fix (a97e5f2)

What actually happened

RestartAsync tore the agent down before it knew it could bring it back up:

_timer.Stop();
IsRunning = false;
await StartAsync();   // Detect() throws while the database is down

When Detect() threw, the agent was left with:

• the timer stopped, so checkState never ran again;
• IsRunning == false, which makes IsStale return false. Even a running timer would have seen nothing wrong. It also makes JasperFxAsyncDaemon.IsRunning false;
• the old loop exiting on its next pass, because it checks IsRunning.

That last point is the irony of the incident. The wedged loop was stuck in a Detect on a dead connection. That Detect errored at 01:48:52 (it was not cancelled). Left alone, that loop would have kept cycling and recovered on its own once the database came back at 01:50:35. Clearing IsRunning is what killed it. The 01:48:53 log line, "has detected a cancellation", was a third, unrelated message, printed for every exit reason.

The change

• Detect first, then swap. StartAsync is split into Detect() + launchAsync(initial). RestartAsync runs Detect() first, bounded by HighWaterStalenessThreshold so a hung connection cannot pin the watchdog. Only once it has a mark does it call launchAsync, which bumps the loop generation and retires the old loop. If the Detect fails, RestartAsync throws and leaves the agent exactly as it found it: same loop, IsRunning unchanged, timer running. The staleness check therefore keeps firing, and the watchdog retries once per HighWaterStalenessThreshold until the database answers. Meanwhile the old loop can still recover by itself. This is the reporter's _restartPending idea taken one step further: there is no pending state to track, because nothing gets torn down until the replacement exists.
• A stop always wins. IsRunning now stays true for the whole restart, so if it reads false after the Detect, StopAsync ran in the meantime. The watchdog's restart then backs off instead of bringing the loop and timer back. The public RestartAsync() keeps its old "start it if it isn't running" behaviour.
• A retired loop never writes (second bug found here). A retired loop's hung Detect can answer minutes later. The loop used to check its generation only at the top of the cycle, so it could publish a reading taken before the restart and move the tracker's mark backwards. markProgressAsync doesn't guard against that, and neither does ShardStateTracker.MarkHighWaterAsync. The loop now checks isCurrentLoop(generation) after every Detect/DetectInSafeZone and inside markProgressAsync.
• Accurate exit log: "detected a cancellation", "was replaced by a restart", or "was stopped", depending on what actually happened.

2. A health check that watches the mark itself (0c95eda)

The #539 heartbeat proves the loop is cycling, not that the mark is getting anywhere. It also only means something while the loop is the thing doing the detecting. The new check watches the mark itself:

• New DaemonSettings.HighWaterProgressStallThreshold: default 2 minutes; null or non-positive disables it. It is also on IReadOnlyDaemonSettings as a default interface member, so an implementer compiled before it existed keeps loading.
• Once the tracker's mark has been still for that long, the watchdog calls IHighWaterDetector.FetchCommittedHighWaterCeilingAsync directly, outside the poll loop, at most once per window. That is Marten's select max(seq_id). If events are committed above the mark, it logs a warning with the mark and ceiling and restarts the agent.
• A quiet store costs one cheap query per window. It uses the committed ceiling, not HighestSequence, so a rolled-back tail of reserved sequence numbers is not a stall. A probe that throws is not a stall either, because the outage path above already handles that. The probe is bounded by HighWaterStalenessThreshold because it holds the remediation guard.
• It can never skip an event. A restart re-reads the mark and never advances it. A gap the detector holds on purpose (Marten's transaction-evidence hold behind an open transaction) stays held through any number of restarts. The warning says so, and its repetition is the signal to go and look for that transaction.
• It only runs while the agent itself believes it is running (the timer is stopped by StopAsync). An externally paused agent, such as PauseHighWaterAgentAsync under ExternallyManaged, is never second-guessed.

Testing

JasperFx (net10.0; no .NET 9 runtime in my environment):

• New HighWaterAgent_restart_failure_Tests:
  • the reporter's repro (fails 3 out of 3 runs on main);
  • the old loop's connection never comes back, so only the watchdog retrying can recover;
  • a retired loop answers late with an old mark;
  • StopAsync lands during a watchdog restart.
• New HighWaterAgent_progress_stall_Tests: a stuck mark is restarted without advancing; a quiet store is left alone; a failing probe is not a stall; null disables the probe entirely.
• Against the original agent, all 3 restart tests and the 4 probe-dependent tests fail. Removing just the stop guard fails the stop-race test.
• EventTests.Daemon.HighWater: 51 out of 51, three times in a row. Full EventTests: 1588 out of 1588.

Marten, against a local pack of this branch (2.90.3-local.1027.2, in place of 2.90.1) on PostgreSQL 16:

• Marten's full DaemonTests suite: 377 out of 377, retries off, net10.0.
• An end-to-end outage harness (throwaway, not committed). It SIGSTOPs every postgres process for 12s, so the loop's in-flight Detect hangs and the watchdog's restart cannot connect. It then thaws, appends 5 more events, and watches the mark:

	Released 2.90.1	This branch
Log sequence during the outage	identical to the issue: restart warning → detect failure → "detected a cancellation" → "Error trying to restart"	restart warning → restart fails → retried each window
After the database returns	mark frozen at 5 for 60s, daemon.IsRunning == false	restart succeeds in under 1s, old loop logs "replaced by a restart", mark reaches 10

• The stall probe, also on real Marten. An open transaction holds sequence 6 while 7..10 commit. The probe warns and restarts once per window, the mark stays at 5 the whole time (no skip), and after the rollback Marten's detector skips the dead gap and the mark reaches 10. Marten's gap clock lives in the detector, so the restarts did not reset it.

🤖 Generated with Claude Code

https://claude.ai/code/session_018E7QLXkj5qDNMPTT8VcaMT