Stop writing a periodic per-shard heartbeat nobody reads (#622)

[jeremydmiller]

Closes #622. Rebased onto main now that #625 (the writer-accumulation fix) has merged — #621 is the bug that multiplied this cost without bound; this is the cost model that remains after it.

What the audit found

EnableExtendedProgressionTracking cost one pooled connection and one transaction per database per node every 5 seconds — ~37 connection acquisitions/sec/node on a 512-shard-database deployment, to keep 6–12 rows per database current, writing into the same rows the progress writer updates. It made a production web app unresponsive (marten#5167).

And nothing reads it. Not JasperFx — the only read of LastHeartbeat anywhere is the writer checking its own input before writing it, and the CLI daemon status grid renders sequence and high-water only. Not CritterWatch, the sole intended consumer, which obtains agent status and heartbeats by reflecting off in-memory objects (CurrentAgents(), the tracker's ShardState) and drops the extended columns it already receives over the wire. Its AgentDown alert fires at heartbeatAge > 60s, on a 30s tick, fed by a 15s poll — a 5s write is a 12× oversample of something nothing downstream can observe below 15s resolution.

The change

Proposal items 1–3 from the issue.

1. Stop the periodic beat. ExtendedProgressionWriter drops non-transition publications by default — outright, rather than queueing them, so a stale beat can't ride along on the next transition write either.

2. Keep the transitional columns. Started/Paused/Stopped writes are untouched: rare, and the data the "durable across a crash" story was actually about.

3. Make the interval configurable. HeartbeatWriteInterval now defaults to TimeSpan.Zero (off) and is reachable from application code through the new DaemonSettings.ExtendedProgressionHeartbeatInterval. Before this it was a hardcoded 5 seconds that no DaemonSettings knob could reach — the field was private on the daemon, exposed by no property, and the only code that set it was the class's own unit test. A positive value restores the old behavior at that cadence; that is the compatibility hatch, not the recommended shape.

The coalescing/batching machinery is unchanged and still fully tested — it just has to be asked for now.

Also

Corrects the comment at the Path B tenant high-water heartbeat (JasperFxAsyncDaemon.cs), which claimed the beat "surfaces the same beat on the live Tracker (and the ExtendedProgression columns)". It never reached the columns — OnNext drops HighWaterMark and AllProjections states, as skips_high_water_mark_and_all_projections_states pins. The issue asked for this regardless of the outcome.

What is deliberately not done

• Per-node liveness rows. The issue's replacement for the per-shard beat already exists on the consuming side (CritterWatch's NodeHeartbeat, #837). Nothing to add here.
• Folding idle-shard liveness into the progress writer's transaction. That's the "if per-shard liveness is wanted despite the above" alternative; it's a larger change and should follow evidence that someone wants the signal.
• The permanently-NULL columns (pause_reason, running_on_node). Left alone — they're free now that they're only written on transition.

Tests

• ExtendedProgressionWriterTests: new coverage for the default (no periodic write even after 5 minutes of publications), transitions still written with the beat off and not carrying a dropped heartbeat along, and a positive interval restoring the beat. The existing periodic-beat tests now opt in explicitly, which is the honest statement of where that behavior lives.
• ExtendedProgressionHeartbeatIntervalTests: end-to-end through a real daemon + shared tracker, proving DaemonSettings.ExtendedProgressionHeartbeatInterval reaches the writer — null and TimeSpan.Zero are off, a positive value is on, transitions persist regardless.

Full EventTests suite passes (693).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Fuk1GybEEmohFmboJuM4Po