High-water health check no longer reads a heartbeat column nothing writes (#5174)

[jeremydmiller]

Closes #5174.

The bug

HighWaterHealthCheckExtensions filtered AllProjectionProgress down to the HighWaterMark rows and used row.LastHeartbeat as its primary staleness signal. That column is never written for those rows:

• ExtendedProgressionWriter.OnNext returns early for ShardState.HighWaterMark / AllProjections — pinned upstream by skips_high_water_mark_and_all_projections_states.
• HighWaterDetector.MarkHighWaterMarkInDatabaseAsync / MarkHighWaterForTenantAsync and BulkEventAppender's high-water upserts write name and last_seq_id only.
• WriteExtendedProgressionAsync is the only statement that sets heartbeat, and it only ever receives shard rows.

So the branch was unreachable in production and the check silently degraded to the gap heuristic — which, under UseTenantPartitionedEvents, is explicitly skipped for per-tenant rows. The per-tenant path asserted nothing. The tests didn't catch it because they seeded heartbeat with raw SQL, passing against a state the daemon never produces.

(JasperFx 2.39.1 already corrected the JasperFxAsyncDaemon doc comment that made this assumption look sound — #622.)

The fix

Not option 1 from the issue (start writing high-water heartbeats): that adds a periodic write stream at exactly the scale #5167 is about, and #622 has just turned the existing periodic beat off by default for cost reasons. Instead, the issue's option 2 — with the better signal it points at:

Per-tenant rows use last_updated. mt_mark_event_progression always sets last_updated = transaction_timestamp(), and TenantedHighWaterCoordinator calls it on every vectorized poll, mark advance or not. Its age is therefore a genuine "the poll loop is cycling" signal — the thing the phantom heartbeat was supposed to provide — at zero extra write cost and with no dependency on EnableExtendedProgressionTracking. This is a real capability the check did not have before.

The store-global row keeps the gap heuristic. There last_updated only moves when the mark advances (persistDetectedMarkAsync returns early when !HasChanged), so it is not a liveness signal and pretending otherwise would false-positive on a quiet store. The docs now state plainly that a store-global loop wedged while caught up is not detectable from the progression table, and point at IProjectionDaemon.HighWaterAgent.IsStale / LastPolledAt for the in-process case.

The heartbeatOnly flag is renamed skipGapHeuristic, which is what it actually meant.

Bonus

The row read no longer goes through AllProjectionProgress. That pulled every projection × tenant row on every probe to keep two of them, and still couldn't see last_updated — ShardState has no field for it. It's now a targeted two-row query.

Tests

DaemonTests.ManualOnly/HealthChecks/HighWaterHealthCheckTests.cs reworked: the heartbeat seeding helper is gone, seedProgressionRowAsync now takes a last_updated. New/changed cases:

• fresh per-tenant poll + mark behind → Healthy
• stale per-tenant poll + mark caught up → Unhealthy (the case the gap heuristic is blind to)
• the_extended_progression_heartbeat_column_is_never_consulted — the pin: with a fresh heartbeat written directly, a stuck store-global mark must still trip the gap heuristic
• a per-tenant row is never gap-assessed
• ExternallyManaged opt-in now asserts via the per-tenant poll signal

23/23 pass on net10.0. Docs updated (docs/events/projections/healthchecks.md), markdownlint + cspell clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_017CTtw2kVRSZKp1p5RTxgAy