Raised by marten#5167: on a 512-shard-database deployment, EnableExtendedProgressionTracking costs one pooled connection and one transaction per database per node every 5 seconds — ~37 connection acquisitions/sec/node — to keep 6-12 rows per database current, and it writes into the same rows the progress writer updates. It made a production web app unresponsive.
Separately filed: a writer-accumulation bug that multiplies this without bound. This issue is the cost model that remains after that's fixed.
What the audit found
I traced every consumer of the persisted extended columns across JasperFx, Marten and CritterWatch. Summary:
Nothing in JasperFx reads any of it. LastHeartbeat, AgentStatus, PauseReason, RunningOnNode — the only read of LastHeartbeat anywhere is ExtendedProgressionWriter.cs:81 checking its own input before writing it. The CLI daemon status grid renders sequence and high-water only. No descriptors, no health checks, no compliance tests.
CritterWatch — the sole intended consumer — never reads the columns either. Grep for agent_status, running_on_node, pause_reason, warning_behind_threshold, critical_behind_threshold across its source returns zero hits, and it has no uses of ReadProjectionProgressAsync/ProjectionProgressRow. It gets all of those values by reflecting off in-memory objects: IProjectionDaemon.CurrentAgents() (EventProgressionPoller.cs:242-246) and the ShardState the tracker publishes (CritterWatchShardObserver.cs:35-39). It does call AllProjectionProgress, and Marten does hydrate the extended fields onto those rows — and CritterWatch uses only ShardName, Sequence, TenantId. The data crosses the wire and is dropped.
The stated justification looks structurally unreachable. IEventDatabase.cs:141 justifies persistence as "a monitoring consumer polling the database (e.g. CritterWatch when the publishing node is down)." But the poller runs inside each monitored node, and since CritterWatch#806 it only polls databases whose agents that node owns. When a node dies nobody polls its databases until the agents redistribute (~5s, advisory locks) — at which point the new owner reads its own live agents in memory. I can't construct a path where the durable column is the source of truth. Happy to be corrected on this one.
The consuming threshold is 60 seconds. CritterWatch's AgentDown alert fires at heartbeatAge > 60s, on a 30s evaluation tick, fed by a 15s poll. A 5s write is a 12× oversample of something nothing downstream can observe below 15s resolution. The poller even synthesises the value when absent — LastHeartbeat: lastHeartbeat ?? now — treating "the agent object answered" as equivalent evidence. And the UI discounts a stale beat whenever derived health disagrees, with a comment saying a stale beat during a quiet stream "is NOT evidence the agent died."
Two of the columns are permanently NULL in the field. pause_reason (nothing in the stack stores a reason string) and running_on_node (only populated from ShardStateTracker.AssignedNodeNumber, which nothing in JasperFx ever assigns — Wolverine stamps it). CritterWatch overwrites RunningOnNode from the Wolverine agent roster anyway.
The signal that would actually indicate a cycling daemon is excluded from persistence. ExtendedProgressionWriter.cs:78 drops HighWaterMark and AllProjections states, pinned by skips_high_water_mark_and_all_projections_states. Meanwhile JasperFxAsyncDaemon.cs:1145-1149 claims the per-cycle tenant high-water beat "surfaces the same beat on the live Tracker (and the ExtendedProgression columns)" — it does not. That comment should be corrected regardless of what happens to this issue.
It causes a second scaling problem downstream. CritterWatch's ShardProgressionHandler.cs:194-210 records 1,007,378 updates and 439 autovacuums over 2.5 hours across ~9,881 rows on a 512-database console, "the leading suspect for the ingest stall in #919" — and notes the heartbeat is specifically the field that defeats its no-op write guard.
For history: the cost was raised and waved through at design time (CritterWatch .claude/docs/alerts.md:629 lists "adds DB writes even when idle" with the alternative of polling in-memory agent status; the recorded answer is "go ahead and do the DB write"). The proposed cadence there was 10s; 5s shipped.
Proposal
Split the periodic beat from the transitional status.
- Stop writing a periodic per-shard heartbeat. Liveness is a node property, and a per-node liveness row already exists on the consuming side — CritterWatch's
NodeHeartbeat (#837): per-node key, 30s cadence, SignalR push, dormant-node reaping at 5 minutes. Combined with the agent assignment grid (already read by the poller for database scoping) and the pre-existing last_updated column, that reconstructs everything any consumer currently renders or alerts on. This takes the reporter from ~37 acquisitions/sec/node to roughly zero in steady state.
- Keep the transitional columns, written on transition only —
agent_status, pause_reason, failure info. The writer already distinguishes these (isTransition on Started/Paused/Stopped). Those writes are rare and preserve the "durable across a crash" story for the data where it means something.
- Make
HeartbeatWriteInterval configurable as the compatibility hatch, not the fix. Today it's hardcoded 5s with no path from application code — the field is private on the daemon and no DaemonSettings knob reaches it.
The one thing genuinely lost is distinguishing "agent loop wedged, node healthy, no events to process" from "quiet stream" — and CritterWatch's UI has already decided not to act on that distinction.
If per-shard liveness is wanted despite the above, the cheaper shape is to fold it into the progress writer's existing transaction: the heartbeat rows are a subset of the rows that writer already updates, so a busy shard costs zero extra connections, and only idle shards need a periodic write. That also dissolves the lock contention marten#5167 measured, since telemetry and progress would stop being two transactions fighting over the same rows.
Raised by marten#5167: on a 512-shard-database deployment,
EnableExtendedProgressionTrackingcosts one pooled connection and one transaction per database per node every 5 seconds — ~37 connection acquisitions/sec/node — to keep 6-12 rows per database current, and it writes into the same rows the progress writer updates. It made a production web app unresponsive.Separately filed: a writer-accumulation bug that multiplies this without bound. This issue is the cost model that remains after that's fixed.
What the audit found
I traced every consumer of the persisted extended columns across JasperFx, Marten and CritterWatch. Summary:
Nothing in JasperFx reads any of it.
LastHeartbeat,AgentStatus,PauseReason,RunningOnNode— the only read ofLastHeartbeatanywhere isExtendedProgressionWriter.cs:81checking its own input before writing it. The CLI daemon status grid renders sequence and high-water only. No descriptors, no health checks, no compliance tests.CritterWatch — the sole intended consumer — never reads the columns either. Grep for
agent_status,running_on_node,pause_reason,warning_behind_threshold,critical_behind_thresholdacross its source returns zero hits, and it has no uses ofReadProjectionProgressAsync/ProjectionProgressRow. It gets all of those values by reflecting off in-memory objects:IProjectionDaemon.CurrentAgents()(EventProgressionPoller.cs:242-246) and theShardStatethe tracker publishes (CritterWatchShardObserver.cs:35-39). It does callAllProjectionProgress, and Marten does hydrate the extended fields onto those rows — and CritterWatch uses onlyShardName,Sequence,TenantId. The data crosses the wire and is dropped.The stated justification looks structurally unreachable.
IEventDatabase.cs:141justifies persistence as "a monitoring consumer polling the database (e.g. CritterWatch when the publishing node is down)." But the poller runs inside each monitored node, and since CritterWatch#806 it only polls databases whose agents that node owns. When a node dies nobody polls its databases until the agents redistribute (~5s, advisory locks) — at which point the new owner reads its own live agents in memory. I can't construct a path where the durable column is the source of truth. Happy to be corrected on this one.The consuming threshold is 60 seconds. CritterWatch's
AgentDownalert fires atheartbeatAge > 60s, on a 30s evaluation tick, fed by a 15s poll. A 5s write is a 12× oversample of something nothing downstream can observe below 15s resolution. The poller even synthesises the value when absent —LastHeartbeat: lastHeartbeat ?? now— treating "the agent object answered" as equivalent evidence. And the UI discounts a stale beat whenever derived health disagrees, with a comment saying a stale beat during a quiet stream "is NOT evidence the agent died."Two of the columns are permanently NULL in the field.
pause_reason(nothing in the stack stores a reason string) andrunning_on_node(only populated fromShardStateTracker.AssignedNodeNumber, which nothing in JasperFx ever assigns — Wolverine stamps it). CritterWatch overwritesRunningOnNodefrom the Wolverine agent roster anyway.The signal that would actually indicate a cycling daemon is excluded from persistence.
ExtendedProgressionWriter.cs:78dropsHighWaterMarkandAllProjectionsstates, pinned byskips_high_water_mark_and_all_projections_states. MeanwhileJasperFxAsyncDaemon.cs:1145-1149claims the per-cycle tenant high-water beat "surfaces the same beat on the live Tracker (and the ExtendedProgression columns)" — it does not. That comment should be corrected regardless of what happens to this issue.It causes a second scaling problem downstream. CritterWatch's
ShardProgressionHandler.cs:194-210records 1,007,378 updates and 439 autovacuums over 2.5 hours across ~9,881 rows on a 512-database console, "the leading suspect for the ingest stall in #919" — and notes the heartbeat is specifically the field that defeats its no-op write guard.For history: the cost was raised and waved through at design time (CritterWatch
.claude/docs/alerts.md:629lists "adds DB writes even when idle" with the alternative of polling in-memory agent status; the recorded answer is "go ahead and do the DB write"). The proposed cadence there was 10s; 5s shipped.Proposal
Split the periodic beat from the transitional status.
NodeHeartbeat(#837): per-node key, 30s cadence, SignalR push, dormant-node reaping at 5 minutes. Combined with the agent assignment grid (already read by the poller for database scoping) and the pre-existinglast_updatedcolumn, that reconstructs everything any consumer currently renders or alerts on. This takes the reporter from ~37 acquisitions/sec/node to roughly zero in steady state.agent_status,pause_reason, failure info. The writer already distinguishes these (isTransitiononStarted/Paused/Stopped). Those writes are rare and preserve the "durable across a crash" story for the data where it means something.HeartbeatWriteIntervalconfigurable as the compatibility hatch, not the fix. Today it's hardcoded 5s with no path from application code — the field is private on the daemon and noDaemonSettingsknob reaches it.The one thing genuinely lost is distinguishing "agent loop wedged, node healthy, no events to process" from "quiet stream" — and CritterWatch's UI has already decided not to act on that distinction.
If per-shard liveness is wanted despite the above, the cheaper shape is to fold it into the progress writer's existing transaction: the heartbeat rows are a subset of the rows that writer already updates, so a busy shard costs zero extra connections, and only idle shards need a periodic write. That also dissolves the lock contention marten#5167 measured, since telemetry and progress would stop being two transactions fighting over the same rows.