WO-8: surface a paused shard's ShardFailure to the assignment plane and health checks

[jeremydmiller]

Summary

WO-8, the Wolverine half of JasperFx/jasperfx#565 (implemented in JasperFx/jasperfx#567). Surface an ApplyEventException-stopped shard to the assignment/observer plane with a reason, so CritterWatch can alert on it and progress doesn't silently flatline. This was deliberately deferred to ship as a coordinated pair with the JasperFx.Events accessor, which has now landed.

What JasperFx.Events gives us

• ISubscriptionAgent.Failure → ShardFailure? — a default interface member (non-breaking), set alongside Status when a shard pauses or stops and cleared on start/replay.
• ShardFailure carries Category (ApplyEvent / EventSerialization / UnknownEventType / ProgressionOutOfOrder / Other), ExceptionType, RootExceptionType, Message, Detail (the full text ShardState.PauseReason has always carried), Event (sequence, event type, stream, tenant, version when known) and OccurredAt. It is a plain serializable record, not an Exception, so it survives the hop to the assignment plane and into a UI.
• ShardStateTracker.CurrentState(...) / TryGetCurrentState(...) / CurrentStates() — a synchronous snapshot for pollers that aren't subscribed.
• ShardState.Failure rides along on the published paused/stopped state, and Marten/Polecat persist the classified fields onto the extended progression row (Implement IEventFailureContext + extended progression failure columns (jasperfx#565) marten#5048, Implement IEventFailureContext + extended progression failure columns (jasperfx#565) polecat#368) so it is readable even when the publishing node is down.

Proposed work in Wolverine

1. Delegate the reason on the wrapper. EventSubscriptionAgent.Status already delegates to the live inner agent (GH-3519); do the same for the reason:

public ShardFailure? Failure => _innerAgent?.Failure;

Expose it on IEventSubscriptionAgent so CritterWatch can read it through the agent surface it already resolves via IEventSubscriptionAgentFamily.FindAgentUriAsync.

2. Make the health check say something actionable. EventSubscriptionAgent.CheckHealthAsync currently returns a fixed string:

if (Status == AgentStatus.Paused)
    return HealthCheckResult.Unhealthy($"Projection {Uri} paused due to errors");

With the failure in hand that becomes the category, the failing event sequence and the root exception type — the difference between an alert an operator can act on and one they have to go dig for.

3. Stop auto-restarting failures that cannot self-heal. This is the part worth the most. The stall detector auto-restarts after MaxConsecutiveStallsBeforeRestart stalled health checks. A shard paused on ApplyEvent, EventSerialization or UnknownEventType will fail on the same event every time it restarts, so the restart loop is pure churn that also keeps resetting the operator's view. Gate the auto-restart on the category: retry Other (a transient database blip is exactly what auto-restart is for), and for the per-event categories surface the failure instead and leave it stopped. ProgressionOutOfOrder means two processes are on the same shard — restarting makes that worse, not better.

4. Carry it into the observer/assignment plane so CritterWatch renders the reason next to the paused shard rather than just the status.

Notes

• Requires the JasperFx.Events version carrying UseNewtonsoftJsonForSerialization causes: InvalidOperationException #567.
• No behavior changes for an agent whose inner agent is null or doesn't track failures — Failure is null and everything reads exactly as it does today.

[jeremydmiller]

Shipped in #3658 (20a5cbd06 on main), alongside the JasperFx.Events 2.34.0 -> 2.36.1 / Marten 9.18.0 -> 9.20.0 / Polecat 5.5.0 -> 5.7.0 bump that carries #565 and #567.

Against the four items proposed above:

1. Delegate the reason on the wrapper. IEventSubscriptionAgent.Failure is a default-null DIM, so existing implementations are unaffected, and EventSubscriptionAgent delegates it to the live inner agent exactly the way Status has since GH-3519 — the daemon sets Failure at the same moment it flips Status, and clears it on start/replay, so caching a copy on the wrapper would let it report a stale reason for a shard that has since recovered. CritterWatch reads it through the same agent surface it already resolves via IEventSubscriptionAgentFamily.FindAgentUriAsync.

2. Actionable health check. The fixed "paused due to errors" string is now the failure category, the failing event's sequence, event type and tenant, and the root exception type — not the ApplyEventException/ShardStopException wrapper around it. It also fires on Stopped, not just Paused, since a shard the daemon stopped outright is no less in need of an alert.

3. No auto-restart for failures that cannot self-heal. The stall detector's auto-restart is now gated on the category. ApplyEvent, EventSerialization and UnknownEventType are bound to a specific event and will die on that identical event on every restart; ProgressionOutOfOrder means two processes are on the same shard, so restarting adds a third contender. Those surface the failure and stay put, with the health check saying explicitly that auto-restart was suppressed and why. Only Other — a database outage, a timeout, the case auto-restart actually exists for — is still retried.

4. Into the observer/assignment plane. IWolverineObserver.AgentPaused(uri, failure) (default no-op) plus a NodeRecordType.AgentPaused node record, fired once per transition into failure by a per-node sweep in the health-check loop.

One deliberate narrowing on item 4, per the analysis in #3637 §4: this landed observe-only. The failure is published and recorded, but a paused shard is not detached or re-assigned on the strength of it. The anti-thrash guards from WO-1..7 (#3619–#3624, #3636) already hold the shard steady, and reacting to the failure by moving the agent would reintroduce exactly the re-assignment churn that wave removed — a shard that fails on the same poison event everywhere gains nothing from being moved, and loses the operator's stable view of where it stopped. If a detach policy is wanted later it should be its own issue with its own thrash analysis.

19 new tests in src/Testing/CoreTests/Agents/paused_shard_failure_surfacing.cs; full CoreTests (2088) and MartenTests Distribution (57) green, wolverine.slnx clean in Release.

Rendering this next to the paused shard in the UI is tracked on the CritterWatch side.