Expose the classified pause/stop reason of a shard (#565)

[jeremydmiller]

Closes #565. Needs to ship as a coordinated set with the Marten and Polecat halves (issues linked below) and Wolverine WO-8.

Problem

When a projection's error policy does not skip, an ApplyEventException routes to SubscriptionAgent.ReportCriticalFailureAsync, which hard-stops execution and sets Status = AgentStatus.Paused. An external supervisor could see that and nothing more:

• ISubscriptionAgent exposed only Status — no reason.
• ShardStateTracker kept its current-state map private; the only public surface was Subscribe (you had to be listening before the transition) or a blocking wait. No synchronous "what is shard X doing right now" snapshot existed.

So Wolverine's EventSubscriptionAgent and CritterWatch behind it could see progress flatline but not say what to do about it — and the operator action is completely different per cause: a poison event needs a code fix or a skip, a corrupt body needs a serializer/data fix, an unknown event type is usually a deployment gap, and an out-of-order progression means two processes are racing the same shard.

What's here

Everything hangs off SubscriptionAgent.ReportCriticalFailureAsync(Exception) — the single funnel every failure path in the daemon already reached (loadNextAsync, both batch executions, the replay executor).

ShardFailureCategory — ApplyEvent, EventSerialization, UnknownEventType, ProgressionOutOfOrder, Other.

IEventFailureContext — implemented by exceptions that can name the single event they failed on, and it declares its own category. ApplyEventException implements it here. The serialization and unknown-event-type exceptions live in the stores (that's where events are read), so Marten and Polecat implement it on theirs; the daemon never sniffs store type names. Only Sequence is guaranteed — a serialization failure is raised while reading a row, before there is an IEvent, which is why every other member is nullable.

EventFailureDetails + ShardFailure — plain serializable values: category, the caught exception type and the root one (the type an operator greps for, the same choice DeadLetterEvent.ExceptionType makes), message, the full detail text, the failing event, timestamp. Deliberately not an Exception: consumers ship this over a wire, persist it, and render it. ShardState.Exception still carries the live exception for in-process observers.

ShardFailure.For walks the whole exception graph, because these arrive wrapped in practice — a ShardStopException around an ApplyEventException, or an AggregateException of several apply failures (lowest failing sequence wins, since that's where the shard actually stopped).

Surface

• ISubscriptionAgent.Failure as a default interface member — non-breaking; a wrapper delegates it the way it already delegates Status. Set alongside Status, cleared on StartAsync/ReplayAsync so a supervisor never alerts on a failure the operator already fixed.
• ShardState.Failure, published on the paused/stopped state. PauseReason keeps carrying the same full exception text it always did (it is ShardFailure.Detail), so string-only consumers are untouched.
• ShardStateTracker.CurrentState(string) / CurrentState(ShardName) / TryGetCurrentState / CurrentStates() — the map was already maintained; it just wasn't readable.

Dead-letter traceability — DeadLetterEvent.Id is now assigned at construction (version 7, time-ordered; stores only generate an id when the value is empty, so nothing about persistence changes) so the id is known before the background retried write lands. DeadLetterEvent.DescribesSameFailureAs(shardName, failure) is the link between the two halves of a per-event failure. Nothing is written to the dead-letter table when a shard pauses: a paused event was not skipped, so a row there would inflate the counts stores report as their "projection is unhealthy" signal and would be rewritten on every restart attempt. If the same event is later skipped, its dead letter lines up on projection name, shard key, sequence and tenant.

Tests

EventTests/Daemon/ShardFailureTests.cs (20 tests): classification per category including the wrapped and aggregate shapes, stream-identity normalization (a string-keyed stream's Guid.Empty reports as null rather than rendering a meaningless Guid), Detail/PauseReason parity, the agent exposing and clearing the reason, the failure riding along on the published state, the tracker snapshot, and the dead-letter correlation (including "unknown tenant doesn't veto a match" and "same sequence in a different tenant is not a match").

The store-side contract is exercised through a FakeStoreEventFailure double that implements IEventFailureContext exactly the way Marten and Polecat will — it doubles as the spec for those PRs.

src/EventTests (621) and src/EventStoreTests (72) pass on net9.0; full solution builds.

Coordinated work

• Marten: Implement IEventFailureContext + extended progression failure columns (jasperfx#565) marten#5048 — implement IEventFailureContext on the read-side exceptions + extended progression columns.
• Polecat: Implement IEventFailureContext + extended progression failure columns (jasperfx#565) polecat#368 — same.
• Wolverine WO-8 consumes ISubscriptionAgent.Failure from EventSubscriptionAgent.

Noticed while testing (not fixed here)

ShardStateTracker.WaitForShardState has a latent race: it checks the state map once and then waits for the next publication, so a state consumed between the check and the watcher subscribing is missed and the wait runs to its timeout. It bit a test in this PR (worked around by polling the new snapshot instead). Worth its own issue.

🤖 Generated with Claude Code