Surface paused-shard failures to the assignment plane + critter stack bump (#3637, #3638, #3519)

[jeremydmiller]

Bumps the critter stack pins and lands the Wolverine half of the WO-8 pause-visibility work that was
deliberately deferred until the JasperFx.Events accessor shipped.

Dependency bump

Package	From	To
JasperFx / JasperFx.Events / *.SourceGenerator	2.34.0	2.36.1
Marten (+ AspNetCore, Newtonsoft)	9.18.0	9.20.0
Polecat	[5.5.0,6.0.0)	[5.7.0,6.0.0)

This is the release carrying JasperFx/jasperfx#565 (ShardFailure), #567 (ISubscriptionAgent.Failure),
#534 and #540 — all three issues below need it.

Closes #3637 / #3638 — surface a paused shard's failure, with a reason

When a projection/subscription shard hits an ApplyEventException under a non-skipping error policy the
daemon pauses it. The anti-thrash guards already kept it from being restart-looped, but nothing in the
assignment plane ever distinguished Paused from Running: progress silently flatlined and the only
trace was a log line on one node. ShardFailure now makes the reason portable, so this ships as the
coordinated pair the two issues describe.

• IEventSubscriptionAgent.Failure — a ShardFailure? default-null DIM;
  EventSubscriptionAgent.Failure delegates to the live inner agent, the same way Status has since
  Managed event-subscription distribution: randomized agent fails first start on multi-(ancillary-)store Marten hosts and wedges in a permanent 30s retry loop #3519. Read-through rather than cached, so a shard that recovers stops reporting a stale reason.
• Health checks say something actionable — the failure category, the sequence and type of the event
  it died on, and the root exception type, instead of the fixed "Projection {uri} paused due to errors". An inner agent that reports no failure reads exactly as it did before.
• Failures that can't self-heal are no longer restarted. A shard stopped on ApplyEvent,
  EventSerialization, UnknownEventType or ProgressionOutOfOrder will die on the identical event on
  every restart, so the Managed event-subscription distribution: randomized agent fails first start on multi-(ancillary-)store Marten hosts and wedges in a permanent 30s retry loop #3519 wedge recovery in NodeAgentController.StartAgentAsync now stands down
  for those and surfaces the reason instead. ProgressionOutOfOrder means two processes are on the same
  shard — restarting adds a third contender. Only Other (a database blip, a timeout — what
  restart-on-stall actually exists for) is still retried.
• Observer / assignment plane — new IWolverineObserver.AgentPaused(Uri, ShardFailure?) default-no-op
  hook plus a NodeRecordType.AgentPaused record carrying the classified reason, so the failure is
  readable from another process and CritterWatch can alert on it (Assert on Publish/SendAsync() if running in MediatorOnly mode #830). A sweep in the health-check loop
  reports locally-owned paused agents on every node, not just the leader — the daemon pauses a shard on
  whichever node owns it. Fires once per transition into failure, and re-arms if the shard recovers and
  later fails anew.

Per the §4 finding on #3637, this takes the observe-only option rather than detaching from
distribution: the anti-thrash guards already hold, and detaching would risk exactly the re-assignment
churn WO-1..7 removed.

Closes #3519 — the wedged first-assignment start

The reported shape: on a multi-store Marten host, one event-subscription agent — a different one on
every boot — failed its very first assignment start and stayed wedged in a permanent 30-second retry loop.
The root causes were daemon-side and land with this bump: the start failure now arrives as a
ShardStartException naming its cause instead of a causeless Exception (#534), and a start that
faults after spinning up the execution pipeline hard-stops the half-started shard rather than orphaning it
(#540), so the next attempt succeeds.

What was left on this side was when that next attempt happens. The node only retried on the next
assignment reevaluation, so the loser of a sub-second startup race sat idle for a full
CheckAssignmentPeriod (30s by default) while its high-water mark climbed. StartAgentAsync now retries a
failed start locally, bounded and configurable:

opts.Durability.AgentStartRetryAttempts = 2;                          // default; 0 restores the old single attempt
opts.Durability.AgentStartRetryDelay = TimeSpan.FromMilliseconds(250); // default, multiplied by attempt number

A failure that outlives the retries is thrown as before with the last cause intact, logged by the existing
per-agent isolation, and picked up on the next reevaluation.

Tests

• paused_shard_failure_surfacing — 15 tests across wrapper delegation, health-check reporting, restart
  suppression by category, and the local sweep (once-per-transition, re-arm after recovery, silence for
  healthy and for reason-less stops).
• agent_start_retry_on_startup_race — 4 tests: recovery from a first-attempt race, giving up after the
  configured attempts with the daemon's reason preserved, opt-out, and no extra attempt for a healthy
  agent.
• Full CoreTests suite green; full wolverine.slnx builds clean against the new pins.

Docs

New "When a Projection Fails" section on the Marten distribution page (categories, what each one means
for an operator, and the three surfaces the reason shows up on) plus "Agent Start Retries", with a
cross-reference from the Polecat page.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_014DkTEuNQQdPp63WLY86KzC