GH-5003 Resume continuous execution after a projection rebuild, even a failed one

[jeremydmiller]

Closes #5003.

A projection rebuild hard-stops the projection's continuous agents and tears down its state before it replays. When the rebuild threw, nothing restarted the shard. On the registered-projection path (GH-3163, used by CritterWatch when no live agent hosts the shard), nothing restarted it even when the rebuild succeeded; it was left for the GH-4193 sweep to notice. The sweep only reports a stopped shard whose pre-rebuild agent carried an Other failure, so that shard stayed stopped until the process restarted.

This builds on the JasperFx 2.87.0 rebuild guard (JasperFx/jasperfx#1009, merged in #1010), which main already pins.

Changes

• EventSubscriptionAgent.RebuildAsync (the live-agent path) resumes continuous execution when the rebuild throws, then rethrows the rebuild's exception. A cancelled rebuild is not resumed. A resume that fails is logged and never hides why the rebuild failed. The progress is already reset by then, so the resumed agent effectively rebuilds from the start in continuous mode.
• EventStoreAgents.TryRebuildRegisteredProjectionAsync (the registered-projection path) now resumes this node's agents for the rebuilt projection, whether the rebuild succeeded or threw. EventStoreAgents remembers the wrappers it builds. It resumes only those on the same daemon, for the same projection and tenant, that weren't stopped on purpose. A wrapper stopped by an operator pause or a reassignment reads Stopped and is skipped.
• IEventSubscriptionAgent.IsRebuilding (default false, so existing implementations are unaffected) reads IProjectionDaemon.IsRebuilding. NodeAgentController's sweep skips a shard while its projection is rebuilding. During the replay the shard reads Stopped with its old failure, which looked exactly like a wedged shard. The A locally-stopped event-subscription agent that keeps its assignment is never recovered: the health sweep defers to StartAgentAsync, which the leader never calls for it #4193 restart could then start a continuous agent on the same progression row as the rebuild (the race described in the Restart a shard the daemon paused on a transient failure #4986 review). The daemon now refuses that start anyway, but this avoids the stop/evict churn that came before the refusal.

Tests

• failed_rebuild_resumes_continuous_execution, live-agent path:
  • a rebuild that throws still resumes and still throws
  • a failed resume doesn't hide the rebuild failure
  • a cancelled rebuild isn't resumed
  • IsRebuilding comes from the daemon
• failed_rebuild_resumes_continuous_execution, registered-projection path:
  • only this node's agents for the rebuilt projection are resumed
  • they're resumed when the rebuild throws too
  • an agent stopped on purpose isn't resumed
  • nothing starts for a projection with no agent on this node
• paused_shard_failure_surfacing:
  • the sweep leaves a rebuilding shard alone, with and without an old Other failure
  • the sweep picks the shard back up once the rebuild is over

With the three production changes disabled, 6 of these fail, which is the expected set.

Verification

• dotnet build wolverine.slnx -c Release -f net9.0: clean.
• CoreTests.Runtime.Agents (net9.0): 485 passed, 6 skipped, 0 failed.
• MartenTests rebuild distribution tests (inline_projection_rebuild, store_scoped_transient_rebuild_3618, event_subscription_agent*) against local Postgres: 11 of 11 passed.

🤖 Generated with Claude Code