Daemon rebuild/rewind lifecycle gaps on coordinator-less hosts: rebuildProjection never restarts the continuous agents it stops; RewindSubscriptionAsync restarts agents that bypass the _agents registry

[jeremydmiller]

JasperFx.Events 2.30.1. Companion to JasperFx/wolverine#3520 (rebuild/rewind leave shards dead under Wolverine-managed distribution); filing the daemon-side halves here.

1. rebuildProjection stops the continuous agents and never restarts them

JasperFxAsyncDaemon.rebuildProjection:

• stopRunningAgents(subscriptionName) — stops the projection's running continuous agents;
• rebuilds via transient agents (buildAgentsForSubscription + rebuildAgent);
• StopAndDrainAsynces the transient agents;
• returns — the agents stopped in step 1 are never brought back.

On a host with the store's own coordinator loop this self-heals (the coordinator resurrects stopped shards), so Marten-daemon users rarely notice. On any host that drives the daemon externally (Wolverine-managed event-subscription distribution being the shipping example) the projection stays stopped forever after every rebuild — frozen progression while the high-water mark climbs. If the no-restart behavior is intentional (caller's job), it deserves a doc note + an explicit "and restart" overload; if not, rebuildProjection should restart what it stopped, through the registered path.

2. RewindSubscriptionAsync restarts agents that bypass the registry

The rewind path does try to resume: after RewindSubscriptionProgressAsync it builds agents via buildAgentsForSubscription(subscriptionName) and calls agent.StartAsync(new SubscriptionExecutionRequest(...)) directly — bypassing tryStartAgentAsync, so the started agents are never AddOrUpdated into _agents. Consequences: StartAgentAsync/TryFind lookups miss them, a second start builds duplicates, stop-by-identity can't reach them, and external wrappers (Wolverine's EventSubscriptionAgent._innerAgent) still reference the stopped instance. Observed live as a rewound subscription frozen while its store's HWM climbed 400+ events.

Repro

Rebuild an async projection while the publisher keeps appending on a coordinator-less host; the progression row freezes at the rebuild ceiling while the HWM climbs. CritterWatch pins it with projection_rebuild_under_load (JasperFx/CritterWatch@02c49251) — the stopped-publisher variant passes vacuously (caught-up and stalled are indistinguishable without event flow), which is worth keeping in mind for daemon tests generally.