Wolverine 6.21.0 / JasperFx.Events 2.30.1 / Marten 9.16.1. Host: CritterWatch's MultiStoreHost sample — a Marten main store (UseWolverineManagedEventSubscriptionDistribution = true) plus two ancillary AddMartenStore<T> stores with plain IntegrateWithWolverine() (the post-#3400 blessed shape), DurabilityMode.Solo, three async shards total: Itinerary:All (main), Incident:All (ancillary snapshot), TelehealthComposite:All (ancillary composite).
Symptom
On every boot, one of the projection agents fails to start on the node's very first event-subscriptions assignment evaluation and then stays wedged in a permanent 30-second retry loop for the life of the process:
fail: Wolverine.Runtime.Agents.NodeAgentController[0]
Error trying to reevaluate agent assignments for 'event-subscriptions' agents
Wolverine.Runtime.Agents.AgentStartingException: Failed trying to start agent event-subscriptions://marten/iincidentsstore/localhost.postgres/incident/all on node …
---> System.Exception: Unable to start a subscription agent for Identity: Incident:All
at JasperFx.Events.Daemon.JasperFxAsyncDaemon`3.StartAgentAsync(ShardName name, CancellationToken token) …:line 614
at Wolverine.Runtime.Agents.EventSubscriptionAgent.StartAsync(…) EventSubscriptionAgent.cs:line 62
at Wolverine.Runtime.Agents.NodeAgentController.StartAgentAsync(…) NodeAgentController.cs:line 182
The victim is randomized per boot: one boot it was Incident:All (31+ consecutive failures, Itinerary/TelehealthComposite fine), the next boot it was Itinerary:All (31 failures, Incident:All zero — but Incident then just sat RegisteredIdle, never started, its progression frozen while its store's high-water climbed thousands of events). Two of three shards dead on that boot; only one shard ran.
No other error is logged — JasperFxAsyncDaemon.tryStartAgentAsync logs caught exceptions at Error level and none appears, so the failing path is one of the silent return false branches (see companion JasperFx issue about the masked failure reason).
Why it smells like a startup race
- It always begins on the first assignment evaluation, immediately after
Wolverine.Configuration.HandlerDiscovery finishes, while the daemons/high-water detection are still coming up.
- The failing shard differs run to run on identical code + data.
- The 30s reevaluation retries the same
StartAgentAsync forever and never succeeds — whatever state the first failure left behind is sticky (the public StartAgentAsync(ShardName) wrapper throws because _agents.TryFind misses right after the string-overload start "succeeded").
Repro
JasperFx/CritterWatch repo → cd src/BffHost && dotnet run (full fleet) → watch the MultiStoreHost resource log; or reduced: any host with a managed-distribution main Marten store + ≥2 ancillary AddMartenStore<T> stores with async shards, booted with a publisher already appending. CW-side tracking issue: JasperFx/CritterWatch#749 (filed alongside).
Possibly related: after RebuildProjection/rewind on any single-store host on this stack, the shard re-registers RegisteredIdle and the controller never restarts it either (no retry loop at all there) — tracked at JasperFx/CritterWatch#747; the common shape is "agent lands Idle and the controller can't/won't bring it (back) up."
Wolverine 6.21.0 / JasperFx.Events 2.30.1 / Marten 9.16.1. Host: CritterWatch's
MultiStoreHostsample — a Marten main store (UseWolverineManagedEventSubscriptionDistribution = true) plus two ancillaryAddMartenStore<T>stores with plainIntegrateWithWolverine()(the post-#3400 blessed shape),DurabilityMode.Solo, three async shards total:Itinerary:All(main),Incident:All(ancillary snapshot),TelehealthComposite:All(ancillary composite).Symptom
On every boot, one of the projection agents fails to start on the node's very first
event-subscriptionsassignment evaluation and then stays wedged in a permanent 30-second retry loop for the life of the process:The victim is randomized per boot: one boot it was
Incident:All(31+ consecutive failures,Itinerary/TelehealthCompositefine), the next boot it wasItinerary:All(31 failures,Incident:Allzero — butIncidentthen just satRegisteredIdle, never started, its progression frozen while its store's high-water climbed thousands of events). Two of three shards dead on that boot; only one shard ran.No other error is logged —
JasperFxAsyncDaemon.tryStartAgentAsynclogs caught exceptions at Error level and none appears, so the failing path is one of the silentreturn falsebranches (see companion JasperFx issue about the masked failure reason).Why it smells like a startup race
Wolverine.Configuration.HandlerDiscoveryfinishes, while the daemons/high-water detection are still coming up.StartAgentAsyncforever and never succeeds — whatever state the first failure left behind is sticky (the publicStartAgentAsync(ShardName)wrapper throws because_agents.TryFindmisses right after the string-overload start "succeeded").Repro
JasperFx/CritterWatchrepo →cd src/BffHost && dotnet run(full fleet) → watch the MultiStoreHost resource log; or reduced: any host with a managed-distribution main Marten store + ≥2 ancillaryAddMartenStore<T>stores with async shards, booted with a publisher already appending. CW-side tracking issue: JasperFx/CritterWatch#749 (filed alongside).Possibly related: after
RebuildProjection/rewind on any single-store host on this stack, the shard re-registersRegisteredIdleand the controller never restarts it either (no retry loop at all there) — tracked at JasperFx/CritterWatch#747; the common shape is "agent lands Idle and the controller can't/won't bring it (back) up."