Skip to content

Daemon start-registration + failure-reason fixes (closes #534, #535; wolverine#3519/#3520) - #536

Merged
jeremydmiller merged 2 commits into
mainfrom
gh-wolverine-3519-3520-daemon-start-registration
Jul 20, 2026
Merged

Daemon start-registration + failure-reason fixes (closes #534, #535; wolverine#3519/#3520)#536
jeremydmiller merged 2 commits into
mainfrom
gh-wolverine-3519-3520-daemon-start-registration

Conversation

@jeremydmiller

@jeremydmiller jeremydmiller commented Jul 20, 2026

Copy link
Copy Markdown
Member

Closes #534 and #535. Companion JasperFx.Events fixes for JasperFx/wolverine#3519 and JasperFx/wolverine#3520 — the daemon-side halves those Wolverine issues reference. The Wolverine-side change for #3520 rides on top of this (JasperFx/wolverine#3523).

#534StartAgentAsync(ShardName) masked the failure reason

tryStartAgentAsync swallowed a faulted start into a bool, so StartAgentAsync(ShardName) could only throw a causeless Exception("Unable to start a subscription agent for …"), filling a retrying caller's log with identical stackless stacks.

  • Stash the caught start exception per shard identity (_lastStartFailures) and attach it as the inner exception when the wrapper throws; clear it on a later successful start.
  • When no exception was captured (the start returned false from a non-throwing path), throw a ShardStartException whose message describes the daemon state that explains the miss — a registered-but-paused agent (with its status), high-water detection not yet running, an unknown shard, or a concurrent-start race.
  • Log the two previously-silent "already running" idempotent return false paths at Debug so the "why did my start return false" trail is unbroken (the side-effect gate and catch paths already logged).
  • Fix the ShardStartException message that said "stop" on a start failure.

#535 — rebuild/rewind lifecycle gaps on coordinator-less hosts

Part 2 (rewind bypasses the registry) — fixed. RewindSubscriptionAsync stopped the running continuous agents, rewound progress, then rebuilt and StartAsynced fresh continuous agents that were never registered in _agents. CurrentAgents() and StartAgentAsync(ShardName) (both read _agents) kept seeing only the pre-rewind agent that stopRunningAgents() had just HardStopped — so the shard was orphaned under Wolverine-managed distribution, and a later restart spun up a duplicate agent on the same progression row. Now each restarted agent is registered under the registry lock (registerStartedAgentAsync), making a subsequent StartAgentAsync idempotent.

Part 1 (rebuild never restarts) — documented as contract. rebuildProjection stops the continuous agents and intentionally does not restart them: a store coordinator resurrects them, and on coordinator-less hosts restoring continuous execution is the driving caller's responsibility (Wolverine's EventSubscriptionAgent.RebuildAsync, JasperFx/wolverine#3523). Restarting here unconditionally would double-start against a store coordinator, so this is a doc-note rather than a behavior change.

Tests

PerTenantStartAgentAsyncTests.rewind_leaves_the_restarted_agent_registered_and_running — starts a store-global agent, rewinds, and asserts exactly one agent is registered afterward, running, and a fresh instance (not the HardStopped pre-rewind one). Verified red before the registration fix, green after. All 240 EventTests.Daemon tests pass on net9.0 and net10.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S


Related issues

wolverine#3520: RewindSubscriptionAsync stopped the running continuous agents,
rewound progress, then rebuilt and StartAsync-ed fresh continuous agents that
were never registered in the daemon's running set (_agents). CurrentAgents() and
StartAgentAsync(ShardName) — which both read _agents — kept seeing only the
pre-rewind agent that stopRunningAgents() had just HardStopped. Under a
store-owned coordinator this was masked; under Wolverine-managed distribution
(no coordinator) it left the shard orphaned, and a later restart through the
registered path spun up a DUPLICATE agent on the same progression row. Register
the restarted agents under the registry lock so the running set reflects reality
and a subsequent restart is idempotent.

wolverine#3519: StartAgentAsync(ShardName) threw a bare, context-free Exception
when the string-overload start returned without throwing yet nothing registered
under the identity — leaving callers (Wolverine's EventSubscriptionAgent) wedged
in a permanent retry loop with no way to see why. Throw a ShardStartException
that surfaces the actual daemon state (a registered-but-paused agent and its
status, high-water detection not yet running, an unknown shard, or a start race
between concurrent agent starts). Also fixes the ShardStartException start/stop
message typo.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
…uild no-restart

#534: tryStartAgentAsync swallowed a faulted start into a bool, so
StartAgentAsync(ShardName) could only throw a causeless "Unable to start". Stash
the caught exception per shard identity (_lastStartFailures) and attach it as the
inner exception when the wrapper throws; clear it on a later successful start. Also
log the two previously-silent "already running" idempotent return-false paths at
Debug so the "why did my start return false" trail is unbroken (the side-effect
gate and catch paths already logged).

#535 (part 1): document that rebuildProjection stops the continuous agents
and intentionally does not restart them — a store coordinator resurrects them, and
on coordinator-less hosts (Wolverine-managed distribution) restoring continuous
execution is the driving caller's responsibility. Restarting here unconditionally
would double-start against a store coordinator.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
@jeremydmiller jeremydmiller changed the title Daemon start-registration fixes for wolverine#3519 + wolverine#3520 Daemon start-registration + failure-reason fixes (closes #534, #535; wolverine#3519/#3520) Jul 20, 2026
@jeremydmiller
jeremydmiller merged commit f0a4ca2 into main Jul 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment