Daemon start-registration + failure-reason fixes (closes #534, #535; wolverine#3519/#3520) - #536
Merged
jeremydmiller merged 2 commits intoJul 20, 2026
Conversation
wolverine#3520: RewindSubscriptionAsync stopped the running continuous agents, rewound progress, then rebuilt and StartAsync-ed fresh continuous agents that were never registered in the daemon's running set (_agents). CurrentAgents() and StartAgentAsync(ShardName) — which both read _agents — kept seeing only the pre-rewind agent that stopRunningAgents() had just HardStopped. Under a store-owned coordinator this was masked; under Wolverine-managed distribution (no coordinator) it left the shard orphaned, and a later restart through the registered path spun up a DUPLICATE agent on the same progression row. Register the restarted agents under the registry lock so the running set reflects reality and a subsequent restart is idempotent. wolverine#3519: StartAgentAsync(ShardName) threw a bare, context-free Exception when the string-overload start returned without throwing yet nothing registered under the identity — leaving callers (Wolverine's EventSubscriptionAgent) wedged in a permanent retry loop with no way to see why. Throw a ShardStartException that surfaces the actual daemon state (a registered-but-paused agent and its status, high-water detection not yet running, an unknown shard, or a start race between concurrent agent starts). Also fixes the ShardStartException start/stop message typo. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
…uild no-restart #534: tryStartAgentAsync swallowed a faulted start into a bool, so StartAgentAsync(ShardName) could only throw a causeless "Unable to start". Stash the caught exception per shard identity (_lastStartFailures) and attach it as the inner exception when the wrapper throws; clear it on a later successful start. Also log the two previously-silent "already running" idempotent return-false paths at Debug so the "why did my start return false" trail is unbroken (the side-effect gate and catch paths already logged). #535 (part 1): document that rebuildProjection stops the continuous agents and intentionally does not restart them — a store coordinator resurrects them, and on coordinator-less hosts (Wolverine-managed distribution) restoring continuous execution is the driving caller's responsibility. Restarting here unconditionally would double-start against a store coordinator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #534 and #535. Companion JasperFx.Events fixes for JasperFx/wolverine#3519 and JasperFx/wolverine#3520 — the daemon-side halves those Wolverine issues reference. The Wolverine-side change for #3520 rides on top of this (JasperFx/wolverine#3523).
#534 —
StartAgentAsync(ShardName)masked the failure reasontryStartAgentAsyncswallowed a faulted start into abool, soStartAgentAsync(ShardName)could only throw a causelessException("Unable to start a subscription agent for …"), filling a retrying caller's log with identical stackless stacks._lastStartFailures) and attach it as the inner exception when the wrapper throws; clear it on a later successful start.falsefrom a non-throwing path), throw aShardStartExceptionwhose message describes the daemon state that explains the miss — a registered-but-paused agent (with its status), high-water detection not yet running, an unknown shard, or a concurrent-start race.return falsepaths at Debug so the "why did my start return false" trail is unbroken (the side-effect gate and catch paths already logged).ShardStartExceptionmessage that said "stop" on a start failure.#535 — rebuild/rewind lifecycle gaps on coordinator-less hosts
Part 2 (rewind bypasses the registry) — fixed.
RewindSubscriptionAsyncstopped the running continuous agents, rewound progress, then rebuilt andStartAsynced fresh continuous agents that were never registered in_agents.CurrentAgents()andStartAgentAsync(ShardName)(both read_agents) kept seeing only the pre-rewind agent thatstopRunningAgents()had just HardStopped — so the shard was orphaned under Wolverine-managed distribution, and a later restart spun up a duplicate agent on the same progression row. Now each restarted agent is registered under the registry lock (registerStartedAgentAsync), making a subsequentStartAgentAsyncidempotent.Part 1 (rebuild never restarts) — documented as contract.
rebuildProjectionstops the continuous agents and intentionally does not restart them: a store coordinator resurrects them, and on coordinator-less hosts restoring continuous execution is the driving caller's responsibility (Wolverine'sEventSubscriptionAgent.RebuildAsync, JasperFx/wolverine#3523). Restarting here unconditionally would double-start against a store coordinator, so this is a doc-note rather than a behavior change.Tests
PerTenantStartAgentAsyncTests.rewind_leaves_the_restarted_agent_registered_and_running— starts a store-global agent, rewinds, and asserts exactly one agent is registered afterward, running, and a fresh instance (not the HardStopped pre-rewind one). Verified red before the registration fix, green after. All 240EventTests.Daemontests pass on net9.0 and net10.0.🤖 Generated with Claude Code
https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
Related issues