GH-3519: restart a projection agent whose shard wedged out from under it - #3550
Merged
Conversation
Follow-up to GH-3535 (start-failure isolation) and GH-3520 (rebuild/rewind resume). On multi-store, Wolverine-managed Marten hosts a projection shard can start successfully and then die underneath its EventSubscriptionAgent wrapper -- a lost first-assignment start race that wedged daemon-side, a daemon-side stop, or an execution-loop fault. Two gaps kept it dead for the process lifetime: 1. EventSubscriptionAgent.Status latched Running once started, so a shard that stopped underneath the wrapper kept reporting Running. NodeAgentController only restarts agents it can see are non-Running, so it saw nothing to fix. 2. NodeAgentController.StartAgentAsync short-circuited on a bare Agents.ContainsKey, treating any registered agent as healthy forever. The recurring Solo reevaluation therefore never resurrected a registered-but-dead agent -- it just sat in the observable 30s retry loop reporting a stale Running while its high-water climbed. Fix both: Status now delegates to the live inner daemon agent once started (and still reads Stopped before start / after an explicit stop), and StartAgentAsync re-drives a registered agent whose shard has Stopped -- stopping it first to release any lingering daemon-side shard state -- while leaving genuinely Running and deliberately Paused (error backoff / blue-green gate) agents untouched. Red-green regression test agent_restart_when_wedged in CoreTests. Also bumps to 6.22.0-alpha.3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the residual of #3519 (following #3535 start-failure isolation and #3520 rebuild/rewind resume).
The wedge
On multi-store, Wolverine-managed Marten hosts (the CritterWatch
MultiStoreHostrepro), a projection shard can start successfully and then die underneath itsEventSubscriptionAgentwrapper — a lost first-assignment start race that wedged daemon-side, a daemon-side stop, or an execution-loop fault. The victim is randomized across boots and stays dead for the process lifetime, sitting in the observable 30s retry loop.Two gaps kept it stuck:
EventSubscriptionAgent.StatuslatchedRunning. Once started it never flipped back, so a shard that stopped underneath the wrapper kept reportingRunning.NodeAgentControlleronly restarts agents it can see are non-Running, so it saw nothing to fix (this is the "agents landing inRegisteredIdlearen't restarted by the controller" symptom from the issue).NodeAgentController.StartAgentAsyncshort-circuited on a bareAgents.ContainsKey. Any registered agent was treated as healthy forever, so the recurring Solo reevaluation never resurrected a registered-but-dead one.The fix
EventSubscriptionAgent.Statusnow delegates to the live inner daemon agent once started (the daemon keeps that status current), and still readsStoppedbefore start / after an explicitStopAsync.NodeAgentController.StartAgentAsyncre-drives a registered agent whose shard hasStopped— stopping it first to release any lingering daemon-side shard state — while leaving genuinelyRunningand deliberatelyPaused(error backoff / blue-green gate) agents untouched (idempotent).Tests
agent_restart_when_wedgedinCoreTests— red-verified without the controller change:Pausedagent aloneFull
wolverine.slnxbuilds clean (0 warnings). Also bumps to6.22.0-alpha.3.🤖 Generated with Claude Code