GH-3520: EventSubscriptionAgent restores continuous execution after Rebuild/Rewind - #3523
Merged
Conversation
…ebuild/Rewind Under Wolverine-managed event-subscription distribution there is no store coordinator to resurrect a shard the daemon stopped. RebuildProjectionAsync stops the continuous agent and never restarts it; RewindSubscriptionAsync restarts it daemon-side but this wrapper's _innerAgent still pointed at the stopped pre-rewind agent while Status still read Running, so NodeAgentController saw nothing to fix. Either way the shard froze at RegisteredIdle while its high-water climbed. RebuildAsync/RewindAsync now resume continuous execution themselves through the registered daemon start path (resumeContinuousAsync) and refresh _innerAgent/Status. resumeContinuousAsync deliberately does not re-invoke OnStarted: a rebuild/rewind is transparent to EventStoreAgents' per-node running-agent count (the wrapper never observed a matching stop), so re-counting would leak the database's tracker subscriptions. The rewind half rides on #536 (rewind registers its restarted agent) so the registered start resolves to that same running agent idempotently instead of spinning up a duplicate on the same progression row - ships in lockstep with that JasperFx bump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
JasperFx.Events 2.30.2 includes #536 (rewind now registers its restarted agent in the daemon's running set). That makes EventSubscriptionAgent.RewindAsync's resumeContinuousAsync resolve to the same running agent idempotently instead of spinning up a duplicate on the same progression row, so the rewind half of GH-3520 is now safe. Bumps JasperFx, JasperFx.Events, JasperFx.Events.SourceGenerator, and JasperFx.SourceGenerator in lockstep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3520. Wolverine-side half of a coordinated fix with JasperFx/jasperfx#536.
The bug
Under Wolverine-managed event-subscription distribution there is no store coordinator (
AddAsyncDaemon) to resurrect a shard the daemon stopped:EventSubscriptionAgent.RebuildAsync→_daemon.RebuildProjectionAsync(...)stops the continuous agent and never restarts it. The wrapper still reportsRunningagainst the now-stopped agent, soNodeAgentControllersees nothing to fix.EventSubscriptionAgent.RewindAsync→_daemon.RewindSubscriptionAsync(...)restarts the agent daemon-side, but the wrapper's_innerAgentstill points at the stopped pre-rewind agent andStatusstill readsRunning.Either way the shard re-registers
RegisteredIdledaemon-side and freezes forever while the high-water mark climbs (JasperFx/CritterWatch#747). An operator "Restart" routed to an agent Wolverine believes is already running no-ops.The fix
RebuildAsync/RewindAsyncnow restore continuous execution themselves through the registered daemon start path (resumeContinuousAsync) and refresh_innerAgent/Status:resumeContinuousAsyncdeliberately does not re-invokeOnStarted: a rebuild/rewind is transparent toEventStoreAgents' per-node running-agent count (the wrapper never observed a matching stop), so re-counting would leak the database's tracker observer subscriptions at the 1→0 transition.Coordination / merge order
RebuildProjectionAsyncleaves the shard stopped-but-registered, so_daemon.StartAgentAsync(_shardName)cleanly starts and re-registers a fresh continuous agent.RewindSubscriptionAsyncleaves an unregistered running agent, so re-callingStartAgentAsyncwould start a second agent on the same progression row. In Memory Queues don't seem to work when IDocumentSession is injected into EndPoint's method #536 registers the rewind's restarted agent, making the wrapper's start resolve to that same running agent idempotently. Merge in lockstep with the JasperFx.Events bump that includes In Memory Queues don't seem to work when IDocumentSession is injected into EndPoint's method #536.Tests
event_subscription_agent_rebuild_rewind_resume(CoreTests) drivesRebuildAsync/RewindAsyncover a substitutedIProjectionDaemonand asserts each resumes through the registered start path (StartAgentAsyncreceived once) and reportsRunning. Both verified red before the fix, green after.🤖 Generated with Claude Code
https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
Related issues