Skip to content

GH-3520: EventSubscriptionAgent restores continuous execution after Rebuild/Rewind - #3523

Merged
jeremydmiller merged 2 commits into
mainfrom
gh-3520-rebuild-rewind-resume
Jul 20, 2026
Merged

GH-3520: EventSubscriptionAgent restores continuous execution after Rebuild/Rewind#3523
jeremydmiller merged 2 commits into
mainfrom
gh-3520-rebuild-rewind-resume

Conversation

@jeremydmiller

@jeremydmiller jeremydmiller commented Jul 20, 2026

Copy link
Copy Markdown
Member

Fixes #3520. Wolverine-side half of a coordinated fix with JasperFx/jasperfx#536.

The bug

Under Wolverine-managed event-subscription distribution there is no store coordinator (AddAsyncDaemon) to resurrect a shard the daemon stopped:

  • EventSubscriptionAgent.RebuildAsync_daemon.RebuildProjectionAsync(...) stops the continuous agent and never restarts it. The wrapper still reports Running against the now-stopped agent, so NodeAgentController sees nothing to fix.
  • EventSubscriptionAgent.RewindAsync_daemon.RewindSubscriptionAsync(...) restarts the agent daemon-side, but the wrapper's _innerAgent still points at the stopped pre-rewind agent and Status still reads Running.

Either way the shard re-registers RegisteredIdle daemon-side and freezes forever while the high-water mark climbs (JasperFx/CritterWatch#747). An operator "Restart" routed to an agent Wolverine believes is already running no-ops.

The fix

RebuildAsync/RewindAsync now restore continuous execution themselves through the registered daemon start path (resumeContinuousAsync) and refresh _innerAgent/Status:

private async Task resumeContinuousAsync(CancellationToken cancellationToken)
{
    _innerAgent = await _daemon.StartAgentAsync(_shardName, cancellationToken);
    Status = AgentStatus.Running;
}

resumeContinuousAsync deliberately does not re-invoke OnStarted: a rebuild/rewind is transparent to EventStoreAgents' per-node running-agent count (the wrapper never observed a matching stop), so re-counting would leak the database's tracker observer subscriptions at the 1→0 transition.

Coordination / merge order

Tests

event_subscription_agent_rebuild_rewind_resume (CoreTests) drives RebuildAsync/RewindAsync over a substituted IProjectionDaemon and asserts each resumes through the registered start path (StartAgentAsync received once) and reports Running. Both verified red before the fix, green after.

🤖 Generated with Claude Code

https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S


Related issues

…ebuild/Rewind

Under Wolverine-managed event-subscription distribution there is no store
coordinator to resurrect a shard the daemon stopped. RebuildProjectionAsync stops
the continuous agent and never restarts it; RewindSubscriptionAsync restarts it
daemon-side but this wrapper's _innerAgent still pointed at the stopped pre-rewind
agent while Status still read Running, so NodeAgentController saw nothing to fix.
Either way the shard froze at RegisteredIdle while its high-water climbed.

RebuildAsync/RewindAsync now resume continuous execution themselves through the
registered daemon start path (resumeContinuousAsync) and refresh _innerAgent/Status.
resumeContinuousAsync deliberately does not re-invoke OnStarted: a rebuild/rewind
is transparent to EventStoreAgents' per-node running-agent count (the wrapper never
observed a matching stop), so re-counting would leak the database's tracker
subscriptions.

The rewind half rides on #536 (rewind registers its restarted agent) so
the registered start resolves to that same running agent idempotently instead of
spinning up a duplicate on the same progression row - ships in lockstep with that
JasperFx bump.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
JasperFx.Events 2.30.2 includes #536 (rewind now registers its restarted
agent in the daemon's running set). That makes EventSubscriptionAgent.RewindAsync's
resumeContinuousAsync resolve to the same running agent idempotently instead of
spinning up a duplicate on the same progression row, so the rewind half of GH-3520
is now safe. Bumps JasperFx, JasperFx.Events, JasperFx.Events.SourceGenerator, and
JasperFx.SourceGenerator in lockstep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018LDiv9GqbQkkAuU1nf4H6S
@jeremydmiller
jeremydmiller merged commit 5b5aefe into main Jul 20, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment