Regression in 6.23.0: a paused per-database projection agent never resumes after RestartAgent (managed event-subscription distribution)

[jeremydmiller]

Summary

In Wolverine 6.23.0, a projection/subscription agent that has been paused and then restarted never comes back. The agent stops correctly on pause, but the restart is accepted and the agent is never re-assigned to any node.

This works in 6.22.0-alpha.3 and fails in 6.23.0 with every other dependency held constant.

Isolation (single variable)

Same test, same clean database, Marten 9.20.0 / JasperFx 2.36.1 / Weasel 9.19.0 / Polecat 5.7.0 held fixed in all runs:

WolverineFx	Result
6.22.0-alpha.3	✅ 3/3 pass (2 runs)
6.23.0	❌ 3/3 fail (2 runs)

So the regression is in Wolverine itself, not the Marten/JasperFx side of the bump.

Setup

• Marten, database-per-tenant (sharded tenancy, 2 shard databases).
• UseWolverineManagedEventSubscriptionDistribution, DurabilityMode.Balanced.
• Each tenant database gets its own projection agent: event-subscriptions://marten/main/localhost.cw_shardN/trip/all.
• Reproduces at nodeCount: 1 and 3 — it is not multi-node-specific.

Sequence

1. Add tenant to shard; wait for that database's Trip agent to be running. ✅
2. Pause the agent (a tenant-scoped pause resolving to that database's agent). ✅ agent stops as expected.
3. Restart the agent. ❌ the agent never reappears in the running-agent set — 30 s timeout.

At the point of failure, agents for other tenants/shards are running normally, e.g.:

tenant-a Trip agent on cw_shard1 should resume after restart (timed out after 30s; expected present=True).
Cluster: node 1: [event-subscriptions://marten/main/localhost.cw_shard2/trip/all]

and on 3 nodes:

tenant-a Trip agent on cw_shard2 should resume after restart (timed out after 30s; expected present=True).
Cluster: node 1: [] | node 2: [event-subscriptions://marten/main/localhost.cw_shard1/trip/all] | node 3: []

So the grid is alive and distributing other agents — only the previously-paused one is never re-assigned.

Suspected area

The agent changes in 6.23.0, most plausibly one of:

• d554b7230 — pending-assignment ledger suppresses duplicate AssignAgent floods (D3/D6)
• a9d4a4de1 — scale-safe chunked, bounded-parallel agent batch starts (D3)
• f703e1794 — ejection hysteresis + leader protection
• 38fe0d317 — re-upsert assignment row for already-running agents (D4)

A plausible mechanism, given the symptom: the pause leaves a record that makes the restart's AssignAgent look like a duplicate/in-flight assignment, so the ledger suppresses it and the agent is never actually started. That would explain why the restart is accepted without error yet nothing starts, and why unrelated agents are unaffected.

Impact

Pause/restart of a projection agent is an operator action surfaced in CritterWatch, so on 6.23.0 an operator can pause a tenant's projection and be unable to resume it without a process restart.

Repro

EventStoreCoordinationTests.TenantPartitionedCoordinationTests in JasperFx/CritterWatch:

• single_node_per_tenant_pause_then_restart
• cluster_per_tenant_pause_then_restart
• per_tenant_pause_is_isolated_from_other_tenants

Happy to help narrow it further or test a candidate fix.

[jeremydmiller]

Root cause confirmed and fix up in #3665.

The suspected mechanism was right — it's the pending-assignment ledger from d554b72 (D3/D6). The ledger only confirmed an entry when a later evaluation saw the agent running and still assigned to the same node. An operator pause breaks that systematically: the first evaluation that could observe the delivered assignment is the same evaluation that detaches the agent for the pause, so the entry was never confirmed, nothing on the stop path clears it, and the post-restart AssignAgent — same agent, same node — was suppressed as still-in-flight.

Two clarifications from the investigation:

• Strictly it isn't "never comes back": the agent resumes when the ledger entry's TTL expires, at 2 × CheckAssignmentPeriod (60s by default). Extending the repro's final wait from 30s to 120s makes it pass at ~62s — which is what pinned the mechanism. So on 6.23.0 the operator impact is a 30–60s dead window per pause/restart cycle rather than a required process restart.
• The pause's own kickstart evaluation runs before the restriction is merged and can briefly start the agent being paused (in this repro it also migrated the shard database's agent universe from store-global to per-tenant, arming the ledger entry in the first place). That ordering churn is now split off as ApplyRestrictionsAsync kickstarts a health check before merging the new restriction, briefly starting the very agent being paused #3666.

#3665 makes delivery alone confirm the entry — OriginalNode matching the pended destination clears it regardless of what the current evaluation decides to do with the agent. Verified against this issue's repro suite (TenantPartitionedCoordinationTests) via a local package overlay: all three tests go from 3/3 failing at the 30s timeout to passing in 8s, with the full 42-test EventStoreCoordinationTests suite green.

🤖 Generated with Claude Code