Fix event-subscription agents left unassigned after node scale-down (#3341)#3346
Merged
Conversation
…3341) Under UseWolverineManagedEventSubscriptionDistribution with a sharded Marten store (MultiTenantedWithShardedDatabases, database-affine assignment), scaling the node count down could leave a subset of shard databases' agents unassigned on the surviving nodes. Those shards' async projections silently stopped advancing — no error logged, no self-heal until a rolling restart. Root cause: a node captures its event-subscription capabilities ONCE at startup (StartLocalAgentProcessingAsync) and persists them. In a sharded store whose databases/tenants are provisioned over time, nodes started at different times hold divergent capability snapshots, so AllNodesHaveSameCapabilities returns false and DistributeByGroupAffinity takes the capability-matching branch. A shard database whose agents ran on a now-departed node, and whose URIs are absent from every surviving node's stale snapshot, produced an empty candidate set with empty per-member CandidateNodes; the per-member fallback then left each member parked (candidate == null) — silently, with no command emitted. Fix: in the candidates.Count == 0 branch, a group whose members are all unassigned and declared by no node is a stale-snapshot artifact (every node can run every shard; the agents are still enumerated by AllKnownAgentsAsync), not a genuine blue/green gap. Assign the whole group to a node so the shard always has a home, kept together to preserve connection-pool affinity. In the mixed- capability case, an unassigned member with no capable node now falls back to the least-loaded node overall instead of being silently stranded. Adds a reproducing unit test (orphaned shard from a departed node with divergent survivor snapshots) and updates the sharded no-capable-node test to assert the member is reassigned rather than parked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 9, 2026
Merged
This was referenced Jul 14, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3341.
Symptom
Under
UseWolverineManagedEventSubscriptionDistributionwith a sharded Marten store(
MultiTenantedWithShardedDatabases, ~512 shard databases, database-affine assignment),scaling the pod count down left a subset of shard databases' event-subscription agents
unassigned on the surviving nodes. Those shards' async projections silently stopped
advancing — no error logged, no self-heal — until a rolling restart redistributed them.
Root cause
A node captures its event-subscription capabilities once at startup
(
StartLocalAgentProcessingAsync→SupportedAgentsAsync()) and persists them. In asharded store whose databases/tenants are provisioned over time, nodes started at
different times hold divergent capability snapshots, so
AllNodesHaveSameCapabilitiesreturns
falseandDistributeByGroupAffinitytakes its capability-matching branch.A shard database whose agents were running on a now-departed node — re-enumerated into
the grid unassigned via
AllKnownAgentsAsync,OriginalNode == null— and whose URIs areabsent from every surviving node's stale snapshot yields:
candidatesempty (no survivor declares the capability; no member has a survivingOriginalNode), andCandidateNodesempty,so the per-member fallback hit
candidate == nulland left every member parked — noAssignAgentcommand emitted, hence the silence.The homogeneous path never strands (candidates = all nodes), which is why this only
appeared on clusters whose nodes' snapshots had diverged (aggravated by rapid successive
scaling).
Fix
In the
candidates.Count == 0branch ofDistributeByGroupAffinity:artifact — every node can run every shard, and the agents are still enumerated as
supported by
AllKnownAgentsAsync. Assign the whole group to one node (kept togetherto preserve the connection-pool affinity this method exists for) instead of parking it.
the least-loaded node overall rather than being silently stranded.
Genuine blue/green placement is unchanged: a real new-version agent is listed by the green
nodes' fresh snapshots, so its candidate set is non-empty and it still lands only on a
capable node.
Tests
a_shard_orphaned_by_a_departed_node_is_reassigned_not_stranded— divergent survivor snapshots + a shard orphaned from a departed node; fails on
main(agent
AssignedNodenull), passes with the fix, and asserts the shard stays co-located.when_no_node_can_host_the_whole_group_members_fall_back_individuallyto assertthe previously-parked member is reassigned rather than silently stranded.
CoreTestsRuntime.Agentssuite green (121 tests).🤖 Generated with Claude Code