Skip to content

Fix event-subscription agents left unassigned after node scale-down (#3341)#3346

Merged
jeremydmiller merged 2 commits into
mainfrom
fix/3341-orphaned-shard-stranding
Jul 8, 2026
Merged

Fix event-subscription agents left unassigned after node scale-down (#3341)#3346
jeremydmiller merged 2 commits into
mainfrom
fix/3341-orphaned-shard-stranding

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #3341.

Symptom

Under UseWolverineManagedEventSubscriptionDistribution with a sharded Marten store
(MultiTenantedWithShardedDatabases, ~512 shard databases, database-affine assignment),
scaling the pod count down left a subset of shard databases' event-subscription agents
unassigned on the surviving nodes. Those shards' async projections silently stopped
advancing — no error logged, no self-heal — until a rolling restart redistributed them.

Root cause

A node captures its event-subscription capabilities once at startup
(StartLocalAgentProcessingAsyncSupportedAgentsAsync()) and persists them. In a
sharded store whose databases/tenants are provisioned over time, nodes started at
different times hold divergent capability snapshots, so AllNodesHaveSameCapabilities
returns false and DistributeByGroupAffinity takes its capability-matching branch.

A shard database whose agents were running on a now-departed node — re-enumerated into
the grid unassigned via AllKnownAgentsAsync, OriginalNode == null — and whose URIs are
absent from every surviving node's stale snapshot yields:

  • group candidates empty (no survivor declares the capability; no member has a surviving OriginalNode), and
  • each member's CandidateNodes empty,

so the per-member fallback hit candidate == null and left every member parked — no
AssignAgent command emitted, hence the silence.

The homogeneous path never strands (candidates = all nodes), which is why this only
appeared on clusters whose nodes' snapshots had diverged (aggravated by rapid successive
scaling).

Fix

In the candidates.Count == 0 branch of DistributeByGroupAffinity:

  • A group whose members are all unassigned and declared by no node is a stale-snapshot
    artifact — every node can run every shard, and the agents are still enumerated as
    supported by AllKnownAgentsAsync. Assign the whole group to one node (kept together
    to preserve the connection-pool affinity this method exists for) instead of parking it.
  • In the mixed-capability case, an unassigned member with no capable node now falls back to
    the least-loaded node overall rather than being silently stranded.

Genuine blue/green placement is unchanged: a real new-version agent is listed by the green
nodes' fresh snapshots, so its candidate set is non-empty and it still lands only on a
capable node.

Tests

  • New reproducing unit test a_shard_orphaned_by_a_departed_node_is_reassigned_not_stranded
    — divergent survivor snapshots + a shard orphaned from a departed node; fails on main
    (agent AssignedNode null), passes with the fix, and asserts the shard stays co-located.
  • Updated when_no_node_can_host_the_whole_group_members_fall_back_individually to assert
    the previously-parked member is reassigned rather than silently stranded.
  • Full CoreTests Runtime.Agents suite green (121 tests).

🤖 Generated with Claude Code

…3341)

Under UseWolverineManagedEventSubscriptionDistribution with a sharded Marten
store (MultiTenantedWithShardedDatabases, database-affine assignment), scaling
the node count down could leave a subset of shard databases' agents unassigned
on the surviving nodes. Those shards' async projections silently stopped
advancing — no error logged, no self-heal until a rolling restart.

Root cause: a node captures its event-subscription capabilities ONCE at startup
(StartLocalAgentProcessingAsync) and persists them. In a sharded store whose
databases/tenants are provisioned over time, nodes started at different times
hold divergent capability snapshots, so AllNodesHaveSameCapabilities returns
false and DistributeByGroupAffinity takes the capability-matching branch. A
shard database whose agents ran on a now-departed node, and whose URIs are
absent from every surviving node's stale snapshot, produced an empty candidate
set with empty per-member CandidateNodes; the per-member fallback then left each
member parked (candidate == null) — silently, with no command emitted.

Fix: in the candidates.Count == 0 branch, a group whose members are all
unassigned and declared by no node is a stale-snapshot artifact (every node can
run every shard; the agents are still enumerated by AllKnownAgentsAsync), not a
genuine blue/green gap. Assign the whole group to a node so the shard always has
a home, kept together to preserve connection-pool affinity. In the mixed-
capability case, an unassigned member with no capable node now falls back to the
least-loaded node overall instead of being silently stranded.

Adds a reproducing unit test (orphaned shard from a departed node with divergent
survivor snapshots) and updates the sharded no-capable-node test to assert the
member is reassigned rather than parked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 78932c5 into main Jul 8, 2026
25 checks passed
This was referenced Jul 9, 2026
This was referenced Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Event-subscription agents left unassigned after node scale-down (shards silently stop projecting)

1 participant