Fix DistributeByGroupAffinity assigning a bumped projection version to nodes that cannot build it

[erdtsieck]

DistributeByGroupAffinity documents itself as mirroring DistributeEvenlyWithBlueGreenSemantics, but on a real blue/green rollout of a multi-database store it does the opposite of that: it assigns the new version's agents to a node that cannot build them, and the new version then never starts anywhere.

Test first — the failing one in this PR is the intersection of two tests that already exist, which is why it slipped through.

What happens

Take one shard database mid-rollout. Its group (group key is the database, EventSubscriptionAgentFamily.DatabaseKeyOf) contains:

members of the group	declared by
Proj/All/v22/{tenant} — previous version, currently running	blue nodes
Proj/All/v23/{tenant} — new version	green nodes
agents of projections whose version did not change	all nodes

AllNodesHaveSameCapabilities is false, so the capability-matching branch runs. No node is capable of the whole group, so members.All(m => m.CandidateNodes.Contains(n)) is false everywhere — and the per-member fallback below is exactly the right behaviour for this.

It never runs. The || members.Any(m => m.OriginalNode == n) grandfathering keeps the incumbent blue node in candidates, so candidates.Count == 0 is false, the fallback is skipped, and:

• incumbent resolves to null (the v23 members are unassigned, so not all members share one AssignedNode);
• "the least-loaded candidate hosts the whole group" assigns every member to the blue node, v23 included;
• EventSubscriptionAgentFamily.BuildAgentAsync throws Unknown event projection or subscription there, because that node's store registers v22, not v23;
• green is never assigned anything, and the new version's tables stay empty.

Node.Assign has no capability check and MatchAgentsToCapableNodesFor is only consulted to build CandidateNodes, so nothing downstream catches it.

Single-database stores are unaffected — they take DistributeEvenlyWithBlueGreenSemantics, which matches per agent.

Where it came from

Both on 2026-07-06, in order:

• afe02783a "Blue/green capability matching inside DistributeByGroupAffinity" — correct.
• 748762a42 "Group placement: minimal disruption + grandfathering for stale capability snapshots" — adds the OriginalNode clause, which subsumes the genuine blue/green case as a side effect.

The grandfathering is right and this PR keeps it. a_group_lands_only_on_a_node_capable_of_running_it and a_node_already_running_a_groups_agents_stays_a_candidate_despite_a_stale_capability_snapshot both still pass; neither covers the case where both conditions hold at once, which is what a rollout looks like at every evaluation for its whole duration rather than transiently.

The fix, in two steps

1. A group is only a single placement unit when its members can actually share a host. Members are sub-partitioned by the set of nodes that declare them, and each partition placed whole. That preserves what the method exists for — affinity inside a version, so a shard database keeps one owner per version and pools scale with databases rather than nodes x databases (#486 / marten#4806). "Just always take the per-member fallback" is not the fix: it scatters one database's agents across every capable node, which is the pool explosion this method was written to prevent.

2. Partitions of one group prefer a node already hosting a sibling partition. Step 1 alone can make the pool budget worse than before. A database's group during a bump has three partitions, not two: previous version (blue-only), new version (green-only), and every projection whose version did not change — and that last one is declared by every node, so it is free to land on a third node and add a third pool set to that database. What a database costs is the number of distinct nodes holding any of its agents, so partitions now prefer a sibling's host, bounded by the same per-node ceiling the incumbent rule already respects.

With homogeneous capabilities there is exactly one partition per group and no sibling to prefer, so the common path is unchanged.

before:  Failed: 1, Passed: 8    (distribute_by_group_affinity)
after:   Passed: 10
         Passed: 287             (all of CoreTests.Runtime.Agents, net9.0 + net10.0)

Both new tests fail without their respective half — the second asserts the number of hosts per database rather than a specific placement, and without the sibling preference four databases across two blue and two green nodes spread onto three nodes each.

Scale context

This is the same 512-shard-database, ~854-tenant deployment as #3746 / #3747 and #594. The rollout shape we need is a second Deployment on the new image that builds the bumped projection version off the load balancer and then takes over serving, which leaves the cluster in mixed-capability state for the whole build — an hour or more, not the seconds a rolling update spends there. The pool arithmetic is why step 2 matters to us specifically: at 512 databases, two owners per database is affordable and three is not.

I have not run this on the cluster yet; the evidence here is the tests and the code path. If it helps, I can put a build of this on our canary (production snapshot, all 512 shards) and report what assignment actually does with a real version bump — that is the measurement loop you asked for, and the one thing this class of bug cannot get from a dev-scale reproduction.

One judgement call worth your eye: the partition key is the ordered set of declaring node ids, so two members share a partition only when the same nodes declare them. That is stricter than strictly necessary — members with overlapping-but-unequal candidate sets could sometimes still share a capable host — but it never assigns to an incapable node, and the sibling preference recovers most of what the strictness would otherwise cost. Happy to relax it to "group by whether a common capable node exists" if you prefer coarser partitions.

[Copilot]

Pull request overview

This PR fixes a blue/green rollout failure mode in AssignmentGrid.DistributeByGroupAffinity where a version-bumped projection agent could be assigned to a node that cannot build/run it, causing the new version to never start. The fix preserves group affinity (to control connection-pool fanout) while safely splitting groups only when mixed capabilities make a single-host placement impossible.

Changes:

• Split a “group” into capability-based partitions under mixed capabilities so incompatible agents (e.g., old vs. bumped projection versions) cannot be co-assigned to an incapable node.
• Prefer placing sibling partitions of the same group onto already-used hosts (within the existing per-node ceiling) to minimize distinct hosts per database and avoid pool explosion.
• Add regression tests covering version-bump splitting and ensuring split groups occupy only as many nodes as the capability split requires.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/Wolverine/Runtime/Agents/AssignmentGrid.Distribution.cs	Partition group placement by candidate-node sets under mixed capabilities and add sibling-host preference to reduce host fanout per group/database.
src/Testing/CoreTests/Runtime/Agents/distribute_by_group_affinity.cs	Add tests reproducing the blue/green version-bump scenario and asserting bounded host count per database during a split.

Suppressed comments (1)

src/Wolverine/Runtime/Agents/AssignmentGrid.Distribution.cs:331

• Private static helper methods in this codebase use camelCase; CapabilityKey is private static but PascalCase. Renaming it will keep naming consistent with other private helpers like canSelfHeal(...) (NodeAgentController.cs:448).

    private static string CapabilityKey(Agent agent) =>
        string.Join(",", agent.CandidateNodes.Select(n => n.AssignedId).OrderBy(id => id));

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[erdtsieck]

Pushed an end-to-end test, because the two unit tests prove the placement rule but not that a real store reaches it.

blue_green_version_bump_assignment in MartenTests/MultiTenancy runs over three real tenant databases and goes through EventSubscriptionAgentFamily.EvaluateAssignmentsAsync rather than DistributeByGroupAffinity directly, so it also covers the two steps between a leader and the placement: the per-store pass selection keyed off IEventStore.DatabaseCardinality, and RetireSupersededAgentsAsync.

The setup is the rollout itself. The blue store registers a projection at V2 and is already running its agents; the green store registers the same projection at V3 plus one other projection unchanged — so each database's group spans a blue-only, a green-only and an every-node partition, the same shape production has. Capabilities come from the family's own SupportedAgentsAsync, the grid is seeded from their union the way NodeAgentController does, and the family that evaluates is the blue one, so the green agents are reachable only through persisted node capabilities. That last detail is the case that breaks.

Verified both ways on a real Postgres:

main:        the_bumped_version_is_assigned_to_the_fleet_that_declares_it  FAIL
             event-subscriptions://marten/main/localhost.bgtenant3/trip/all/v3
             may only run on a node that declares it
with the fix: Passed: 2

It also pins down the premise the whole rollout rests on, which nothing covered before: the two fleets advertise the bumped projection under disjoint identities and share the agents of the projection that was not bumped. That one passes on main too — it is the thing that makes the fix meaningful rather than the thing the fix changes.

Two notes for review:

• The pool bound stays in the unit test, deliberately. I tried asserting "exactly one owner per version per database" here too and it fails for a legitimate reason: with three databases over four nodes the per-node ceiling binds before the sibling preference does, and spreading a group's third partition to balance load is the intended trade. The bound only holds where agents outnumber nodes the way they do in a real cluster, which the unit test models and this one cannot.
• Most of the diff in AssignmentGrid.Distribution.cs is indentation. The body moved inside a foreach (var members in partitions), so ~120 of those lines are whitespace-only. git diff -w main is 3 files / ~90 real lines if that is easier to read.

[erdtsieck]

Two updates, one of them a correction to something I claimed above.

The end-to-end test now models the shape the bug lives in

My first version used AddSingleTenantDatabase — one tenant per database — which exercises the version split but not what group affinity is actually for. It now uses three databases carrying three tenants each, with Events.TenancyStyle = Conjoined and UseTenantPartitionedEvents = true, so IEventStore.DistributesAgentsPerTenant is on and there is an agent per (database, tenant, projection version). Each database's group is nine agents across three capability classes instead of three, and the failure on main now reads with the full grammar:

event-subscriptions://marten/main/localhost.bgshard1/trip/all/v3/bgshard1-alpha
may only run on a node that declares it

It also asserts the shape itself, because without that a change which stopped distributing per tenant would shrink every group back to one agent per projection and the file would keep passing while testing something much weaker.

Correction: the bound is one host per capability class, not two

I wrote above that "at 512 databases, two owners per database is affordable and three is not". The second half is a fair description of our connection budget; the first half overstated what this PR guarantees.

What the partitioning actually bounds is one host per distinct capability set in the group — during a single-version bump that is three (previous-version-only, new-version-only, and the projections that did not change, which every node declares). The sibling preference collapses that to two whenever the shared partition can join one of the other two, but it is bounded by the same per-node ceiling the incumbent rule respects, so under load it legitimately spills to a third node to keep the distribution balanced. Two is the common case, three is the guarantee.

That matters for how the fix should be read: the value is not "exactly two" but that the count is bounded by capability classes at all. The per-member fallback — the other way to fix the assignment — bounds it by nothing: it would place each agent individually, so a database's nine agents could land on nine nodes and each one opens a pool on that shard.

each_database_keeps_one_owner_per_version_and_no_more is therefore deliberately not in the end-to-end test; the exact-two assertion in distribute_by_group_affinity holds for that test's fixed arithmetic, where the ceiling does not bind. If you would rather see the general bound asserted somewhere, "at most one host per capability class" is the honest form and I am happy to add it.