Make database affinity a property of the database across agent families (GH-3785)

[jeremydmiller]

Addresses #3785. Builds on #3792; relates to #3753.

The problem, as measured

Assignment honoured database affinity within the event-subscriptions family (#3792 / marten#4806) and within the durability family — but not between them. On the reporting cluster (5 pods, 512 shard databases, ~8,700 agents): 73% of shard databases had durability and projections on different nodes, costing ~425 connections held by durability owners against databases they otherwise never open — on a server that had just spent half an hour at 2,393 of 2,400 max_connections.

The fix

1. AssignmentGrid.DistributeEvenlyWithAffinity — an even distribution where any agent with a preferred node goes there regardless of the even spread. Preferred placements are deliberately not ceiling-bounded: they piggyback on the projection family's own balanced distribution, and co-location is the point even when it costs strict evenness. The remainder (databases with no projections — e.g. the main store) spreads evenly counting only itself, so no-affinity agents don't crowd onto whichever nodes hold no projections.

2. DurabilityProjectionAffinity — joins a wolverinedb:// agent URI to the node owning that database's event-subscriptions:// agents in the current pass. The two families describe the same physical database through different pipelines (Weasel descriptor vs Marten database descriptor), so the join keys on the database name when unambiguous, falling back to normalized server comparison only when two servers carry the same name. A miss is never wrong, only not-better — the agent falls back to today's even spread. During a blue/green split the durability agent follows the larger side, deterministically, so it doesn't flap between the two version owners.

3. NodeAgentController now explicitly evaluates the durability family last — previously true only by Dictionary insertion-order accident, and the affinity can only see the event-subscription assignments if they're already in the shared grid.

An agent running away from its preferred node is moved (an ordinary ReassignAgent) — the one-time migration that converges an existing cluster. The settled co-located state is a fixed point, pinned by test.

Testing

Nine unit tests in durability_follows_projection_affinity (placement, one-time migration, fixed point, even fallback, name-ambiguity disambiguation, port-spelling tolerance, blue/green larger-side follow, family-ordering guarantee) — the placement and ordering tests verified red against the pre-change code.

Plus the integration test a unit test cannot substitute for: durability_projection_affinity_real_stores runs both real URI pipelines (a multi-database Marten store on one side, Weasel-described PostgresqlMessageStores on the other) against the same three tenant databases and asserts the join actually engages — because a spelling divergence between the two pipelines makes the affinity silently never fire, which looks exactly like the feature working, minus the benefit.

• CoreTests: 2245 passed
• MartenTests.Distribution + MartenTests.MultiTenancy: 105 passed
• dotnet build wolverine.slnx -c Release: clean

What to measure on the canary

The #3785 numbers make this directly verifiable: the durability-owner-on-a-different-node count should drop from ~375/446 toward zero over one convergence cycle (expect a one-time wave of durability-agent reassignments on first deploy), and the shard server's connection count should shed roughly the ~425 durability-owner connections.

🤖 Generated with Claude Code

https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA