Managed distribution: per-tenant agents + database-affine assignment for sharded stores (#3280, marten#4806)#3281
Conversation
…partitioned stores EventStoreAgents.SupportedAgentsAsync now expands each async shard into one agent URI per (database, tenant) when the store reports DistributesAgentsPerTenant, registering the per-tenant ShardName (ForTenant) so ResolveShardNameAsync round-trips it and the daemon starts a per-tenant agent. Databases with no tenants keep the store-global agent. Fixes co-located tenants on a shard sharing one high-water (lagging tenant skipped). See JasperFx#3280.
There was a problem hiding this comment.
Pull request overview
This PR updates Wolverine’s managed event-subscription distribution to correctly create per-tenant async projection agents when using a store that requires per-tenant distribution (intended for sharded databases + tenant-partitioned events, per #3280), preventing co-located tenants from incorrectly sharing a single high-water mark.
Changes:
- Precomputes the set of async projection shard names once per
SupportedAgentsAsynccall. - Expands async shard agent URIs to one agent per (database, tenant, shard) when
_store.DistributesAgentsPerTenantand the database descriptor has known tenants; otherwise preserves the existing store-global agent behavior. - Registers per-tenant
ShardNameentries into the shard-name cache soResolveShardNameAsynccan round-trip agent URIs back to shard definitions.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (_store.DistributesAgentsPerTenant && database.TenantIds.Count > 0) | ||
| { | ||
| foreach (var tenantId in database.TenantIds.Distinct()) | ||
| { | ||
| var tenantShard = shardName.ForTenant(tenantId); | ||
| _shardNames = _shardNames.AddOrUpdate(tenantShard.RelativeUrl, tenantShard); | ||
| list.Add(EventSubscriptionAgentFamily.UriFor(_store.Identity, id, tenantShard)); | ||
| } | ||
| } |
| // wolverine#3280: under sharded databases + per-tenant event partitioning, multiple tenants are | ||
| // co-located in one shard database and each draws its own event sequence, so a single store-global | ||
| // agent per (shard, database) cannot track them — fan out one agent per (shard, tenant) instead. | ||
| // For any other store (or a database with no tenants yet) keep the one store-global agent. | ||
| void addAgentsForShard(DatabaseDescriptor database, DatabaseId id, ShardName shardName) |
… agents per tenant Covers SupportedAgentsAsync emitting one agent URI per tenant when IEventStore.DistributesAgentsPerTenant is true and the database has assigned tenants, and falling back to per-shard when it is not.
…inity) When a sharded store sets IEventStore.GroupAgentAssignmentsByDatabase, EventSubscriptionAgentFamily assigns event-subscription agents grouped by their shard database (URI database segment) via a new greedy least-loaded DistributeByGroupAffinity, keeping every agent of a database on one node and balancing total agent count across nodes. Each node then opens connection pools only to the databases it owns. Default path (even distribution) unchanged. Unit test covers the grouping + balance. See JasperFx/marten#4806.
DistributeByGroupAffinity gains a maxNodesPerGroup bound: a shard database's agents are placed on its k = min(bound, size, nodes) least-loaded nodes and greedily balanced, so a heavy database parallelizes across up to N nodes while its server connection ceiling stays N x pool. EventSubscriptionAgentFamily passes the largest MaxNodesPerDatabaseForAgents among affine stores. maxNodesPerGroup=1 is the prior strict-affinity behavior. Tests cover fan-out + small-group cases. See JasperFx/marten#4806.
…KeyOf Covers the branch selection in EventSubscriptionAgentFamily (affine grouping when a store opts in, even blue/green spread otherwise), that the family plumbs the store's MaxNodesPerDatabaseForAgents bound into the grid, DatabaseKeyOf URI grouping/fallback, and the EventStoreAgents pass-through of both flags. Fast unit tests over an AssignmentGrid with NSubstitute stores. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
New section under Projection/Subscription Distribution documenting the opt-in database-affine assignment driven by Marten's UseDatabaseAffineAgentAssignment / DatabaseAffineAgentFanout, surfaced via IEventStore.GroupAgentAssignmentsByDatabase and MaxNodesPerDatabaseForAgents. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
MartenIntegration gains UseDatabaseAffineAgentAssignment + DatabaseAffineAgentFanout, bridged onto the Marten event store (main store only) via the existing MartenOverrides IConfigureMarten hook, so the database-affine distribution can be configured where you opt into Wolverine-managed distribution rather than only on StoreOptions.Events. Doc updated to show the IntegrateWithWolverine surface. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
Guards the MartenOverrides bridge (resolving MartenIntegration and applying the settings to StoreOptions.Events) against a DI-resolution regression: asserts UseDatabaseAffine- AgentAssignment / DatabaseAffineAgentFanout set on IntegrateWithWolverine land on Events, and that defaults are unchanged when not configured. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
DistributeByGroupAffinity loses the maxNodesPerGroup bound: a group (shard database) is always placed whole on the least-loaded node, largest-first, deterministic. The family/EventStoreAgents fan-out pass-throughs and the MartenIntegration fanout setting go with it — the mix was speculative and never needed >1; a minimal surface is easier to accept upstream. Tests + docs updated. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
erdtsieck
changed the title
…ntegration test - Per-tenant fan-out now dedupes tenant ids case-insensitively (matching TryResolveTenantDatabaseIdAsync) and skips blank ids, so a malformed usage descriptor cannot yield duplicate or invalid per-tenant agents. Unit-tested. - New integration test sharded_tenant_partitioned_distribution: the JasperFx#3280 regression end-to-end — MultiTenantedWithShardedDatabases + UseTenantPartitionedEvents with two tenants CO-LOCATED on one shard database under managed distribution asserts one agent per tenant (":all/<tenant>" URIs) and that both tenants' events project against their own high-water marks. Tenants are provisioned before host start; adding a database's FIRST tenants at runtime leaves the empty-phase store-global agent running alongside the new per-tenant agents until assignments reconcile — noted in the test as a known transition edge. Runs green locally against the companion JasperFx#482 + marten#4799 builds; goes green in CI once those merge (stated merge order). Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
…on' into feat/per-tenant-agent-distribution
|
Addressed both Copilot findings in a38bce8:
One observation from writing the test, noted there as a known transition edge: when a database gains its first tenants while the host is running, the store-global agent from the empty-database phase keeps running alongside the new per-tenant agents until assignments reconcile. The test provisions tenants before host start; retiring the stale agent on that transition may deserve a small follow-up. |
|
Thank you for this work @erdtsieck — the per-(shard, tenant) fan-out and the test coverage here are solid, and we're going to reuse them. Closing this PR only because the design underneath it is shifting on the JasperFx side and it's cleaner to restart than rebase:
The 🤖 Generated with Claude Code |
|
One addition for the successor PR, from running this at production scale (500 tenants / 512 sharded databases, field notes on #486): a complement to the "retire stale store-global agents on the first-tenant transition" bullet in the close-out. The transition edge has two sides, and never-creating is cleaner than retiring-after-the-fact:
Isolated commit (fan-out change + unit test that asserts an empty per-tenant database yields zero agents): erdtsieck/wolverine@dfeccbe97 on branch |
…y-cardinality Per-tenant agent fan-out + automatic database-affine assignment by store cardinality (supersedes #3281)
Wolverine piece of #3280 and JasperFx/marten#4806. Depends on JasperFx/jasperfx#482 and pairs with JasperFx/marten#4799.
Per-tenant agents (#3280)
EventStoreAgents.SupportedAgentsAsyncfans out one agent per (shard, tenant) when the store reportsDistributesAgentsPerTenant— fixing sharded + tenant-partitioned stores where co-located tenants shared one high-water mark and a lagging tenant's events were skipped. Store-global and database-per-tenant stores keep one agent per (shard, database), unchanged.Database-affine assignment (marten#4806, opt-in)
With per-tenant agents across many shard databases, the default even spread makes every node open connection pools to (nearly) every database — pools grow as nodes × databases and exhaust a shared server's
max_connections(observed in production-scale testing: Postgres 53300 at ~9% DB CPU — the ceiling was connections, not hardware). When a store reportsGroupAgentAssignmentsByDatabase:AssignmentGrid.DistributeByGroupAffinity(scheme, groupKey): every agent sharing a group key (the[store type]/[store name]/[database]prefix of the agent Uri) is placed whole on the least-loaded node, largest group first, with deterministic tie-breaks — so a node only connects to the databases it owns and the connection budget scales with the topology (databases), not the deployment (replicas).EventSubscriptionAgentFamily.EvaluateAssignmentsAsyncuses it when any registered store opts in; otherwise the existingDistributeEvenlyWithBlueGreenSemanticspath is untouched. (The affine path does not apply blue/green capability matching — called out in code + docs.)IntegrateWithWolverine(m => m.UseDatabaseAffineAgentAssignment = true)flows through toStoreOptions.Events.UseDatabaseAffineAgentAssignmentfor the main store.Tests & docs
Unit tests for the per-tenant fan-out, grid affinity (grouping, spreading, balancing around a heavy database), family branch selection, the Uri group key, and the IntegrateWithWolverine bridge; the distribution guide documents the opt-in.
Merge order: #482 → Marten #4799 → this.