Per-tenant agent fan-out + automatic database-affine assignment by store cardinality (supersedes #3281)#3328
Merged
Conversation
…partitioned stores EventStoreAgents.SupportedAgentsAsync now expands each async shard into one agent URI per (database, tenant) when the store reports DistributesAgentsPerTenant, registering the per-tenant ShardName (ForTenant) so ResolveShardNameAsync round-trips it and the daemon starts a per-tenant agent. Databases with no tenants keep the store-global agent. Fixes co-located tenants on a shard sharing one high-water (lagging tenant skipped). See #3280.
… agents per tenant Covers SupportedAgentsAsync emitting one agent URI per tenant when IEventStore.DistributesAgentsPerTenant is true and the database has assigned tenants, and falling back to per-shard when it is not.
…inity) When a sharded store sets IEventStore.GroupAgentAssignmentsByDatabase, EventSubscriptionAgentFamily assigns event-subscription agents grouped by their shard database (URI database segment) via a new greedy least-loaded DistributeByGroupAffinity, keeping every agent of a database on one node and balancing total agent count across nodes. Each node then opens connection pools only to the databases it owns. Default path (even distribution) unchanged. Unit test covers the grouping + balance. See JasperFx/marten#4806.
DistributeByGroupAffinity gains a maxNodesPerGroup bound: a shard database's agents are placed on its k = min(bound, size, nodes) least-loaded nodes and greedily balanced, so a heavy database parallelizes across up to N nodes while its server connection ceiling stays N x pool. EventSubscriptionAgentFamily passes the largest MaxNodesPerDatabaseForAgents among affine stores. maxNodesPerGroup=1 is the prior strict-affinity behavior. Tests cover fan-out + small-group cases. See JasperFx/marten#4806.
…KeyOf Covers the branch selection in EventSubscriptionAgentFamily (affine grouping when a store opts in, even blue/green spread otherwise), that the family plumbs the store's MaxNodesPerDatabaseForAgents bound into the grid, DatabaseKeyOf URI grouping/fallback, and the EventStoreAgents pass-through of both flags. Fast unit tests over an AssignmentGrid with NSubstitute stores. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
New section under Projection/Subscription Distribution documenting the opt-in database-affine assignment driven by Marten's UseDatabaseAffineAgentAssignment / DatabaseAffineAgentFanout, surfaced via IEventStore.GroupAgentAssignmentsByDatabase and MaxNodesPerDatabaseForAgents. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
MartenIntegration gains UseDatabaseAffineAgentAssignment + DatabaseAffineAgentFanout, bridged onto the Marten event store (main store only) via the existing MartenOverrides IConfigureMarten hook, so the database-affine distribution can be configured where you opt into Wolverine-managed distribution rather than only on StoreOptions.Events. Doc updated to show the IntegrateWithWolverine surface. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
Guards the MartenOverrides bridge (resolving MartenIntegration and applying the settings to StoreOptions.Events) against a DI-resolution regression: asserts UseDatabaseAffine- AgentAssignment / DatabaseAffineAgentFanout set on IntegrateWithWolverine land on Events, and that defaults are unchanged when not configured. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
DistributeByGroupAffinity loses the maxNodesPerGroup bound: a group (shard database) is always placed whole on the least-loaded node, largest-first, deterministic. The family/EventStoreAgents fan-out pass-throughs and the MartenIntegration fanout setting go with it — the mix was speculative and never needed >1; a minimal surface is easier to accept upstream. Tests + docs updated. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
…on test - Per-tenant fan-out now dedupes tenant ids case-insensitively (matching TryResolveTenantDatabaseIdAsync) and skips blank ids, so a malformed usage descriptor cannot yield duplicate or invalid per-tenant agents. Unit-tested. - New integration test sharded_tenant_partitioned_distribution: the #3280 regression end-to-end — MultiTenantedWithShardedDatabases + UseTenantPartitionedEvents with two tenants CO-LOCATED on one shard database under managed distribution asserts one agent per tenant (":all/<tenant>" URIs) and that both tenants' events project against their own high-water marks. Tenants are provisioned before host start; adding a database's FIRST tenants at runtime leaves the empty-phase store-global agent running alongside the new per-tenant agents until assignments reconcile — noted in the test as a known transition edge. Runs green locally against the companion #482 + marten#4799 builds; goes green in CI once those merge (stated merge order). Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
…s it from store cardinality Drops MartenIntegration.UseDatabaseAffineAgentAssignment, the MartenOverrides bridge onto StoreOptions.Events, the EventStoreAgents.GroupAgentAssignmentsByDatabase passthrough (the revised JasperFx.Events contract no longer carries that member), the opt-in branch in EventSubscriptionAgentFamily.EvaluateAssignmentsAsync, the flag-specific test suites, and the opt-in docs section. Database-affine assignment comes back in the next commits keyed off IEventStore.DatabaseCardinality instead of an explicit flag. Epic: JasperFx/jasperfx#486. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… per store EventSubscriptionAgentFamily.EvaluateAssignmentsAsync now partitions the scheme's agents per owning store (StoreKeyOf reproduces the _stores key from the agent URI's type/name prefix): every store whose DatabaseCardinality is StaticMultiple or DynamicMultiple gets its agents distributed with group affinity by DatabaseKeyOf, while all remaining agents keep the even blue/green distribution — no opt-in flag. AssignmentGrid grows Func<Uri,bool>-filtered overloads of DistributeEvenly, DistributeEvenlyWithBlueGreenSemantics, DistributeByGroupAffinity, AllNodesHaveSameCapabilities, MatchAgentsToCapableNodesFor and AvailableAgentsForScheme; the existing signatures delegate to them with an all-pass filter. Filtered passes count per-node load only over their own agent subset so independent passes of one scheme never detach each other's agents. Docs updated to describe the automatic behavior. Epic: JasperFx/jasperfx#486. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Group placement now mirrors DistributeEvenlyWithBlueGreenSemantics' capability semantics: homogeneous capabilities keep the capability-blind placement; when node capabilities differ, a group's candidate nodes are those capable of every member (via MatchAgentsToCapableNodesFor), the least-loaded candidate hosts the whole group with the existing tie-breaks, and when no node can host the whole group each member falls back individually to its least-loaded capable node — an agent no node declares a capability for is left exactly as the even path leaves it. Epic: JasperFx/jasperfx#486. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…shard The assignment grid is seeded from every node's ActiveAgents, so when a database gains its first tenant at runtime the old store-global agent is still present and running in the grid even though SupportedAgentsAsync now enumerates per-tenant agents instead — the distribution would keep it assigned and it would process events concurrently with its replacements (double-processing). EvaluateAssignmentsAsync now first detaches any agent of the scheme that (a) is no longer in the supported set and (b) shares a tenant-neutral key — store/database/projection/shardKey/version, tenant slot stripped per the ShardName.RelativeUrl grammar — with a supported agent, and excludes it from both distribution passes; a detached running agent is exactly what TryBuildAssignmentCommand turns into StopRemoteAgent. The inverse transition (tenant removed -> its per-tenant agent superseded by the remaining scoping) retires the same way. Deliberately narrower than 'not in the supported set': blue/green rollouts legitimately run agents the evaluating node's store doesn't enumerate (new projections/versions matched via node capabilities), and a version bump changes the version slot, not the tenant slot, so those keep running untouched. Epic: JasperFx/jasperfx#486. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ale retirement Replaces the flag-based family suite with event_subscription_family_cardinality_assignment (affinity auto-on for Static/DynamicMultiple stores, even spread for Single/None, mixed-store independence, blue/green capabilities honored per pass, node-loss failover regrouping on survivors), adds event_subscription_family_stale_agent_retirement (store-global agent stopped when a database gains its first tenants, per-tenant agent stopped when its tenant is removed, unrelated/other-version agents left running), tenant_neutral_key_of grammar tests, and capability-aware DistributeByGroupAffinity tests (group lands only on a capable node; per-member fallback parks agents no node declares). Epic: JasperFx/jasperfx#486. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…automatically Two shard databases with two co-located tenants each across a two-node cluster: the Marten store's multi-database cardinality alone (no opt-in) makes managed distribution assign each database's per-(shard, tenant) agents together on one node, and spread the two database groups one per node. Requires the Marten side's DistributesAgentsPerTenant override, so like sharded_tenant_partitioned_distribution this goes green once the companion Marten release ships. Epic: JasperFx/jasperfx#486. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lity snapshots Two even-path semantics the group-affinity placement was missing, both exposed by MartenTests' MultiTenantContext suites (spread_databases_out_via_host): 1. Grandfathering: node capability snapshots are persisted once at node startup, so a node that started before its store's tenant databases were provisioned declares no event-subscription capabilities even though it runs those agents. The even paths leave running agents in place regardless of capabilities; a node already running part of a group is now a candidate for that group. 2. Minimal disruption with a ceiling: the even paths only move agents off a node when it exceeds ceil(agents/nodes). Group placement reshuffled every group from scratch each evaluation, so a stale-capability node could be starved permanently once an intermediate evaluation moved its groups away. The node currently running a WHOLE group now keeps it unless that pushes it past the ceiling; only over-ceiling groups move (to the least-loaded candidate, same tie-breaks). The no-candidate fallback likewise leaves already-assigned members in place. Epic: JasperFx/jasperfx#486. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JasperFx 2.20.0 (#487): IEventStore.DistributesAgentsPerTenant + hardened per-tenant StartAgentAsync. Marten 9.13.0-alpha.2 (marten#4856): DocumentStore reports DistributesAgentsPerTenant for sharded + tenant-partitioned stores — required by the two end-to-end distribution tests. (alpha.1 was a burned version: nuget.org serves stale 2026-06-27 bits for it, without the override.) Marten 9.13 also moved the storage-operation contract (marten#4820/#4821/ #4861 storage-dialect extraction): ConfigureCommand now takes Weasel.Postgresql.ICommandBuilder + Weasel.Storage.IStorageSession, and NoDataReturnedCall lives in Weasel.Storage — both envelope operations adapted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jeremydmiller
added a commit
that referenced
this pull request
Jul 7, 2026
…urn/failover e2e, #4862 regression pin (#486) (#3334) * Consume Marten 9.13.0-alpha.3 + JasperFx 2.21.0 (epic #486 WS0/WS1) Marten 9.13.0-alpha.3 carries the per-tenant distribution work this PR's e2e tests exercise: marten#4864 (broadened DistributesAgentsPerTenant gate + single-DB descriptor TenantIds), marten#4865 (ExternallyManaged honored), marten#4869 (per-tenant high-water advancement — unblocks the failover floor-continuity test), and marten#4870 (native per-tenant distribution). JasperFx/JasperFx.Events pins move 2.20.0 -> 2.21.0 to match what Marten alpha.3 already pulls transitively — keeps the explicit pins from drifting below the resolved graph. No envelope-operation contract drift from marten#4872 (Weasel 9.12 metadata binders): Wolverine.Marten's StoreIncomingEnvelope/StoreOutgoingEnvelope compile unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * E2E: runtime tenant churn under Wolverine-managed distribution (#486 WS1) Sharded + UseTenantPartitionedEvents, single node: with tenant-a's per-tenant agent already projecting, add tenant-b to the same shard database at RUNTIME via Advanced.AddTenantToShardAsync. Within the assignment-evaluation window (CheckAssignmentPeriod = 1s) the leader re-enumerates the store usage, fans out the ":all/tenant-b" agent, tenant-b's fresh seq-1.. events project, its HighWaterMark-independent PartitionedCounter:All:tenant-b progression row lands in the shard database, and no store-global (tenant-less) agent ever runs alongside the per-tenant agents (the #3328 stale-agent-retirement edge). Tenant REMOVAL is deliberately not covered end-to-end: Marten 9.13.0-alpha.2 exposes RemoveTenantAsync/DisableTenantAsync (ShardedTenancy.cs:378/:470), but a running store's usage never shrinks - MartenDatabase.TenantIds is add-only (Fill at ShardedTenancy.cs:211/:364) and DescribeDatabasesAsync copies that in-memory list, so Wolverine keeps enumerating the removed tenant's agent until a process restart. The Wolverine half of the removal transition stays pinned by the running_per_tenant_agent_is_stopped_when_its_tenant_is_removed unit test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * E2E (pinned/skipped): failover preserves per-tenant progression floors (#486 WS1) Two nodes, sharded + UseTenantPartitionedEvents, two tenants co-located on one shard database: append N events per tenant, record each tenant's progression floor, kill the node owning the affine agent group, assert the survivor picks both agents up and resumes tenant-b exactly from its floor (doc == N+M, no skips, no rewind) after M more post-failover events. SKIPPED, pinning a Marten 9.13.0-alpha.2 product bug found by this test: the failover mechanics work (survivor restarts both agents, floors never rewind, tenant-a intact), but per-tenant high-water detection FREEZES at the first persisted HighWaterMark:<tenant> row - HighWaterDetector.loadPerTenantStatistics (src/Marten/Events/Daemon/HighWater/HighWaterDetector.cs:260): var currentMark = lastSeqId > 0 ? lastSeqId : lastValue; Once the row exists, CurrentMark is pinned to it and max(seq_id) is never consulted again, so ANY second batch of events per tenant never projects (not failover-specific; sibling tests all append one batch per tenant and never see it). Empirically confirmed: tenant-b at max seq_id 11 committed, store-global HighWaterMark advanced to 11 (global detection + per-tenant poll trigger both firing), HighWaterMark:tenant-b frozen at 6. Unskip on the Marten-side fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Unskip the failover floor-continuity test — fixed by marten#4869 in 9.13.0-alpha.3 The skip pinned marten#4867: per-tenant high-water detection froze at the first persisted HighWaterMark:<tenant> row, so the M post-failover tenant-b events never projected. Marten 9.13.0-alpha.3 advances per-tenant CurrentMark past the persisted row (marten#4869), which makes the final leg pass. The class comment keeps a condensed history of the pinned bug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Regression test for marten#4862: single-DB tenant-partitioned per-tenant distribution (#486 WS1) Promotes the WS1 single-database fact-finding probe (found the bug 2026-07-06) into the marten#4862 regression test. Against Marten 9.13.0-alpha.2 a single-database store reported DistributesAgentsPerTenant = false, so managed distribution ran ONE store-global agent whose progression floor (driven to the leading tenant's max per-tenant seq) permanently skipped a lagging tenant's later events at per-tenant seq 1..N — the wolverine#3280 failure mode on one database. Marten 9.13.0-alpha.3 (marten#4864) broadens the gate to any tenant-partitioned store, so the startup expectation flips to per-tenant fan-out (2 tenants -> 2 agents) and the lagging-tenant projection assertions now pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Flip single-DB distribution granularity tests to per-tenant fan-out (marten#4862/#4864) With Marten 9.13.0-alpha.3, DistributesAgentsPerTenant is true for ANY tenant-partitioned store, so one async projection over a single tenant-partitioned database now fans out one subscription agent per managed tenant instead of a single store-global "/all" agent whose lone progression floor could silently skip a lagging tenant. - tenant_partitioned_distribution_granularity: 1 agent -> 3 (one per registered tenant, default tenant included), agent URIs carry the tenant in the shard path - tenant_partitioned_distribution_multinode: 2 store-global agents -> 6 per-tenant agents (2 projections x 3 tenants), spread 3+3 across the cluster and all reassigned to the survivor on node loss - AssignmentWaiter.WritePersistedActualsAsync no longer throws KeyNotFoundException when the database holds a node record the waiter isn't tracking (ghost rows from a hard-stopped prior run); it reports the untracked node instead so the TimeoutException diagnostic survives Part of #486 WS1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Ancillary stores: dispatch message-store shape on database cardinality, not tenancy type name Marten 9.13.0-alpha.3 (marten#4864) gives a single-database store with UseTenantPartitionedEvents a TenantPartitionedSingleDatabaseTenancy — a DefaultTenancy subclass, chosen precisely so 'is DefaultTenancy' dispatch keeps working. The ancillary IntegrateWithWolverine seam compared Tenancy.GetType().Name == "DefaultTenancy" instead, so such a store was misrouted to the multi-tenanted message-database path and host startup failed with "no configured connectivity for the required master PostgreSQL message database" after 20 broker retries (the CIMarten failure on this PR). Both the Marten and Polecat ancillary seams now dispatch on Tenancy.Cardinality == DatabaseCardinality.Single, mirroring the main-store integration (WolverineOptionsMartenExtensions), which is why only ancillary stores were affected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 9, 2026
This was referenced Jul 14, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #3281 — all of @erdtsieck's commits carried with authorship intact, plus a redesigned distribution layer on top. Closes #3280. Also addresses JasperFx/marten#4806 and JasperFx/marten#4798 (the Wolverine leg). Part of the per-tenant event partitioning epic JasperFx/jasperfx#486; final car of the train after #487 (JasperFx 2.20.0) and marten#4856 (Marten 9.13.0-alpha.2).
What it does
1. Per-(shard, tenant) agent fan-out (from #3281, unchanged): when a store reports
DistributesAgentsPerTenant(Marten: sharded databases +UseTenantPartitionedEvents),EventStoreAgents.SupportedAgentsAsyncemits one agent per (shard × tenant) instead of one store-global agent per (shard × database) — co-located tenants with independent event sequences each get their own high-water tracking, closing the #3280 data-loss path where a lagging tenant's appends fell below the shared mark and were skipped.2. Database-affine agent assignment — automatic, per store, no configuration. The
UseDatabaseAffineAgentAssignmentopt-in from #3281 is gone. Instead, each store's agents are distributed by its own topology, derived from the existingIEventStore.DatabaseCardinality:StaticMultiple/DynamicMultiple→ group affinity: all agents for the same shard database land on one node (AssignmentGrid.DistributeByGroupAffinity, largest-group-first bin packing). Connection pools scale with databases, not nodes × databases (the marten#4806 production report: 128 shard DBs → Postgres 53300 exhaustion at 9% CPU).Single/None→ today's even + blue/green distribution, byte-for-byte.3. Capability-aware group placement. Group affinity honors blue/green node capabilities with the same semantics as the even path: candidate nodes must be capable of the group's agents; grandfathering keeps a node already running part of a group as a candidate when its startup capability snapshot predates newly-added tenant databases; minimal-disruption keeps a group where it runs unless that node exceeds its fair share.
4. Stale-agent retirement. When a shard's tenant scoping changes at runtime (a database gains its first tenant → per-tenant agents supersede the store-global agent, or a tenant is removed), the assignment evaluation now stops the superseded running agents — closing a concurrent double-processing window. Retirement is keyed on tenant-neutral shard identity, so blue/green agents the leader doesn't enumerate are deliberately untouched.
Behavior change (release note)
Existing Wolverine + multi-database Marten apps (including plain database-per-tenant, no event partitioning) move from per-agent even spreading to whole-database placement on upgrade. This matches what Marten's native HotCold
MultiTenantedProjectionDistributoralready does (one database's shards stay together), and is the connection-conserving behavior. Adding nodes beyond the database count no longer buys projection parallelism for multi-DB stores.Tests
CoreTests(grid/agent): fan-out, cardinality gating, mixed stores, failover (node loss → groups reassign), stale retirement, capability-aware placement — plus all of @erdtsieck's original suites, adapted to the no-flag world.MartenTests.Distributionend-to-end against published packages (JasperFx 2.20.0, Marten 9.13.0-alpha.2): @erdtsieck'ssharded_tenant_partitioned_distribution(Managed event-subscription distribution: no per-tenant agents under MultiTenantedWithShardedDatabases + UseTenantPartitionedEvents (Phase 3) → co-located tenants share one high-water, lagging tenant's events skipped #3280 regression) and the newsharded_two_databases_affine_colocation(2 shard DBs × 2 tenants × 2 nodes: each database's agents co-located, databases spread across nodes).An epic side-note: these e2e tests caught that the first
9.13.0-alpha.1publish served stale June bits (burned version number on nuget.org) — hence the alpha.2 pin.Follow-ups (deliberately out of scope)
EvaluateAssignmentsAsynccallsSupportedAgentsAsynconce more per cycle for retirement; consider caching for very large stores.🤖 Generated with Claude Code