Per-tenant agent distribution + database-affine assignment surface (wolverine#3280, marten#4806)#482
Closed
erdtsieck wants to merge 6 commits into
Closed
Conversation
This was referenced Jun 30, 2026
… + StartAgentAsync per-tenant fan-out Adds IEventStore.DistributesAgentsPerTenant (default false) so a node-distributed daemon can fan out one agent per (shard, tenant) for sharded + tenant-partitioned stores. Fixes the blocker that StartAgentAsync(string) could only resolve store-global shard identities from AllShards(): it now parses a per-tenant identity (<proj>:All:<tenant>), resolves the base shard, activates the tenant in the high-water coordinator, and starts a per-tenant agent (the shape buildPerTenantContinuousAgents already uses). See JasperFx/wolverine#3280. Refs: ZOD-2330
erdtsieck
force-pushed
the
feat/per-tenant-agent-distribution
branch
from
June 30, 2026 09:26
eb14c70 to
6863843
Compare
When Wolverine-managed distribution starts a per-tenant projection agent, the agent had no high-water mark until the next global poll tick, so a freshly assigned tenant's projections did not advance. StartAgentAsync now runs pollTenantHighWaterAsync once the tenant agent has started, so progression begins immediately. Covered by the marten regression per_tenant_agent_start_routes_high_water.
… assignment Default interface member (=> false) so existing IEventStore implementers are unaffected. Signals that a node-distributed daemon should co-locate a shard database's per-(shard,tenant) agents on one node, so pools scale with databases not nodes x databases. Opt-in; see JasperFx/marten#4806.
Complements GroupAgentAssignmentsByDatabase: an int (default 1) capping how many nodes a single shard database's agents may fan out across. 1 = strict affinity; higher trades connections for parallelism on heavy databases. See JasperFx/marten#4806.
DistributesAgentsPerTenant / GroupAgentAssignmentsByDatabase default to false and MaxNodesPerDatabaseForAgents to 1 on a store that overrides none of them, following the AllDatabasesDefaultTests bare-store convention. See JasperFx/marten#4806 and JasperFx/wolverine#3280. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
Slims the upstream surface to the essential opt-in: GroupAgentAssignmentsByDatabase keeps a shard database's agents on one node (pools scale with databases, not nodes x databases). The bounded fan-out knob was speculative — per-database load is small under per-tenant partitioning and rebuild throughput is server-IO-bound, so a mix never proved necessary. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX
erdtsieck
changed the title
This was referenced Jul 6, 2026
Merged
Member
|
Thank you @erdtsieck — this design and nearly all of the code carry forward as-is. Superseded by #487, which contains every commit from this branch with your authorship intact, plus three review-driven revisions on top:
Tracked under the per-tenant partitioning epic #486. 🤖 Generated with Claude Code |
jeremydmiller
added a commit
that referenced
this pull request
Jul 6, 2026
… + hardened per-tenant StartAgentAsync (supersedes #482) (#487) * Per-tenant agent distribution: IEventStore.DistributesAgentsPerTenant + StartAgentAsync per-tenant fan-out Adds IEventStore.DistributesAgentsPerTenant (default false) so a node-distributed daemon can fan out one agent per (shard, tenant) for sharded + tenant-partitioned stores. Fixes the blocker that StartAgentAsync(string) could only resolve store-global shard identities from AllShards(): it now parses a per-tenant identity (<proj>:All:<tenant>), resolves the base shard, activates the tenant in the high-water coordinator, and starts a per-tenant agent (the shape buildPerTenantContinuousAgents already uses). See JasperFx/wolverine#3280. Refs: ZOD-2330 * Poll per-tenant high water right after starting a per-tenant agent When Wolverine-managed distribution starts a per-tenant projection agent, the agent had no high-water mark until the next global poll tick, so a freshly assigned tenant's projections did not advance. StartAgentAsync now runs pollTenantHighWaterAsync once the tenant agent has started, so progression begins immediately. Covered by the marten regression per_tenant_agent_start_routes_high_water. * Add GroupAgentAssignmentsByDatabase surface for database-affine agent assignment Default interface member (=> false) so existing IEventStore implementers are unaffected. Signals that a node-distributed daemon should co-locate a shard database's per-(shard,tenant) agents on one node, so pools scale with databases not nodes x databases. Opt-in; see JasperFx/marten#4806. * Add MaxNodesPerDatabaseForAgents (bounded fan-out) to IEventStore Complements GroupAgentAssignmentsByDatabase: an int (default 1) capping how many nodes a single shard database's agents may fan out across. 1 = strict affinity; higher trades connections for parallelism on heavy databases. See JasperFx/marten#4806. * Test defaults for the new agent-distribution IEventStore members DistributesAgentsPerTenant / GroupAgentAssignmentsByDatabase default to false and MaxNodesPerDatabaseForAgents to 1 on a store that overrides none of them, following the AllDatabasesDefaultTests bare-store convention. See JasperFx/marten#4806 and JasperFx/wolverine#3280. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX * Drop MaxNodesPerDatabaseForAgents; database-affine assignment is strict Slims the upstream surface to the essential opt-in: GroupAgentAssignmentsByDatabase keeps a shard database's agents on one node (pools scale with databases, not nodes x databases). The bounded fan-out knob was speculative — per-database load is small under per-tenant partitioning and rebuild throughput is server-IO-bound, so a mix never proved necessary. See JasperFx/marten#4806. Claude-Session: https://claude.ai/code/session_01LXFw8u9F71jLhMTFNi9DyX * Drop GroupAgentAssignmentsByDatabase from IEventStore Database-affine agent grouping is not a per-store opt-in after all: Wolverine's managed distribution will always group a store's agents by database when the store reports a multi-database DatabaseCardinality (StaticMultiple/DynamicMultiple) — surface IEventStore already exposes. No new contract needed; DistributesAgentsPerTenant stays. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Harden per-tenant StartAgentAsync: prime-then-start + exact-identity precedence + loud failure Three fixes to the wolverine#3280 per-tenant start branch: 1. Prime the tenant's high-water ceiling BEFORE starting the agent (Activate -> poll -> start), mirroring StartAllAsync. Starting first ran DetermineStartingPositionAsync against ceiling 0, which rewound a SubscribeFromPresent subscription's progression row to 0 and replayed the tenant's entire history with side effects. The post-start poll is now redundant and removed. 2. Exact AllShards() identity matches win before tenant parsing, so a registered shard identity that happens to parse as tenant-bearing cannot be hijacked into the fan-out branch. 3. A tenant-bearing identity against a store without per-tenant high-water tracking now throws instead of silently starting a phantom agent seeded from the store-global mark (double-processing risk). A failed tenant start also resyncs the polled-tenant set so the coordinator stops persisting high-water rows for a tenant with no agent on this node. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Behavioral coverage for per-tenant StartAgentAsync Seven daemon-level tests against a real JasperFxAsyncDaemon with stubbed store/detector/loader, including the SubscribeFromPresent regression pin: the from-present floor must derive from the primed per-tenant ceiling, never from 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Anne Erdtsieck <anne.erdtsieck@topicus.nl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Foundational piece of the per-tenant agent distribution work for sharded + tenant-partitioned event stores. Addresses JasperFx/wolverine#3280 and carries the store-agnostic surface for JasperFx/marten#4806.
Problem. Under node-distributed daemons (Wolverine-managed event-subscription distribution), agents are distributed per (shard × database). With
MultiTenantedWithShardedDatabases+UseTenantPartitionedEvents, multiple tenants are co-located in one shard database, each drawing its own event sequence — a single store-global agent (and high-water mark) per database cannot track them, so a lagging tenant's appends fall below the shared mark and are skipped.What's here (default interface members — zero impact on stores that don't opt in):
IEventStore.DistributesAgentsPerTenant(defaultfalse): signals that a node-distributed daemon must fan out one agent per (shard, tenant). Store-global, single-database per-tenant partitioning, and database-per-tenant behavior is unchanged.JasperFxAsyncDaemon.StartAgentAsync(string)detects a tenant-bearing identity (ShardName.TryParse→ non-nullTenantId), resolves the base shard, activates the tenant in the high-water coordinator, and starts a per-tenant agent that advances against its own mark and persists its own<proj>:All:<tenant>progression row — polling the per-tenant high water right after start.IEventStore.GroupAgentAssignmentsByDatabase(defaultfalse): opt-in signal for database-affine assignment (marten#4806) — a distribution-aware host keeps all of a shard database's per-tenant agents together on one node, so connection pools scale with the number of shard databases instead of nodes × databases (which otherwise exhausts a shared server'smax_connections).AgentDistributionDefaultTests(same bare-store convention asAllDatabasesDefaultTests).Companion PRs (merge order): this first → Marten #4799 (surfaces both signals + per-tenant high-water and bulk-import work) → Wolverine #3281 (per-tenant fan-out + affine distribution).