WS1 per-tenant distribution e2e completion: Marten 9.13.0-alpha.3, churn/failover e2e, #4862 regression pin (jasperfx#486)#3334
Merged
Conversation
Marten 9.13.0-alpha.3 carries the per-tenant distribution work this PR's e2e tests exercise: marten#4864 (broadened DistributesAgentsPerTenant gate + single-DB descriptor TenantIds), marten#4865 (ExternallyManaged honored), marten#4869 (per-tenant high-water advancement — unblocks the failover floor-continuity test), and marten#4870 (native per-tenant distribution). JasperFx/JasperFx.Events pins move 2.20.0 -> 2.21.0 to match what Marten alpha.3 already pulls transitively — keeps the explicit pins from drifting below the resolved graph. No envelope-operation contract drift from marten#4872 (Weasel 9.12 metadata binders): Wolverine.Marten's StoreIncomingEnvelope/StoreOutgoingEnvelope compile unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…WS1) Sharded + UseTenantPartitionedEvents, single node: with tenant-a's per-tenant agent already projecting, add tenant-b to the same shard database at RUNTIME via Advanced.AddTenantToShardAsync. Within the assignment-evaluation window (CheckAssignmentPeriod = 1s) the leader re-enumerates the store usage, fans out the ":all/tenant-b" agent, tenant-b's fresh seq-1.. events project, its HighWaterMark-independent PartitionedCounter:All:tenant-b progression row lands in the shard database, and no store-global (tenant-less) agent ever runs alongside the per-tenant agents (the #3328 stale-agent-retirement edge). Tenant REMOVAL is deliberately not covered end-to-end: Marten 9.13.0-alpha.2 exposes RemoveTenantAsync/DisableTenantAsync (ShardedTenancy.cs:378/:470), but a running store's usage never shrinks - MartenDatabase.TenantIds is add-only (Fill at ShardedTenancy.cs:211/:364) and DescribeDatabasesAsync copies that in-memory list, so Wolverine keeps enumerating the removed tenant's agent until a process restart. The Wolverine half of the removal transition stays pinned by the running_per_tenant_agent_is_stopped_when_its_tenant_is_removed unit test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#486 WS1) Two nodes, sharded + UseTenantPartitionedEvents, two tenants co-located on one shard database: append N events per tenant, record each tenant's progression floor, kill the node owning the affine agent group, assert the survivor picks both agents up and resumes tenant-b exactly from its floor (doc == N+M, no skips, no rewind) after M more post-failover events. SKIPPED, pinning a Marten 9.13.0-alpha.2 product bug found by this test: the failover mechanics work (survivor restarts both agents, floors never rewind, tenant-a intact), but per-tenant high-water detection FREEZES at the first persisted HighWaterMark:<tenant> row - HighWaterDetector.loadPerTenantStatistics (src/Marten/Events/Daemon/HighWater/HighWaterDetector.cs:260): var currentMark = lastSeqId > 0 ? lastSeqId : lastValue; Once the row exists, CurrentMark is pinned to it and max(seq_id) is never consulted again, so ANY second batch of events per tenant never projects (not failover-specific; sibling tests all append one batch per tenant and never see it). Empirically confirmed: tenant-b at max seq_id 11 committed, store-global HighWaterMark advanced to 11 (global detection + per-tenant poll trigger both firing), HighWaterMark:tenant-b frozen at 6. Unskip on the Marten-side fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
….13.0-alpha.3 The skip pinned marten#4867: per-tenant high-water detection froze at the first persisted HighWaterMark:<tenant> row, so the M post-failover tenant-b events never projected. Marten 9.13.0-alpha.3 advances per-tenant CurrentMark past the persisted row (marten#4869), which makes the final leg pass. The class comment keeps a condensed history of the pinned bug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ant distribution (#486 WS1) Promotes the WS1 single-database fact-finding probe (found the bug 2026-07-06) into the marten#4862 regression test. Against Marten 9.13.0-alpha.2 a single-database store reported DistributesAgentsPerTenant = false, so managed distribution ran ONE store-global agent whose progression floor (driven to the leading tenant's max per-tenant seq) permanently skipped a lagging tenant's later events at per-tenant seq 1..N — the wolverine#3280 failure mode on one database. Marten 9.13.0-alpha.3 (marten#4864) broadens the gate to any tenant-partitioned store, so the startup expectation flips to per-tenant fan-out (2 tenants -> 2 agents) and the lagging-tenant projection assertions now pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…marten#4862/#4864) With Marten 9.13.0-alpha.3, DistributesAgentsPerTenant is true for ANY tenant-partitioned store, so one async projection over a single tenant-partitioned database now fans out one subscription agent per managed tenant instead of a single store-global "/all" agent whose lone progression floor could silently skip a lagging tenant. - tenant_partitioned_distribution_granularity: 1 agent -> 3 (one per registered tenant, default tenant included), agent URIs carry the tenant in the shard path - tenant_partitioned_distribution_multinode: 2 store-global agents -> 6 per-tenant agents (2 projections x 3 tenants), spread 3+3 across the cluster and all reassigned to the survivor on node loss - AssignmentWaiter.WritePersistedActualsAsync no longer throws KeyNotFoundException when the database holds a node record the waiter isn't tracking (ghost rows from a hard-stopped prior run); it reports the untracked node instead so the TimeoutException diagnostic survives Part of #486 WS1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y, not tenancy type name Marten 9.13.0-alpha.3 (marten#4864) gives a single-database store with UseTenantPartitionedEvents a TenantPartitionedSingleDatabaseTenancy — a DefaultTenancy subclass, chosen precisely so 'is DefaultTenancy' dispatch keeps working. The ancillary IntegrateWithWolverine seam compared Tenancy.GetType().Name == "DefaultTenancy" instead, so such a store was misrouted to the multi-tenanted message-database path and host startup failed with "no configured connectivity for the required master PostgreSQL message database" after 20 broker retries (the CIMarten failure on this PR). Both the Marten and Polecat ancillary seams now dispatch on Tenancy.Cardinality == DatabaseCardinality.Single, mirroring the main-store integration (WolverineOptionsMartenExtensions), which is why only ancillary stores were affected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 9, 2026
Merged
This was referenced Jul 14, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final Wolverine piece of #486 WS1 (per-tenant event-partitioning distribution). Builds on #3328/#3330 and the Marten 9.13.0-alpha.3 / JasperFx 2.21.0 releases.
What's here
9.13.0-alpha.3+ JasperFx2.21.0— picks up marten#4864 (per-tenant fan-out gate broadened to any tenant-partitioned store), marten#4865 (ExternallyManagedhonored), and marten#4869 (per-tenant HWM freeze fix).runtime_tenant_churn_under_managed_distribution): adding a tenant at runtime fans out its subscription agent within the assignment window under Wolverine-managed distribution.failover_preserves_per_tenant_progression_floors): un-skipped now that marten#4869 fixes the per-tenant HWM freeze — a surviving node resumes each tenant from its own progression floor.single_db_tenant_partitioned_distribution): a lagging tenant's low per-tenant seq_ids are projected (not silently skipped by a store-global floor) on a single tenant-partitioned database.tenant_partitioned_distribution_granularity,..._multinode): these previously pinned the old per-(shard × database) behavior — one store-global/allagent. Under alpha.3 a single-DB tenant-partitioned store fans out one agent per tenant (single-database stores keep even blue/green spread; database-affine grouping stays a multi-database concern).AssignmentWaiter.WritePersistedActualsAsyncreports an untracked persisted node instead of throwingKeyNotFoundException(ghost node rows from a hard-stopped prior run were masking the real timeout diagnostic).Verification
Full
MartenTests.Distributionfolder green locally against Postgres (31 tests), plus the fullwolverine.slnxRelease build.Part of the #486 epic (WS1). Remaining WS1 residue — tenant-removal e2e — is blocked on marten#4868 and intentionally not in this PR.
🤖 Generated with Claude Code