Marten.ScaleTesting: multi-node native-HotCold daemonload + leadership failover (#4883, epic jasperfx#486 WS6)#4894
Merged
Conversation
--databases N pools the tenants across N scaletest_dl_shard_* databases via MultiTenantedWithShardedDatabases (explicit round-robin placement, one daemon per shard). ShardConnectionSampler groups pg_stat_activity by datname so the --max-connections gate is enforced PER DATABASE, plus per-tenant catch-up verification on every shard and a database-affine placement check (per-tenant sequence in exactly the home shard). Local-scale result (25 tenants x 2 shards x 2 projections, 60s): shard peaks 16/11, all 25 tenants caught up, zero placement violations. Single-database path (--databases 1, the default) unchanged. Addresses #4882 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XNfeNz73Q3EdiTgSeDbo96
daemonload-multinode coordinator launches N daemonload-node child processes (multiple OS processes of this binary = multi-node without cluster infra), each an IHost running AddAsyncDaemon(HotCold) over one shared tenant-partitioned store on one DaemonLockId. Coordinator appends under load, samples pg_stat_activity per node x database (NodeConnectionSampler), optionally kills the leader (--kill-leader-after-seconds) and verifies a survivor takes over, then confirms per-tenant catch-up to each tenant's own ceiling. --max-connections-per-node gates each node's footprint. MultiNodeStore centralizes the shared config so coordinator and nodes can't drift. NodeProcess wraps launch/NODE_READY handshake/graceful stdin-close stop/kill. Reuses DaemonLoadCommand catch-up probe. Local-scale result (2 nodes, 8 tenants x 2 projections, kill leader at 10s of 30s): leader node0 killed -> node1 took over, 8/8 tenants caught up, 0 append failures, per-node peak 9-10 connections. Native HotCold single-DB = one hot leader (per-database lock). Remaining WS6 multi-node work: cross-node simultaneous distribution over sharded multi-DB topology + Wolverine-managed mode (wolverine#3328; harness has no Wolverine dependency). Documented in README. Addresses #4883 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XNfeNz73Q3EdiTgSeDbo96
jeremydmiller
force-pushed
the
feat/4883-multinode-daemonload
branch
from
July 7, 2026 19:36
b0967b9 to
1b7b0fd
Compare
jeremydmiller
force-pushed
the
feat/4882-sharded-daemonload
branch
from
July 7, 2026 19:36
f8d8f7a to
b76029b
Compare
…-daemonload # Conflicts: # src/Marten.ScaleTesting/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #4883 (epic #486 WS6). Stacked on #4890 (
feat/4882-sharded-daemonload) → #4887 (feat/4684-batch-instrumentation) → master. Review/merge in that order; this PR's own diff is the top commit.What's here — native HotCold multi-node
Multi-node without cluster infra = multiple OS processes of this same binary:
daemonload-multinode(coordinator) provisions one tenant-partitioned store, launches--nodeschild processes, appends continuously, samplespg_stat_activitygrouped by per-node Application Name × database, optionally kills the leader, verifies per-tenant catch-updaemonload-node(internal worker) — anIHostrunningAddAsyncDaemon(DaemonMode.HotCold)over the shared store on oneDaemonLockId; idles until the coordinator closes its stdin (graceful) or kills it (failover). Signals readiness with aNODE_READYstdout handshakeMultiNodeStorecentralizes the shared store config so coordinator and nodes contend on a byte-identical projection set + lock;NodeProcesswraps launch/handshake/stop/kill;NodeConnectionSamplerdoes the per-node×database attributionWS6 assertions the run makes
Under native HotCold + one tenant-partitioned database,
MultiTenantedProjectionDistributorlocks the database's whole per-tenant agent set (#491 fan-out) to ONE leader at a time. So: single-leader exclusivity, failover to a survivor when the leader is killed, per-tenant catch-up to every tenant's own ceiling afterward, governed per-node footprint (--max-connections-per-node).Measured (local reduced scale)
daemonload-multinode --nodes 2 --tenants 8 --projections 2 --duration-seconds 30 --kill-leader-after-seconds 10 --wipescaletest-multinode-node0scaletest-multinode-node1The killed leader's per-tenant agents moved to the survivor and every tenant still reached its own ceiling — no stall.
Full-scale rerun for the epic record:
--nodes 3 --tenants 100 --projections 2 --duration-seconds 120 --kill-leader-after-seconds 30 --max-connections-per-node 16.Remaining (listed in issue #4883, documented in README)
--databasesand is the natural follow-up.DaemonMode.ExternallyManaged) — the harness references onlyMarten.csproj, not Wolverine, so this mode needs either a Wolverine package addition here or a sibling harness.🤖 Generated with Claude Code
https://claude.ai/code/session_01XNfeNz73Q3EdiTgSeDbo96