Marten.ScaleTesting: rebuild-load governor sweep + load-test-informed defaults (#4884, epic jasperfx#486 WS6/WS3)#4896
Merged
Conversation
--databases N pools the tenants across N scaletest_dl_shard_* databases via MultiTenantedWithShardedDatabases (explicit round-robin placement, one daemon per shard). ShardConnectionSampler groups pg_stat_activity by datname so the --max-connections gate is enforced PER DATABASE, plus per-tenant catch-up verification on every shard and a database-affine placement check (per-tenant sequence in exactly the home shard). Local-scale result (25 tenants x 2 shards x 2 projections, 60s): shard peaks 16/11, all 25 tenants caught up, zero placement violations. Single-database path (--databases 1, the default) unchanged. Addresses #4882 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XNfeNz73Q3EdiTgSeDbo96
daemonload-multinode coordinator launches N daemonload-node child processes (multiple OS processes of this binary = multi-node without cluster infra), each an IHost running AddAsyncDaemon(HotCold) over one shared tenant-partitioned store on one DaemonLockId. Coordinator appends under load, samples pg_stat_activity per node x database (NodeConnectionSampler), optionally kills the leader (--kill-leader-after-seconds) and verifies a survivor takes over, then confirms per-tenant catch-up to each tenant's own ceiling. --max-connections-per-node gates each node's footprint. MultiNodeStore centralizes the shared config so coordinator and nodes can't drift. NodeProcess wraps launch/NODE_READY handshake/graceful stdin-close stop/kill. Reuses DaemonLoadCommand catch-up probe. Local-scale result (2 nodes, 8 tenants x 2 projections, kill leader at 10s of 30s): leader node0 killed -> node1 took over, 8/8 tenants caught up, 0 append failures, per-node peak 9-10 connections. Native HotCold single-DB = one hot leader (per-database lock). Remaining WS6 multi-node work: cross-node simultaneous distribution over sharded multi-DB topology + Wolverine-managed mode (wolverine#3328; harness has no Wolverine dependency). Documented in README. Addresses #4883 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XNfeNz73Q3EdiTgSeDbo96
…4884, epic #486 WS6/WS3) rebuildload subcommand load-tests N projections x many partitioned tenants rebuilding concurrently and sweeps the three governor knobs (MaxConcurrentRebuildsPerDatabase cap, MaxConcurrentEventLoads/ BatchWritesPerDatabase) + EnableExtendedProgressionTracking, measuring wall-clock, peak/idle connections and mt_event_progression contention per configuration via the existing ConnectionSampler + ProgressionLock Sampler. Outer cap enforced by SemaphoreSlim(cap) mirroring ProjectionHost.RebuildProjectionsWithCapAsync (#463). Prints a comparison table + advisory recommendation; never changes a shipped default (measurement only). Local-scale finding (20 tenants x 8 projections x 1200 events, pool 100 => default cap 12): peak conns ~= min(cap, projections)+1 (outer cap is the dominant connection driver, validating the two-layer model); zero progression waiters at every cap; wall-clock flattens past cap=4; extended progression tracking within noise. Recommendation: CONFIRM the max(1, MaxPoolSize/8) cap + governor default of 4. Derivation documented in rebuilding.md. --databases > 1 (sharded rebuild sweep) noted as a follow-up; the governor tuning evidence lives on a single database. Addresses #4884 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XNfeNz73Q3EdiTgSeDbo96
jeremydmiller
force-pushed
the
feat/4883-multinode-daemonload
branch
from
July 7, 2026 19:36
b0967b9 to
1b7b0fd
Compare
jeremydmiller
force-pushed
the
feat/4884-rebuild-load-tuning
branch
from
July 7, 2026 19:36
3ef4046 to
3b61a06
Compare
…oad-tuning # Conflicts: # src/Marten.ScaleTesting/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #4884 (epic #486 WS6/WS3 closeout). Stacked on #4894 → #4890 → #4887 → master; review/merge in that order. This PR's own diff is the top commit.
What's here — governor tuning by evidence
rebuildloadload-tests N projections × many partitioned tenants rebuilding concurrently and sweeps the three governor knobs so their shipped defaults can be confirmed — or a change recommended — without ever mutating a shipped default (measurement tool only, per the epic's WS3 constraint).For each configuration in the cross-product sweep it:
MaxConcurrentEventLoadsPerDatabase,MaxConcurrentBatchWritesPerDatabase) + optionalEnableExtendedProgressionTracking,SemaphoreSlim(cap)— the exact shape ofProjectionHost.RebuildProjectionsWithCapAsync(Address Dictionary query generation #463), whose shipped default ismax(1, MaxPoolSize / 8),pg_stat_activity+mt_event_progressionlock-waits throughout (reusing the Phase E: Per-batch measurement instrumentation in Marten.ScaleTesting harness #4684ConnectionSampler/ProgressionLockSampler),then prints a per-config comparison + an advisory recommendation, and documents the derivation in
docs/events/projections/rebuilding.md.Measured (local reduced scale — 20 tenants × 8 projections × 1200 events, MaxPoolSize=100 ⇒ default cap 12)
* = shipped default cap for this poolFindings (local scale, indicative):
MaxPoolSize/8fraction keeps headroom.mt_event_progressionwaiters at every cap — the concurrent-rebuild cap creates no progression-row contention at this scale.EnableExtendedProgressionTracking: no measurable rebuild overhead (within noise).Recommendation: CONFIRM the shipped
max(1, MaxPoolSize/8)rebuild cap and the load/write governor default of 4. No defaults changed. Documented inrebuilding.mdwith the explicit caveat to re-run at production pool + event volume before deviating.Remaining
--databases > 1) — currently clamps to single-DB with a note; the governor tuning evidence lives on one database, and the sharded repeat wires onto Marten.ScaleTesting: daemonload over pooled sharded databases — gate connections at O(databases) (epic jasperfx#486 WS6) #4882's shard provisioning as a follow-up.JasperFxAsyncDaemon) is tracked separately.🤖 Generated with Claude Code
https://claude.ai/code/session_01XNfeNz73Q3EdiTgSeDbo96