Cap concurrent projection rebuilds per database (MaxConcurrentRebuildsPerDatabase)

[jeremydmiller]

Background

Projection rebuild today fans out per (tenant, projection) cell with no concurrency cap. On a 100-tenant store with UseTenantPartitionedEvents, calling rebuild can spawn 100 concurrent rebuild agents, each holding a session, pre-fetching snapshots, streaming events through its own Block(10) slice fan-out (AggregationRunner.cs:59), and writing batches back. Multiplied through: ~1000 concurrent operations against one PG instance. Connection pool blown, buffer cache thrashing, lock contention on mt_event_progression, write amplification across the partition set.

We need a configurable cap on concurrent rebuilds, scoped per database.

Scope

This applies only to rebuild — not continuous catch-up. Continuous is steady-state per-tenant; rebuild is the batchy, expensive operation that warrants throttling.

Design

New config knob

On StoreOptions.Events (Marten side; mirrored / surfaced however JasperFx-side options work):

/// <summary>
/// Maximum number of concurrent projection rebuilds per database during a rebuild operation.
/// null = derived default (see remarks). Applies to rebuild only; continuous catch-up is unaffected.
/// </summary>
/// <remarks>
/// Derived default: max(1, NpgsqlConnectionPoolSize / 8). e.g. a 100-connection pool defaults
/// to 12 concurrent rebuilds; a 20-connection pool defaults to 2. The fraction is conservative
/// to leave room for application traffic during rebuild windows.
/// </remarks>
public int? MaxConcurrentRebuildsPerDatabase { get; set; }

Name is deliberately broader than MaxConcurrentTenantRebuilds — it caps the cross product of (tenants × projections × shards) at the per-database level, not just tenants. Same knob covers single-tenant multi-projection rebuilds, multi-tenant single-projection rebuilds, and the cross product.

Where the throttle lives

Inside the per-database rebuild orchestrator. The CLI (ProjectionController.ExecuteRebuilds, src/JasperFx.Events/CommandLine/ProjectionController.cs:74) already iterates databases sequentially via foreach (var database in selection.DatabaseIdentifiers). Within each database, _host.TryRebuildShardsAsync(...) fans out — this is where the SemaphoreSlim(N) (or Parallel.ForEachAsync with MaxDegreeOfParallelism = N) wraps the per-(tenant, projection, shard) rebuild dispatch.

Each rebuild cell keeps its existing daemon machinery (per-tenant high-water, per-tenant progression row, internal Block(10) slice fan-out). The cap just bounds how many cells run concurrently.

Per-database scoping

Under MultiTenantedWithShardedDatabases, rebuilds on database A and database B run independently — the cap is enforced within each database, not store-global. The CLI's sequential database loop preserves this naturally; the per-database fan-out is where the cap applies.

Default value rationale

Derived from NpgsqlConnectionPoolSize / 8 (with floor of 1) instead of a fixed default. Reasoning:

• A fixed N=4 is a footgun on a beefy DB (under-utilizes) and unsafe on a small one (over-utilizes).
• Pool size is the closest thing to a "DB capacity" signal we have at config time.
• 1/8 fraction leaves headroom for application traffic and the 10-wide intra-rebuild Block fan-out (cap=4 + 10 sub-workers = 40 concurrent ops max, still well below typical pool sizes).
• Users with rebuild-dominated workloads can override; users on shared/small DBs get safe defaults.

Required analysis: interaction with EnableExtendedProgressionTracking

EnableExtendedProgressionTracking (Marten EventGraph.cs:238) reshapes the progression table schema (EventProgressionTable.cs:46) and the read statements (ProjectionProgressStatement.cs:40,48, ShardStateSelector.cs:47), and combines with UseOptimizedProjectionRebuilds. The rebuild throttle work needs to:

• Trace exactly what extended progression tracking changes in the rebuild path (extra columns? per-tenant progression segregation? extra writes per batch?).
• Confirm whether extended tracking changes the per-rebuild DB cost profile in a way that should influence the default cap derivation.
• Confirm the throttle composes correctly when extended tracking is on (no double-counting of progression-row writes, no race on the new columns).
• If extended tracking changes the resource shape materially, surface that in the docs ("with extended tracking on, lower the cap to N/2" or similar).

This is an investigation deliverable, not a guess — the implementer should read the four files above end-to-end before settling the default formula.

CLI command update

ProjectionController.ExecuteRebuilds (src/JasperFx.Events/CommandLine/ProjectionController.cs:74) is the entry point for dotnet run -- projections rebuild. It needs to:

1. Pass the cap through to _host.TryRebuildShardsAsync(...) (or let the host pull it from store options).
2. Optionally accept a --max-concurrent flag to override the configured default for one-off operational rebuilds.
3. Log the effective cap at rebuild start so operators see what's enforced.
4. Display per-database rebuild progress with the cap visible — useful when an operator wants to confirm the throttle is actually limiting throughput.

Documentation tasks

Add a new section to marten/docs/events/projections/rebuilding.md (or the equivalent JasperFx docs location):

• What the throttle does — cap on concurrent rebuild cells per database during a rebuild operation. Not continuous catch-up.
• How to configure — StoreOptions.Events.MaxConcurrentRebuildsPerDatabase = N.
• Default derivation — pool size / 8, with examples for common pool sizes.
• When to override — rebuild-dominated workloads (increase), shared / small DBs (decrease).
• Interaction with EnableExtendedProgressionTracking — pin whatever the analysis above concludes.
• Interaction with the intra-rebuild Block(10) slice fan-out — explain the two-layer concurrency model (outer cap × inner slice workers).
• CLI override — --max-concurrent flag on projections rebuild.

Cross-reference from the multi-tenancy and UseTenantPartitionedEvents docs.

Out of scope (separate follow-up)

A CritterWatch-side issue will track the long-running rebuild orchestration story: CritterWatch discovers all known stores/databases, accepts rebuild commands, queues them with awareness of which database has capacity, persists state so rebuilds survive node restart / failure, and can restart partial rebuilds from progression-row state. That issue depends on this one (CritterWatch needs the per-database cap to be honored) but is its own deliverable. Link will be added when filed.

Acceptance

• StoreOptions.Events.MaxConcurrentRebuildsPerDatabase exists, defaults to null (derived), accepts integer overrides.
• TryRebuildShardsAsync (or its successor) honors the cap.
• projections rebuild CLI honors the configured cap and accepts --max-concurrent override.
• Regression test: rebuild 8 projections across 32 tenants on a single DB with cap=4 — at no point exceed 4 concurrent rebuild cells. Verify via instrumentation (e.g., Interlocked.Increment on a counter inside the rebuild lambda).
• Docs updated per the documentation tasks above.
• EnableExtendedProgressionTracking interaction analyzed and documented.

Non-goals

• Not adaptive throttling. Static cap is predictable; adaptive is hard to get right (back-off signal selection is fraught, can amplify cascade failures). Defer until we have evidence the static cap is insufficient.
• Not an events/sec ceiling. Different shape of throttle, different failure modes, defer.
• Not continuous catch-up. Different resource profile, different ergonomics; if we need it later, it's a separate config knob.
• Not the cross-process/cross-node orchestration story. That's CritterWatch's job (see Out of scope).

Related

• marten#4666 — the load test harness that should be extended to validate this throttle works under the 20M-event composite-projection rebuild scenario.
• marten#4679 / jasperfx/jasperfx#419 — concurrent ShardName / progression-row identity work; this throttle should land after that since it changes the per-tenant rebuild path that's being audited.

[jeremydmiller]

Companion CritterWatch issue for the long-running cross-node rebuild orchestration story: https://github.com/JasperFx/CritterWatch/issues/309 — depends on this throttle landing first.