Marten.ScaleTesting: composite-projection rebuild load test harness on a Telehealth-derived domain

[jeremydmiller]

Background

We need a long-running load test harness for Marten 9's async daemon projection rebuilds. Driving factors:

• Recent reports of concurrency violations on the shared DocumentSessionBase under daemon fan-out (parent: PR Fix #4657: bypass session VersionTracker from projection StoreProjection (async daemon race) #4658, follow-up plan in Closed-shape DocumentStorage: eliminate runtime if/then via type splits (ConcurrencyMode, then hierarchy) #4659 and the daemon-thread-safety synthesis tracked separately).
• Need to verify that fixes are durable under realistic event interleaving and tenant fan-out at the 20M+ event scale — not just at the unit-test scale of multi_stage_projections.cs (~thousands of events).
• Need a stable engine to drive projection-rebuild optimization work after the safety fixes ship.

The harness is interactive / dev-box only. No CI integration. No NuGet package. Pure tooling for us.

Goal

A new src/Marten.ScaleTesting/ project — JasperFx.CommandLine console runner — that:

1. Seeds 20M+ events across N tenants under conjoined multi-tenancy, with events realistically interleaved across streams (not stream-by-stream linear).
2. Runs a composite projection rebuild against that data through three stages: single-stream snapshots → multi-stream with enrichment → multi-stream with cross-stage enrichment.
3. Gate 1 (correctness): completes without concurrency violations / NullReferenceException / dictionary-modification errors.
4. Gate 2 (optimization engine): emits enough metrics to compare runs across master commits so we can measure the impact of optimization work.

Domain — lift Telehealth, don't share

Telehealth at src/DaemonTests/TeleHealth/ (12 files: Appointments.cs, Boards.cs, ProviderShift.cs, plus reference types Specialty / RoutingReason / Patient / Provider) is the seed domain. Copy the relevant types into src/Marten.ScaleTesting/Domain/ rather than introducing a ProjectReference back into DaemonTests — the test project should stay self-contained, and we want freedom to extend the domain without affecting test fixtures.

Existing Telehealth projections we reuse:

• AppointmentProjection (single-stream snapshot, CacheLimitPerTenant = 1000)
• Board and ProviderShift (single-stream snapshots)
• AppointmentMetricsProjection (custom IProjection)
• AppointmentDetailsProjection, BoardSummaryProjection, AppointmentByExternalIdentifierProjection (multi-stream with enrichment)

Composite topology (3 stages)

Stage	Projection	Type	Source
1	AppointmentProjection	Single-stream	Existing (lifted)
1	Board	Single-stream	Existing (lifted)
1	ProviderShift	Single-stream	Existing (lifted)
1	AppointmentMetricsProjection	Custom IProjection	Existing (lifted)
2	AppointmentDetailsProjection	Multi-stream + enrichment	Existing (lifted)
2	BoardSummaryProjection	Multi-stream	Existing (lifted)
3	NEW ProviderUtilizationProjection	Multi-stream + enrichment	Reads stage-2 AppointmentDetailsProjection + BoardSummaryProjection output
3	NEW TenantDailyRollupProjection	Multi-stream + enrichment	Reads stage-2 output; per-tenant daily aggregate; exercises the conjoined boundary

Wired via the existing API:

public class TelehealthCompositeProjection : CompositeProjection
{
    public TelehealthCompositeProjection()
    {
        Snapshot<Appointment>(stageNumber: 1);
        Snapshot<Board>(stageNumber: 1);
        Snapshot<ProviderShift>(stageNumber: 1);
        Add<AppointmentMetricsProjection>(stageNumber: 1);

        Add<AppointmentDetailsProjection>(stageNumber: 2);
        Add<BoardSummaryProjection>(stageNumber: 2);

        Add<ProviderUtilizationProjection>(stageNumber: 3);
        Add<TenantDailyRollupProjection>(stageNumber: 3);
    }
}

Focus on the single-pass rebuild path (JasperFx 2.5.0+ CompositeReplayExecutor) rather than per-projection rebuild. Single-projection rebuild is a follow-up if we need it.

Conjoined multi-tenancy

Lift the configuration from src/DaemonTests/Composites/multi_stage_projections.cs:246-254:

opts.Events.TenancyStyle = TenancyStyle.Conjoined;
opts.Policies.AllDocumentsAreMultiTenantedWithPartitioning(x =>
    x.ByHash(Enumerable.Range(1, bucketCount).Select(i => $"b_{i}").ToArray()));
opts.Advanced.DefaultTenantUsageEnabled = false;

Defaults: 50 tenants, 8 hash buckets, ~400K events/tenant for 20M total. CLI knobs override.

Event seeding architecture

The hard part. The repo has zero bulk event seeding infrastructure — BulkInsertAsync is document-only, and TripStream.RandomStreams (src/DaemonTests/TestingSupport/TripStream.cs:23-32) fills streams linearly. We have to build:

Producer model:

• Independent IEnumerable<EventBatch> generators per logical stream (Appointment N, Board M, ProviderShift K).
• A weighted-random k-way merge across all generators, with per-stream-type weights tuned for realism (Appointments dominate; Boards open then accumulate alerts slowly; ProviderShifts cluster around board open/close).
• Producer pushes batches to a Channel<EventBatch>; multiple writer tasks consume and call IDocumentSession.Events.Append per stream within a batch then SaveChangesAsync.
• Deterministic seeding via (seed, tenantSeed) so runs are repeatable across machines.
• Idempotent: check mt_streams for existing event count and skip seeding if already complete.

Tenancy seeding:

• Tenants pinned per stream for its lifetime (real-world is tenant-stable).
• Reference data (Patient, Provider, RoutingReason, Specialty) seeded via existing BulkInsertAsync(tenantId, ...) pattern from the existing Telehealth fixture.

Throughput estimate: based on EventAppenderPerfTester results — roughly 5K events/sec/writer × 8 writers ≈ 40K events/sec → 20M / 40K ≈ ~8 min on a dev box. PG-bottlenecked; the harness must print continuous throughput so we can tune.

CLI shape

Modelled on src/EventAppenderPerfTester/TestCommand.cs (JasperFx.CommandLine):

marten-scaletest seed       --tenants 50 --events 20000000 --buckets 8 --seed 42
marten-scaletest rebuild    --projection composite --report metrics.json
marten-scaletest validate   --baseline baseline.json
marten-scaletest stress     --tenants 50 --events 20000000

stress = seed + rebuild + validate chained. Each subcommand standalone for iterative dev use.

Metrics (Phase C, for the optimization engine)

Cheap counters + histograms first, no external observability dependencies:

• Events processed, batches processed, per-projection update counts, per-tenant event counts (counters)
• events/sec per shard (rolling 10-second window)
• Per-batch processing wall-clock (System.Diagnostics.Metrics Histogram)
• Memory (Process.WorkingSet64 polled every 5s → CSV)

One JSON output file per run with all of the above + run parameters. Diffable across runs — this is the engine for "did my optimization actually help."

Validation strategy

validate subcommand:

• Run the same composite projection through a single-shard, single-threaded rebuild to produce a "known good" baseline aggregate snapshot.
• Run again under default fan-out → compare aggregate-by-aggregate.
• Mismatch = correctness bug. Match = fan-out is safe for this run.

This is how we get a real signal beyond "it didn't throw."

Phased delivery

Phase A — Domain & seeder (~1 week)

• New src/Marten.ScaleTesting/ project + CLI scaffolding (model on EventAppenderPerfTester).
• Lift Telehealth events, aggregates, and existing projections into src/Marten.ScaleTesting/Domain/.
• Build the producer + interleaving merge + bulk writer.
• Implement seed subcommand. Target: 20M events in <15 min on a dev box.

Acceptance: seed runs to completion at 20M events; rerun is no-op (idempotent); mt_events row count matches the requested total exactly.

Phase B — Composite topology & rebuild (~3-4 days)

• Wire the 4+2+2 composite (stages 1+2+3) per the table above.
• Write the 2 new stage-3 projections.
• Implement rebuild subcommand. Focus on single-pass rebuild path.

Acceptance: rebuild completes without crash for at least 50 consecutive runs at 5M events; reaches the 20M target at least once on the dev box.

Phase C — Validation & metrics (~3-4 days)

• validate subcommand: dump aggregate state, compare to single-threaded baseline snapshot.
• JSON metrics sink + a small compare-runs helper.
• stress subcommand chains it all together.

Acceptance: validate correctly flags a deliberately-introduced bug (e.g., drop random events from one projection); two runs of stress on identical input produce metrics within 5% of each other.

Phase D — Use it (the real point)

• Establish a master-commit baseline.
• Drive the daemon-thread-safety fixes (the synthesis plan from Closed-shape DocumentStorage: eliminate runtime if/then via type splits (ConcurrencyMode, then hierarchy) #4659's follow-up) against the harness — each fix should hold the crash gate AND not regress rebuild time.

Non-goals

• Not a microbenchmark — MartenBenchmarks/ already covers per-method timings.
• Not a NuGet package — internal tool only.
• Not wired into CI — interactive use on dev box.
• Not sharded-PG or distributed — single PG instance is the realistic baseline.
• Not per-projection rebuild for now — single-pass composite rebuild is the priority. Per-projection rebuild can be a follow-up if it surfaces interesting bugs.
• Not a redesign of Telehealth — we lift, copy, and extend; don't modify the original at src/DaemonTests/TeleHealth/.

Risks

• Seed time becomes the bottleneck. Mitigation: parametrize event count so daily iteration uses 2M and full runs use 20M.
• Telehealth's natural event mix doesn't stress the right hot paths. Mitigation: tunable mix weights via CLI; option to add a synthetic high-fanout stream type if needed.
• Single-pass rebuild has its own bugs surfaced by scale. Acceptable — that's part of why we're building this.

Related

• PR Fix #4657: bypass session VersionTracker from projection StoreProjection (async daemon race) #4658 / Issue Async daemon: StoreProjection() races on session VersionTracker under parallel slice processing #4657 — StoreProjection race that motivated the audit
• Issue Closed-shape DocumentStorage: eliminate runtime if/then via type splits (ConcurrencyMode, then hierarchy) #4659 — closed-shape DocumentStorage type splits (the surface this harness will exercise after the splits land)
• Forthcoming daemon-thread-safety synthesis issue — this harness is the regression check for it

[jeremydmiller]

Success!