EventProjectionScenario's up-front wipe is a no-op for composite projections — read models survive while their events are deleted

[erdtsieck]

Found while trying the harness from the 03-08 blog post against a real CompositeProjectionFor(...). Adjacent to #5127 but not one of its 11 items, and not #5126.

Summary

EventProjectionScenario's up-front wipe is a no-op for composite projections. Event data is deleted, the composite's read models are not — so every scenario after the first starts against the previous scenario's documents, now orphaned from any events.

Why

ProjectionScenario.Execute (9.22.2) / DeleteExistingDataAsync (master) derives the wipe list from StorageTypes:

await _store.Advanced.Clean.DeleteAllEventDataAsync(ct);
foreach (var storageType in _store.Options.Projections.All.SelectMany(x => x.Options.StorageTypes))
{
    await _store.Advanced.Clean.DeleteDocumentsByTypeAsync(storageType, ct);
}

A CompositeProjection never populates its own Options.StorageTypes (nor Options.CleanUps) from its member projections. For a store whose entire read side is one composite over ~10 members, the full registry is:

invoices (Marten.Events.Projections.CompositeProjection)
    StorageTypes : []
    CleanUps     : []

So the loop iterates nothing.

Two things compound it:

1. StorageTypes is documented as a schema-building hint — "used to help build out schema objects if the async daemon is started before the rest of the application" — not as a teardown list. Teardown-on-rebuild goes through AsyncOptions.CleanUps, which is why composite rebuilds are fine: the composite delegates teardown to its members internally, and neither list is surfaced on the composite itself. Anything reasoning about a composite from the outside sees a projection that writes nothing.
2. Because the events are deleted, the leftover documents cannot be reconstructed or noticed by a subsequent rebuild.

Reproduction

Store: one CompositeProjectionFor("invoices", …), stage-1 snapshot + stage-2 read models, StreamIdentity.AsString, conjoined tenancy. Two scenarios in sequence:

var invoiceId = Guid.NewGuid().ToString();

await store.Advanced.EventProjectionScenario(scenario =>
{
    scenario.StartStream<Invoice>(invoiceId, Creation(invoiceId, 1_000_000m));
    scenario.DocumentShouldExist<InvoiceOverviewItem>(invoiceId);
}, ct);

// Second scenario, touching nothing related. If the wipe worked, the first
// scenario's read model would be gone by the time this one asserts.
await store.Advanced.EventProjectionScenario(scenario =>
{
    var unrelatedId = Guid.NewGuid().ToString();
    scenario.StartStream<Invoice>(unrelatedId, Creation(unrelatedId, 1m));

    scenario.DocumentShouldExist<InvoiceOverviewItem>(unrelatedId);
    scenario.DocumentShouldExist<InvoiceOverviewItem>(invoiceId, leftover =>
        leftover.ClaimedAmount.ShouldBe(1_000_000m));   // PASSES — the leak
}, ct);

// ...and its events really are gone, so events and read models are out of sync.
await using var session = store.QuerySession();
(await session.Events.FetchStreamAsync(invoiceId, token: ct)).ShouldBeEmpty();   // PASSES

Plus the direct assertion:

var storageTypes = store.Options.Projections.All.SelectMany(x => x.Options.StorageTypes).ToList();

storageTypes.ShouldNotContain(typeof(Invoice));               // PASSES
storageTypes.ShouldNotContain(typeof(InvoiceOverviewItem));   // PASSES
storageTypes.ShouldNotContain(typeof(InsurerInvoice));        // PASSES

Control group — the same harness against a store of 17 ordinary (non-composite) projections. Every one reports both lists, e.g.

EigenPrestatieMetTarieven (…EigenPrestatieMetTarievenProjection)
    StorageTypes : [EigenPrestatieMetTarieven]
    CleanUps     : [DeleteDocuments { DocumentType = …EigenPrestatieMetTarieven }]

and there a second scenario genuinely starts from an empty slate. So this is specific to CompositeProjectionFor(...), not a general failure of the wipe.

Observed on Marten 9.22.2 (JasperFx.Events 2.37.2). The StorageTypes-derived wipe and the empty composite StorageTypes are both unchanged on master, so I'd expect it to reproduce there too.

Impact

Silent, and it undercuts exactly the guarantee the docs lead with ("deletes all event data plus storage for every registered projection before it runs"). Tests stay accidentally correct as long as they use unique ids per scenario, but any unfiltered Query<T>() inside AssertAgainstProjectedData sees prior scenarios' rows — which is how I hit it: an ordered Query<InvoiceOverviewItem>() asserting [100m, 200m] returned [90m, 100m, 100m, 200m, 250m].

Suggested fix

Two independent parts, either of which fixes the symptom:

1. Have the wipe use the teardown rules rather than StorageTypes. Options.CleanUps is the list that already means "what to delete when this projection is torn down", and running it per projection is closer to the guarantee the docs state. It also stops the harness depending on a property whose documented purpose is schema building.
2. Have CompositeProjection surface the union of its members' StorageTypes / CleanUps. Worth doing regardless of (1): a composite currently claims to write no documents, which is wrong for any external consumer, including the schema pre-build that StorageTypes exists for.

Happy to open a PR for either if you have a preference on which.

[jeremydmiller]

Confirmed on master, and your diagnosis of the mechanism is exactly right. CompositeProjectionFor registers only the composite in Projections.All (ProjectionOptions.cs:326); members land in ProjectionStage._projections and nowhere else, and every member-registration path terminates in a bare list append (ProjectionStage.Add, ProjectionStage.cs:31) that reads nothing from the member's Options and writes nothing to the composite's. So the wipe loop genuinely iterates nothing.

Three things to save you writing the wrong PR, since you offered.

Fix (1) doesn't fix it

You wrote that either fix addresses the symptom. It doesn't. StorageTypes has exactly one writer in the whole codebase — AsyncOptions.DeleteViewTypeOnTeardown (AsyncOptions.cs:98-112) — and it adds to both lists:

public void DeleteViewTypeOnTeardown(Type type)
{
    _cleanups.Add(new DeleteDocuments(type));
    StorageTypes.Add(type);
}

So StorageTypes isn't an independent schema list that happens to be reused for teardown; it's a by-product of teardown registration. Switching the wipe to CleanUps gains only FlatTableProjection's DeleteDataInTableOnTeardown entries — and for a composite both lists are empty either way, exactly as your diagnostic output shows. Fix (2) is the only one that works.

Fix (1) is still a reasonable cleanup on its own terms, and your point that the harness shouldn't be leaning on a property documented as a schema hint stands. It's just not this bug's fix.

Fix (2) is right, but not for the reason given

You argue a composite "claims to write no documents, which is wrong for any external consumer, including the schema pre-build that StorageTypes exists for." Schema pre-build is actually fine: it goes through AllPublishedTypes() → PublishedTypes(), and CompositeProjection overrides PublishedTypes() to fan into its members, deliberately dropping the base's Concat(Options.StorageTypes):

// JasperFx.Events/Projections/Composite/CompositeProjection.cs:74-77
public override IEnumerable<Type> PublishedTypes()
{
    return Stages.SelectMany(stage => stage.Projections.SelectMany(x => x.PublishedTypes()));
}

So composites already report their types correctly to everything except this wipe. Which also means populating composite.Options.StorageTypes will surface only in the wipe loop — that's sufficient, and it's the right change, but it isn't the broader repair it looks like. (Changing that override to also concat StorageTypes would just double-count member types; harmless, since the downstream consumers are Distinct()/Contains-guarded, but pointless.)

The place to do it is CompositeProjectionFor, after configure?.Invoke(composite), where the member list is final. There's a vestigial ProjectionOptions parent constructor parameter at CompositeProjection.cs:45 that's accepted and discarded — plausibly where this was once intended.

One caveat to handle in the same PR: a composite's own Options.CleanUps and TeardownDataOnRebuild are currently ignored by rebuild entirely (the composite branch never calls teardownProjectionStorage for the composite itself). Populating those lists would leave the harness honoring them while rebuild doesn't. Covered in #5175.

"Composite rebuilds are fine" holds for only two of the four member kinds

Rebuild teardown loops composite.AllProjections() and reads each member's own CleanUps. That works for Snapshot<T> and Add(IProjectionSource, …). It does not work for AddProjectionWithServices<T> or a raw Add(IProjection, …): both wrap the member in a source with a fresh, empty AsyncOptions and never adopt the inner projection's options the way ProjectionWrapper/ScopedProjectionWrapper do via replaceOptions. Those members survive a rebuild and get double-written into. Filed as #5175, and it's more urgent than this issue since it corrupts real rebuilds rather than tests.

Your control group had a blind spot

The 17 projections that behaved correctly were presumably aggregations. JasperFxAggregationProjectionBase's constructor registers teardown (:58); JasperFxEventProjectionBase's does not — it only calls RegisterPublishedType. So a plain EventProjection reports both lists empty too, and the wipe is a no-op for those as well, composite or not. Already known and worked around rather than fixed (there's a Marten test carrying an explicit Options.DeleteViewTypeOnTeardown<LegLog>() with a comment spelling out the double-write consequence). Filed upstream as #626.

On the PR

Fix (2) in CompositeProjectionFor is a good, self-contained PR and very welcome — it fixes Marten and Polecat at once, since both harnesses read Options.StorageTypes identically. If you'd rather not take on #5175's caveat, keep them separate and I'll sequence them.

The longer-term shape I'd like is a general "sources this projection owns" member on IProjectionSource — a composite yields its members, everything else yields itself — so the rebuild teardown and this wipe share one traversal and the is CompositeProjection type test in DocumentStore.EventStore.cs goes away. That's a JasperFx change and shouldn't block your fix.

Thanks for the writeup, and for testing the harness against a real composite — that's how this surfaced at all.