Compliance suite: flat-table event projections — needs a table-inspection capability on the fixture

[jeremydmiller]

Part of #5118 (compliance wave growth), epic #5110.

Unparks the one candidate explicitly deferred in wave 2. Flat-table event projections (an EventProjection writing into a plain relational table rather than a document) are tested in both repos and were held back because asserting the result means reading a table, and the compliance fixture has no table-inspection capability — the products' schema prefixes differ (mt_ vs pc_), as do their DDL and their SQL dialects.

What each store tests today

Marten	Polecat
Projections/Flattened/*	Projections/flat_table_projection_tests.cs
Projections/EventProjections/* (flat-table cases)	Projections/using_event_projection_for_flat_tables.cs

The design question this issue has to settle

The suite needs to assert "row exists in table T with columns X=…, Y=…" without speaking either dialect. Options:

A. A narrow query capability on the fixture. One member, something like:

public abstract Task<IReadOnlyList<IDictionary<string, object?>>> QueryTableAsync(
    string tableName, CancellationToken token);

The fixture resolves the schema/prefix and runs the dialect-appropriate select *. Smallest surface; the suite never sees SQL. Downside: it is the first fixture member that is a raw data-access escape hatch, and it invites reuse for things that should stay product-specific.

B. Assert through the projection's own read path. Only viable if both products expose a supported way to read a flat-table projection's rows, which today they do not.

C. Assert on schema objects rather than data — that the table and columns were created with the right shape, via each product's schema-diagnostics API. Weaker, but it covers the part that actually drifts (the DDL the projection generates) and needs no data access.

Recommendation is A, deliberately narrow (table name in, rows out, no predicates, no SQL from the suite), with the fixture owning prefix resolution — but this is the decision the issue exists to make, so record the rationale wherever it lands.

Scope once the capability lands

1. Flat-table projection creates its table with the declared columns and types
2. Insert / update / delete per event type
3. Column mapping from event body members, including a computed column
4. Async and inline lifecycles produce the same rows
5. Rebuild truncates and repopulates

Acceptance

• Capability decision recorded in the issue and in the fixture's XML docs
• Suite enrolled in both stores
• Originals retired in both repos

[jeremydmiller]

Decision: option A, with a correction to the premise

Recording this per the issue's own acceptance criteria. Option A (narrow table-query capability on the fixture), with two findings that change the shape of the work.

The stated blocker was the smaller half

Table inspection is one fixture member and genuinely portable — both repos already hand-roll the same query. What actually blocks a shared suite is the constructor: FlatTableProjection is a parallel implementation in each product, and the bases take different arguments (Marten a SchemaNameSource enum, Polecat a literal schema name), so no single base(...) call satisfies both. Declaring the primary key is per-product for the same reason.

This is not the first non-shared base the library has absorbed — ComplianceEventProjection and ComplianceStringPartyProjectionBase already alias per-product types. The difference is that those have compatible (parameterless) constructors.

Resolved with a partial class: the library owns the table name, projection name and every event mapping in ConfigureMappings(); each consumer supplies a ~5-line partial carrying the constructor and the PK column. partial is already established practice in this package.

One objection in the issue body is dead

"their DDL differs" no longer holds. Weasel.Core.ITable now carries a dialect-neutral column surface — AddColumn(name, Type) / AddPrimaryKeyColumn(name, Type) routed through a per-dialect GetDatabaseTypeFor (typeof(Guid) → uuid on PG, UNIQUEIDENTIFIER on SQL Server), TableBase.cs:221-231.

It is deliberately not used. These sources compile inside every consumer, so a Weasel reference in one suite becomes a Weasel reference for the whole package, and a JasperFx.Events store is not obliged to be built on Weasel. The PK declaration stays in the per-consumer shim instead. Noted here because it is the obvious path for a future lift (#5153/#5154 class) and the reason against it is non-obvious.

Capability decision, as required

QueryTableAsync(string tableName, CancellationToken) → all rows as case-insensitive column/value maps. Table name in, every row out — no predicates, no ordering, no SQL from the suite; the fixture owns schema resolution and dialect. Suites filter in memory on the identity they appended under, so it cannot drift into a general query escape hatch. Case-insensitive keys are load-bearing: PostgreSQL folds undelimited identifiers to lower case, SQL Server preserves declared casing.

Also adds a SupportsFlatTableProjections gate — the first gate that exists for a store still being built (see #5156). It gates behavior, not compilation: a store still needs the type and mapping API to exist before it can enroll.

Scope delivered

FlatTableProjectionCompliance, 8 tests, covering all five scope items. Marten 8/8 and full EventSourcingTests 1691/0/7.

Originals retired — less than this issue assumed

Only the 5 duplicated behavior tests in flat_table_projection_with_stream_id_identifier_end_to_end.cs. Deliberately kept, because compliance explicitly does not cover them:

• table_should_be_built / functions_are_built — PG DDL and the mt_upsert_* functions, i.e. storage layout
• flat_table_projection_with_stream_key_identifier_end_to_end.cs — string stream identity (compliance covers Guid only; a candidate for a later wave)
• flat_table_projection_with_event_member_identifier_end_to_end.cs — PK derived from an event member, not covered
• Bug_4255, Bug_4290_4291, StatementMapTests, FlatTableProjectionTests — regressions and unit tests of Marten internals

Two cross-store divergences found

Writing the suite surfaced real disagreement — see the follow-up comment.

[jeremydmiller]

Two cross-store divergences the suite found

Both surfaced while making the rebuild test actually discriminating, and the first is why the second was nearly missed.

1. Rebuild does not empty the flat table on Polecat — a real product gap

Marten empties the projected table before replaying; Polecat does not, so a row whose events the replay can no longer see survives the rebuild.

Root cause is a shared contract with one consumer. AsyncOptions.DeleteDataInTableOnTeardown(...) → AsyncOptions.CleanUps is JasperFx-owned (JasperFx.Events/Projections/AsyncOptions.cs:119), and grepping for it finds:

• Marten: registers the table (Marten/Events/Projections/Flattened/FlatTableProjection.cs:228) and consumes the list (Marten/Events/Daemon/AsyncOptionsExtensions.cs:12,40)
• Polecat: zero references, either side

Polecat's TeardownProjectionStateAsync (Polecat/DocumentStore.EventStore.cs:597) instead derives teardown targets from source.PublishedTypes(). A flat-table projection publishes no document types, so publishedTableNames is empty and nothing is wiped.

The fix looks small and rides the DELETE FROM {tableName} loop already in that method: fold Options.CleanUps' DeleteTableData entries into the tables list. Worth noting the gap is wider than flat tables — any custom teardown rule registered through the shared AsyncOptions surface is currently ignored on Polecat.

2. Clean erases flat-table rows on Polecat but not Marten

Polecat's DeleteAllEventDataAsync / DeleteAllDocumentsAsync also erase flat-table projection data — it has explicit tests pinning that (clean_all_event_data_erases_flat_table_projection_data). Marten's leave the rows.

Lower stakes (test-support API), but it is a genuine disagreement, and it had a sharp edge here: the first draft of the rebuild test orphaned a row by deleting event data, which made the assertion pass vacuously on Polecat — the row was gone before the rebuild ran. The suite now archives the stream instead, and the reason is recorded in a comment so nobody reintroduces it.

Arguably Marten is the weaker of the two: a test-support Clean that leaves stale projection rows behind is a footgun. Not filed as an issue pending a view on which behavior is intended.

Method note

The suite passed 8/8 on both stores before either of these was visible. Both only appeared after asking whether each assertion could fail — the rebuild test as first written could not, because Map overwrites rather than accumulates, so replay-onto-surviving-rows and empty-then-replay produce identical values. Worth remembering for suites that assert projection state: a green rebuild test proves nothing unless something in it is unrecreatable by the replay.