Compliance wave 7: projection rebuild/catch-up and the dead letter path (2.44.0)

[jeremydmiller]

The last two event-sourcing suites in the backlog — marten#5150 and marten#5149. Library 24 → 26 suites, 199 → 216 tests.

Both needed zero seam additions — the backlog had them misfiled

They were grouped under "needs a seam addition." Checking the shared surfaces rather than the issue text says otherwise:

• IProjectionDaemon already declares the entire rebuild surface — seven RebuildProjectionAsync overloads, both CatchUpAsync forms, PrepareForRebuildsAsync — and the fixture's StartDaemonAsync() already hands one back.
• The error policy and dead letters are both already shared — IEventStore<TOperations, TQuerySession>.ContinuousErrors for the former, IEventStore.AllDatabases() → IEventDatabase.QueryDeadLetterEventsAsync for the latter. I verified both products override both rather than inheriting the empty-array DIMs before writing a line.

One wrinkle worth knowing generally: ContinuousErrors is on the generic IEventStore<,> while the fixture exposes the non-generic IEventStore. The suite casts, which is safe precisely because it is generic over the same pair the store closes over — that reaches the whole generic store surface without adding seam.

RebuildAndCatchUpCompliance — 11 tests

AsyncDaemonCompliance already smoke-tests catch-up and a basic rebuild, so this covers what's easy to get subtly wrong: teardown-before-replay, idempotency, agreement between the by-view-type and by-name overloads, isolation from a second projection, catch-up without a full daemon run, and PrepareForRebuildsAsync.

The teardown test is the one that matters. A rebuild that replays onto surviving rows looks correct for every stream whose events still exist, and is wrong only for rows the replay can no longer produce — so it passes any test that checks a live aggregate. That exact divergence was a real product gap found by the flat-table suite (polecat#415). So this plants a document the replay cannot possibly produce and requires the rebuild to remove it.

The by-name test plants the orphan too, deliberately: a named rebuild that silently matched no projection would otherwise pass a state assertion that was already true.

DeadLetterCompliance — 6 tests

A poison event whose Apply always throws, which needs no store-specific hook. Pins that a skip policy actually skips — the shard survives and keeps projecting events after the fault, not just before it — and that the skipped event is recorded rather than lost, with enough on the row to identify the event and the failure.

Corrected mid-flight: ExceptionMessage carries the wrapping ApplyEventException's own message, not the inner reason, while ExceptionType names the inner exception. Asserting the inner type is what the shared ctor actually promises; searching the message for the thrown text does not work, and was my first draft.

Deliberately not asserted: the stop-on-error policy. How a shard pauses, stops or faults is expressed differently enough between the products that encoding one shape would pin an implementation rather than a promise.

⚠️ Disclosure — an intermittent failure I could not reproduce

a_rebuild_reproduces_the_projected_state failed twice on Polecat early in development, both times at ~31s, and has passed every run since — 7 consecutive full 216-test compliance runs, including one immediately after a container restart.

It is not the rebuild shard timeout (that defaults to 5 minutes — checked). I have no error text and no reproduction, so I am not guess-fixing it or quietly hardening the test to make it go away. Recording it so it doesn't look clean: if this resurfaces in either CI, this is the first sighting.

Verification

Marten 17/17 on the two new suites, full EventSourcingTests running. Polecat 17/17, full compliance 216/216, full suite running.

Closes JasperFx/marten#5149
Closes JasperFx/marten#5150

🤖 Generated with Claude Code

https://claude.ai/code/session_01VpDCvJcBDZerieJB4JEHde

[jeremydmiller]

Refinement on the disclosed intermittent failure

Both sightings were Polecat only. Since writing the PR body:

• Marten: 5 consecutive full compliance runs, 216/216 each, ~14s per run. Zero sightings.
• Polecat: 7 consecutive full compliance runs, 216/216 each, ~100s+ per run. Zero sightings since the two early ones.

The asymmetry is itself informative. The suite logic is identical on both stores, so a fault confined to one of them points at either a Polecat-side race under concurrency or SQL Server timing, rather than anything wrong with the test — which is the main reason I did not want to "fix" it blind by hardening the assertion.

Still no error text and still no reproduction, so this remains a first sighting to watch rather than a diagnosis.