Fix #5065: make the recording message outbox thread safe

[jeremydmiller]

Closes #5065. The issue's premise turns out to be wrong, and so was my first diagnosis on it — this is not a drain-timing flake, and not the gate bug I initially reported. It is a thread-safety defect in the test harness.

Why the "asserts before the daemon drained" reading is wrong

It fails 4 out of 4 runs on clean master in about a second. A drain race does not do that.

What the state actually shows

Instrumenting the blue leg — which has no opt-in and no gate at all, so a gate boundary cannot be involved:

fired=19 distinct=19 progress=20 trackerSeq=20 batches=1 perBatch=[19] docs=20
fired=18 distinct=18 progress=20 trackerSeq=20 batches=1 perBatch=[18] docs=20
fired=19 distinct=19 progress=20 trackerSeq=20 batches=1 perBatch=[19] docs=20

Self-contradictory for any "events were missed" theory:

• progress=20 / trackerSeq=20 — the projection committed all 20
• docs=20 — all 20 aggregates were built, so every slice ran and RaiseSideEffects fired for each
• batches=1 perBatch=[19], distinct == fired — the one batch holds 19 messages, no duplicates

Every slice ran, yet a message is missing from the list: a lost List<T>.Add. That also explains the wandering shortfall — 19/20, 18/20, 7/8, 4/8 — which is what made it read as timing.

The contention is measured, not inferred

Instrumenting PublishAsync with thread ids and an in-flight counter:

threads=10  maxConcurrent=5
threads=10  maxConcurrent=8
threads=10  maxConcurrent=4

Up to 8 concurrent publishers across 10 threads into a single batch, for one single-stream projection catching up over 20 streams. So the lock in this PR is not papering over a timing bug — the contended access is directly observed. ProjectionUpdateBatch.CurrentMessageBatch is properly double-check-locked, so batch creation was never the problem.

The fix

RecordingMessageOutbox / RecordingMessageBatch in Aggregations/side_effects_in_aggregations.cs appended to unsynchronized List<T> for both Batches and Messages. The sibling copy in Composites/composite_rebuild_suppresses_side_effects.cs already locks both — a known hazard that this copy never picked up.

Also documents the requirement on IMessageBatch. That is the part worth keeping beyond the test: it is the interface Marten users implement, nothing said PublishAsync is called concurrently, and in a real outbox the symptom is silently dropped messages rather than a visibly flaky test.

Verification

• 10 consecutive runs of the class, 2/2 green every time — against 4/4 red immediately before.
• Full DaemonTests on net9.0 is now 257/257. It was 256/257, with this as the only failure — so DaemonTests has no known red left.
• Builds clean on net9.0 and net10.0. mdsnippets gate clean: 22 drifted doc files, exactly the pre-existing master baseline, none of it from this change (neither file contributes snippets).

No upstream issue needed

I had reported this as a JasperFx hand-off; that was wrong and I have corrected the issue thread. Nothing here is in JasperFx — the harness is Marten's, IMessageBatch is Marten's, and CurrentMessageBatch is Marten's and already correct.

🤖 Generated with Claude Code