Composite synthetic events from EventProjection stage 1: intentional gap or missing pattern? #4432
Unanswered
ppiela-dev
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I am new to both event sourcing and the Marten framework. My team is investigating building a financial accounting application on Marten. Our initial investigation has gone well, very impressed with Marten, but we are running into a problem building a data pipeline using composite projections. We have constraints on performance and scalability that are moving us to async projections using sql tables. Our choices have resulted in the problem described below. We have investigated several options, but have not found a viable solution. I am hoping that we can get some help from the community to find a better solution or maybe ideas on how to reframe the design problem.
Context. Multi-tenant CQRS service, Marten 8.37. Two async projections in the same domain:
EventProjection. Each domain event produces N documents (one detail row per child of the parent entity, e.g. one per period).SingleStreamProjection<T, Id>is shape-incompatible (one-stream → N-docs).QueueSqlCommand.Both run as separate async shards today. We have a documented cross-shard race: downstream may process an event before upstream has materialized its rows → summary computed against stale upstream state → silent wrong values until the next event for that parent.
What we've validated empirically (test below):
CompositeProjectionForwithSingleStreamProjection<T, Id>as stage 1 + downstreamIProjectionas stage 2 ⇒ syntheticUpdated<T>/ProjectionDeleted<T, TId>events delivered to stage 2 ✓CompositeProjectionForwithEventProjectionas stage 1 + downstreamIProjectionas stage 2 ⇒ no synthetic events emitted; stage 2 sees only the raw domain eventsQuestion. Is the lack of synthetic-event emission from
EventProjection'sops.Store/ops.Delete/ops.DeleteWherecallsintentional? Is there a Marten-blessed pattern for "downstream projection consumes upstream projection's materialized output, where upstream is an
EventProjectionwith one-stream → N-doc fan-out"?Constraints we cannot relax:
Inline(per-event fan-out is too heavy for batch-style write operations at our scale target).What we've already considered and rejected: merging projections (monolith), re-deriving from event payloads in downstream (drift risk), Inline upstream (scale), polling
mt_event_progressionfrom inside downstream (hacky).All reactions