Fix #4788 — repopulate mt_natural_key on projection rebuild (supersedes #4789)#4793
Merged
Conversation
…ral_key lookup table
The natural-keys lookup table (mt_natural_key_{aggregate}) is maintained by
NaturalKeyProjection, an auto-registered inline projection whose ApplyAsync
iterates the StreamActions queued during the current SaveChanges and writes
ON CONFLICT upserts per matching event. That dispatch fires correctly on the
normal append path but the async-daemon rebuild path appends no streams — it
replays already-persisted events from mt_events through the projection's
aggregator and stores the rebuilt snapshots, so the inline projection chain
is never given anything to iterate. Net effect: after rebuild the parent
projection's documents are repopulated, but the mt_natural_key table stays
exactly as teardown left it (empty), and any FetchLatest<T, TNaturalKey>
afterwards misses.
Fix — two new hooks in Marten, no JasperFx changes required (the existing
abstractions — IAggregateProjection.NaturalKeyDefinition, EventRange.Events
on every rebuild page, and ProjectionBatch.SessionForTenant routing writes
into the batch's work-tracker — already give Marten everything it needs):
1. teardownProjectionStorage now also wipes the natural-key table when the
source projection carries a NaturalKeyDefinition, so the rebuild starts
from a clean slate (mirrors the doc-table TRUNCATE).
2. StartProjectionBatchAsync, in Rebuild mode, looks up the source by
range.ShardName.Name and — when it has a NaturalKeyDefinition — feeds
range.Events through a new NaturalKeyProjection.QueueUpsertsForEvents
method per tenant. The IDocumentOperations comes from
projectionBatch.SessionForTenant(...) which returns a
ProjectionDocumentSession whose ISessionWorkTracker IS the
ProjectionUpdateBatch — so the upserts flush inside the same transaction
as the rebuilt snapshots, not the bare session's unit-of-work (which the
batch's ExecuteAsync never touches).
NaturalKeyProjection refactor: extracted queueUpsertSql to take
(streamId, streamKey, tenantId, innerValue) instead of a StreamAction so the
inline path (StreamAction-driven) and the rebuild path (IEvent-driven) share
the SQL builder. ApplyAsync behavior is byte-for-byte unchanged.
Test: ytqsl's Bug_4788_natrual_key_table_not_populated_on_rebuild repro
(authored in commits 27ecadd / 12f16db / 6322a63 on this branch) now
passes. Broader regression sweep on net10: 34/34 existing natural-key tests
+ 8/8 EventSourcingTests rebuild tests + 31/31 DaemonTests rebuild tests
all green.
Closes #4788
Supersedes #4789
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Jun 30, 2026
Merged
jeremydmiller
added a commit
that referenced
this pull request
Jul 5, 2026
#4789) (#4793) (#4795) * test: repro rebuild projection with natural key * fix: query by natural key * updated file name * Fix #4788: rebuild a [NaturalKey] aggregate also rebuilds the mt_natural_key lookup table The natural-keys lookup table (mt_natural_key_{aggregate}) is maintained by NaturalKeyProjection, an auto-registered inline projection whose ApplyAsync iterates the StreamActions queued during the current SaveChanges and writes ON CONFLICT upserts per matching event. That dispatch fires correctly on the normal append path but the async-daemon rebuild path appends no streams — it replays already-persisted events from mt_events through the projection's aggregator and stores the rebuilt snapshots, so the inline projection chain is never given anything to iterate. Net effect: after rebuild the parent projection's documents are repopulated, but the mt_natural_key table stays exactly as teardown left it (empty), and any FetchLatest<T, TNaturalKey> afterwards misses. Fix — two new hooks in Marten, no JasperFx changes required (the existing abstractions — IAggregateProjection.NaturalKeyDefinition, EventRange.Events on every rebuild page, and ProjectionBatch.SessionForTenant routing writes into the batch's work-tracker — already give Marten everything it needs): 1. teardownProjectionStorage now also wipes the natural-key table when the source projection carries a NaturalKeyDefinition, so the rebuild starts from a clean slate (mirrors the doc-table TRUNCATE). 2. StartProjectionBatchAsync, in Rebuild mode, looks up the source by range.ShardName.Name and — when it has a NaturalKeyDefinition — feeds range.Events through a new NaturalKeyProjection.QueueUpsertsForEvents method per tenant. The IDocumentOperations comes from projectionBatch.SessionForTenant(...) which returns a ProjectionDocumentSession whose ISessionWorkTracker IS the ProjectionUpdateBatch — so the upserts flush inside the same transaction as the rebuilt snapshots, not the bare session's unit-of-work (which the batch's ExecuteAsync never touches). NaturalKeyProjection refactor: extracted queueUpsertSql to take (streamId, streamKey, tenantId, innerValue) instead of a StreamAction so the inline path (StreamAction-driven) and the rebuild path (IEvent-driven) share the SQL builder. ApplyAsync behavior is byte-for-byte unchanged. Test: ytqsl's Bug_4788_natrual_key_table_not_populated_on_rebuild repro (authored in commits 27ecadd / 12f16db / 6322a63 on this branch) now passes. Broader regression sweep on net10: 34/34 existing natural-key tests + 8/8 EventSourcingTests rebuild tests + 31/31 DaemonTests rebuild tests all green. Closes #4788 Supersedes #4789 --------- Co-authored-by: Jeremy D. Miller <jeremydmiller@yahoo.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Jul 6, 2026
This was referenced Jul 13, 2026
This was referenced Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #4789 (test contribution from @ytqsl preserved as the first three commits on this branch — same author, same email — followed by the implementation commit).
Root cause
NaturalKeyProjection(auto-registered for any aggregate carrying[NaturalKey]) maintains themt_natural_key_{aggregate}lookup table from an inlineApplyAsyncthat iterates theStreamActionsqueued during the currentSaveChangesand emits ON CONFLICT upserts per matching event.That dispatch fires correctly on the normal append path, but the async-daemon rebuild path appends no streams — it replays already-persisted events from
mt_eventsthrough the projection's aggregator and stores the rebuilt snapshots, so the inline projection chain is never given anything to iterate. Net effect: afterRebuildProjectionAsync, the parent projection's documents are repopulated but themt_natural_keytable stays exactly as teardown left it (empty), and any subsequentFetchLatest<T, TNaturalKey>misses.The PR description on #4789 said "expect the root cause and fix to land in JasperFx.Events" — after walking the dispatch chain I went a different way. JasperFx already exposes everything Marten needs:
IAggregateProjection.NaturalKeyDefinitionEventRange.Eventson every rebuild pageProjectionBatch.SessionForTenant(tenantId)returning aProjectionDocumentSessionwhoseISessionWorkTrackerIS theProjectionUpdateBatchSo this PR is Marten-only — no JasperFx changes needed. If you'd prefer the fix to live further upstream I'm happy to redo it, but I wanted to flag that the existing abstractions covered the case cleanly.
Changes
Two new hooks in Marten plus a small refactor:
teardownProjectionStorage— when the source projection carries aNaturalKeyDefinition, also queue aDELETE FROM mt_natural_key_Xso the rebuild starts from a clean slate (mirrors the doc-table teardown).StartProjectionBatchAsync— inShardExecutionMode.Rebuild, look up the source byrange.ShardName.Name, and when it has aNaturalKeyDefinitionfeedrange.Eventsthrough a newNaturalKeyProjection.QueueUpsertsForEventsmethod (per tenant). Operations routed throughprojectionBatch.SessionForTenant(...)so the upsert SQL flushes alongside the rebuilt snapshots in the same batch transaction — not on the bare session's unit-of-work, whichProjectionBatch.ExecuteAsyncnever touches.NaturalKeyProjection— extracted the upsert SQL builder to take(streamId, streamKey, tenantId, innerValue)instead of aStreamAction, so the inline path (StreamAction-driven) and the rebuild path (IEvent-driven) share the SQL.ApplyAsyncbehavior is byte-for-byte unchanged.Test plan
Bug_4788_natrual_key_table_not_populated_on_rebuild(commits27ecadda6/12f16dbef/6322a6353on this branch) — was red on master, now green.FetchForWriting/*natural*,Bug_4197,Bug_4199, conjoined + archived suites)Closes #4788