Fix #4788 — repopulate mt_natural_key on projection rebuild (ported to V8)#4795
Merged
Merged
Conversation
…supersedes JasperFx#4789) (JasperFx#4793) * test: repro rebuild projection with natural key * fix: query by natural key * updated file name * Fix JasperFx#4788: rebuild a [NaturalKey] aggregate also rebuilds the mt_natural_key lookup table The natural-keys lookup table (mt_natural_key_{aggregate}) is maintained by NaturalKeyProjection, an auto-registered inline projection whose ApplyAsync iterates the StreamActions queued during the current SaveChanges and writes ON CONFLICT upserts per matching event. That dispatch fires correctly on the normal append path but the async-daemon rebuild path appends no streams — it replays already-persisted events from mt_events through the projection's aggregator and stores the rebuilt snapshots, so the inline projection chain is never given anything to iterate. Net effect: after rebuild the parent projection's documents are repopulated, but the mt_natural_key table stays exactly as teardown left it (empty), and any FetchLatest<T, TNaturalKey> afterwards misses. Fix — two new hooks in Marten, no JasperFx changes required (the existing abstractions — IAggregateProjection.NaturalKeyDefinition, EventRange.Events on every rebuild page, and ProjectionBatch.SessionForTenant routing writes into the batch's work-tracker — already give Marten everything it needs): 1. teardownProjectionStorage now also wipes the natural-key table when the source projection carries a NaturalKeyDefinition, so the rebuild starts from a clean slate (mirrors the doc-table TRUNCATE). 2. StartProjectionBatchAsync, in Rebuild mode, looks up the source by range.ShardName.Name and — when it has a NaturalKeyDefinition — feeds range.Events through a new NaturalKeyProjection.QueueUpsertsForEvents method per tenant. The IDocumentOperations comes from projectionBatch.SessionForTenant(...) which returns a ProjectionDocumentSession whose ISessionWorkTracker IS the ProjectionUpdateBatch — so the upserts flush inside the same transaction as the rebuilt snapshots, not the bare session's unit-of-work (which the batch's ExecuteAsync never touches). NaturalKeyProjection refactor: extracted queueUpsertSql to take (streamId, streamKey, tenantId, innerValue) instead of a StreamAction so the inline path (StreamAction-driven) and the rebuild path (IEvent-driven) share the SQL builder. ApplyAsync behavior is byte-for-byte unchanged. Test: ytqsl's Bug_4788_natrual_key_table_not_populated_on_rebuild repro (authored in commits 27ecadd / 12f16db / 6322a63 on this branch) now passes. Broader regression sweep on net10: 34/34 existing natural-key tests + 8/8 EventSourcingTests rebuild tests + 31/31 DaemonTests rebuild tests all green. Closes JasperFx#4788 Supersedes JasperFx#4789 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Yossi T <yossi.tamari@live.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test: repro rebuild projection with natural key
fix: query by natural key
updated file name
Fix Natural keys table is not populated when rebuilding a snapshot #4788: rebuild a [NaturalKey] aggregate also rebuilds the mt_natural_key lookup table
The natural-keys lookup table (mt_natural_key_{aggregate}) is maintained by NaturalKeyProjection, an auto-registered inline projection whose ApplyAsync iterates the StreamActions queued during the current SaveChanges and writes ON CONFLICT upserts per matching event. That dispatch fires correctly on the normal append path but the async-daemon rebuild path appends no streams — it replays already-persisted events from mt_events through the projection's aggregator and stores the rebuilt snapshots, so the inline projection chain is never given anything to iterate. Net effect: after rebuild the parent projection's documents are repopulated, but the mt_natural_key table stays exactly as teardown left it (empty), and any FetchLatest<T, TNaturalKey> afterwards misses.
Fix — two new hooks in Marten, no JasperFx changes required (the existing abstractions — IAggregateProjection.NaturalKeyDefinition, EventRange.Events on every rebuild page, and ProjectionBatch.SessionForTenant routing writes into the batch's work-tracker — already give Marten everything it needs):
teardownProjectionStorage now also wipes the natural-key table when the source projection carries a NaturalKeyDefinition, so the rebuild starts from a clean slate (mirrors the doc-table TRUNCATE).
StartProjectionBatchAsync, in Rebuild mode, looks up the source by range.ShardName.Name and — when it has a NaturalKeyDefinition — feeds range.Events through a new NaturalKeyProjection.QueueUpsertsForEvents method per tenant. The IDocumentOperations comes from projectionBatch.SessionForTenant(...) which returns a ProjectionDocumentSession whose ISessionWorkTracker IS the ProjectionUpdateBatch — so the upserts flush inside the same transaction as the rebuilt snapshots, not the bare session's unit-of-work (which the batch's ExecuteAsync never touches).
NaturalKeyProjection refactor: extracted queueUpsertSql to take (streamId, streamKey, tenantId, innerValue) instead of a StreamAction so the inline path (StreamAction-driven) and the rebuild path (IEvent-driven) share the SQL builder. ApplyAsync behavior is byte-for-byte unchanged.
Test: ytqsl's Bug_4788_natrual_key_table_not_populated_on_rebuild repro (authored in commits 27ecadd / 12f16db / 6322a63 on this branch) now passes. Broader regression sweep on net10: 34/34 existing natural-key tests
Closes #4788
Supersedes #4789