Skip to content

Fix #4788 — repopulate mt_natural_key on projection rebuild (ported to V8)#4795

Merged
jeremydmiller merged 1 commit into
JasperFx:8.0from
ytqsl:fix/4788-natural-key-rebuild
Jul 5, 2026
Merged

Fix #4788 — repopulate mt_natural_key on projection rebuild (ported to V8)#4795
jeremydmiller merged 1 commit into
JasperFx:8.0from
ytqsl:fix/4788-natural-key-rebuild

Conversation

@ytqsl

@ytqsl ytqsl commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

The natural-keys lookup table (mt_natural_key_{aggregate}) is maintained by NaturalKeyProjection, an auto-registered inline projection whose ApplyAsync iterates the StreamActions queued during the current SaveChanges and writes ON CONFLICT upserts per matching event. That dispatch fires correctly on the normal append path but the async-daemon rebuild path appends no streams — it replays already-persisted events from mt_events through the projection's aggregator and stores the rebuilt snapshots, so the inline projection chain is never given anything to iterate. Net effect: after rebuild the parent projection's documents are repopulated, but the mt_natural_key table stays exactly as teardown left it (empty), and any FetchLatest<T, TNaturalKey> afterwards misses.

Fix — two new hooks in Marten, no JasperFx changes required (the existing abstractions — IAggregateProjection.NaturalKeyDefinition, EventRange.Events on every rebuild page, and ProjectionBatch.SessionForTenant routing writes into the batch's work-tracker — already give Marten everything it needs):

  1. teardownProjectionStorage now also wipes the natural-key table when the source projection carries a NaturalKeyDefinition, so the rebuild starts from a clean slate (mirrors the doc-table TRUNCATE).

  2. StartProjectionBatchAsync, in Rebuild mode, looks up the source by range.ShardName.Name and — when it has a NaturalKeyDefinition — feeds range.Events through a new NaturalKeyProjection.QueueUpsertsForEvents method per tenant. The IDocumentOperations comes from projectionBatch.SessionForTenant(...) which returns a ProjectionDocumentSession whose ISessionWorkTracker IS the ProjectionUpdateBatch — so the upserts flush inside the same transaction as the rebuilt snapshots, not the bare session's unit-of-work (which the batch's ExecuteAsync never touches).

NaturalKeyProjection refactor: extracted queueUpsertSql to take (streamId, streamKey, tenantId, innerValue) instead of a StreamAction so the inline path (StreamAction-driven) and the rebuild path (IEvent-driven) share the SQL builder. ApplyAsync behavior is byte-for-byte unchanged.

Test: ytqsl's Bug_4788_natrual_key_table_not_populated_on_rebuild repro (authored in commits 27ecadd / 12f16db / 6322a63 on this branch) now passes. Broader regression sweep on net10: 34/34 existing natural-key tests

  • 8/8 EventSourcingTests rebuild tests + 31/31 DaemonTests rebuild tests all green.

Closes #4788
Supersedes #4789


…supersedes JasperFx#4789) (JasperFx#4793)

* test: repro rebuild projection with natural key

* fix: query by natural key

* updated file name

* Fix JasperFx#4788: rebuild a [NaturalKey] aggregate also rebuilds the mt_natural_key lookup table

The natural-keys lookup table (mt_natural_key_{aggregate}) is maintained by
NaturalKeyProjection, an auto-registered inline projection whose ApplyAsync
iterates the StreamActions queued during the current SaveChanges and writes
ON CONFLICT upserts per matching event. That dispatch fires correctly on the
normal append path but the async-daemon rebuild path appends no streams — it
replays already-persisted events from mt_events through the projection's
aggregator and stores the rebuilt snapshots, so the inline projection chain
is never given anything to iterate. Net effect: after rebuild the parent
projection's documents are repopulated, but the mt_natural_key table stays
exactly as teardown left it (empty), and any FetchLatest<T, TNaturalKey>
afterwards misses.

Fix — two new hooks in Marten, no JasperFx changes required (the existing
abstractions — IAggregateProjection.NaturalKeyDefinition, EventRange.Events
on every rebuild page, and ProjectionBatch.SessionForTenant routing writes
into the batch's work-tracker — already give Marten everything it needs):

1. teardownProjectionStorage now also wipes the natural-key table when the
   source projection carries a NaturalKeyDefinition, so the rebuild starts
   from a clean slate (mirrors the doc-table TRUNCATE).

2. StartProjectionBatchAsync, in Rebuild mode, looks up the source by
   range.ShardName.Name and — when it has a NaturalKeyDefinition — feeds
   range.Events through a new NaturalKeyProjection.QueueUpsertsForEvents
   method per tenant. The IDocumentOperations comes from
   projectionBatch.SessionForTenant(...) which returns a
   ProjectionDocumentSession whose ISessionWorkTracker IS the
   ProjectionUpdateBatch — so the upserts flush inside the same transaction
   as the rebuilt snapshots, not the bare session's unit-of-work (which the
   batch's ExecuteAsync never touches).

NaturalKeyProjection refactor: extracted queueUpsertSql to take
(streamId, streamKey, tenantId, innerValue) instead of a StreamAction so the
inline path (StreamAction-driven) and the rebuild path (IEvent-driven) share
the SQL builder. ApplyAsync behavior is byte-for-byte unchanged.

Test: ytqsl's Bug_4788_natrual_key_table_not_populated_on_rebuild repro
(authored in commits 27ecadd / 12f16db / 6322a63 on this branch) now
passes. Broader regression sweep on net10: 34/34 existing natural-key tests
+ 8/8 EventSourcingTests rebuild tests + 31/31 DaemonTests rebuild tests
all green.

Closes JasperFx#4788
Supersedes JasperFx#4789

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Yossi T <yossi.tamari@live.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ytqsl ytqsl changed the title Fix #4788 — repopulate mt_natural_key on projection rebuild (supersedes #4789) (#4793) Fix #4795 — repopulate mt_natural_key on projection rebuild (#4788 for v8) Jun 29, 2026
@ytqsl ytqsl changed the title Fix #4795 — repopulate mt_natural_key on projection rebuild (#4788 for v8) Fix #4788 — repopulate mt_natural_key on projection rebuild (ported to V8) Jun 29, 2026
@jeremydmiller
jeremydmiller merged commit 8c90f6a into JasperFx:8.0 Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants