Fix #4591: serialize concurrent DCB tag-boundary appends via side table#4595
Merged
Conversation
Pre-fix, `AssertDcbConsistency` emitted a `SELECT EXISTS(... FROM mt_events ... WHERE seq_id > :lastSeen AND <tag predicate>)` as a separate non-locking statement before the INSERTs at READ COMMITTED. Two truly-concurrent fetch→save sessions both ran the EXISTS check before either committed, both saw no conflict, and both committed when only one should have. The reproduction in PR #4594 routinely observed 6 of 8 racers commit when the contract demands exactly one. The fix is per the prior research-write-up: a new `mt_dcb_tag_version` side table keyed by `(tag_table, tag_value, tenant_id)` whose `version` column converts the predicate read into a row-level write conflict. How it serializes (READ COMMITTED, no SERIALIZABLE, no advisory locks): - Every tagged event append (boundary or not) queues a `DcbTagVersionBumpOperation` that INSERTs the row at version=1 (first time) or `ON CONFLICT DO UPDATE SET version = mt_dcb_tag_version.version + 1` (every subsequent time). This is the *producer* side — it makes the side table reflect every commit, not only boundary saves. - `FetchForWritingByTags` adds a second batched SELECT that captures each tag's current `version` (or 0 if no row yet) and stamps it onto the returned `EventBoundary` via `DcbTagVersionAssertion`. - At `SaveChangesAsync` time, the assertion emits `INSERT … ON CONFLICT DO UPDATE SET version = version + 1 WHERE version = $captured RETURNING 1` per captured tag, sorted by `(tag_table, tag_value)` for deterministic lock-acquisition order. The row-level write lock plus the captured-version predicate is the serialization point: any save with a stale captured value matches zero rows on RETURNING and surfaces `DcbConcurrencyException`. The `AssertDcbConsistency` class is gone — the side-table mechanism replaces it for both `DcbStorageMode.HStore` and `DcbStorageMode.TagTables`. `EventStore.Dcb.FetchForWritingByTags` is now a thin delegate over `FetchForWritingByTagsHandler` so the two entry points (direct and `BatchedQuery`) can't drift on the boundary semantics. Regression test: - `Bug_4591_truly_concurrent_dcb_tag_appends_both_commit` is now a `[Theory]` over `DcbStorageMode` (HStore + TagTables). 8 racers, each fetches via its own session, barrier-syncs on a TCS so every fetch captures the same boundary, then races into SaveChangesAsync. Asserts exactly one commit + `Racers - 1` `DcbConcurrencyException` throws. Both variants pass. Caveats / known follow-ups (separate task): - Test currently lives in `EventSourcingTests`; it's slow (~100 ms per Theory case, runs concurrent sessions) and per the user's note should move to its own project alongside `DaemonTests` in a follow-up so it doesn't bloat the EventSourcingTests run. - `DcbTagVersionTable` does not yet set heap `fillfactor` (Weasel.Postgresql doesn't expose `WITH (fillfactor = N)` on the `Table` type). Tracked as a Weasel follow-up; column-list keeps `version` unindexed so HOT updates remain eligible regardless. - The side table is never truncated. Growth is bounded by the count of distinct (tag_table, tag_value, tenant_id) tuples — see docs/events/dcb.md for the long-lived-tag guidance. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Jeremy's review: the side-table fix lands as a required, mandatory schema migration in a point release, so the migration callout needs to be impossible to miss for users on AutoCreate.None / db-patch pipelines. - migration-guide.md: new "Key Changes in 9.4.0" section at the top with an action-required Badge, the exact db-patch / db-apply commands, and a section explaining why the fix lands in 9.4 rather than waiting for 10.0 (it's a correctness bug affecting both DcbStorageMode.HStore and DcbStorageMode.TagTables — the racy SELECT-then-INSERT pattern was identical in both branches of the old AssertDcbConsistency). - docs/events/dcb.md: warning callout at the head of the "How the boundary check serializes" section pointing back at the migration guide. Also corrected the SQL shape to match what shipped (INSERT … ON CONFLICT DO UPDATE … WHERE … RETURNING 1, not a bare UPDATE) and added the producer-bump paragraph that explains why non-boundary appends now also bump the row. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Jun 2, 2026
This was referenced Jun 3, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4591.
This PR adds a new schema object:
mt_dcb_tag_versionin your event-store schema, created automatically whenever any DCB tag types are registered. The new fetch and save paths reference it, so users onAutoCreate.Nonemust rundb-patch/db-applybefore deploying 9.4 or saves with tagged events will fail withrelation "<schema>.mt_dcb_tag_version" does not exist. Auto-apply users (default) get the table on first save, no action needed. Full migration write-up atdocs/migration-guide.md→ 9.4.0 section on this branch.This lands as a required migration in a point release because it's a correctness fix — the racy
SELECT-then-INSERTpattern affected bothDcbStorageMode.HStoreandDcbStorageMode.TagTables(the predicate shape differed; the race didn't). Per-save cost: one extraINSERT … ON CONFLICT DO UPDATEper distinct(tag_type, tag_value)tuple referenced by the batch — typically one or two rows per tagged save.The race
AssertDcbConsistencyemitted aSELECT EXISTS(... FROM mt_events ... WHERE seq_id > :lastSeen AND <tag predicate>)as a separate non-locking statement before the INSERTs at READ COMMITTED. Two truly-concurrent fetch→save sessions both ran the EXISTS check before either committed, both saw no conflict, and both committed when only one should have. The reproduction in #4594 routinely observed 6 of 8 racers commit when the contract demands exactly one.The constraint Jeremy set was no
SERIALIZABLE, no advisory locks.The fix —
mt_dcb_tag_versionside tableA new side table keyed by
(tag_table, tag_value, tenant_id)whoseversioncolumn converts the predicate read into a row-level write conflict.DcbTagVersionBumpOperationthat inserts the row atversion=1first time orON CONFLICT DO UPDATE SET version = mt_dcb_tag_version.version + 1every subsequent time. This is what makes the side table reflect every commit, not only boundary saves — a plainsession.Events.Append(streamId, taggedEvent)correctly invalidates any in-flight boundary that captured the prior version.FetchForWritingByTagsadds a second batchedSELECTthat reads each tag's currentversion(or 0 if no row yet) and stamps it onto the returnedEventBoundaryviaDcbTagVersionAssertion.INSERT … ON CONFLICT DO UPDATE SET version = version + 1 WHERE version = $captured RETURNING 1per captured tag, sorted by(tag_table, tag_value)for deterministic lock-acquisition order. The row-level write lock plus the captured-version predicate is the serialization point: any save with a stale captured value matches zero rows onRETURNINGand surfacesDcbConcurrencyException.Works on
READ COMMITTEDfor bothDcbStorageMode.HStoreandDcbStorageMode.TagTables. TheAssertDcbConsistencyclass is gone.EventStore.Dcb.FetchForWritingByTagsis now a thin delegate overFetchForWritingByTagsHandlerso the two entry points (direct andBatchedQuery) can't drift on the boundary semantics.Regression test
Bug_4591_truly_concurrent_dcb_tag_appends_both_commitis now a[Theory]overDcbStorageMode(HStore + TagTables). 8 racers, each fetches via its own session, barrier-syncs on aTaskCompletionSourceso every fetch captures the same boundary, then races intoSaveChangesAsync. Asserts exactly one commit +Racers - 1DcbConcurrencyExceptionthrows. Both variants pass.Follow-ups (tracked separately, not in this PR)
DaemonTestsso it doesn't bloat the EventSourcingTests run (it spins concurrent sessions). Left inEventSourcingTestsfor this PR.DcbTagVersionTabledoes not yet setWITH (fillfactor = N)on the heap — Weasel.Postgresql'sTabletype doesn't currently expose that. Tracked as a Weasel follow-up; the column list leavesversionunindexed so HOT updates remain eligible regardless.(tag_table, tag_value, tenant_id)tuples —docs/events/dcb.mddocuments the long-lived-tag guidance.🤖 Generated with Claude Code