Ignore non-shard progression rows when waiting for non-stale data (#5161) - #5164
Merged
Conversation
…ale data WaitForNonStaleProjectionDataAsync's store-global bar required EVERY row returned by AllProjectionProgress to reach the initial event sequence. That set is the whole of mt_event_progression, which also holds rows that are not projection shards and have no reason to track the sequence: high-water bookkeeping (HighWaterAllocationFence, #5108), and residue left by projections that are no longer registered. Nothing advances rows like those, so the wait could never complete and the caller timed out even though every real shard had finished its work. Apply the bar only to rows that represent progress this store is actually making — the store-global high water mark and its per-tenant HighWaterMark:{tenant} form, plus each registered shard identity and its {shard}:{tenant} form, matching ShardName.Compose. Everything else is ignored, which also stops the next piece of bookkeeping from reintroducing this. The count bar moves to the filtered set too, keeping projectionsCount's "registered shards + 1 for the high water mark" accounting intact. The per-tenant branch already keyed off shardIdentities and was immune; this brings the fallback in line so the two cannot drift again. Ignoring residue from unregistered projections is a second, related win: a stale row from a projection that is no longer configured used to block the wait forever, which is the shape that surfaced this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5161.
Problem
WaitForNonStaleProjectionDataAsync's store-global bar required every row returned byAllProjectionProgressto reach the initial event sequence:That set is the whole of
mt_event_progression, which also holds rows that are not projection shards and have no reason to track the event sequence:HighWaterAllocationFence(Dead sequence gaps + idle-in-transaction sessions: defaults stall the high-water mark permanently; SkipStaleGapsDespiteLiveTransactionsAfter has two weaknesses #5108) records an observed sequence allocation and legitimately sits below the mark;Either one holds the wait open forever, so the caller times out even though every real shard has finished its work.
HighWaterMarkonly escaped because it happens to be the mark.Fix
Apply the bar only to rows that represent progress this store is actually making, matching the
ShardName.Composegrammar:HighWaterMark:{tenant}form{shard}:{tenant}formEverything else is ignored. The count bar moves to the filtered set too, which keeps
projectionsCount's "registered shards + 1 for the high water mark" accounting intact.The per-tenant branch already keyed off
shardIdentitiesand was immune; this brings the fallback into line so the two cannot drift again — which is what @jeremydmiller suggested on the issue.Second, related win
Ignoring residue from unregistered projections fixes a broader shape of the same bug: a stale row from a projection that is no longer configured used to block the wait forever. That is exactly how this surfaced — leftover rows in a shared dev database from a branch that has since merged.
Tests
Bug_5161_wait_ignores_non_shard_progression_rowscatches a store up through a real daemon, then inserts two rows that nothing will ever advance —HighWaterAllocationFenceand a retiredSomeRetiredProjection:All— and asserts the wait still returns promptly.Verified RED against master's helper: 11-second timeout, passing here in ~1s.
All net10.0.
🤖 Generated with Claude Code