Skip to content

Ignore non-shard progression rows when waiting for non-stale data (#5161) - #5164

Merged
jeremydmiller merged 1 commit into
masterfrom
fix/5161-iscaughtup-shard-filter
Aug 3, 2026
Merged

Ignore non-shard progression rows when waiting for non-stale data (#5161)#5164
jeremydmiller merged 1 commit into
masterfrom
fix/5161-iscaughtup-shard-filter

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #5161.

Problem

WaitForNonStaleProjectionDataAsync's store-global bar required every row returned by AllProjectionProgress to reach the initial event sequence:

return rows.Count >= projectionsCount && rows.All(x => x.Sequence >= initial.EventSequenceNumber);

That set is the whole of mt_event_progression, which also holds rows that are not projection shards and have no reason to track the event sequence:

Either one holds the wait open forever, so the caller times out even though every real shard has finished its work. HighWaterMark only escaped because it happens to be the mark.

Fix

Apply the bar only to rows that represent progress this store is actually making, matching the ShardName.Compose grammar:

  • the store-global high water mark and its per-tenant HighWaterMark:{tenant} form
  • each registered shard identity and its {shard}:{tenant} form

Everything else is ignored. The count bar moves to the filtered set too, which keeps projectionsCount's "registered shards + 1 for the high water mark" accounting intact.

The per-tenant branch already keyed off shardIdentities and was immune; this brings the fallback into line so the two cannot drift again — which is what @jeremydmiller suggested on the issue.

Second, related win

Ignoring residue from unregistered projections fixes a broader shape of the same bug: a stale row from a projection that is no longer configured used to block the wait forever. That is exactly how this surfaced — leftover rows in a shared dev database from a branch that has since merged.

Tests

Bug_5161_wait_ignores_non_shard_progression_rows catches a store up through a real daemon, then inserts two rows that nothing will ever advance — HighWaterAllocationFence and a retired SomeRetiredProjection:All — and asserts the wait still returns promptly.

Verified RED against master's helper: 11-second timeout, passing here in ~1s.

Suite Result
DaemonTests 274/274
EventSourcingTests 1627/1627
TenantPartitionedEventsTests 238/238
MultiTenancyTests 159/159

All net10.0.

🤖 Generated with Claude Code

…ale data

WaitForNonStaleProjectionDataAsync's store-global bar required EVERY row
returned by AllProjectionProgress to reach the initial event sequence. That
set is the whole of mt_event_progression, which also holds rows that are
not projection shards and have no reason to track the sequence: high-water
bookkeeping (HighWaterAllocationFence, #5108), and residue left by
projections that are no longer registered. Nothing advances rows like
those, so the wait could never complete and the caller timed out even
though every real shard had finished its work.

Apply the bar only to rows that represent progress this store is actually
making — the store-global high water mark and its per-tenant
HighWaterMark:{tenant} form, plus each registered shard identity and its
{shard}:{tenant} form, matching ShardName.Compose. Everything else is
ignored, which also stops the next piece of bookkeeping from
reintroducing this.

The count bar moves to the filtered set too, keeping projectionsCount's
"registered shards + 1 for the high water mark" accounting intact.

The per-tenant branch already keyed off shardIdentities and was immune;
this brings the fallback in line so the two cannot drift again.

Ignoring residue from unregistered projections is a second, related win:
a stale row from a projection that is no longer configured used to block
the wait forever, which is the shape that surfaced this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 7c00ea1 into master Aug 3, 2026
10 checks passed
@jeremydmiller
jeremydmiller deleted the fix/5161-iscaughtup-shard-filter branch August 3, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WaitForNonStaleProjectionDataAsync counts the HighWaterAllocationFence row as a projection shard, so the wait can never complete

1 participant