WaitForNonStaleProjectionDataAsync counts the HighWaterAllocationFence row as a projection shard, so the wait can never complete

[jeremydmiller]

Summary

WaitForNonStaleProjectionDataAsync's store-global fallback requires every row in mt_event_progression to reach the initial event sequence — including the HighWaterAllocationFence bookkeeping row, which is not a projection shard and legitimately lags. When that row is present and behind, the wait can never succeed and the test times out.

Surfaced as EventSourcingTests.Aggregation.ancillary_store_enrichment_tests.enrichment_from_ancillary_store_resolves_entity_and_maps_to_projection, but the helper is shared, so any test calling WaitForNonStaleProjectionDataAsync / WaitForNonStaleData can hit it.

Evidence

System.TimeoutException : The projections timed out before reaching the initial sequence of 18

-------------------------------------------
| Shard Name                 | Sequence   |
-------------------------------------------
| HighWaterAllocationFence   |          13|
| HighWaterMark              |          18|
| Order:All                  |          18|
-------------------------------------------

   at Marten.Events.TestingExtensions.WaitForNonStaleDataAsync(...) in src/Marten/Events/AsyncProjectionTestingExtensions.cs:line 279
   at Marten.Events.TestingExtensions.WaitForNonStaleProjectionDataAsync(...) in src/Marten/Events/AsyncProjectionTestingExtensions.cs:line 156
   at EventSourcingTests.Aggregation.ancillary_store_enrichment_tests.enrichment_from_ancillary_store_resolves_entity_and_maps_to_projection() in src/EventSourcingTests/Aggregation/ancillary_store_enrichment_tests.cs:line 67

The real work is done — HighWaterMark and the only projection shard Order:All both reached 18. Only the fence row, at 13, holds the wait open.

Root cause

AsyncProjectionTestingExtensions.cs, the non-per-tenant fallback inside isCaughtUp:

if (!perTenant || tenantHighWater.Count == 0)
{
    return rows.Count >= projectionsCount && rows.All(x => x.Sequence >= initial.EventSequenceNumber);
}

rows is everything AllProjectionProgress returns, so rows.All(...) sweeps in bookkeeping rows alongside actual projection shards. HighWaterMark happens to satisfy it (it is the mark), but the allocation-fence row tracks a different thing and has no reason to equal the current sequence.

The fence row arrived with the #4953 high-water liveness work (PR #5057, shipped 9.21.0); this helper predates it and was never taught to skip it.

Suggested fix

Exclude non-projection bookkeeping rows from both the rows.All(...) bar and the projectionsCount comparison — i.e. filter rows down to actual projection/subscription shard names before checking, the way the per-tenant branch already keys off shardIdentities. A shared "is this a real shard row" predicate would keep the two branches from drifting again.

Reproduction / scope

• Reproduces in isolation on clean master at 77efabc0f:
  dotnet test src/EventSourcingTests/EventSourcingTests.csproj --framework net10.0 --filter "FullyQualifiedName~ancillary_store_enrichment_tests"
• Not caused by any in-flight branch — verified in a separate worktree at that commit with no local changes.
• Intermittent: the same test passed in a full-suite run earlier the same day, so whether the fence row is behind at the moment of the wait is timing-dependent. That makes it a latent flake in CI rather than a hard failure.

[jeremydmiller]

Related: #5162 tracks a second test — Bug_2666_create_or_apply_but_not_both_should_apply.projection_with_create_and_ctor_should_use_create_on_event_match — that flakes under full-suite load only and calls the same WaitForNonStaleData helper. Its failure output was never captured, so it is unproven whether it is the same defect, but fixing this issue is the cheapest way to find out.

[jeremydmiller]

This one is mine, and it is not a master bug — apologies for the false trail.

HighWaterAllocationFence does not exist on master. It is a row introduced by #5163 (still open), and it got into your database because I ran that branch's test suites — DaemonTests, EventSourcingTests, TenantPartitionedEventsTests — against the shared marten_testing database while building it. That left 76 fence rows scattered across 76 schemas, including ancillary_store_enrichment_tests. A clean-master checkout then reads those leftover rows through AllProjectionProgress, master has no reason to filter a row it has never heard of, and WaitForNonStaleProjectionDataAsync blocks on a row nothing is updating any more. That also explains the intermittency: whether the stale row happens to sit below the current sequence depends entirely on what wrote that schema last.

I have deleted all 76. The reproduction should stop firing.

The underlying concern in your analysis is still right, though, and worth recording. Once #5163 merges the row exists for everyone, so the helper genuinely must not treat it as a shard. #5163 already handles it at the source: ProjectionProgressStatement excludes the reserved name, so AllProjectionProgress — which is what isCaughtUp calls at AsyncProjectionTestingExtensions.cs:246 — never sees it. There is a regression test pinning exactly that (the_fence_row_is_not_reported_as_projection_progress).

That said, your suggested fix is still the more robust shape and I would not object to it landing separately. Filtering at the query keeps the fence row out of the one path I knew about; a shared "is this a real shard row" predicate in isCaughtUp would stop the next piece of bookkeeping from doing the same thing to the rows.All(...) bar. The two branches of that method have already drifted once.

Suggest closing this as not-a-bug unless you want to keep it open for that hardening. Either way, the lesson for me is that running a branch that adds a progression row against the shared dev database leaves landmines for every other checkout — I should have cleaned up after myself.

[jeremydmiller]

Closing as not-a-bug on master — the trigger was 76 leftover HighWaterAllocationFence rows my #5163 test runs wrote into the shared dev database, since deleted.

Taking the hardening you suggested forward separately: isCaughtUp should filter to real projection/subscription shard rows rather than trusting everything AllProjectionProgress returns, so the next piece of high-water bookkeeping cannot do this again.