#4712 + #4717 — per-tenant partitioning daemon: safe-harbor high-water + per-tenant progression (JasperFx 2.9.4)

[jeremydmiller]

Fixes #4712. Follow-up to #4705.

Problem

Under UseTenantPartitionedEvents on a sharded conjoined store, composite projection rebuilds hang. The daemon logs a SafeHarborTime of 0001-01-01 (≈ DateTime.MinValue + the 3s stale threshold), the gap-skip becomes a no-op, and the store-global high-water agent loops forever — silently freezing some composite rebuilds.

Root cause (the #4705 bug class, one query that was missed)

The store-global HighWaterStatisticsDetector reads select last_value from mt_events_sequence for HighestSequence. Under per-tenant partitioning the store-global mt_events_sequence is never advanced (each tenant draws seq_ids from its own mt_events_sequence_{suffix}), so HighestSequence reads 1 while the true mark is far higher → the agent treats the store as perpetually Stale. And because no store-global HighWaterMark progression row is read, HighWaterStatistics.Timestamp is left at default(DateTimeOffset) = 0001-01-01 — the source of the bogus SafeHarborTime.

Fix (mirrors the #4705 FetchHighestEventSequenceNumber change)

• Read coalesce(max(seq_id), 0) from mt_events when UseTenantPartitionedEvents.
• Stamp Timestamp from that first result (which always returns a row), so it can never be left at 0001-01-01 when the progression row is absent.
• Non-partitioned stores keep reading last_value from mt_events_sequence.

Test

Bug_4712_safe_harbor_high_water drives HighWaterDetector.Detect directly under per-tenant partitioning. Before the fix: HighestSequence=1, Timestamp=0001-01-01 (with CurrentMark=40). After: HighestSequence=40 and a real timestamp. Deterministic single-DB single-tenant repro (the detector-level seam; the sharded multi-composite hang is the downstream symptom). Non-partitioned high-water detection tests unchanged (10/10).

🤖 Generated with Claude Code

[jeremydmiller]

Follow-up: #4717 per-tenant progression (consume JasperFx 2.9.4)

Building on the #4712 fix, this PR now also adopts the per-tenant continuous-progression work shipped in JasperFx.Events 2.9.4 and resolves #4717.

• Bumped JasperFx* 2.9.2 → 2.9.4 (per-tenant continuous agents fan out one per (shard, tenant); also carries the projections-rebuild subscription fix, HiLoSequence Increment is not used #438).
• HighWaterDetector.MarkHighWaterForTenantAsync — Marten implements the new IHighWaterDetector hook to persist a durable HighWaterMark:<tenant> row.
• Un-skipped Bug_4717_per_tenant_progression — now green: two tenants of different heights, composite + standalone, each get per-tenant <Projection>:All:<tenant> rows and per-tenant high-water rows at their own height.
• sharded_daemon_per_shard_progression (×2) now poll for the per-tenant rows/docs. WaitForNonStaleData's caught-up check counts store-global shards, so it can return before a per-tenant agent commits on a partitioned store — making the immediate assert racy. Per-tenant progression itself is correct; only the test wait was racy.

Verification: full TenantPartitionedEventsTests 186/186 on clean DBs; the sharded tests are stable across repeated runs (including under within-run shard accumulation).

Investigation note: the earlier "sharded regression" / "1-event-per-tenant commit bug" turned out to be a flaky WaitForNonStaleData + dirty-DB artifact — not a product bug. Per-tenant progression is production-correct, so no JasperFx 2.9.5 was needed. Hardening WaitForNonStaleData to be per-tenant-aware (so the public helper is reliable on partitioned stores) is a worthwhile separate follow-up.