Supported per-tenant projection lag read anchored on registered sources at their current version

[jeremydmiller]

Downstream ask: marten#5170 — a 512-shard-database / ~854-tenant deployment has no supported way to ask "how far behind is projection X, at its current version, for tenant T on this database?" and had to re-derive it from raw mt_event_progression row names, hitting two bugs on the way.

The read there is that this belongs in JasperFx rather than Marten: the correlation is provider-neutral, needs nothing but pieces IEventDatabase already exposes, and Polecat wants the same thing.

Why this keeps getting reinvented

The "anchor on registered sources at their current version, and treat a missing row as fully behind rather than caught up" semantic already exists in three independent implementations:

1. JasperFxAsyncDaemon.resolvePriorVersionProgressAsync (src/JasperFx.Events/Daemon/JasperFxAsyncDaemon.cs:548-576) — reads AllProjectionProgress, ShardName.TryParses each row, filters parsed.Version < name.Version so a prior version's row can't be mistaken for current progress. That's the blue/green side-effect gate.
2. Marten's WaitForNonStaleDataAsync (src/Marten/Events/AsyncProjectionTestingExtensions.cs:188-265) — same idea, plus per-tenant high-water pairing. That block carries four separate bug-fix comments: marten#4761 (a store-global bar is wrong when each tenant has its own sequence), its follow-up (a single :All agent records no per-tenant rows at all), marten#5161 (non-shard bookkeeping rows never advance and must be excluded), marten#4797 (sharded fan-out).
3. The reporter's application code, which independently hit two of the same four.

None of that is discoverable from the row names, and every caller that wants a readiness probe or a status endpoint has to get all of it right.

Two things already point at this API

• IEventDatabase.FetchHighestEventSequenceNumber's own doc comment (src/JasperFx.Events/IEventDatabase.cs:71-77) describes it as "the 'head' an explorer subtracts projection progress from to render lag." The concept is already half-acknowledged in the surface.
• ShardState.WarningBehindThreshold / ShardState.CriticalBehindThreshold (src/JasperFx.Events/Daemon/ShardState.cs:116,122) are declared and persisted into the extended progression columns, and grep finds no consumer anywhere that computes "behind by N" against them. This is the missing read side of columns that already ship.

Proposed shape

public readonly record struct ProjectionLag(
    ShardName Shard,              // Name / ShardKey / Version / TenantId all reachable
    string? DatabaseIdentifier,
    long Sequence,                // 0 when the current version has no progression row
    long HighWaterMark,           // that tenant's mark, not a store-global one
    bool HasProgressionRow)
{
    public long Lag => Math.Max(0, HighWaterMark - Sequence);
    public bool IsCaughtUp => HasProgressionRow && Sequence >= HighWaterMark;
}

with a default implementation on IEventDatabase:

Task<IReadOnlyList<ProjectionLag>> FetchProjectionLagAsync(CancellationToken token);
Task<IReadOnlyList<ProjectionLag>> FetchProjectionLagAsync(ShardName name, CancellationToken token);

Notes on the shape, against the version proposed downstream:

• Carry the ShardName, don't flatten to (name, version, tenant). Flattening drops ShardKey, so a projection sliced across custom shard keys yields N rows that all collapse to the same tuple, and composite stages (StageNumber / CompositeParentName) vanish. Born broken for sliced projections.
• HasProgressionRow as a real field, not a sentinel Sequence == 0. Conflating "never started" with "at zero" is precisely the ambiguity that latched the reporter's readiness probe green during a version bump, while the previous version's row still sat at the old mark.
• DatabaseIdentifier — ShardState already carries one (ShardState.cs:175). Without it a fan-out across 512 databases concatenates into an unattributable list.
• Overloads take ShardName, not string? tenantId. ShardName already carries TenantId, so a tenant-scoped read is just a tenant-qualified shard name. Preferred direction over the string? tenantId overload pattern from Per-tenant partitioning #209: JasperFx.Events contracts (composite projections, ShardName tenant grammar, admin overloads, daemon fan-out) #407 — that pattern is what produced the suffix-matching in marten#5171.
• Anchor on ProjectionGraph.AllShards() (src/JasperFx.Events/Projections/ProjectionGraph.cs:43, built at :496-503). It's already keyed by the versioned shard identity, so "registered sources at their current version" is a property of the existing registry rather than something new.

No new SQL: this is an in-memory correlation over the single AllProjectionProgress round trip that already exists, plus the registry.

Acceptance criterion

Rewrite Marten's WaitForNonStaleDataAsync isCaughtUp on top of this. If the API can't express it, the shape is wrong — and doing it stops the logic forking a fourth time. resolvePriorVersionProgressAsync is a second candidate consumer.

Edge case to document, because the API can't fix it

The tenant set is discovered from the HighWaterMark:{tenant} rows themselves. A tenant that exists but has never had a high-water row written produces no row at all — invisible, rather than "fully behind". For a readiness probe that's the same failure mode the API is meant to eliminate, just relocated. It should be stated in the XML docs, or the tenant list cross-referenced where the store can supply one.

Depends on the HighWaterMark:{tenant} parsing fix (#618) — a lag implementation that round-trips high-water rows through ShardName today would silently attribute every tenant's mark to the store-global one.