No supported way to read per-tenant projection lag for the current version on a sharded store

[erdtsieck]

The gap

There is no supported way to ask a Marten database how far behind each async projection's current version is, per tenant, when the store uses sharded/per-tenant databases.

Advanced.ProjectionProgressFor and Advanced.AllProjectionProgress throw under a sharded tenancy — there is no store-global high-water and no default tenant to resolve. IMartenDatabase.AllProjectionProgress(token) does work per database, but it hands back raw ShardState rows whose names are the framework's own encoding:

HighWaterMark:<tenant>
<source>:V<n>:All:<tenant>

So a caller that wants "is projection X's current version caught up for tenant T on this database" has to:

1. know that the high-water row is prefixed HighWaterMark: and split the tenant off it;
2. know that a projection shard identity is <name>:V<version>:All with the tenant appended, and split that apart;
3. know that a V{n} segment is absent for version 1;
4. know that a missing row for the current version means "has not started", not "caught up" — the distinction that matters during a version bump, when the previous version's row still sits at the old mark;
5. and know to compare against that tenant's own high-water, not a global one.

We have all five of those in application code, and getting (4) wrong is what made our readiness signal latch early on a rollout. Points 1-3 are re-implementing ShardName parsing that JasperFx already owns — the same knowledge ShardName.TryParse and EventSubscriptionAgentFamily.TenantNeutralKeyOf encode internally.

What I would like

Something that answers the question per database, with the version and tenant semantics inside Marten rather than in every caller:

// on IMartenDatabase, or via Advanced
Task<IReadOnlyList<ProjectionLag>> FetchProjectionLagAsync(CancellationToken token);

public readonly record struct ProjectionLag(
    string ProjectionName,
    uint Version,          // the version the store currently registers
    string? TenantId,       // null for a store-global shard
    long Sequence,          // 0 when the current version has no progression row yet
    long HighWaterMark);    // that tenant's high-water, not a global one

Anchored on the registered async sources at their current version, so a lingering previous-version row is not mistaken for progress and an absent row reads as fully behind. That single guarantee is the part that is easy to get wrong and impossible to discover from the row names.

Two smaller things in the same area, if they are cheap:

• ShardName's parsing is public enough to build an identity but not to take one apart. A public way to get (name, version, tenant) out of a shard identity would remove the string splitting from callers.
• Wolverine's EventSubscriptionAgentFamily.DatabaseKeyOf is internal, so a test that reasons about agent placement per database has to copy the URI grammar. Making it public would let consumers group agents by database without duplicating that.

Where this comes from

A 512-shard-database, ~854-tenant deployment. We use this to gate a readiness probe, to drive an admin projection-status endpoint, and now to decide when a warm-up job may exit — three callers, all of which had to learn the same encoding. Happy to open a PR if you tell me where you would want it to live (IMartenDatabase extension, Advanced, or a small service); the shape above is what our own helper converged on after two bugs, so I have a decent idea of the edge cases, but the API placement is your call.

[jeremydmiller]

Thanks — this is a well-diagnosed issue, and the core of it is accepted. I've split it into three tracked issues across the repos that own each piece, so I'm closing this one as decomposed rather than declined. Details, including one thing you can act on immediately:

Something to check in your code now

ShardName.TryParse is public, along with Name, ShardKey, Version, TenantId and Identity — so the string splitting in points 1-3 is already avoidable for the projection shard rows. But it is a trap on exactly the rows you're also parsing:

ShardName.TryParse("HighWaterMark:acme", out var parsed);
// returns true
parsed.Name;      // "HighWaterMark"
parsed.ShardKey;  // "acme"  <- the tenant id, in the shard-key slot
parsed.TenantId;  // null    <- tenant lost
parsed.Identity;  // "HighWaterMark"  <- does not round-trip the input

TryParse special-cases only the bare HighWaterMark constant; the per-tenant form falls through to the generic two-segment branch, and the constructor then collapses the identity back to the store-global constant. No exception, no false — it just claims to be the store-global mark for every tenant. If you switch your high-water parsing to TryParse you'd silently mis-attribute marks, so keep doing the manual prefix strip there for now. Filed as JasperFx/jasperfx#618, and it blocks the lag API below.

Your point about Advanced throwing is out of date

Advanced.AllProjectionProgress and ProjectionProgressFor no longer throw under sharded tenancy. That was #4797, fixed in 10f56fd7e and shipped in 9.13.0 — both now fan out across every database the store knows about when no tenant id is given (concatenating rows, and taking the max sequence respectively). You're on something older. That doesn't undermine the request, but it may remove a workaround you're carrying.

Ask 1: accepted, in JasperFx — JasperFx/jasperfx#619

Placed there rather than in Marten because the correlation is provider-neutral: it needs nothing but AllProjectionProgress plus the registered shard list, both already on IEventDatabase, so Polecat gets it from the same implementation. It's an in-memory correlation over a round trip that already happens — no new SQL.

Your read on why this is worth owning is right, and stronger than the issue states. The "anchor on registered sources at their current version, and treat a missing row as fully behind" semantic already exists in three independent implementations: the blue/green side-effect gate in JasperFxAsyncDaemon.resolvePriorVersionProgressAsync, Marten's own WaitForNonStaleDataAsync, and your application code. Marten's copy carries four separate bug-fix comments — #4761, its follow-up, #5161, #4797 — and you independently hit two of the same four. That's the argument for the API.

Four changes to your proposed shape, all in #619:

• Carry the ShardName, don't flatten to (ProjectionName, Version, TenantId). The flattened tuple drops ShardKey, so a projection sliced across custom shard keys yields N rows collapsing to one tuple, and composite stages vanish. Your deployment wouldn't hit it, but the API would ship broken for sliced projections.
• HasProgressionRow as a real field, not Sequence == 0. That's your point 4, and a sentinel zero is the same ambiguity in a new costume.
• A DatabaseIdentifier — ShardState already carries one, and without it a 512-database fan-out concatenates into an unattributable list.
• Overloads take ShardName, not string? tenantId. ShardName already carries TenantId, so a tenant-scoped read is just a tenant-qualified shard name.

Also documented there: the edge case the API can't fix. The tenant set is discovered from the HighWaterMark:{tenant} rows themselves, so a tenant that exists but has never had a high-water row written produces no row at all — invisible rather than "fully behind". For a readiness probe that's your original failure mode relocated, so it's worth knowing about wherever this ends up.

Yes to the PR — please take it to #619. One acceptance criterion: rewrite Marten's WaitForNonStaleDataAsync isCaughtUp on top of the new API. If it can't express that, the shape is wrong, and doing it is what stops this logic forking a fourth time.

Ask 3: JasperFx/wolverine#3819

DatabaseKeyOf → public. Worth noting Wolverine's own test suite already copies that grammar verbatim in one place while calling the internal in two others, so you're not the only one paying for it.

One more, found while looking

#5171 — the per-tenant progression filter builds an unescaped LIKE pattern from the tenant id. _ is both a valid PG identifier character and a LIKE single-character wildcard, so a read scoped to acme_corp also matches acmeXcorp and friends. If any of your ~854 tenant ids contain underscores and you use the tenant-scoped overloads, that one may be affecting you today.

Closing here — follow #619 for the API.