Should the high-water progression identity grammar be a compliance suite? Two stores compose it independently and nothing pinned it

[jeremydmiller]

Part of #5118 (compliance wave growth), epic #5110. Surfaced by #5176 and the cleanup in #5178.

The observation

#618 (JasperFx 2.39.0) changed how ShardName composes the identity of a high-water shard: the tenant slot is no longer discarded, so a store-global high-water shard still collapses to the literal "HighWaterMark" while a tenant-scoped one composes to "HighWaterMark:{tenant}".

That change deterministically broke a Marten test (Bug_4785_delete_projection_progress_by_shard_name) and broke nothing in Marten's runtime, because Marten had always persisted that exact grammar through its own constants in Marten/Events/Daemon/HighWater/HighWaterShardIdentity.cs — StoreGlobal, PerTenantPrefix, PerTenant(tenantId). The upstream change brought ShardName into line with Marten rather than the other way round.

The interesting part is what it revealed: Polecat independently composes the same grammar, at Polecat/Events/Daemon/PolecatHighWaterDetector.cs:289:

var highWaterPrefix = ShardState.HighWaterMark + ":";

So "HighWaterMark" / "HighWaterMark:{tenant}" is not a Marten convention. It is a shared expectation about progression-row identities that two stores depend on independently, that JasperFx now also encodes in ShardName, and that no test pinned anywhere until #5178 added one on the Marten side.

Why this is worth pinning cross-store rather than per-product

The progression SQL in both products is keyed by string equality on the name column. There is no parsing, no normalization, no fallback. HighWaterShardIdentity's own doc comment states the consequence:

both sides need to agree on this shape exactly, and any drift (a missing colon, a different separator) silently desyncs the writers from the readers because the SQL is keyed by string equality on name

A desync here does not throw. The writer keeps writing rows under one name and the reader keeps finding nothing under another, so the high-water mark looks permanently stuck at zero — which is the failure signature of several already-filed bugs in this area (#5108, #5125, #5061). A grammar change is therefore a silent cross-store break, and silent breaks are precisely what the compliance library exists to make loud.

The question to settle

Does the high-water identity grammar belong in JasperFx.Events.ComplianceTests, and if so, at what level?

Option A — assert the grammar only. A pure-composition test with no store involved: ShardName for a high-water shard, with and without a tenant, produces the documented strings. Cheap, no seam addition, but it tests JasperFx against itself and arguably belongs in jasperfx's own unit tests rather than a compliance suite consumers run.

Option B — assert the round trip through storage. The store writes a high-water progression row, and reading progress back finds it under the expected identity, for both the store-global and per-tenant shapes. This is what actually protects users, because it fails when either side of the writer/reader pair drifts, not just when the composition helper changes. Needs a seam member to read progression rows by identity — plausibly the shared IEventDatabase.AllProjectionProgress rather than anything new, which is worth checking first.

Option C — leave it per-product. The grammar is arguably an implementation detail of progression storage, and each store is entitled to its own. Against this: both stores already chose the same detail, JasperFx has now encoded it in a shared type, and CritterWatch reads these rows across stores.

My read is B if the existing shared progression surface can reach it, otherwise A, with C rejected — the moment two independent implementations converge on a string grammar and a third component parses it, it has stopped being an implementation detail. But B's seam cost is the deciding factor and I have not measured it, so this is a question rather than a recommendation.

Related

• Per-tenant progression filter uses an unescaped LIKE: tenant ids containing '_' match other tenants' rows #5171 — the per-tenant progression read filter over-matches via unescaped LIKE on this same grammar. Open, and a live example of the grammar being handled inconsistently within one product.
• Split the high-water assertions in Bug_4785 and cover the per-tenant delete path #5178 — pins the grammar and the per-tenant delete path on the Marten side, which is the stopgap regardless of how this is decided.

🤖 Generated with Claude Code

[jeremydmiller]

Decision: no — not a compliance suite

Jeremy's call: the mechanics are too different between the stores for this to be a shared behavioural suite.

That is consistent with what #5178 actually found. Marten composes the grammar through its own constants (Marten/Events/Daemon/HighWater/HighWaterShardIdentity.cs — StoreGlobal / PerTenantPrefix / PerTenant), Polecat composes it independently at PolecatHighWaterDetector.cs:289, and JasperFx now encodes a third view of it in ShardName. Three independent compositions of the same string is not a behaviour the stores owe their users in common — it is an implementation detail that happens to be observable, and pinning it cross-store would freeze three sets of internals rather than one contract.

Option C from the original framing (leave per-product) is therefore the outcome, not the rejected one.

What stays

The coverage added in #5178 remains the right home for this in Marten:

• high_water_identity_grammar_agrees_between_shard_name_and_marten — the JasperFx↔Marten contract check, no database
• deleting_one_tenants_high_water_row_leaves_a_lookalike_tenant_alone — the acme_corp vs acmeXcorp discriminator, where the differing character is _, a PG LIKE wildcard

Polecat keeps its own equivalent. The live related bug, #5171 (the per-tenant progression read filter using unescaped name like '%:' || tenant), is unaffected by this decision and stays open on its own merits.

Closing.