Skip to content

Extended progression opt-in is broken under AddMarten (SetEventStoreInstrumentation clobbers it), docs snippet throws InvalidCastException, and mt_event_progression.last_updated freezes at insert for daemon shards #4981

Description

@jeremydmiller

Three defects around extended progression tracking / mt_event_progression, found while building the clean-room repro for #537 (the daemon never persists heartbeat/agent_status — see that issue for the write-path gap itself). All verified on Marten 9.16.1 / JasperFx.Events 2.30.1 against Postgres 17.

1. SetEventStoreInstrumentation silently clobbers a direct EnableExtendedProgressionTracking = true

The documented Marten-side toggle does not work under AddMarten:

builder.Services.AddMarten(opts =>
{
    opts.Connection(cs);
    opts.Events.EnableExtendedProgressionTracking = true;   // documented; silently lost
    opts.Projections.Add(new CounterProjection(), ProjectionLifecycle.Async);
}).AddAsyncDaemon(DaemonMode.Solo);

Observed: store.Options.Events.EnableExtendedProgressionTracking == false at runtime, and the extended columns are never added to mt_event_progression.

Cause: AddMarten always registers SetEventStoreInstrumentation (MartenServiceCollectionExtensions.cs), whose IConfigureMarten.Configure runs at store build and unconditionally overwrites:

options.EventGraph.EnableExtendedProgressionTracking = ExtendedProgressionEnabled; // adapter default: false
options.EventGraph.AppendObserver = AppendObserver;                                // adapter default: null

So the only opt-in that works under DI is mutating the DI-registered IEventStoreInstrumentation singleton (which is what CritterWatch does). Same clobber applies to AppendObserver, and to SetEventStoreInstrumentation<T> for ancillary stores. The adapter should OR-combine / only apply when explicitly set.

2. Docs opt-in snippet throws InvalidCastException

docs/events/projections/async-daemon.md shows:

((IEventStoreInstrumentation)opts.Events).ExtendedProgressionEnabled = true;

EventGraph implements IEventStoreOptions / IReadOnlyEventStoreOptions but not IEventStoreInstrumentation, so this throws Unable to cast object of type 'Marten.Events.EventGraph' to type 'JasperFx.Events.IEventStoreInstrumentation' (verified on 9.16.1; master's EventGraph.cs still doesn't implement it). The docs also say EnableExtendedProgressionTracking "continues to work" — which item 1 shows is untrue under AddMarten.

3. last_updated is frozen at insert time for daemon shard rows

With an async projection advancing for 2 minutes:

Counters:All    seq=48   last_updated=2:24:19 PM
Counters:All    seq=391  last_updated=2:24:19 PM   <- 2 minutes later, frozen
HighWaterMark   seq=394  last_updated=2:26:19 PM   <- advances fine

Cause: the daemon writes shard progress via InsertProjectionProgress (insert into ... (name, last_seq_id) values (?, ?)last_updated filled by table default) and UpdateProjectionProgress (update ... set last_seq_id = ? where name = ? and last_seq_id = ?) — neither touches last_updated. Only HighWaterDetector goes through mt_mark_event_progression(), whose upsert does set last_updated = transaction_timestamp(). If freezing is by design it's a trap for anyone using last_updated as an advance signal (it looks exactly like a stalled projection); if not, UpdateProjectionProgress should set it.

Repro

Full standalone program in #537 (Marten 9.16.1, AddAsyncDaemon(DaemonMode.Solo), one async SingleStreamProjection, continuous appends, progression dumped every 15s).

Refs: #537, #519, CritterWatch#750.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions