Daemon shutdown doesn't drain extended-progression writer or barrier the batch-write throttle → late telemetry clobbers rows + ObjectDisposedException on SemaphoreSlim

[jeremydmiller]

Summary

Async daemon shutdown disposes two shutdown primitives without first awaiting the in-flight work that still touches them:

1. The ExtendedProgressionWriter's background drain is fire-and-forgotten, so a "Stopped" heartbeat queued during shutdown can execute after the daemon is considered stopped and overwrite a later, deliberate write to mt_event_progression.
2. The shared BatchWriteThrottle SemaphoreSlim is disposed while an in-flight batch commit is still mid-finally, so its Release() throws ObjectDisposedException.

Both are surfaced downstream in Marten as marten#5022. The test-only mitigation there (marten#5023) sidesteps the race by not running a live-then-stopped daemon; this issue tracks the product-side root cause, which lives here in JasperFx.Events.

Symptom 1 — undrained shutdown heartbeat clobbers a later write

ExtendedProgressionWriter.DisposeAsync() completes the block but returns without awaiting the drain:

// src/JasperFx.Events/Daemon/ExtendedProgressionWriter.cs:124-130
public ValueTask DisposeAsync()
{
    // Lets any queued final writes (e.g. the Stopped state published during shutdown) drain
    // in the background rather than dropping them on the floor
    _block.Complete();
    return ValueTask.CompletedTask;   // <-- not awaited
}

And the daemon fire-and-forgets even that:

// src/JasperFx.Events/Daemon/JasperFxAsyncDaemon.cs:231-235
// Completes the writer's queue so a final Stopped write can drain in the background
_ = _extendedProgression.DisposeAsync();   // <-- discarded task
_deadLetterBlock.Dispose();
_loadThrottle?.Dispose();
_batchWriteThrottle?.Dispose();

So a Stopped ShardState posted to the writer during shutdown drains on a background thread after Dispose() returns and the daemon is reported stopped. Any subsequent explicit IEventDatabase.WriteExtendedProgressionAsync(...) (a test, or a redeploy/rebuild that touches the same shard row) can be overtaken by that late Stopped write — the row is silently rolled back to agent_status = 'Stopped'. This is the flaky should be "Paused" but was "Stopped" failure in marten#5022.

Note StopAllAsync() (JasperFxAsyncDaemon.cs:868) drains agents (StopAndDrainAsync) and the dead-letter block, but never drains _extendedProgression — the writer is only ever "completed" from the sync Dispose().

Symptom 2 — Release() on a disposed SemaphoreSlim

_batchWriteThrottle?.Dispose() (JasperFxAsyncDaemon.cs:235) runs during Dispose() with no barrier against batch executions still unwinding. An in-flight commit reaches its finally and releases the now-disposed shared throttle:

// src/JasperFx.Events/Daemon/GroupedProjectionExecution.cs:351-355
finally
{
    writeThrottle?.Release();          // <-- line 353: ObjectDisposedException on the disposed semaphore
    await batch.DisposeAsync().ConfigureAwait(false);
}

where writeThrottle == range.Agent.BatchWriteThrottle, the daemon-owned shared semaphore (SubscriptionAgent.BatchWriteThrottle → JasperFxAsyncDaemon._batchWriteThrottle). The same writeThrottle?.Release()-in-finally pattern against the shared throttle exists in AggregationRunner.cs:568 and ProjectionExecution.cs:197, so all three execution paths can hit it.

System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Threading.SemaphoreSlim'.
   at System.Threading.SemaphoreSlim.Release(Int32 releaseCount)
   at JasperFx.Events.Daemon.GroupedProjectionExecution.applyBatchOperationsToDatabaseAsync(...)
     GroupedProjectionExecution.cs:line 353

Root cause

Shutdown tears down primitives on the synchronous Dispose() path, which cannot await either (a) the extended-progression writer's drain or (b) the quiescence of in-flight batch executions that release the shared throttle. Disposal races the work that still depends on those primitives.

Suggested directions

1. Await the writer drain. Make ExtendedProgressionWriter.DisposeAsync() actually await completion — await _block.WaitForCompletionAsync() (exists at src/JasperFx/Blocks/Block.cs:110, which calls Complete() then awaits) — instead of returning ValueTask.CompletedTask.
2. Drain the writer on the async stop path, before disposing throttles. In StopAllAsync() — after agents StopAndDrainAsync and the dead-letter drain, where the daemon is genuinely quiesced — await _extendedProgression.DisposeAsync() (or a dedicated DrainAsync) so no Stopped write is still in flight when control returns. Keep the sync Dispose() fire-and-forget only as the last-resort path for callers that never called StopAllAsync().
3. Barrier the throttle. Don't Dispose() _batchWriteThrottle (and _loadThrottle / _rebuildBudget) until in-flight batch executions have released it — i.e. dispose after agent quiescence on the async path — and/or guard the three writeThrottle?.Release() finally sites against ObjectDisposedException (treat a disposed throttle as a no-op release, since a disposed throttle means the daemon is already gone). Ordering is the real fix; the guard is defensive belt-and-braces.

Notes

• Discovered reviewing marten#5008 (batched extended-progression telemetry, pairs with ExtendedProgressionWriter heartbeats don't scale under per-tenant agent fan-out: one unbatched connection round-trip per (shard, tenant) per ~10s drove a test cluster to its connection ceiling #553). #5008's production UPDATE … FROM unnest batch write is correct; only shutdown ordering is at fault.
• Downstream: marten#5022, test-only mitigation marten#5023.
• Related: ExtendedProgressionWriter heartbeats don't scale under per-tenant agent fan-out: one unbatched connection round-trip per (shard, tenant) per ~10s drove a test cluster to its connection ceiling #553.