fix(#574): stop fanning StopAllAsync out over disposed daemons at shutdown (marten#5055) - #575
Merged
Merged
Conversation
…tdown (marten#5055) At pod shutdown a second Pause/Stop pass (double AddAsyncDaemon hosted-service registration, user pause + host stop, Wolverine quiesce + host stop) hit StopAllAsync on daemons the first pass had already disposed. _semaphore.WaitAsync(_cancellation.Token) then threw ObjectDisposedException off the disposed CancellationTokenSource, and ProjectionCoordinatorBase.PauseAsync logged one "Error while trying to stop daemon agents" per daemon on every shutdown. - JasperFxAsyncDaemon: idempotent Dispose() sets a volatile _disposed flag; StopAllAsync no-ops when disposed, including the window where Dispose races an in-flight stop between the flag check and the token access. - ProjectionCoordinatorBase.PauseAsync: ObjectDisposedException from a daemon stop now lands at Debug, not Error — nothing is left to stop. - ProjectionCoordinatorBase.StopAsync: new ClearResolvedDaemons() seam purges the subclass daemon cache after disposal, so a second StopAsync has nothing to fan out over and a later ResumeAsync/daemon accessor resolves fresh daemons instead of handing back dead instances. Marten and Polecat must implement ClearResolvedDaemons() when they bump. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #574. JasperFx.Events side of JasperFx/marten#5055.
Problem
At Kubernetes pod shutdown, a second Pause/Stop pass (double
AddAsyncDaemonhosted-service registration, user pause + host stop, Wolverine quiesce + host stop) fansStopAllAsync()out over daemons the first pass already disposed.StopAllAsyncopened withawait _semaphore.WaitAsync(_cancellation.Token), and reading.Tokenoff the disposedCancellationTokenSourcethrewObjectDisposedException— whichProjectionCoordinatorBase.PauseAsynclogged asError while trying to stop daemon agents, once per daemon, on every shutdown.There was also a latent second bug: after
StopAsync, the subclass daemon caches still handed back the disposed daemons toResumeAsyncand the daemon accessors.Changes
JasperFxAsyncDaemon:Dispose()is now idempotent and sets avolatile bool _disposed;StopAllAsync()early-returns when disposed, and additionally catchesObjectDisposedExceptionaround the semaphore entry to cover aDispose()racing an in-flight stop. A disposed daemon has nothing left to stop, so a silent no-op is the correct behavior.ProjectionCoordinatorBase.PauseAsync: anObjectDisposedExceptionfrom a daemon stop now logs at Debug ("already disposed... benign during shutdown") instead of Error — same philosophy as the ProjectionCoordinatorBase.executeAsync: terminate on cancellation / disposed data source instead of re-polling (shutdown drain race) #499/#499: terminate coordinator loop on disposed data source / wrapped cancellation #500 disposed-data-source handling inexecuteAsync.ProjectionCoordinatorBase.StopAsync: after disposing the resolved daemons, calls a newprotected abstract void ClearResolvedDaemons()seam so the subclass cache is purged. A secondStopAsyncthen has nothing to fan out over, and a laterResumeAsync/daemon accessor resolves fresh daemons instead of dead instances.ClearResolvedDaemons()is abstract, soProjectionCoordinatorBasesubclasses must add it when they bump:ProjectionCoordinator(andExplicitProjectionCoordinator) should clear the_daemonsImHashMap. The Marten follow-up (cache clearing +AddAsyncDaemonidempotence) is tracked separately on marten#5055.ProjectionCoordinatorshould clear its daemon dictionary the same way.Tests
New coverage in
EventTests/Daemon:DisposedDaemonStopAllTests— realJasperFxAsyncDaemon:StopAllAsyncafterDispose()(and after a stop→dispose cycle) no-ops instead of throwing;Dispose()is idempotent.ProjectionCoordinatorBaseTests—StopAsyncpurges the resolved-daemon cache; a secondStopAsyncneither throws nor logs an Error (verified against a fake daemon that reproduces the pre-fixObjectDisposedException); pausing over an already-disposed daemon logs Debug, not Error../build.sh test-eventsand./build.sh test-event-storeboth pass locally.🤖 Generated with Claude Code