#5022: kill the shutdown-telemetry race in extended_progression_batch_write - #5023
Merged
Merged
Conversation
This was referenced Jul 22, 2026
jeremydmiller
added a commit
that referenced
this pull request
Jul 22, 2026
…tdown drain (#5024) (#5025) Bumps JasperFx.Events 2.33.1 -> 2.34.0 to consume #555 (the two remaining tenant-scoped explorer read paths) and #557 (the extended-progression shutdown-drain fix), then implements/proves both from Marten's side. #5021 — the second pair of explorer reads #503 left untenanted: - Override IEventStore.QueryByTagsAsync(tags, tenantId, ct) in DocumentStore.EventStoreExplorer.cs with the same two-model scoping PR #5020 established for the stream reads: a conjoined single-database store bounds the outer mt_events scan with an e.tenant_id predicate on the AllowAnyTenant explorer session (the tag sub-selects match by the globally-unique seq_id, so the outer predicate discards cross-tenant seq_ids); a database-per-tenant / sharded store opens the session against the tenant's own database. Null delegates to the tenant-less overload, byte-identical to today. - Honour EventQuery.TenantId in QueryEventStore.QueryEventsAsync: a set tenant scopes the query with TenantIsOneOf (overriding the session's own filter); a null tenant is left untouched so the pre-existing per-tenant paging contract is unchanged. - Explorer tests: same tag / same event under two tenants on a conjoined store reads each tenant's slice in isolation. #5024 Phase 3 — consume + prove the #557 shutdown fix: - Bug_5024_extended_progression_survives_shutdown drives the real live-then-stopped daemon path (#5023 deliberately stopped doing this to de-flake #5008): start a live daemon, catch it up, StopAllAsync(), then issue an explicit WriteExtendedProgression and assert the row is NOT clobbered by a late "Stopped" heartbeat (Symptom 1) and no ObjectDisposedException reached the log on shutdown (Symptom 2). Deterministic because the upstream drain is now awaited; green across repeated runs. Claude-Session: https://claude.ai/code/session_01JQ9NHbg31EWJmrQK9EN6i8 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_write The new extended_progression_batch_write tests (#553 / #5008) started a live daemon, waited for it to catch up, then StopAllAsync()'d it before asserting on the mt_event_progression rows. Daemon shutdown itself emits a "Stopped" extended-progression heartbeat asynchronously and does not drain it before the daemon is considered stopped, so it races back in AFTER the explicit WriteExtendedProgressionAsync batch write and clobbers the rows under assertion (the intermittent `should be "Paused" but was "Stopped"` CI failure). The helper's own comment had it backwards: stopping the daemon was the source of the racing write, not a guard against it. Nothing in these tests needs a running daemon. Seed the two committed progression rows directly via mt_mark_event_progression so the only writer left against those rows is the WriteExtendedProgressionAsync call under test. This removes the race (and the shutdown-path ObjectDisposedException) entirely and makes the tests deterministic, unblocking #5008. Test-only change (issue #5022 direction 1). The product-side drain of in-flight shutdown heartbeats before disposing the JasperFx.Events shutdown semaphore (Symptom 2) lives in JasperFx.Events.Daemon.GroupedProjectionExecution and is tracked separately as the cross-repo fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jeremydmiller
force-pushed
the
fix/5022-drain-shutdown-telemetry-race
branch
from
July 22, 2026 13:05
54db6da to
6ffc3e7
Compare
This was referenced Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5022.
Problem
DaemonTests.extended_progression_batch_write(#5008 / #553) fails intermittently in CI:Its setup helper started a live daemon, waited for it to catch up, then
StopAllAsync()'d it before asserting on themt_event_progressionrows. Daemon shutdown itself emits a"Stopped"extended-progression heartbeat asynchronously and does not drain it before the daemon is considered stopped — so it races back in after the explicitWriteExtendedProgressionAsyncbatch write and clobbers the rows under assertion. The helper's comment ("Stop the daemon so nothing else races telemetry writes") had it backwards: stopping the daemon was the source of the racing write.The same shutdown path also produces the
ObjectDisposedExceptionon a disposedSemaphoreSlimnoted in the issue (Symptom 2).Fix (test-only — issue direction 1)
Nothing in these tests needs a running daemon. Seed the two committed progression rows directly via
mt_mark_event_progressionand exerciseWriteExtendedProgressionAsyncwith no daemon running, so the only writer left against those rows is the call under test. This removes the race entirely and makes the tests deterministic.#5008's production code is unchanged — only its new test is touched.
Verification
dotnet test -f net9.0 --filter FullyQualifiedName~extended_progression_batch_write— 3/3 passing across 5 consecutive runs (previously flaky).Out of scope
The product-side fix (Symptom 2 — draining in-flight shutdown heartbeats before disposing the shutdown semaphore) lives in
JasperFx.Events.Daemon.GroupedProjectionExecution, cross-repo in jasperfx, and is tracked separately.🤖 Generated with Claude Code