Skip to content

fix: migrate NodeChurnSpec to async TestKit APIs to fix shutdown-timeout flakiness - #8286

Merged
Aaronontheweb merged 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/node-churn-spec-async-shutdown
Jun 24, 2026
Merged

Aaronontheweb merged 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/node-churn-spec-async-shutdown

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Member

Problem

NodeChurnSpec intermittently fails on CI with:

System.TimeoutException : Failed to stop [NodeChurnSpec] within [00:00:05]. ActorSystem is being forcefully shut down.
   at Akka.TestKit.TestKitBase.Shutdown(...)

(Most recently surfaced on the Microsoft.NET.Test.Sdk 17.9.0 → 18.7.0 bump in #8282, which is purely an environmental/test-runner timing change — not a product regression.)

Root cause

The spec creates two transient ActorSystems per round and tears them down with the synchronous Shutdown(node, verifySystemShutdown:true) helper, which does system.Terminate().Wait(5s). That blocks a thread-pool thread for up to its 5s budget while the coordinated-shutdown pipeline (run-coordinated-shutdown-when-down = on → cluster leave/down + remoting teardown + scheduler stop) needs that same thread pool to make progress. On slower/contended CI agents this sync-over-async self-starvation pushes a normally-fast shutdown past the hardcoded 5s default and the verify path throws.

Fix

Migrate the spec to the async, task-returning TestKit APIs and stop blocking on shutdown:

  • [MultiNodeFact] void → async Task (already the dominant pattern; sibling specs LeaderElectionSpec / ClusterAccrualFailureDetectorSpec in this project are async)
  • Within/AwaitMembersUp/EnterBarrier/AwaitAssert/ExpectNoMsg → WithinAsync/AwaitMembersUpAsync/EnterBarrierAsync/AwaitAssertAsync/ExpectNoMsgAsync
  • Replace the blocking per-system Shutdown(...) loop with concurrent async termination:
    await Task.WhenAll(systems.Select(s => s.Terminate())).WaitAsync(30.Seconds());
    This frees the thread (removing the self-starvation), terminates both systems in parallel, and preserves the verify-shutdown semantics (WaitAsync throws TimeoutException if a system fails to stop in time). Same idiom already used by QuickRestartSpec and DistributedPubSubRestartSpec.

No production code is touched — this is a test-only change.

Verification

  • Builds clean (Release, 0 warnings / 0 errors).
  • Ran NodeChurnSpec locally 4 consecutive times — Passed! Failed: 0, Passed: 3 every run (~44–48s each).

Notes

This is one of two flaky-test fixes split out from the #8282 investigation. The CircuitBreakerSpec timing fragility (same family of cold-start/thread-pool root cause) will be addressed in a separate PR.

…out flakiness

NodeChurnSpec intermittently failed with "Failed to stop [NodeChurnSpec] within
[00:00:05]" on slower CI agents. The synchronous Shutdown(node,
verifySystemShutdown:true) helper blocks a thread-pool thread inside Task.Wait()
for its 5s budget while the coordinated-shutdown pipeline itself needs the thread
pool to make progress — a sync-over-async self-starvation.

Migrate the spec to the async, task-returning TestKit APIs (WithinAsync,
AwaitMembersUpAsync, EnterBarrierAsync, AwaitAssertAsync, ExpectNoMsgAsync) and
replace the blocking per-system Shutdown loop with a concurrent
`await Task.WhenAll(systems.Select(s => s.Terminate())).WaitAsync(30s)`, which frees
the thread and preserves verify-shutdown semantics (throws TimeoutException if a
system fails to stop). Same idiom as QuickRestartSpec / DistributedPubSubRestartSpec.

Verified locally across 4 consecutive runs (3/3 node roles pass each, ~45s).
@Aaronontheweb

Copy link
Copy Markdown
Member Author

Basically this addresses a ton of legacy sync-over-async issues from the original TestKit / MNTR designs

@Aaronontheweb
Aaronontheweb merged commit 4b69aa5 into akkadotnet:dev Jun 24, 2026
11 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/node-churn-spec-async-shutdown branch June 24, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant