Repository navigation
Fix intermittent timeout in ParallelTestActorDeadlockSpec test - #7974
Merged
Conversation
Fixed race condition causing ~1/40 test runs to fail with timeout. Root cause: Under high parallelism (40 concurrent TestKits), thread pool starvation could delay actor PreStart() execution, causing the test to timeout before the "ping" message was sent. Changes: - Removed unnecessary inner Task.Run that added extra thread pool queueing - Increased ExpectMsgAsync timeouts from 2s to 5s to account for legitimate scheduling delays under heavy parallel load The double Task.Run pattern was creating two levels of thread pool queueing, increasing the likelihood of scheduling delays when running 40 concurrent actor systems. The 2-second timeout was too aggressive for this level of parallelism - logs showed actual delays of ~5 seconds due to thread pool contention and timer drift under load. Test now passes consistently with all 40 instances completing in ~13 seconds. Related: akkadotnet#7770
Member
Author
|
A tad skeptical about this AI-suggested fix so I'm going to run CI/CD a few times to be certain |
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
Aaronontheweb
deleted the
claude-wt-Akka.TestKit.Tests.TestActorRefTests.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
branch
January 11, 2026 03:49
Aaronontheweb
added a commit
to Aaronontheweb/akka.net
that referenced
this pull request
Sep 21, 2026
…e budgets The spec starts 16 TestKits with Task.Run, and that is deliberate: it is what Akka.Hosting.TestKit does (the TestKit is built inside a StartActors callback on a host-startup continuation), so the bodies must stay on ThreadPool threads or the spec stops guarding the condition it exists for. Why it flaked: building a TestKit blocks its caller more than once (LoggingBus.StartDefaultLoggers waits up to akka.logger-startup-timeout, TestKitBase waits for the TestActor's PreStart), and every one of those waits is for work that runs on the same pool. On a two-core agent the pool injects roughly one thread per 500 ms, so sixteen blocked pool threads starve it. Measured at two cores over 160 TestKits: construction up to 5.3 s (the loggers hit their 5 s timeout), and a PingerActor up to 3.1 s to exist after ActorOf returns, against a flat 5 s ExpectMsgAsync budget. akkadotnet#7974, akkadotnet#8016 and akkadotnet#8023 raised that budget or lowered the count; the wait they were sizing was never made explicit. - ActorOf returns a RepointableActorRef before the actor exists. The spec now resolves the PingerActor first, then expects its ping, so the wait is on the thing that is actually slow. - The per-step budgets are 10 s and Dilated, so akka.test.timefactor applies. Construction plus one wait still fits under the 20 s [Fact] timeout, which remains the deadlock detector. - Teardown is await using, so Terminate().Wait no longer blocks a pool thread during the shutdown cascade. - The spec passed $"test-{id}" to the single-string TestKit overload, which parses its argument as HOCON: every system was named "test". It now uses the (Config, name, output) overload, so the log output is attributable. - One Stopwatch line per TestKit is kept for the next investigator. Validation on a build pinned to two cores: 30 runs, 0 failures; worst wait for the actor 3.1 s of the 10 s budget, ping within 26 ms once it exists. Akka.TestKit.Tests passes in full. Follow-ups not made here: TestKit construction blocking on pool work is the real cause and belongs in TestKitBase (an async construction path, or the test loggers on a dispatcher that is not the shared pool); the ManualResetEventSlim created with `using var` in CreateInitialTestActor is kept by InternalTestActor and would be Set() after disposal on a TestActor restart.
Aaronontheweb
added a commit
that referenced
this pull request
Sep 30, 2026
…e budgets (#8595) The spec starts 16 TestKits with Task.Run, and that is deliberate: it is what Akka.Hosting.TestKit does (the TestKit is built inside a StartActors callback on a host-startup continuation), so the bodies must stay on ThreadPool threads or the spec stops guarding the condition it exists for. Why it flaked: building a TestKit blocks its caller more than once (LoggingBus.StartDefaultLoggers waits up to akka.logger-startup-timeout, TestKitBase waits for the TestActor's PreStart), and every one of those waits is for work that runs on the same pool. On a two-core agent the pool injects roughly one thread per 500 ms, so sixteen blocked pool threads starve it. Measured at two cores over 160 TestKits: construction up to 5.3 s (the loggers hit their 5 s timeout), and a PingerActor up to 3.1 s to exist after ActorOf returns, against a flat 5 s ExpectMsgAsync budget. #7974, #8016 and #8023 raised that budget or lowered the count; the wait they were sizing was never made explicit. - ActorOf returns a RepointableActorRef before the actor exists. The spec now resolves the PingerActor first, then expects its ping, so the wait is on the thing that is actually slow. - The per-step budgets are 10 s and Dilated, so akka.test.timefactor applies. Construction plus one wait still fits under the 20 s [Fact] timeout, which remains the deadlock detector. - Teardown is await using, so Terminate().Wait no longer blocks a pool thread during the shutdown cascade. - The spec passed $"test-{id}" to the single-string TestKit overload, which parses its argument as HOCON: every system was named "test". It now uses the (Config, name, output) overload, so the log output is attributable. - One Stopwatch line per TestKit is kept for the next investigator. Validation on a build pinned to two cores: 30 runs, 0 failures; worst wait for the actor 3.1 s of the 10 s budget, ping within 26 ms once it exists. Akka.TestKit.Tests passes in full. Follow-ups not made here: TestKit construction blocking on pool work is the real cause and belongs in TestKitBase (an async construction path, or the test loggers on a dispatcher that is not the shared pool); the ManualResetEventSlim created with `using var` in CreateInitialTestActor is kept by InternalTestActor and would be Set() after disposal on a TestActor restart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixed race condition causing ~1/40 test runs to fail with timeout in
ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock.Root Cause
Under high parallelism (40 concurrent TestKits), thread pool starvation could delay actor
PreStart()execution, causing the test to timeout before the "ping" message was sent to TestActor.Changes Made
Task.Run- The test had a doubleTask.Runpattern that added extra thread pool queueing and increased scheduling delaysExpectMsgAsynctimeouts from 2s to 5s - Accounts for legitimate scheduling delays under heavy parallel load. Logs showed actual delays of ~5 seconds due to thread pool contention and timer driftTechnical Details
The double
Task.Runpattern was creating two levels of thread pool queueing:When running 40 concurrent actor systems, this extra queueing increased the probability that
PreStart()execution would be delayed beyond the 2-second timeout window.Verification
Test Plan
Fixes #7770