Skip to content

Fix intermittent timeout in ParallelTestActorDeadlockSpec test - #7974

Merged
Aaronontheweb merged 6 commits into
akkadotnet:devfrom
Aaronontheweb:claude-wt-Akka.TestKit.Tests.TestActorRefTests.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
Jan 11, 2026
Merged

Aaronontheweb merged 6 commits into
akkadotnet:devfrom
Aaronontheweb:claude-wt-Akka.TestKit.Tests.TestActorRefTests.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Member

Summary

Fixed race condition causing ~1/40 test runs to fail with timeout in ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock.

Root Cause

Under high parallelism (40 concurrent TestKits), thread pool starvation could delay actor PreStart() execution, causing the test to timeout before the "ping" message was sent to TestActor.

Changes Made

  • Removed unnecessary inner Task.Run - The test had a double Task.Run pattern that added extra thread pool queueing and increased scheduling delays
  • Increased ExpectMsgAsync timeouts from 2s to 5s - Accounts for legitimate scheduling delays under heavy parallel load. Logs showed actual delays of ~5 seconds due to thread pool contention and timer drift

Technical Details

The double Task.Run pattern was creating two levels of thread pool queueing:

// OLD - Anti-pattern
async Task RunOneTestKit()
{
    await Task.Run(async () =>  // Unnecessary second Task.Run
    {
        // test code
    });
}

// NEW - Direct execution  
async Task RunOneTestKit()
{
    // test code runs directly
}

When running 40 concurrent actor systems, this extra queueing increased the probability that PreStart() execution would be delayed beyond the 2-second timeout window.

Verification

  • Test now passes consistently with all 40 instances completing in ~13 seconds
  • Ran test multiple times with no failures
  • All instances complete successfully without timeouts

Test Plan

  • Test passes locally on .NET 8.0
  • Verified fix addresses the race condition
  • No changes to production code, only test improvements

Fixes #7770

Fixed race condition causing ~1/40 test runs to fail with timeout.

Root cause: Under high parallelism (40 concurrent TestKits), thread pool
starvation could delay actor PreStart() execution, causing the test to
timeout before the "ping" message was sent.

Changes:
- Removed unnecessary inner Task.Run that added extra thread pool queueing
- Increased ExpectMsgAsync timeouts from 2s to 5s to account for legitimate
  scheduling delays under heavy parallel load

The double Task.Run pattern was creating two levels of thread pool queueing,
increasing the likelihood of scheduling delays when running 40 concurrent
actor systems. The 2-second timeout was too aggressive for this level of
parallelism - logs showed actual delays of ~5 seconds due to thread pool
contention and timer drift under load.

Test now passes consistently with all 40 instances completing in ~13 seconds.

Related: akkadotnet#7770
@Aaronontheweb Aaronontheweb added akka-testkit Akka.NET Testkit issues tests labels Dec 22, 2025
@Aaronontheweb

Copy link
Copy Markdown
Member Author

A tad skeptical about this AI-suggested fix so I'm going to run CI/CD a few times to be certain

Aaronontheweb and others added 5 commits December 22, 2025 12:33
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
…s.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock
@Aaronontheweb
Aaronontheweb merged commit dacadd1 into akkadotnet:dev Jan 11, 2026
11 checks passed
@Aaronontheweb
Aaronontheweb deleted the claude-wt-Akka.TestKit.Tests.TestActorRefTests.ParallelTestActorDeadlockSpec.Parallel_TestKit_startup_should_not_deadlock branch January 11, 2026 03:49
Aaronontheweb added a commit to Aaronontheweb/akka.net that referenced this pull request Sep 21, 2026
…e budgets

The spec starts 16 TestKits with Task.Run, and that is deliberate: it is
what Akka.Hosting.TestKit does (the TestKit is built inside a StartActors
callback on a host-startup continuation), so the bodies must stay on
ThreadPool threads or the spec stops guarding the condition it exists for.

Why it flaked: building a TestKit blocks its caller more than once
(LoggingBus.StartDefaultLoggers waits up to akka.logger-startup-timeout,
TestKitBase waits for the TestActor's PreStart), and every one of those
waits is for work that runs on the same pool. On a two-core agent the pool
injects roughly one thread per 500 ms, so sixteen blocked pool threads
starve it. Measured at two cores over 160 TestKits: construction up to
5.3 s (the loggers hit their 5 s timeout), and a PingerActor up to 3.1 s to
exist after ActorOf returns, against a flat 5 s ExpectMsgAsync budget.
akkadotnet#7974, akkadotnet#8016 and akkadotnet#8023 raised that budget or lowered the count; the wait
they were sizing was never made explicit.

- ActorOf returns a RepointableActorRef before the actor exists. The spec
  now resolves the PingerActor first, then expects its ping, so the wait is
  on the thing that is actually slow.
- The per-step budgets are 10 s and Dilated, so akka.test.timefactor
  applies. Construction plus one wait still fits under the 20 s [Fact]
  timeout, which remains the deadlock detector.
- Teardown is await using, so Terminate().Wait no longer blocks a pool
  thread during the shutdown cascade.
- The spec passed $"test-{id}" to the single-string TestKit overload, which
  parses its argument as HOCON: every system was named "test". It now uses
  the (Config, name, output) overload, so the log output is attributable.
- One Stopwatch line per TestKit is kept for the next investigator.

Validation on a build pinned to two cores: 30 runs, 0 failures; worst wait
for the actor 3.1 s of the 10 s budget, ping within 26 ms once it exists.
Akka.TestKit.Tests passes in full.

Follow-ups not made here: TestKit construction blocking on pool work is the
real cause and belongs in TestKitBase (an async construction path, or the
test loggers on a dispatcher that is not the shared pool); the
ManualResetEventSlim created with `using var` in CreateInitialTestActor is
kept by InternalTestActor and would be Set() after disposal on a TestActor
restart.
Aaronontheweb added a commit that referenced this pull request Sep 30, 2026
…e budgets (#8595)

The spec starts 16 TestKits with Task.Run, and that is deliberate: it is
what Akka.Hosting.TestKit does (the TestKit is built inside a StartActors
callback on a host-startup continuation), so the bodies must stay on
ThreadPool threads or the spec stops guarding the condition it exists for.

Why it flaked: building a TestKit blocks its caller more than once
(LoggingBus.StartDefaultLoggers waits up to akka.logger-startup-timeout,
TestKitBase waits for the TestActor's PreStart), and every one of those
waits is for work that runs on the same pool. On a two-core agent the pool
injects roughly one thread per 500 ms, so sixteen blocked pool threads
starve it. Measured at two cores over 160 TestKits: construction up to
5.3 s (the loggers hit their 5 s timeout), and a PingerActor up to 3.1 s to
exist after ActorOf returns, against a flat 5 s ExpectMsgAsync budget.
#7974, #8016 and #8023 raised that budget or lowered the count; the wait
they were sizing was never made explicit.

- ActorOf returns a RepointableActorRef before the actor exists. The spec
  now resolves the PingerActor first, then expects its ping, so the wait is
  on the thing that is actually slow.
- The per-step budgets are 10 s and Dilated, so akka.test.timefactor
  applies. Construction plus one wait still fits under the 20 s [Fact]
  timeout, which remains the deadlock detector.
- Teardown is await using, so Terminate().Wait no longer blocks a pool
  thread during the shutdown cascade.
- The spec passed $"test-{id}" to the single-string TestKit overload, which
  parses its argument as HOCON: every system was named "test". It now uses
  the (Config, name, output) overload, so the log output is attributable.
- One Stopwatch line per TestKit is kept for the next investigator.

Validation on a build pinned to two cores: 30 runs, 0 failures; worst wait
for the actor 3.1 s of the 10 s budget, ping within 26 ms once it exists.
Akka.TestKit.Tests passes in full.

Follow-ups not made here: TestKit construction blocking on pool work is the
real cause and belongs in TestKitBase (an async construction path, or the
test loggers on a dispatcher that is not the shared pool); the
ManualResetEventSlim created with `using var` in CreateInitialTestActor is
kept by InternalTestActor and would be Set() after disposal on a TestActor
restart.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

akka-testkit Akka.NET Testkit issues tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestKit deadlock during parallel test execution with TestActor initialization

2 participants