Skip to content

fix: pre-warm thread pool in CircuitBreakerSpecBase to fix cold-start latch flakiness - #8287

Closed
Aaronontheweb wants to merge 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/circuit-breaker-spec-threadpool-warmup
Closed

Aaronontheweb wants to merge 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/circuit-breaker-spec-threadpool-warmup

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Member

Problem

Several async CircuitBreakerSpec tests fail intermittently on CI with a bare:

Assert.True() Failure
Expected: True
Actual:   False

Most recently on the Microsoft.NET.Test.Sdk 17.9.0 → 18.7.0 bump (#8282) — e.g. Must_reopen_on_exception_in_call (CircuitBreakerSpec.cs:283, HalfOpen latch) and Must_throw_exceptions_when_called_before_reset_timeout (line 311, Open latch). The Assert.True(false) is a CountdownEvent latch wait (latch.Wait(2s)) returning false.

Root cause

CircuitBreaker signals the test latches from its transition listeners, which run fire-and-forget on the thread pool (AtomicState.Enter → NotifyTransitionListeners → Task.Factory.StartNew, deliberately un-awaited so listeners never block state transitions). The half-open transition additionally depends on the HashedWheelTimer reset firing — also pool-driven. Meanwhile the test thread blocks in CountdownEvent.Wait.

The .NET thread pool injects new workers above the minimum at only ~1 per 500ms. On a cold/contended CI agent at process start, the fire-and-forget breaker call, the transition-listener signal, and the timer continuation can be starved behind that ramp and miss the fixed 2s latch budget. This is environmental timing, not a product bug — the SDK bump simply changed the test-host's startup footprint enough to expose a pre-existing latent race. (Confirmed: akka.test.timefactor is 1.0 on CI, so Dilated(...) is a no-op and the latch budget is a hard 2s.)

Fix

Pre-warm the thread pool in CircuitBreakerSpecBase so those work items get a worker immediately instead of waiting on throttled injection:

ThreadPool.GetMinThreads(out var minWorker, out var minIo);
ThreadPool.SetMinThreads(Math.Max(minWorker, Environment.ProcessorCount * 2), minIo);

This is the same remedy already used in the repo for the identical cold-start root cause (InMemoryPersistenceSpecConfig.EnsureThreadPoolWarmed()). Math.Max guarantees a previously-raised minimum is never lowered.

Test-only change — no public API touched (notably TestBreaker and its CountdownEvent latches are unchanged, per extend-only design) and no timeout values changed.

Verification

  • Builds clean (Release, 0 warnings / 0 errors).
  • Full Akka.Tests.Pattern suite green (Total: 71, Failed: 0).
  • The two previously-failing tests pass 25/25 in a tight loop.

Local runs can't reproduce CI's cold-start starvation (the dev-box pool warms instantly), so local results confirm no regression; the fix is grounded in the root-cause analysis and the proven in-repo precedent.

Notes

Second of two flaky-test fixes split out from the #8282 investigation; the multi-node NodeChurnSpec fix is #8286. The broader observation that akka.test.timefactor is never raised on CI (making every Dilated(...) a no-op) is left as a separate, optional CI-config change.

… latch flakiness

The async CircuitBreakerSpec tests intermittently failed (e.g.
Must_reopen_on_exception_in_call, Must_throw_exceptions_when_called_before_reset_timeout)
with "Assert.True() Failure" when a CountdownEvent latch wait timed out.

The breaker signals these latches from fire-and-forget callbacks dispatched via
Task.Factory.StartNew, and the half-open transition also depends on the HashedWheelTimer
reset firing — all thread-pool work. The .NET thread pool injects new workers above the
minimum at only ~1 per 500ms, so on a cold/contended CI agent those work items can be
starved past the 2s latch timeout. Surfaced most recently by the Microsoft.NET.Test.Sdk
18.x test-host change (akkadotnet#8282).

Pre-warm the pool in the spec base (ThreadPool.SetMinThreads(ProcessorCount*2), guarded by
Math.Max so an existing minimum is never lowered), matching the existing
InMemoryPersistenceSpecConfig.EnsureThreadPoolWarmed() remedy for the identical root cause.
Test-only; no public API or timeout changes.

Verified: full Akka.Tests.Pattern suite green; the two previously-failing tests pass
25/25 in a tight loop.
@Aaronontheweb

Copy link
Copy Markdown
Member Author

Pretty gross code fix.

@Aaronontheweb
Aaronontheweb deleted the fix/circuit-breaker-spec-threadpool-warmup branch June 24, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant