Skip to content

De-flake ClusterShardingLeaseSpec: join the cluster in InitializeAsync instead of blocking a pool worker in the constructor - #8558

Merged
Aaronontheweb merged 2 commits into
devfrom
fix/sharding-lease-spec-async-join
Sep 10, 2026
Merged

Aaronontheweb merged 2 commits into
devfrom
fix/sharding-lease-spec-async-join

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Sep 9, 2026 •

Copy link
Copy Markdown
Member

What changes

ClusterShardingLeaseSpec, the base that all fifteen lease tests and variants run through, no longer joins the cluster from its constructor with a blocking wait. The join moves into InitializeAsync and awaits Cluster.JoinAsync under a dilated 30 s cancellation bound, which completes on the node's own MemberUp and parks no thread. Sharding starts with StartAsync. A DisposeAsync bridge calls Dispose() so xUnit v3 still runs the TestKit teardown; once #8545 lands this becomes an override that chains to the base, and a comment says so.

The facts pass TestActor explicitly on every region tell, because the implicit sender reads a thread-static cell and a fact resumes on a different thread after InitializeAsync. The nested expect inside the lease-lost recovery loop gets a 1 s inner budget so the outer 10 s loop retries about ten times instead of three. The whole file moves to the async TestKit API per the maintainer's rule: every fact is async Task, and every wait is the Async form. No assertion changes.

Why

Build 131196 (Windows unit tests) failed PersistenceClusterShardingLeaseSpec.Cluster_sharding_with_lease_should_recover_if_lease_lost, but not in the lease logic. The constructor asserted that the node had reached Up inside a blocking AwaitAssert with a flat 3 s default, polling every 100 ms. MemberUp arrived at 3.462 s, a 462 ms miss, after 3.17 s of log silence. The cluster daemon's startup needs about six dispatches on the default dispatcher, which is the thread pool, and the constructor's blocking wait held one of the two workers a 2-vCPU agent starts with. xUnit v3 never raises the pool floor for this assembly because it disables parallelization. The lease, shard-stop, and persistence paths have no race; the analysis ruled each out.

A product item from the analysis is not in this PR: Shard.ReleaseLeaseIfNeeded blocks in PostStop with a synchronous wait where Pekko fires and forgets.

How it was checked

dotnet build src/contrib/cluster/Akka.Cluster.Sharding.Tests -c Release -warnaserror clean. All fifteen tests and variants run three times: 15 passed each time. A grep for the synchronous TestKit forms over the file returns nothing. BREAKING_CHANGES_V1.6.md untouched; this is test-only.

Second commit

From the adversarial review: the _ = TestActor touch in InitializeAsync was redundant, because the ambient-context attribute already pins the actor cell on the test thread before each fact, and the touch only left a live cell on a returning pool worker. Removed with its comment. The spec passes 15 of 15. #8545 is stacked on this PR and converts this spec's lifecycle methods to overrides once the TestKit gains the virtuals.

…c instead of blocking a pool worker in the constructor

The constructor's blocking wait for Up against a flat 3s default missed
by 462ms on CI, because the join's six dispatches queued behind the
parked worker on a starved thread pool. Cluster formation now happens
in IAsyncLifetime.InitializeAsync via JoinAsync/StartAsync, which waits
on the real MemberUp signal without parking a thread. The file is also
migrated to the async TestKit API per the repo's standing rule.
…ient-context attribute already pins the cell
@Aaronontheweb
Aaronontheweb force-pushed the fix/sharding-lease-spec-async-join branch from ea12ef2 to 0649029 Compare September 10, 2026 15:27
@Aaronontheweb
Aaronontheweb merged commit 408e008 into dev Sep 10, 2026
15 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/sharding-lease-spec-async-join branch September 10, 2026 15:46
Aaronontheweb added a commit that referenced this pull request Sep 12, 2026
…c instead of blocking a pool worker in the constructor (#8558)

* De-flake ClusterShardingLeaseSpec: join the cluster in InitializeAsync instead of blocking a pool worker in the constructor

The constructor's blocking wait for Up against a flat 3s default missed
by 462ms on CI, because the join's six dispatches queued behind the
parked worker on a starved thread pool. Cluster formation now happens
in IAsyncLifetime.InitializeAsync via JoinAsync/StartAsync, which waits
on the real MemberUp signal without parking a thread. The file is also
migrated to the async TestKit API per the repo's standing rule.

* ClusterShardingLeaseSpec: drop the redundant TestActor touch; the ambient-context attribute already pins the cell

(cherry picked from commit 408e008)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant