fix: make ClusterShardingQueriesSpec stats/region-state queries converge-then-assert (async)

[Aaronontheweb]

Summary

Makes the flaky multi-node spec ClusterShardingQueriesSpec.Querying_cluster_sharding_specs stable by converting its query assertions to converge-then-assert and to async TestKit methods. Test-only — no production code under Akka.Cluster.Sharding/ is touched.

The flake

Observed CI failure:

Expected regions.Values.Select(i => i.Stats.Count).Sum() to be 4, but found 3

The GetClusterShardingStats (and GetShardRegionState) reads were one-shot: region.Tell(query) → probe.ExpectMsg<...>() → .Sum().Should().Be(4) with no retry. That read is a point-in-time snapshot that races sharding's internal shard-region-query-timeout (3s on the Second/Third regions). Under CI load a Second/Third Shard actor can momentarily miss that internal timeout and be reported in Failed instead of Stats, transiently undercounting the Stats sum from 4 to 3. It clears on the next query (warm mailbox).

The product is correct by design — sharding stats/state queries return partial results when a shard doesn't answer within the query timeout. The defect was purely the one-shot assertion racing convergence.

The fix

• Wrap the stats query + both sum assertions, and each GetShardRegionState assertion block, in AwaitAssertAsync (converge-then-assert) that re-issues the query until the deterministic steady state is reached (Stats sum == 4, Failed sum == NumberOfShards / regions.Count).
• Use a fresh TestProbe per attempt so a late ClusterShardingStats / CurrentShardRegionState reply from a prior iteration can't pollute the next ExpectMsg.
• This mirrors the existing converge-then-assert pattern already used by the sibling ClusterShardingGetStatsSpec for the identical GetClusterShardingStats → Sum()==4 assertion.
• Converted the spec to async TestKit (per project convention): the [MultiNodeFact] is now async Task, using RunOnAsync, AwaitClusterUpAsync, WithinAsync, AwaitAssertAsync, ExpectMsgAsync, ReceiveWhileAsync, and EnterBarrierAsync — no .Result/.Wait(), no synchronous ExpectMsg/AwaitAssert/Within.

Intentionally unchanged

• Busy's shard-region-query-timeout = 0ms — this is the feature under test (its 2 shards deterministically report as Failed, keeping Stats sum at 4 and Failed sum at NumberOfShards / regions.Count). Kept as-is.
• Entity/shard setup, NumberOfShards, the id % NumberOfShards mapping, and the rebalance / min-nr-of-members config are all unchanged.

Validation

This is a multi-node (MNTR) spec, which cannot be run locally without the multi-node test runner, so it was compile-verified only:

dotnet build src/contrib/cluster/Akka.Cluster.Sharding.Tests.MultiNode/Akka.Cluster.Sharding.Tests.MultiNode.csproj \
  -c Release -warnaserror --framework net10.0
# Build succeeded. 0 Warning(s) 0 Error(s)

The actual multi-node behavior is validated by CI's multi-node runner.

Backport

Candidate to backport to v1.5 (the same spec exists there).