De-flake ShardingBufferAdapterSpec: re-send the cold messages with a fresh probe per attempt under a sharding-derived budget

[Aaronontheweb]

Stacked on #8569.

What changes

ShardingBufferAdapterSpec re-sends its first message through each region instead of waiting once on a probe's flat default.

• A helper sends the cold message under AwaitAssertAsync with a fresh probe per attempt. The budget is the shard start timeout plus the updating-state timeout from the sharding settings, about 15 s, which is the cold path plus one shard restart cycle; each attempt waits the retry interval, about 2 s, so the loop iterates instead of spending the budget in one wait. A fresh probe per attempt keeps a late reply from one attempt from satisfying the next.
• The warm-phase sends stay single sends, bounded by the updating-state timeout instead of the flat default, and check that the reply came from the same entity as the cold phase, so a shard restart between the phases cannot make the two counter snapshots describe different buffer histories. Passivation is off and the warm phase writes nothing to the remember-entities store, so that identity check holds.
• The counter assertions are unchanged and still snapshot after the sends.
• StartShard created both regions with the test system's settings; each region now uses its own system's. AfterAll no longer shuts the test system down twice, which also flips teardown to B then A, the order in which B's graceful leave can complete while A is still leader. The one blocking shutdown left in AfterAll, and the implicit one the TestKit's dispose runs after it, both wait on TestKit.Xunit: implement the async dispose chain so a derived DisposeAsync no longer leaks the ActorSystem; de-flake StreamRefsSpec #8545, which gives the TestKit an async dispose chain; a TODO marks the spot.

Why

Build 131348, Windows unit tests, on a PR that changed only the serialization generator: the first message through region A was not echoed within the probe's 5 s default. The message had been destroyed mid-wait. The healthy cold path on that same agent took 35 ms, then the whole test process stalled for about 5.8 s, both actor systems missing their heartbeat timers by identical amounts at identical instants. During the stall the shard's remember-entities write timed out, the region restarted the shard, and its buffered messages died with the instance with no dead letter. The write acknowledgement arrived 5 ms after the shard gave up. Waiting longer could not have helped; sharding is at-most-once, so the request has to be re-sent.

Two product findings came out of this. The shard arms its remember-entities write timeout with the read setting, 2 s instead of the documented 5 s, since #6479 in 2023, so the ddata store gets none of the retries it sizes against the 5 s value; that is fixed separately. And a shard restart drops its buffer silently; that is #8572. The stall itself is the thread-pool scheduler pattern on #8549, now with a fourth data point there.

The generator change on the triggering PR is not involved: the only generated types in Akka.Remote are Artery's, this spec runs the classic transport, and nothing in the default serialization bindings is generated.

How it was checked

dotnet build src/contrib/cluster/Akka.Cluster.Sharding.Tests -c Release -warnaserror clean. The spec run five times: passed each time, about 4 s per run. The grep for synchronous TestKit calls, blocking waits, and sleeps on the file returns only the one Shutdown(_sysB) that #8545 blocks. Test-only change. No ledger entry.

Second commit

From the adversarial review: the comment on the identity check said ActorPath.Equals includes the uid. It does not; IActorRef equality compares the uid, ActorPath alone compares address and names. The assertion was right, the reason was wrong, and a reader who trusted it could have swapped in a path comparison and silently lost the check. Comment corrected, nothing else. A related TestKit finding from the same analysis, AwaitAssert reading the clock twice so a stall between the reads turns into a thrown ArgumentOutOfRangeException, is recorded on #8565.