De-flake DistributedPubSubRestartSpec: the restarted node makes first contact, and the survivor waits on it

[Aaronontheweb]

What changes

DistributedPubSubRestartSpec reverses who makes first contact after the restart.

• The restarted third system's own Shutdown actor pings a forwarder on first from its PreStart, so "the ping arrived" and "the actor exists" are one fact. It repeats the ping every 500 ms on a scheduler timer until first acks it, and cancels the timer on the ack and in PostStop. A single ping is not enough on its own: it rides the ordinary lane, and Artery's handshake stage holds one element there while the handshake to the new incarnation is gating. If that stage stops first, the held element is lost.
• The restarted node's Shutdown actor acknowledges a kill and then terminates its system 5 s later on a sliding timer, re-armed by every repeated kill, instead of terminating inline. The acknowledgement has to leave on a live transport, and the timer has to outlive one full retry cycle of the kill loop, 2 s of waiting plus a 500 ms interval, so a lost ack still gets a second attempt at a live target.
• The forwarder on first acks every ping straight back to its sender and hands the signal to a probe. first waits on that probe once, with a 60 s bound, and ignores later pings. Only then does it send the shutdown command, with ActorSelection.Tell and a fresh probe per attempt over a 20 s loop, because only a selection message crosses a quarantined association.
• third asserts, right after its restart and inside the block that owns the new system, that it is bound to the pinned port, so a failed assertion still shuts that system down and the next failure on the Linux lane settles whether the listener came up where it should.
• The conductor shutdown uses the async overload.
• The barrier budget rises to 180 s with the arithmetic in a comment: a 30 s shutdown cap, the 60 s wait, and the 20 s loop are 110 s, and the barrier clock arms when the first node arrives. The best-effort wait for the new system's termination on third stays at 120 s. Its clock starts after third creates the Shutdown actor, whose start is what sends the ready ping, so first's 30 s shutdown cap and its ready wait are already spent by then. What remains of first's pipeline is the 20 s kill loop, its last 2 s attempt, and the 5 s terminate timer, about 27 s, so 120 s carries about 90 s of margin.

The earlier fixes in this file stay: the port pin, the transport failure-detector bound, and the double-sampled delta count. The ResolveOne with a fresh temporary actor is replaced by the selection send, which carries the same freshness on the probe.

Why

This spec failed six times in three days on the Artery lanes (builds 131137, 131156, 131161, 131174, 131187, 131188), always node first, always a ResolveOne against the restarted node timing out after 17 attempts inside a 45 s window. It has an open issue since 2016 and ten fix commits since 2025, some of which reversed each other.

None of them changed who makes first contact. first's association to third's address can only learn third's new identity from a frame third sends: promotion runs from the inbound handshake stage, and after an identity change the ordinary lane needs a second inbound event because the association state resets its handshake flag. There is no cluster event or reachability change to await, because the fresh system joins itself by design. The old test started its 45 s clock the moment the conductor's shutdown call returned, before third had begun its 8 to 13 s graceful shutdown, about 6 s before its actors existed, and up to 21 s of re-association. The window overlapped the downtime almost entirely. Now the survivor waits on the one signal that means the restart is complete, and that signal is also what heals the association. The measured worst case for that sequence is 12.9 s of shutdown, 6 s before the actors exist, and 21 s of re-association, 39.9 s in total, so the 60 s bound leaves about 20 s of margin.

Two Artery defects made the re-association slow and are fixed in #8554: the reconnect backoff was bypassed, measured at five control reconnects per second against a configured one, and a handshake stage could lose an element it had taken from the control queue. Two more are filed as issues, #8550 and #8551. The spec does not depend on any of them, because a lost ping is resent, not fatal.

How it was checked

dotnet build src/contrib/cluster/Akka.Cluster.Tools.Tests.MultiNode -c Release -warnaserror clean. The spec run locally through the multi-node adapter at the current head: classic transport, 3 of 3 passed in 10 s; Artery, the lane that fails on CI, 3 of 3 passed twice, 28 s each. The grep for synchronous TestKit calls on the file returns only comments, the local Shutdown actor class, and the pre-existing Shutdown(ActorSystem) override.

Third commit: what this PR's own CI run showed

Build 131332, Linux Artery, all three node logs read from the failed-run artifact. The ready ping did its job: the restarted third bound the pinned port 5 s after the kill, created its Shutdown actor at 57.947, and first had the ping and sent the kill within 30 ms. Then third replied and called Terminate in the same handler. Artery aborted its outbound streams before any flush, the acknowledgement's stream among them, so the ack never left. third's process exited with a pass 50 ms later, and first retried the kill against a closed port for 20 s and failed. The local runs passed because the ack won that race on a fast machine.

The Shutdown actor now replies and arms a sliding terminate timer, cancelled in PostStop, the same shape as the Subject actor in #8557. The Artery flush fix in #8554 would usually carry the ack out as well, but this spec must not depend on it. Run three times on each transport after the change: 3 of 3 passed each time, 11 to 27 s classic, 12 to 13 s Artery. A temporary log line, removed before the commit, confirmed the ack leaving 2.0 s before Artery's teardown on third.

Fourth commit: what the adversarial review found

The first version of the timer was 2 s, shorter than the 2.5 s retry cycle it exists to outlive, so a lost ack would still have killed the target half a second before the next attempt. It is 5 s now, with the arithmetic in the comment. The termination wait had been raised to 155 s on a sum that double-counted first's already-spent waits; it is back at 120 s with the corrected arithmetic above. And one synchronous call had survived the migration, the Node(to) lookup in the join helper, now NodeAsync. Run three times on each transport after the change: 3 of 3 passed each time, 20 to 33 s.

[Aaronontheweb]

LGTM - this spec has consistently been the worst offender in the entire test suite for years and the addition of the Artery MNTR specs exposed some additional cases where things like quarantine didn't pierce it correctly.