DistributedData: Replicator accepts writes from a member it first saw as Leaving or Exiting

[Aaronontheweb]

What changes

Replicator.IsKnownNode now also accepts a member the replicator has seen in any membership state and a member in its exiting set. Before, it accepted only members it had seen reach Up, WeaklyUp, or Joining, plus itself.

Why

The replicator subscribes to cluster events with InitialStateAsEvents, which replays each member at its current status. A replicator that starts after a member has already reached Leaving receives MemberLeft and never MemberUp. Only MemberUp added a member to the known set, so every Write, Read, and DeltaPropagation that member sent was dropped as an unknown node for as long as it stayed in the cluster. A delete is a Write of a tombstone, so deletes were dropped too. Gossip and status messages were not affected: they are gated on the destination system uid and never reach this check.

Cluster sharding hits this during a coordinated shutdown. When the oldest node leaves, the departing shard coordinator writes its final state through the replicator on the next-oldest node. If that replicator started while the oldest node was already leaving, it drops the write, and the coordinator burns its full 5 s updating-state-timeout before the singleton hands over. That was 5.0 of the 5.6 seconds ClusterShardingRegistrationCoordinatedShutdownSpec waited on the Artery lane, where it failed four times in two days. Pekko has the same gap. The test-side fix for that spec is #8544; this PR removes the waste itself.

Quorum and majority sizing are untouched. IsKnownNode has one call site, the inbound gate, and every majority reads the up, exiting, and age-ordered sets directly. This is a bug fix that restores intended behavior, so it is not recorded in the 1.6 breaking-changes ledger.

How it was checked

Akka.DistributedData builds with warnings as errors. Akka.DistributedData.Tests: 193 passed, 1 skipped as before. Akka.API.Tests: 18 passed, no approval change; the new state is private. The new ReplicatorKnownNodeSpec has three cases: a write from a member first seen as Leaving is accepted, a write from a member first seen as Exiting is accepted, and a write from an address never seen in any member event is still rejected. The first two fail on the old code with the exact "Ignoring message [Write] ... unknown node" log line.

Second commit: the spec's teardown

From the adversarial review: the spec shut its peer system down with a blocking call in AfterAll, and named it the same as the test system. Each fact now shuts the peer down with ShutdownAsync in a finally block, so teardown runs on success or failure without pinning a thread. The class deliberately declares no DisposeAsync, because the TestKit gains an async dispose chain in #8545 and a bare method of that name on a derived class would then hide it and fail the build with warnings as errors. The peer is named with a -peer suffix; it self-joins its own cluster and the test system never joins it, so nothing depends on the names matching. Run three times: 3 passed each time.

[Aaronontheweb]

Opened #8582 as an alternative implementation of the same fix, for comparison.

The approach is the same — stop dropping Write/gossip messages from a member the Replicator first observed in a terminal status — but it avoids introducing new membership state. Both sets it needs are already maintained and already pruned on MemberRemoved:

• _exitingNodes is populated by ReceiveMemberExiting and is currently never read by IsKnownNode
• MemberLeft / MemberDowned already land in _leader via the generic ReceiveOtherMemberEvent path

private bool IsKnownNode(Address node) => _nodes.Contains(node) || _weaklyUpNodes.Contains(node) ||
                                          _joiningNodes.Contains(node) || _exitingNodes.Contains(node) ||
                                          _leader.Any(x => x.Address == node) || _selfAddress == node;

That's a 2-line production change versus a new field with new populate/cleanup sites.

I also confirmed the _seenNodes design in this PR is doing real work that a naive _exitingNodes extension would not: _exitingNodes is read by the removed-node pruning path (_nodes.Except(_exitingNodes)), so putting Leaving/Downed nodes into it would make them ineligible as pruning replicas. Routing those statuses through _leader avoids that side effect entirely.

Verified against the spec from this PR, with one addition. Running the three facts here unchanged against the alternative:

Fact	Baseline dev	With #8582
First seen as Leaving	❌ timeout	✅
First seen as Exiting	❌ timeout	✅
Never seen (gate still closed)	✅	✅

I added a fourth fact for MemberDowned, which shares the generic ReceiveOtherMemberEvent path with MemberLeft and is the third status a late subscriber can be handed. It fails on baseline and passes with the fix. Worth covering here too regardless of which implementation lands.

Full suite on #8582: 194 passed, 1 skipped, 0 failed.

One note relevant to both PRs: Apache Pekko retains the original four-term isKnownNode and has no equivalent of _seenNodes. It only avoids the symptom incidentally — it adds to nodes on MemberUp and removes only on MemberRemoved, so a member that went Up → Leaving stays in nodes for its whole dying phase. That means Pekko's answer for a late-starting Replicator still depends on when that Replicator started, so this looks worth an upstream issue either way.

[Aaronontheweb]

Closing in favor of #8582, which fixes the same gate by reusing the sets the Replicator already maintains instead of adding a new one, and adds the Downed case to the spec.