Akka.Cluster: a Down node must never be leader, so it can't remove itself and linger outside the cluster

[Aaronontheweb]

close #8651

Problem

MembershipState.LeaderOf is missing a pair of parentheses in the branch that runs when some members are unreachable:

.Where(m => m.Status != MemberStatus.Down && reachability.IsReachable(m.UniqueAddress) || m.UniqueAddress == SelfUniqueAddress)

&& binds tighter than ||, so this reads (not Down && reachable) || self. A node that is Down in its own view still counts as a leader candidate.

That becomes a real failure when every member is Down, for example after SBR's down-all-when-unstable or after a DownAll decision arrives through gossip:

1. The downed node is the only candidate left, so it becomes leader of its own view.
2. Unreachable Down members don't block convergence, so LeaderActionsOnConvergence runs.
3. Other observers have marked this node unreachable, so it is in AllUnreachableOrTerminated. The leader removes it as an unreachable Down member: Leader is removing unreachable node [<self>].
4. ShutdownSelfWhenDown only acts on Down. Self now reads as the Removed placeholder, so the node never shuts down.
5. The next local gossip update stamps self as seen while self is no longer in Members. Every outgoing gossip then fails with ArgumentException: Unknown address [<its own address>] in cluster message.

The process keeps running outside the cluster until someone restarts it.

Fix

.Where(m => m.Status != MemberStatus.Down && (reachability.IsReachable(m.UniqueAddress) || m.UniqueAddress == SelfUniqueAddress))

A Down node is never leader, so it never runs leader actions and never removes itself. It stays in Members as Down, and the existing ShutdownSelfWhenDown path shuts it down. The all-reachable branch already filtered out Down members, so this makes both branches agree.

Tests

• New GossipSpec.A_gossip_must_not_have_Down_self_as_leader_when_some_members_are_unreachable. The existing Down-leader test only covered the all-reachable branch. The new test covers the unreachable branch, and also checks that a self that is not Down still counts as leader when others see it as unreachable.
• The first commit adds the test alone and fails on current v1.5: Expected State(g1, aDown).Leader to be <null>, but found UniqueAddress: (akka.tcp://sys@a:2552, 0).
• With the fix, all 369 Akka.Cluster.Tests pass on net10.0.

Notes

• No public API or wire-format change.
• dev has the same line and needs a forward-port.
• Related: Akka.Cluster: Gossip serialization fails with 'Unknown address in cluster message' due to Seen/Members inconsistency #8009 and Gossip protocol can propagate corrupted Seen state - implement defensive fix and tombstones #8015 share the Unknown address ... in cluster message symptom, through different paths.

[Aaronontheweb]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).