Repository navigation
Akka.Cluster: a Down node must never be leader, so it can't remove itself and linger outside the cluster (forward-port of #8650) - #8652
Merged
Aaronontheweb merged 2 commits intoSep 28, 2026
Conversation
…some members are unreachable MembershipState.LeaderOf lets a Down self count as a leader candidate whenever the cluster has unreachable members: `&&` binds tighter than `||`, so the self check skips the Down filter. Once every member is Down (e.g. SBR down-all-when-unstable), the downed node becomes leader of its own view, removes itself as an unreachable member in LeaderActionsOnConvergence, and never shuts down. It then fails to serialize every outgoing gossip with "Unknown address ... in cluster message", reporting its own address. The existing Down-leader test only covers the all-reachable branch. The new test covers the unreachable branch, and also checks that a self that is not Down still counts as a leader candidate when other members see it as unreachable. (cherry picked from commit 18bea7b)
…e unreachable
Add the missing parentheses in MembershipState.LeaderOf so the Down
filter applies to self too:
m.Status != Down && (reachable(m) || m == self)
Before, a Down self stayed a leader candidate. After a down-all
decision it could become leader of its own view, remove itself in
LeaderActionsOnConvergence, and then never shut down, because
ShutdownSelfWhenDown only acts on Down and self now reads as Removed.
(cherry picked from commit 378ddb9)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
close #8651
Forward-port of #8650 to
dev. Both commits were cherry-picked from thev1.5branch and applied cleanly.Problem
MembershipState.LeaderOfis missing parentheses in the branch that runs when some members are unreachable:&&binds tighter than||, so a node that isDownin its own view still counts as a leader candidate. When every member isDown, for example after SBR'sdown-all-when-unstable, the downed node becomes leader of its own view and removes itself as an unreachable member inLeaderActionsOnConvergence.ShutdownSelfWhenDownonly acts onDown, and self now reads asRemoved, so the node never shuts down and keeps running outside the cluster. See #8651 for the full walkthrough.Fix
A
Downnode is never leader, so it stays inMembersasDownand the existingShutdownSelfWhenDownpath shuts it down.Tests
GossipSpec.A_gossip_must_not_have_Down_self_as_leader_when_some_members_are_unreachable. It fails on currentdevwithout the fix: 1 failed, 35 passed inGossipSpec.Akka.Cluster.Testspass on net10.0.Notes
BREAKING_CHANGES_V1.6.md: this restores the intended behavior, since the all-reachable branch already excludedDownmembers.