Akka.Cluster: gossip removal tombstones - stop resurrecting removed members

BREAKING_CHANGES_V1.6.md

@@ -41,6 +41,7 @@ of `Behavior`, `Wire`, `API` (combine with `+`).
 | Status | PR / Branch | Component | Type | Change | Migration |
 |--------|-------------|-----------|------|--------|-----------|
 | Planned | [#8324](https://github.com/akkadotnet/akka.net/pull/8324) | `Akka.Routing` | Behavior | `ConsistentHash<T>` no longer retains the `SortedDictionary` passed to its public constructor — the ring is snapshotted into internal sorted arrays. Mutating that dictionary after construction no longer affects the instance (previously the aliasing was inconsistent: `IsEmpty` and `operator +`/`-` read it live, while `NodeFor` froze it after the first lookup). A `null` dictionary now throws `ArgumentNullException` from the constructor instead of surfacing later as a `NullReferenceException`. No public API removed; the ring built by `ConsistentHash.Create` is byte-identical. (#8293) | None for normal use — `ConsistentHash.Create` already builds the dictionary fully before constructing, so routers/receptionists are unaffected. If you call the `ConsistentHash(SortedDictionary, int)` constructor directly, populate the dictionary before passing it and don't rely on post-construction mutation being visible. |
+| Planned | `fix/gossip-removal-tombstones` | `Akka.Cluster` | Wire + Behavior | Cluster gossip now carries a removal tombstone for every member the leader removes (new `repeated Tombstone tombstones = 7` on the `Gossip` proto message, previously the one vacant field number). A member the leader removed can no longer be put back into the gossip by a peer that has not caught up, which previously blocked convergence permanently and needed a full cluster restart to clear. Tombstones expire after the new `akka.cluster.prune-gossip-tombstones-after` setting (default `24h`), pruned by the leader on a converged tick and reclaimed for good: gossip reception keeps the tombstones of whichever gossip wins the causal comparison, so a peer that has not pruned yet cannot hand a pruned tombstone back. Tombstones are only unioned across the two gossips when neither descends from the other -- equal clocks, or concurrent clocks, where each side may hold a removal the other has not heard about. The field is additive and proto3 ignores unknown fields in both directions, so rolling upgrades are safe; an older node parses the gossip fine but drops the tombstones when it re-emits it, and if that stripped gossip is causally newer than an upgraded node's, the upgraded node adopts it and loses those tombstones too -- so the protection only holds between upgraded nodes and the fix is not fully in effect until every node is upgraded. | No action required. Raise `akka.cluster.prune-gossip-tombstones-after` if the cluster must survive partitions longer than 24 hours, or lower it if a cluster with heavy join/leave churn needs the gossip message kept small -- each tombstone adds a full address plus a timestamp to every gossip message until it expires. |
 | Planned | `feature/default-bounded-shard-rebalancing` | `Akka.Cluster.Sharding` | Behavior | The default `rebalance-absolute-limit` is now `20`, selecting the bounded shard allocation strategy instead of the legacy threshold-based strategy. | To retain the legacy strategy temporarily, explicitly set `akka.cluster.sharding.least-shard-allocation-strategy.rebalance-absolute-limit = 0`. Review `rebalance-threshold` and `max-simultaneous-rebalance`, which do not apply while the bounded strategy is active. |
 | Planned | `fix/artery-inbound-quarantine-check` | `Akka.Remote` (Artery) | Behavior | Quarantine is now enforced on the INBOUND path too. Previously Artery only gated outbound sends -- an envelope arriving FROM a uid this system has quarantined was still delivered, and the quarantined peer was only notified once, proactively, at the moment `Quarantine()` was called. A new `InboundQuarantineCheckStage`, woven into the inbound pipeline right after handshake, now drops every inbound envelope (ordinary or control, including system messages) whose origin uid is quarantined, and reactively re-sends a `Quarantined` control notice to the origin for each drop (except for a heartbeat or the peer's own `Quarantined` notice, to avoid a reply storm). The existing one-shot proactive notice in `Quarantine()` is unchanged. Additionally, an ordinary/large outbound stream that terminates while its association is quarantined no longer wedges permanently: the materialize-once gate is released (timer-driven auto-reconnect stays suppressed), so a quarantine-piercing `ActorSelection` send -- or any send after a new incarnation's handshake lifts the quarantine -- re-materializes the stream on demand and can reach a restarted peer at the same address. | No action required -- this is a bug fix restoring the documented "no further communication" guarantee of quarantine and the documented new-incarnation piercing behavior; code that (incorrectly) depended on a quarantined peer's replies still arriving is unsupported. |
 | Planned | `fix/artery-daemonmsgcreate-control-stream` | `Akka.Remote` (Artery) | Behavior | Remote deployment's `DaemonMsgCreate` now travels over Artery's CONTROL stream (as a plain envelope, no delivery/ack sequencing) instead of the ordinary stream, ordering it ahead of the `Watch` that remote deployment sends immediately afterwards. Previously the two rode independent, unordered TCP connections and `Watch` systematically arrived first, so the receiver replied `DeathWatchNotification(existenceConfirmed: false)` for a not-yet-created actor and the deployer reaped the freshly-deployed routee before its `Supervise` registration landed, emptying cluster router pools. Additionally (Pekko parity), inbound ordinary messages addressed to a remote-deployed recipient that has not been created yet are no longer dead-lettered immediately: the resolve is retried on a bounded schedule (20 attempts x 50ms, buffered per recipient in FIFO order) so first messages that arrive ahead of the in-flight `DaemonMsgCreate` are delivered once the actor exists; paths that never resolve are banned and dead-letter as before. | No action required -- this is a bug fix restoring correct create-before-watch ordering; code that (incorrectly) depended on the old race is unsupported. |

Directory.Build.props

@@ -41,6 +41,7 @@
     <ProduceReferenceAssembly>true</ProduceReferenceAssembly>
     <FsCheckVersion>3.3.3</FsCheckVersion>
     <FsCheck3Version>3.3.3</FsCheck3Version>
+    <CsCheckVersion>4.8.0</CsCheckVersion>
     <HoconVersion>2.0.3</HoconVersion>
     <ConfigurationManagerVersion>6.0.1</ConfigurationManagerVersion>
     <MicrosoftLibVersion>[6.0.*,)</MicrosoftLibVersion>

src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCluster.DotNet.verified.txt

@@ -213,6 +213,7 @@ namespace Akka.Cluster
         public System.Collections.Immutable.ImmutableDictionary<string, int> MinNrOfMembersOfRole { get; }
         public int MonitoredByNrOfMembers { get; }
         public System.TimeSpan PeriodicTasksInitialDelay { get; }
+        public System.TimeSpan PruneGossipTombstonesAfter { get; }
         public System.TimeSpan? PublishStatsInterval { get; }
         public int ReduceGossipDifferentViewProbability { get; }
         public System.TimeSpan? RetryUnsuccessfulJoinAfter { get; }
@@ -257,6 +258,7 @@ namespace Akka.Cluster
         public override string ToString() { }
         public static Akka.Cluster.Member HighestPriorityOf(Akka.Cluster.Member m1, Akka.Cluster.Member m2) { }
         public static System.Collections.Immutable.ImmutableHashSet<Akka.Cluster.Member> PickHighestPriority(System.Collections.Generic.IEnumerable<Akka.Cluster.Member> a, System.Collections.Generic.IEnumerable<Akka.Cluster.Member> b) { }
+        public static System.Collections.Immutable.ImmutableHashSet<Akka.Cluster.Member> PickHighestPriority(System.Collections.Generic.IEnumerable<Akka.Cluster.Member> a, System.Collections.Generic.IEnumerable<Akka.Cluster.Member> b, System.Collections.Immutable.IImmutableSet<Akka.Cluster.UniqueAddress> tombstones) { }
         public static Akka.Cluster.Member PickNextTransition(Akka.Cluster.Member a, Akka.Cluster.Member b) { }
         public static System.Collections.Immutable.ImmutableSortedSet<Akka.Cluster.Member> PickNextTransition(System.Collections.Generic.IEnumerable<Akka.Cluster.Member> a, System.Collections.Generic.IEnumerable<Akka.Cluster.Member> b) { }
     }

src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCluster.Net.verified.txt

@@ -213,6 +213,7 @@ namespace Akka.Cluster
         public System.Collections.Immutable.ImmutableDictionary<string, int> MinNrOfMembersOfRole { get; }
         public int MonitoredByNrOfMembers { get; }
         public System.TimeSpan PeriodicTasksInitialDelay { get; }
+        public System.TimeSpan PruneGossipTombstonesAfter { get; }
         public System.TimeSpan? PublishStatsInterval { get; }
         public int ReduceGossipDifferentViewProbability { get; }
         public System.TimeSpan? RetryUnsuccessfulJoinAfter { get; }
@@ -257,6 +258,7 @@ namespace Akka.Cluster
         public override string ToString() { }
         public static Akka.Cluster.Member HighestPriorityOf(Akka.Cluster.Member m1, Akka.Cluster.Member m2) { }
         public static System.Collections.Immutable.ImmutableHashSet<Akka.Cluster.Member> PickHighestPriority(System.Collections.Generic.IEnumerable<Akka.Cluster.Member> a, System.Collections.Generic.IEnumerable<Akka.Cluster.Member> b) { }
+        public static System.Collections.Immutable.ImmutableHashSet<Akka.Cluster.Member> PickHighestPriority(System.Collections.Generic.IEnumerable<Akka.Cluster.Member> a, System.Collections.Generic.IEnumerable<Akka.Cluster.Member> b, System.Collections.Immutable.IImmutableSet<Akka.Cluster.UniqueAddress> tombstones) { }
         public static Akka.Cluster.Member PickNextTransition(Akka.Cluster.Member a, Akka.Cluster.Member b) { }
         public static System.Collections.Immutable.ImmutableSortedSet<Akka.Cluster.Member> PickNextTransition(System.Collections.Generic.IEnumerable<Akka.Cluster.Member> a, System.Collections.Generic.IEnumerable<Akka.Cluster.Member> b) { }
     }

src/core/Akka.Cluster.Tests/Akka.Cluster.Tests.csproj

@@ -17,6 +17,7 @@
     <PackageReference Include="xunit.v3" Version="$(Xunit3Version)" />
     <PackageReference Include="xunit.runner.visualstudio" Version="$(Xunit3RunnerVersion)" />
     <PackageReference Include="FsCheck.Xunit.v3" Version="$(FsCheck3Version)" />
+    <PackageReference Include="CsCheck" Version="$(CsCheckVersion)" />
     <PackageReference Include="FluentAssertions" Version="$(FluentAssertionsVersion)" />
     <PackageReference Include="Fsharp.Core" Version="$(FsharpVersion)" />
   </ItemGroup>

src/core/Akka.Cluster.Tests/ClusterConfigSpec.cs

@@ -37,6 +37,7 @@ public void Clustering_must_be_able_to_parse_generic_cluster_config_elements()
             settings.PeriodicTasksInitialDelay.Should().Be(1.Seconds());
             settings.GossipInterval.Should().Be(1.Seconds());
             settings.GossipTimeToLive.Should().Be(2.Seconds());
+            settings.PruneGossipTombstonesAfter.Should().Be(TimeSpan.FromHours(24));
             settings.HeartbeatInterval.Should().Be(1.Seconds());
             settings.MonitoredByNrOfMembers.Should().Be(9);
             settings.HeartbeatExpectedResponseAfter.Should().Be(1.Seconds());