Cluster.Sharding: a coordinator hand-over only stops the local ShardRegion when the node is leaving

[Aaronontheweb]

What changes

ShardCoordinator.HandleTerminate no longer treats every termination message from its ClusterSingletonManager as a node shutdown. It still stops the coordinator on every hand-over, which is what lets the singleton manager complete the hand-over. It only tears the node's own ShardRegion down with it when that region is already handing off, which is the coordinated-shutdown path, or when the node itself is leaving: the actor system is terminating, CoordinatedShutdown is running, or the self member is Leaving, Exiting, or Down. On a hand-over that is not a shutdown the region keeps running and the coordinator logs one debug line saying so.

A new single-node spec, ShardCoordinatorHandOverSpec, sends the singleton's termination message the way the manager does and asserts that the coordinator stops and the region does not. On dev it fails with "Expected no messages, instead we received Terminated [.../system/sharding/Entity]".

No public API, wire, or configuration change, so no ledger row; this restores the intent of a block that was written for the leaving case.

Why

Fixes #8583. ClusterShardingMinMembersPerRoleNotConfiguredSpec has failed six times in two days on PRs that do not touch sharding, on both transports. The chain is the same in every log:

1. While the cluster forms with min-nr-of-members = 2, two nodes act as leader on disjoint views and assign conflicting up-numbers. Two members of the same role both reach Oldest for the coordinator singleton.
2. Gossip merges, the younger one hands over (Oldest -> WasOldest -> HandingOver) and stays Up. The manager sends the coordinator its termination message, as it does on every hand-over.
3. The coordinator reads that as "this node is shutting down", sends its own live region GracefulShutdown, and defers its stop until the region is gone.
4. ClusterShardingGuardian watches the region and removes it from ClusterSharding's cache when it stops. From then on ClusterSharding.Get(sys).ShardRegion(typeName) throws on that node for the life of the process.

Steps 1 and 2 are by design and match Pekko line for line: the TakeOverFromMe / HandOverToMe exchange exists for exactly this, and it settled in about half a second in every log. Step 3 is the fault. The branch that sends GracefulShutdown to a live local region came from JVM Akka PR 30338 in 2021, a latency optimization for rolling updates whose own issue text says things work fine without it. Before that a termination message was a bare stop, which is still what the DData and persistent coordinators do on their other stop paths. So the change routes a benign hand-over back to what the coordinator did for six years before that optimization, and keeps the optimization on every real shutdown route.

The same gate also covers LeaseLost in the singleton manager, which sends the same termination message on a route that is not a shutdown at all.

Neither #8359 nor #8580 is involved: in every new log the racing pair is two joiners, never the self-joiner, and the up-number assignment, leader choice, and convergence check are unchanged and identical to upstream.

Safety on the real shutdown routes

Route	What the gate reads when the termination message arrives
CoordinatedShutdown.Run, any reason	ShutdownReason is set by the compare-and-swap at the top of Run, before any phase
Cluster.Leave(self), region drained first	the region is already in the hand-off set; that branch is unchanged and unconditional
Cluster.Leave(self), peer's HandOverToMe beats our own shutdown phase	self is Leaving. Leaving is part of the convergence set, so the leader can only move us to Exiting after we have seen that gossip, and a peer only becomes oldest on MemberExited
self downed by a peer	Down, or a stale Up; a stale read costs nothing because the region already skips its own drain when self is Down
actor system terminating	Cluster.IsTerminated

Removed is deliberately not in the check: the read view synthesizes a Removed self member whenever self is absent from its snapshot, including before the node has joined, and a real removal stops the singleton manager outright.

How it was checked

dotnet build src/contrib/cluster/Akka.Cluster.Sharding.Tests -c Release -warnaserror clean. The new spec fails on dev with the region's Terminated and passes with the change, ten of ten. The graceful-shutdown, leaving, and coordinated-shutdown specs in the unit project pass, and the multi-node ClusterShardingGracefulShutdownOldestSpec, ClusterShardingGracefulShutdownSpec, and ClusterShardingLeavingSpec pass; the first of those is the upstream gate for the optimization this touches. ClusterShardingMinMembersPerRoleNotConfiguredSpec, the spec that surfaced the fault, passes. The full Akka.Cluster.Sharding.Tests suite passes. Akka.API.Tests passes with no approval change.

[Aaronontheweb]

LGTM - needs backport to v1.5