ClusterClient: arm the reconnect deadline per establishing phase via IWithTimers

[Aaronontheweb]

Fixes #8508.

Problem

ClusterClient armed a one-shot ReconnectTimeout on every message processed while establishing — the arming code sat in the Establishing receive delegate, which runs per message, and only the ActorIdentity success branch cancelled (and only its own timer). Every other message leaked one. Strays landing in Active dead-lettered harmlessly; a stray landing after re-entry into Establishing hit Context.Stop(Self) and killed a healthy client mid-reconnect. Masked in production because reconnect-timeout defaults to off; both ClusterClientHandoverSpec CI failures (builds 130927/130934) show the exact arithmetic — the fatal timer armed by the message before the previous successful connect, firing ~600ms into the new establishing phase.

Fix — IWithTimers, per review direction

Raw-scheduler cancelables are replaced by keyed timers (matching ClusterClientDiscovery, which already uses them):

• The reconnect deadline is armed once per entry into the establishing phase (PreStart, Reestablish()) under a fixed key and cancelled on the transition to Active. A fixed key means re-arming replaces rather than accumulates; timers auto-cancel with the actor; and the timer wrapper discards messages from cancelled/replaced timers even if already in the mailbox — closing the race that a plain Cancel() cannot (it can't recall a delivered message).
• Heartbeat and refresh-contacts moved to the same keyed discipline: one timer model for the actor, no manual PostStop cleanup left.
• reconnect-timeout = off (the default) arms nothing.

Fail-first proof

Both new tests fail on the unfixed actor:

1. Deterministic kill: client Active for 7s, receptionist stopped — the leaked timer (armed 10s earlier, in the previous phase) kills the client 2.997s into a fresh 10-second window. Same subtraction as both CI failures.
2. Zero strays: an establish → active cycle on unfixed code publishes four unhandled ReconnectTimeouts; fixed, none.

Measured effect at the leak level

Same 10-round artery ClusterClientHandoverSpec soak under CPU load, counting stray ReconnectTimeout dead letters on the client node:

before: 5,6,6,5,6,0,6,6,6,6  = 52
after:  0,0,0,0,0,0,0,0,0,0  =  0

Handover itself unchanged (connects=2 every run, both before and after). The soak passes either way on a fast box — the kill needs a slow agent — so the deterministic tests carry the proof and the soak carries no-regression plus the leak observable.

API and behavior notes (flagged, not buried)

• Approval delta on the sealed class: +IWithTimers, +Timers { get; set; }, +PreStart override, −PostStop override (it existed only to cancel the hand-managed timers; on a sealed class it was unreachable from consumer code). Recorded in BREAKING_CHANGES_V1.6.md.
• Secondary behavior change, documented in the same entry: a refresh-contacts tick queued under an old cadence is discarded when the interval changes on a state transition, instead of being delivered against the new one — the same stale-timer class as the bug itself.

Verification

Check	Result
New ClusterClientReconnectTimeoutSpec	2/2, both fail-first-proven
Full Akka.Cluster.Tools.Tests	100/100
Akka.API.Tests	18/18 after the flagged approval update
ClusterClientHandoverSpec artery ×10 under load / classic ×4	14/14, zero strays
-warnaserror on both touched projects	0 warnings

With this fix, ClusterClientHandoverSpec comes off the flake ledger as resolved — it was a true positive all along.

[Aaronontheweb]

LGTM