Skip to content

Fix ClusterClientDiscovery: preserve contact-point subscriptions across rediscovery - #8426

Merged
Aaronontheweb merged 2 commits into
akkadotnet:devfrom
Aaronontheweb:fix/clusterclient-discovery-subscription-survival
Jul 24, 2026
Merged

Aaronontheweb merged 2 commits into
akkadotnet:devfrom
Aaronontheweb:fix/clusterclient-discovery-subscription-survival

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Member

Closes #8425.

Problem

With initial-contacts discovery enabled, ClusterClientDiscovery supervises the real ClusterClient as a child and recreates that child from scratch on every rediscovery (when the current child exhausts its reconnect-timeout and stops). The contact-point subscriber list lives on the child, and the supervisor kept no record of it — so a SubscribeContactPoints subscriber silently stopped receiving ContactPoints / ContactPointAdded / ContactPointRemoved events once the client rediscovered. GetContactPoints was unaffected (the supervisor forwards each query to the current child); only the durable subscription was dropped.

Fix

ClusterClientDiscovery now tracks contact-point subscribers at the supervisor level and re-establishes them on each newly-created child:

  • SubscribeContactPoints / UnsubscribeContactPoints are recorded in a supervisor-level subscriber set (and DeathWatched, so a terminated subscriber is dropped — auto-unsubscribe), then still forwarded to the current child so live behavior is unchanged.
  • When a new child is created on rediscovery, the supervisor re-sends SubscribeContactPoints on each tracked subscriber's behalf, so the child registers them and replies with the current snapshot.

No public API change; internal supervisor behavior only.

Testing

ClusterClientDiscoverySpec is reworked to verify contact points through the client's own subscription event stream (SubscribeContactPoints → ContactPoints/ContactPointAdded/ContactPointRemoved) instead of polling GetContactPoints. Its second and third phases exercise rediscovery after a graceful down and a hard shutdown, so the subscription must survive the child being recreated for the test to observe the new node — i.e. the spec both de-flakes the old orphaned inner-timeout poll and directly covers this fix. Verified locally, 4/4 across repeated runs.

…ss rediscovery

ClusterClientDiscovery supervises the real ClusterClient as a child and recreates that child from
scratch on every rediscovery (when the current child exhausts its reconnect-timeout and stops). The
contact-point subscriber list lives on the child, so a SubscribeContactPoints subscriber silently
stopped receiving ContactPoints/ContactPointAdded/ContactPointRemoved once the client rediscovered —
the fresh child had no record of it.

Track contact-point subscribers at the supervisor level and re-subscribe them onto each newly-created
child (on the subscriber's behalf, so the child registers them and replies with the current snapshot).
DeathWatch each subscriber so a terminated subscriber is dropped from tracking (auto-unsubscribe).
Subscribe/Unsubscribe are still forwarded to the current child, so live behavior is unchanged; only the
across-rediscovery gap is closed.
…ents

Replace the three GetContactPoints polling blocks (each with an orphaned, hand-tightened 1s inner
ExpectMsg that spuriously timed out on loaded CI agents) with a single event-driven helper: subscribe
via SubscribeContactPoints and wait on the client's own ContactPoints/ContactPointAdded/ContactPointRemoved
stream until the contact points settle to exactly the expected node. No per-attempt reply timeout to race.

This also covers the ClusterClientDiscovery subscription-survival fix: the second and third phases exercise
rediscovery after a graceful down and a hard shutdown, so the subscription must survive the child being
recreated for the helper to observe the new node. Verified locally, 4/4 across repeated runs.
@Aaronontheweb
Aaronontheweb merged commit 810c00a into akkadotnet:dev Jul 24, 2026
11 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/clusterclient-discovery-subscription-survival branch July 24, 2026 19:15
Aaronontheweb added a commit that referenced this pull request Aug 11, 2026
…ss rediscovery (#8426)

* Fix ClusterClientDiscovery: preserve contact-point subscriptions across rediscovery

ClusterClientDiscovery supervises the real ClusterClient as a child and recreates that child from
scratch on every rediscovery (when the current child exhausts its reconnect-timeout and stops). The
contact-point subscriber list lives on the child, so a SubscribeContactPoints subscriber silently
stopped receiving ContactPoints/ContactPointAdded/ContactPointRemoved once the client rediscovered —
the fresh child had no record of it.

Track contact-point subscribers at the supervisor level and re-subscribe them onto each newly-created
child (on the subscriber's behalf, so the child registers them and replies with the current snapshot).
DeathWatch each subscriber so a terminated subscriber is dropped from tracking (auto-unsubscribe).
Subscribe/Unsubscribe are still forwarded to the current child, so live behavior is unchanged; only the
across-rediscovery gap is closed.

* ClusterClientDiscoverySpec: verify contact points via subscription events

Replace the three GetContactPoints polling blocks (each with an orphaned, hand-tightened 1s inner
ExpectMsg that spuriously timed out on loaded CI agents) with a single event-driven helper: subscribe
via SubscribeContactPoints and wait on the client's own ContactPoints/ContactPointAdded/ContactPointRemoved
stream until the contact points settle to exactly the expected node. No per-attempt reply timeout to race.

This also covers the ClusterClientDiscovery subscription-survival fix: the second and third phases exercise
rediscovery after a graceful down and a hard shutdown, so the subscription must survive the child being
recreated for the helper to observe the new node. Verified locally, 4/4 across repeated runs.
This was referenced Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ClusterClientDiscovery drops contact-point subscriptions across rediscovery

1 participant