linkerd fails to find Lease component for service-mirror

[hawkw]

Discussed in #11503

^{Originally posted by lboutin-mwm October 18, 2023}
Hello,

I use the linkerd multicluster plugin to ease communications between some of my clusters and deploy it using its helm chart.

I recently upgraded the stack to stable-2.14.1 (from stable-2.14.0) and one of my clusters is unable to find the lease created when linking two clusters:

This is the result of linkerd multicluster check on my central cluster:

linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
        * ml-demos-us
        * ml-staging-us
√ remote cluster access credentials are valid
        * ml-demos-us
        * ml-staging-us
√ clusters share trust anchors
        * ml-demos-us
        * ml-staging-us
√ service mirror controller has required permissions
        * ml-demos-us
        * ml-staging-us
√ service mirror controllers are running
        * ml-demos-us
        * ml-staging-us
√ probe services able to communicate with all gateway mirrors
        * ml-demos-us
        * ml-staging-us
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match

Status check results are √

and this is the result of the same command on my faulty cluster:

linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
        * tools-eu
√ remote cluster access credentials are valid
        * tools-eu
√ clusters share trust anchors
        * tools-eu
√ service mirror controller has required permissions
        * tools-eu
√ service mirror controllers are running
        * tools-eu
× probe services able to communicate with all gateway mirrors
        failed to get the service-mirror component Lease for target cluster tools-eu: leases.coordination.k8s.io "service-mirror-write-tools-eu" not found
    see https://linkerd.io/2.14/checks/#l5d-multicluster-gateways-endpoints for hints
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match

Status check results are ×

For some reason it seems to work one way and not the other.
The lease does in fact exist:

kctl get leases -n linkerd-multicluster
NAME                            HOLDER                                             AGE
service-mirror-write-tools-eu   linkerd-service-mirror-tools-eu-7854857ffc-j6c8k   44m

and I can get its description just fine:

Name:         service-mirror-write-tools-eu
Namespace:    linkerd-multicluster
Labels:       <none>
Annotations:  <none>
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2023-10-18T15:28:29Z
  Managed Fields:
    API Version:  coordination.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:acquireTime:
        f:holderIdentity:
        f:leaseDurationSeconds:
        f:leaseTransitions:
        f:renewTime:
    Manager:         controller
    Operation:       Update
    Time:            2023-10-18T16:14:20Z
  Resource Version:  178043801
  UID:               99550f50-0d88-4076-a7fb-1abcc07c7386
Spec:
  Acquire Time:            2023-10-18T15:28:29.396070Z
  Holder Identity:         linkerd-service-mirror-tools-eu-7854857ffc-j6c8k
  Lease Duration Seconds:  30
  Lease Transitions:       0
  Renew Time:              2023-10-18T16:14:20.345628Z
Events:                    <none>

All of the aforementioned clusters are running in GKE with version 1.25.12-gke.500 of k8s

These are the commands I used to link both clusters:
linkerd --context=tools-eu multicluster link --cluster-name tools-eu | kubectl --context=ml-demos-us apply -f -
linkerd --context=ml-demos-us multicluster link --cluster-name ml-demos-us | kubectl --context=tools-eu apply -f -

Only one of three clusters connected to the central one is affected and there was no issue with the earlier 2.14.0 version of linkerd.
It also does seem like the mirroring and connection between clusters is working, this might just only be an error with the check command