ExternalWorkload's Endpoint not getting a resolved IP to VM

[SamKirsch10]

Hello. Thanks for everyone's help on slack for all the initial debugging! I am trying to mostly follow this. Our POC is located on a legacy GKE cluster. This caveat means it is NOT VPC-native (and thus direct VM to pod IP communication will not work). Because of this, I have done two additional things to get to where I am now:

1. I exposed the destination pod's destination and policy containers via an internal LB service. This basically copied the existing service the proxy was going to target, and just exposed it via an internal LB. I added this internal IP resolution to the /etc/hosts file of the server

internal lb(which ends up getting IP 10.156.0.82):

apiVersion: v1
kind: Service
metadata:
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
    component: destination
  name: linkerd-ext-lb
  namespace: linkerd
spec:
  ports:
    - name: dst
      port: 8086
      protocol: TCP
      targetPort: 8086
    - name: policy
      port: 8090
      protocol: TCP
      targetPort: 8090
  selector:
    linkerd.io/control-plane-component: destination
  sessionAffinity: None
  type: LoadBalancer

/etc/hosts:

10.156.0.82 linkerd-dst-headless.linkerd.svc.cluster.local
10.156.0.82 linkerd-policy.linkerd.svc.cluster.local

2. I then setup a dnsmasq deployment in our GKE cluster to expose the .svc.cluster.local lookups and added it as a condition to the VM's dnsmasq. SO, after the proxy started up and connected, it could then successfully lookup K8s DNS entries.

The following is what was deployed for the VM to connect in.

spire stuff:

./bin/spire-server entry create -parentID spiffe://root.linkerd.cluster.local/agent \
      -spiffeID spiffe://root.linkerd.cluster.local/external-workload -selector unix:uid:$(id -u root)

confirmed with:

./bin/spire-server entry show -spiffeID "spiffe://root.linkerd.cluster.local/external-workload"
Found 1 entry
Entry ID         : 4fa420bd-a7ab-4245-9d54-1306f1d7d6e9
SPIFFE ID        : spiffe://root.linkerd.cluster.local/external-workload
Parent ID        : spiffe://root.linkerd.cluster.local/agent
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : unix:uid:0

apiVersion: workload.linkerd.io/v1beta1
kind: ExternalWorkload
metadata:
  name: external-workload
  namespace: linkerd
  labels:
    location: vm
    app: legacy-app
    workload_name: external-workload
spec:
  meshTLS:
    identity: "spiffe://root.linkerd.cluster.local/external-workload"
    serverName: "external-workload.linkerd.cluster.local"
  workloadIPs:
    - ip: 10.156.0.81
  ports:
  - port: 8080
    name: http
status:
  conditions:
    - type: Ready
      status: "True"
      lastTransitionTime: "2024-01-24T11:53:43Z"
      reason: "Alive"
      message: "This workload is alive"
---
apiVersion: v1
kind: Service
metadata:
  name: external-workload
  namespace: linkerd
spec:
  type: ClusterIP
  selector:
    workload_name: external-workload
  ports:
    - port: 8080
      protocol: TCP
      name: http
---
apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
  name: external-workloads
  namespace: linkerd
  annotations:
    config.linkerd.io/default-inbound-policy: allow
spec:
  externalWorkloadSelector:
    matchLabels:
      workload_name: external-workload
  port: http
  proxyProtocol: HTTP/1

I am now able to curl from the VM into the cluster via a K8s endpoint (an example echo server endpoint that's mesh'd):

curl -v echo-server.default.svc.cluster.local:10360
*   Trying 10.67.242.22:10360...
* Connected to echo-server.default.svc.cluster.local (10.67.242.22) port 10360 (#0)
> GET / HTTP/1.1
> Host: echo-server.default.svc.cluster.local:10360
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< x-trace-id: 0ee03a8af4c4037663d18280b44c6f9c
< date: Mon, 22 Jul 2024 13:05:46 GMT
< content-length: 205
<
{
    "headers": {
        "accept": "*/*",
        "host": "echo-server.default.svc.cluster.local:10360",
        "user-agent": "curl/7.88.1"
    },
    "method": "GET",
    "path": "/",
    "query": ""
* Connection #0 to host echo-server.default.svc.cluster.local left intact
}

However, I am not able to curl from the echo-server (get 504 and eventually 503s). Running kubectl -n linkerd get endpints, you can see the external-workload has no endpoint. I'd assume this needs to be the VM itself (which is 10.156.0.81 from the ExtnernalWorkload object).

NAME                        ENDPOINTS                           AGE
external-workload           <none>                              2d23h
kube-dns-int-lb             10.64.23.14:5353                    2d21h
linkerd-dst                 10.64.0.17:8086                     5d
linkerd-dst-headless        10.64.0.17:8086                     5d
linkerd-ext-lb              10.64.0.17:8086,10.64.0.17:8090     3d19h
linkerd-gateway             10.64.4.16:4191,10.64.4.16:4143     4d22h
linkerd-identity            10.64.18.21:8080                    5d
linkerd-identity-headless   10.64.18.21:8080                    5d
linkerd-policy              10.64.0.17:8090                     5d
linkerd-policy-validator    10.64.0.17:9443                     5d
linkerd-proxy-injector      10.64.18.25:8443                    5d
linkerd-sp-validator        10.64.0.17:8443                     5d
metrics-api                 10.64.11.26:8085                    4d23h
tap                         10.64.11.28:8088,10.64.11.28:8089   4d23h
tap-injector                10.64.11.16:8443                    4d23h
web                         10.64.0.12:9994,10.64.0.12:8084     4d23h

If I try and dig deeper into that object with kubectl -n linkerd get endpoints external-workload -o yaml, I don't get much:

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2024-07-19T13:15:26Z"
  creationTimestamp: "2024-07-19T13:15:26Z"
  name: external-workload
  namespace: linkerd
  resourceVersion: "960348867"
  uid: 6338ac3d-9d5a-46dd-a693-59091dd9bfa4

I've turned on debug logging on the policy and destination container's to see if there's anything that pops out, but I do not see much. What parts of this deployment resolve the finalized Endpoint?

[kflynn]

Hey @SamKirsch10! Several things come immediately to mind here:

0. Your external workload is a Pod in a GKE cluster? If that's true, Linkerd multicluster is likely to be much simpler than mesh expansion -- have you looked into that?
1. Can you include the output of kubectl get externalworkload -n linkerd external-workload -o yaml ?
2. I've actually never tried putting an ExternalWorkload in the linkerd namespace – generally speaking, linkerd is for Linkerd itself, and things for your application should go in some other namespace. (I don't know that this would break things, but if you put app resources in the linkerd namespace you're going to regret it -- maybe not today, maybe not tomorrow, but soon, and for the rest of your life. 😇)
3. I don't see you talking about running the proxy on your external workload, but this ties back to point 0...

[mateiidavid]

@SamKirsch10 ahhh, so I think what you're seeing (EndpointSlice and Endpoints) are all the k8s generated ones. Since you have a label selector on your service, they're generated and will be filled by the core Kubernetes controller when there's a pod backing the Service. You can see in the example above the endpointslice is managed by the k8s controller, not us.

Since you provided your manifests, I gave this a quick go in my local environment.

✦ 2 :; k apply -f tmp-repro.yaml
externalworkload.workload.linkerd.io/external-workload created
service/external-workload created
server.policy.linkerd.io/external-workloads created

matei ~/w/linkerd2 matei/prototype-cluster-agnostic-svc*​​ ≡
✦ 2 :; k get endpointslices -n linkerd
NAME                                       ADDRESSTYPE   PORTS     ENDPOINTS     AGE
linkerd-identity-fzvsl                     IPv4          8080      10.23.0.7     34d
linkerd-identity-headless-4cl95            IPv4          8080      10.23.0.7     34d
linkerd-proxy-injector-7zc9c               IPv4          8443      10.23.0.9     34d
linkerd-dst-headless-vn69q                 IPv4          8086      10.23.0.8     34d
linkerd-sp-validator-5wt56                 IPv4          8443      10.23.0.8     34d
linkerd-policy-validator-8zxgt             IPv4          9443      10.23.0.8     34d
linkerd-policy-k6x25                       IPv4          8090      10.23.0.8     34d
linkerd-dst-r8rtt                          IPv4          8086      10.23.0.8     34d
linkerd-external-external-workload-2kr95   IPv4          8080      10.156.0.81   7s

Do you have any endpoint slices that are prefixed with linkerd-external-?

✦ 2 :; k get endpointslices -n linkerd linkerd-external-external-workload-2kr95 -o yaml
addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
- addresses:
  - 10.156.0.81
  conditions:
    ready: false
    serving: false
    terminating: false
  targetRef:
    kind: ExternalWorkload
    name: external-workload
    namespace: linkerd
    uid: 0c59b7f6-233c-4ba4-a8de-4e3c739e3e31
kind: EndpointSlice
metadata:
  creationTimestamp: "2024-07-23T14:20:13Z"
  generateName: linkerd-external-external-workload-
  generation: 1
  labels:
    endpointslice.kubernetes.io/managed-by: linkerd-external-workloads-controller
    kubernetes.io/service-name: external-workload
  name: linkerd-external-external-workload-2kr95
  namespace: linkerd
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Service
    name: external-workload
    uid: d5fae9cf-9f8d-4f86-9ef1-0e5392149455
  resourceVersion: "2449058"
  uid: 8a2919ab-fbf4-4ffc-bb8e-265347ca9369
ports:
- name: http
  port: 8080
  protocol: TCP

This is what I see in my cluster, for example. The 'owner' label is different here and indicates linkerd's controller is responsible for writing state to the slice.

[SamKirsch10]

Ah! Good catch @mateiidavid. I do have one of those!

$ k get endpointslice linkerd-external-external-workload-xfk76 -o yaml
addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
- addresses:
  - 10.156.0.81
  conditions:
    ready: true
    serving: true
    terminating: false
  targetRef:
    kind: ExternalWorkload
    name: external-workload
    namespace: default
    uid: 806bb7f3-8b02-42b4-b2b3-0a5b09aa06b3
kind: EndpointSlice
metadata:
  creationTimestamp: "2024-07-23T12:39:06Z"
  generateName: linkerd-external-external-workload-
  generation: 2
  labels:
    endpointslice.kubernetes.io/managed-by: linkerd-external-workloads-controller
    kubernetes.io/service-name: external-workload
  name: linkerd-external-external-workload-xfk76
  namespace: default
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Service
    name: external-workload
    uid: 78db0f64-4a6a-4c48-a4a9-f70ba31d6529
  resourceVersion: "966264034"
  uid: ef8ad140-73d5-443a-8cb1-0edb8a739a2b
ports:
- name: http
  port: 8080
  protocol: TCP

[SamKirsch10]

And now me curling the external-workload svc is working! Weird that it's now working, but it is

[SamKirsch10]

This was all incredibly helpful information for us, and now I can continue to go down the rabbit hole a bit to expand the PoC. Thanks x1000000 @mateiidavid & @kflynn

[mateiidavid]

Great! Happy it all worked out. Perhaps the state wasn't synchronised yet or something. Let us know if you find any issues or if you have any questions.