Linkerd is altering status on HTTPRoutes it doesn't own

[quyenhoang96]

What is the issue?

When I config Envoy-Gateway https://github.com/envoyproxy/gateway.
Then I apply HTTPRoute, the control plan of linkerd watch resource of envoy-gateway, and change it. So it make the envoy-gateway error. Help me how to fix it.

How can it be reproduced?

When I config Envoy-Gateway https://github.com/envoyproxy/gateway.
Then I apply HTTPRoute, the control plan of linkerd watch resource of envoy-gateway, and change it. So it make the envoy-gateway error. Help me how to fix it.

Logs, error output, etc

When I config Envoy-Gateway https://github.com/envoyproxy/gateway.
Then I apply HTTPRoute, the control plan of linkerd watch resource of envoy-gateway, and change it. So it make the envoy-gateway error. Help me how to fix it.

output of linkerd check -o short

• name: linkerd-crds
  repository: "https://helm.linkerd.io/stable"
  version: 1.8.0
• name: linkerd-control-plane
  repository: https://helm.linkerd.io/stable
  version: 1.16.6

Environment

• name: linkerd-crds
  repository: "https://helm.linkerd.io/stable"
  version: 1.8.0
• name: linkerd-control-plane
  repository: https://helm.linkerd.io/stable
  version: 1.16.6

Possible solution

No response

Additional context

When I config Envoy-Gateway https://github.com/envoyproxy/gateway.
Then I apply HTTPRoute, the control plan of linkerd watch resource of envoy-gateway, and change it. So it make the envoy-gateway error. Help me how to fix it.

Would you like to work on fixing this bug?

yes

[kflynn]

I'm going to resolve this as a dup of envoyproxy/gateway#2248 – I'm involved with Envoy Gateway, too, so I think we can track it in just the one project. 🙂

[kflynn]

Upon investigation... yeah, we think that this is a Linkerd bug. 🙁 Reopening this issue.

How to reproduce

Start with an empty cluster.

1. Install Linkerd.

Install Linkerd without messing with Gateway API CRDs:

linkerd install --crds --set enableHttpRoutes=false | k apply -f -
linkerd install --set enableHttpRoutes=false | k apply -f -

At this point Linkerd will not start running, which surprised me, but the next step fixes that.

2. Install Envoy Gateway.

This is from the Envoy Gateway quickstart:

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 \
             -n envoy-gateway-system --create-namespace

(Note that eg here is an abbreviation for Envoy Gateway, not "for example".)

3. Wait for Linkerd and Envoy Gateway to be running.

kubectl rollout status -n envoy-gateway deploy
kubectl rollout status -n linkerd deploy

4. Create a GatewayClass, Gateway, and HTTPRoute.

Again from the Envoy Gateway quickstart:

kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default

5. Observe the bug.

kubectl get httproutes.gateway.networking.k8s.io backend -o yaml

The status will show

status:
  parents: []

instead of the proper status from Envoy Gateway.

Hypothesis

It appears that Envoy Gateway correctly notices that our HTTPRoute has an entry in parentRefs pointing to Envoy Gateway, so it does all the reconciliation that it needs to and then updates the HTTPRoute's status.

Linkerd correctly sees the status update, but then incorrectly decides to update the status itself. I think that what Linkerd is trying to do is an empty update with the Merge strategy, but it appears that Merge incorrectly overwrites the parents list with the (defaulted-to-?)empty list from Linkerd.

Overall, I think that Linkerd shouldn't be attempting any update of a route that it doesn't own, and I think that maybe that starts with altering the check in policy-controller/k8s/status/src/index.rs at line 129?

                    } else if id.gkn.group == k8s_gateway_api::HttpRoute::group(&()) {

[kflynn]

Part of this fix is also going to be to add conformance tests for mesh/gateway-controller interaction, as tracked in kubernetes-sigs/gateway-api#2630.)

[quyenhoang96]

Hi @kflynn
Thank you for your assistance. Currently we are deploying linkerd + envoy-gateway for production. Can you push this bug to priority?