rewrite annotation not working if multiple ingresses use the same backend

[skoczko]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Given two ingresses that point to the same service but only one uses the rewrite annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin-1
  annotations:
    konghq.com/rewrite: /get
spec:
  ingressClassName: kong
  rules:
  - host: httpbin.example.com
    http:
      paths:
      - path: /rewrite
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 8000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin-2
  annotations:
    konghq.com/strip-path: "true"
spec:
  ingressClassName: kong
  rules:
  - host: httpbin.example.com
    http:
      paths:
      - path: /regular
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 8000

Requests both to httpbin.example.com/rewrite and httpbin.example.com/regular seem to randomly apply the same rewrite. This means that request to httpbin.example.com/rewrite will occasionally faily with 404 from httpbin and the request httpbin.example.com/regular will occasionally rewrite to /get while other times serve the httpbin root.

Expected Behavior

The rewrite is only and consistently applied for httpbin.example.com/rewrite

Steps To Reproduce

Use curl and request `httpbin.example.com/rewrite` and `httpbin.example.com/regular` 10 times each.

Kong Ingress Controller version

2.12.1

Kong version: 3.4

Kubernetes version

k version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.25.11

Anything else?

No response

[skoczko]

I belive this #5021 is a manifestation of the same bug.

[skoczko]

Note that I have tested the same setup with two different httpbin deployments (httpbin1 and httpbin2) and the problem is gone so the issue is caused by using the same backend on two ingresses.

[mflendrich]

Thank you @skoczko for your report and investigation. It is confirmed that KIC should apply the request-transformer plugin on the route and not on the service level. I'm going to close this issue as a duplicate and we'll track the fix in #5021, as it currently has more low-level details about the root cause that will be helpful for the resolution.