Rewrite annotation causing KIC sync loop

[jeffrey-arndt]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Setup:

2 Ingress objects pointing at the same service
1 Ingress with the rewrite annotation
1 Ingress without the annotation

We see that the request transformer plugin is applied at the Service level and the configuration flip flops creating and deleting the plugin in a loop

Video attached showing the behavior:

https://drive.google.com/file/d/1FYwZaSOwT-Lvin9NzTcFN2mnFxds1UAR/view?usp=sharing

Expected Behavior

The the Ingress objects should not conflict causing a scenario where the plugin is deleted/recreated over and over

Additionally, I believe it would be better that the request transformer plugin is applied at the route level. This would allow for more granular configuration

Steps To Reproduce

1) Stand up a KIC + Proxy and enable the rewrite feature gate
  env:
    feature_gates: RewriteURIs=true

2) Stand up an echo service
3) Create 2 Ingress objects:

echo "
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: echo1
     namespace: kong
     annotations:
       konghq.com/strip-path: 'true'
   spec:
     ingressClassName: kong
     rules:
     - host: kong.example
       http:
         paths:
         - path: /echo
           pathType: ImplementationSpecific
           backend:
             service:
               name: echo
               port:
                 number: 1027
   " | kubectl apply -f -

echo "
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: echo2
     namespace: kong
     annotations:
       konghq.com/rewrite: /api/foo/REWRITE
       konghq.com/strip-path: 'true'
   spec:
     ingressClassName: kong
     rules:
     - host: kong.example
       http:
         paths:
         - path: /rewrite
           pathType: ImplementationSpecific
           backend:
             service:
               name: echo
               port:
                 number: 1027
   " | kubectl apply -f -

3) Verify via Kong Manager the plugin is being delete/recreated. Check the KIC logs to see repeated sync succesful messages looping

Kong Ingress Controller version

KIC 2.12 / Gateway 3.4.0.0

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:13:28Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}

Anything else?

No response

[programmer04]

Rewrite is attached in KIC code on service level instead of route level

kubernetes-ingress-controller/internal/dataplane/translator/subtranslator/ingress.go

Lines 444 to 474 in 9ad403f

	// MaybeRewriteURI appends a request-transformer plugin if the value of konghq.com/rewrite annotation is valid.
	func MaybeRewriteURI(service *kongstate.Service, rewriteURIEnable bool) error {
	rewriteURI, exists := annotations.ExtractRewriteURI(service.Parent.GetAnnotations())
	if !exists {
	return nil
	}
	if !rewriteURIEnable {
	return fmt.Errorf("konghq.com/rewrite annotation not supported when rewrite uris disabled")
	}
	if rewriteURI == "" {
	rewriteURI = "/"
	}
	config, err := generateRewriteURIConfig(rewriteURI)
	if err != nil {
	return err
	}
	service.Plugins = append(service.Plugins, kong.Plugin{
	Name: kong.String("request-transformer"),
	Config: kong.Configuration{
	"replace": map[string]string{
	"uri": config,
	},
	},
	})
	return nil
	}

It leads to conflicts as described #5140 (comment)

[programmer04]

@jeffrey-arndt yesterday releases

• v2.12.2
• v3.0.1

contain a fix for the reported bug

[herself]

I'm seeing the same issue with KIC 3.2 + Kong 3.7 on K8s 1.28 and getting the same behaviour.

If I have two Ingresses with the same backing service, the rewrite gets applied randomly on requests. When I change the backing services to different ones, the rewrite works properly again.

Should I open a new bug? Or will this be tracked here? Any additional data I can provide?

[pmalek]

@herself 👋

Yes, please create a new issue specifically mentioning the version that you use. Can you also try with 3.3 if the issue persists?

[herself]

Thanks - unfortunately, it seems to persist on 3.3 as well. Opened #6542, thank you for guidance!