Ingress addon: `Error: server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name` #11121

sherbang · 2021-04-17T00:16:27Z

Steps to reproduce the issue:

Minikube v1.19.0 docker driver on PopOS 20.10

Using the ingress plugin, installing an ingress always fails.

Exact same steps and yaml works in v1.18.1

Doesn't matter if the ingress is defined with apiVersion: networking.k8s.io/v1 or apiVersion: extensions/v1beta1 (yaml files used to test each work in v1.18.1)

Suspect that ansible/awx-operator#205 is the same issue.

Failing ingress definition:

- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.global-static-ip-name":"not-used","networking.gke.io/managed-certificates":"not-used"},"labels":{"app":"cwacc-webapp"},"name":"cwacc-webapp","namespace":"gozynta-dev"},"spec":{"rules":[{"host":"cwacc.minikube.local","http":{"paths":[{"backend":{"service":{"name":"cwacc-webapp","port":{"number":5000}}},"path":"/","pathType":"Prefix"}]}}],"tls":[{"hosts":["cwacc.minikube.local"]}]}}
      kubernetes.io/ingress.global-static-ip-name: not-used
      networking.gke.io/managed-certificates: not-used
    labels:
      app: cwacc-webapp
    name: cwacc-webapp
    namespace: gozynta-dev
  spec:
    rules:
    - host: cwacc.minikube.local
      http:
        paths:
        - backend:
            service:
              name: cwacc-webapp
              port:
                number: 5000
          path: /
          pathType: Prefix
    tls:
    - hosts:
      - cwacc.minikube.local
kind: List
metadata: {}

Full output of failed command:

kubectl error: Error from server (InternalError): error when creating "gozynta-dev/profiles/cwacc": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": an error on the server ("") has prevented the request from succeeding

Error in ingress-controller: E0417 00:03:30.320457 7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name cwacc-webapp in namespace gozynta-dev"

Full output of minikube start command used, if not already included:

😄  [cwacc] minikube v1.19.0 on Debian bullseye/sid
✨  Automatically selected the docker driver. Other choices: kvm2, ssh, none
❗  Your cgroup does not allow setting memory.
    ▪ More information: https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities
👍  Starting control plane node cwacc in cluster cwacc
🔥  Creating docker container (CPUs=3, Memory=4096MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.5 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "cwacc" cluster and "default" namespace by default
✅  minikube profile was successfully set to cwacc
Switched to context "cwacc".
minikube ssh --profile cwacc sudo sysctl fs.inotify.max_user_watches=1048576
fs.inotify.max_user_watches = 1048576
minikube addons enable ingress --profile cwacc
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

Optional: Full output of minikube logs command:

The text was updated successfully, but these errors were encountered:

btalbot · 2021-04-20T20:23:45Z

Same exact failure on macos 11.2.3 with virtualbox dirver as well

robrich · 2021-04-22T05:29:13Z

kubernetes/ingress-nginx#5401 looks like a similar stream, and references errors in the last few days using Minikube.

I worked around it by disabling the Minikube ingress addon and using the ingress-nginx helm chart from Artifact Hub. Now it starts up correctly but doesn't route, and it never gets an address. Minikube 1.19 currently uses ingress-nginx 0.44.0 and Helm's latest is 0.45.0. Maybe it was a bug in ingress-nginx 0.44.0?

balnbibarbi · 2021-04-22T07:37:44Z

Confirm, disabling the Minikube ingress addon and using the ingress-nginx helm chart from Artifact Hub fixes this.
For both the Ingress and LoadBalancer to get an IP address, I have to minikube tunnel.
But avoiding this was my motivation for trying ingress-nginx, so this isn't ideal.

btalbot · 2021-04-22T17:58:13Z

Deleting the ValidatingWebhookConfiguration/ingress-nginx-admission also allows the minikube ingress controller to function but without that safety net of course. Reasonable work-around until the bug is fixed though.

medyagh · 2021-04-24T16:28:21Z

thank you everyone for reporting this bug, this seems to be side effect of this PR #10879 that we tried to update the ingress to new upstream version, unfortunately our integration test did not catch this problem causing by adding the validating webhooks,
Please stay tuned, we will create a fix for this and hopefully by adding an integration test this should never happen again.

Thank you for your patience

and thank you @joshuamataaraya @btalbot @bpb54321 @dgoldssfo for reporting this issue in the other issues, please follow this one for centralized place to gather information

medyagh · 2021-04-24T16:28:29Z

CC: @govargo could this be related to upstream bug kubernetes/ingress-nginx#5401 ?

govargo · 2021-04-24T20:23:06Z

Sorry for this unablement to use ingress addon. I investigated why this happens.

CC: @govargo could this be related to upstream bug kubernetes/ingress-nginx#5401 ?

Yes. This is the issue of the admission webhook.
Currently, upstream ingress-nginx admission controller doesn't allow Ingress of networking.k8s.io/v1.
The closest issue is kubernetes/ingress-nginx#7066

I confirmed the admission controller of ingress-nginx and found out it doesn't support v1 version.
It only supports networking.k8s.io/v1beta version.
https://github.com/kubernetes/ingress-nginx/blob/59922afae3331ee76e44dd93f26c6ea0b2309986/internal/admission/controller/main.go#L46-L52

// this support v1beta1 only. v1 version is not supported
var (
	ingressResource = metav1.GroupVersionKind{
		Group:   networking.GroupName,
		Version: "v1beta1",
		Kind:    "Ingress",
	}
)

As a result of further investigation, It used to support networking.k8s.io/v1 in ingress-nginx v0.40.0 by kubernetes/ingress-nginx#6187 in the past.
However, it was removed in v0.41.0 by kubernetes/ingress-nginx#6366.
(I don't know why this was changed, because the reason wasn't written in the PR description.)

govargo · 2021-04-24T20:28:38Z

And I also confirmed the ingress can route with these yaml files.

# pod and service
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
  namespace: default
spec:
  containers:
  - name: nginx
    image: nginx:alpine
    ports:
    - containerPort: 80
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: nginx
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  sessionAffinity: None
  type: ClusterIP

This worked because the version is v1beta1

# ingress v1beta1
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
  labels:
    integration-test: ingress
spec:
  rules:
  - host: nginx.example.com
    http:
      paths:
      - path: "/"
        backend:
          serviceName: nginx
          servicePort: 80

This doesn't work because the version is v1. the admission controller always rejects networking.k8s.io/v1 request

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
  labels:
    integration-test: ingress
spec:
  rules:
  - host: nginx.example.com
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

Error with v1 version

"failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name nginx-ingress in namespace default"

Current integration test of ingress addon tests only networking.k8s.io/v1.

minikube/test/integration/testdata/nginx-ing.yaml

Lines 1 to 4 in 15cede5

    
           apiVersion: networking.k8s.io/v1beta1 
        
           kind: Ingress 
        
           metadata: 
        
             name: nginx-ingress

This is because our integration test did not catch this problem. It should include networking.k8s.io/v1 version test.

govargo · 2021-04-24T20:31:57Z

I'll add workaround to current ingress addon so that we can use v1 ingress resource.
And I'll add integration test one more pattern about networking.k8s.io/v1.
I'll send PR soon.

govargo · 2021-04-24T21:59:49Z

Further investigation and confirmation, I found out the difference between upstream ingress-nginx manifests and our manifests.
As I described above, the ingress-nginx controller validates only networking.k8s.io/v1beta1 not inculding networking.k8s.io/v1.
However, our manifests currently include both networking.k8s.io/v1beta1 and networking.k8s.io/v1.

minikube/deploy/addons/ingress/ingress-dp.yaml.tmpl

Lines 183 to 200 in f854a08

    
           apiVersion: admissionregistration.k8s.io/v1 
        
           kind: ValidatingWebhookConfiguration 
        
           metadata: 
        
             labels: 
        
               app.kubernetes.io/name: ingress-nginx 
        
               app.kubernetes.io/instance: ingress-nginx 
        
               app.kubernetes.io/component: admission-webhook 
        
               addonmanager.kubernetes.io/mode: Reconcile 
        
             name: ingress-nginx-admission 
        
           webhooks: 
        
             - name: validate.nginx.ingress.kubernetes.io 
        
               matchPolicy: Equivalent 
        
               rules: 
        
                 - apiGroups: 
        
                     - networking.k8s.io 
        
                   apiVersions: 
        
                     - v1beta1 
        
                     - v1

It might have been included while updating ingress addon to v0.44.0.
The validation check for the v1 version of Ingress should be removed from ValidatingWebhookConfiguration.

I updated the PR to remove this check for networking.k8s.io/v1.
(Of course, the networking.k8s.io/v1 should be validated if possible,since networking.k8s.io/v1beta1 is deprecated version in current k8s version. I added the TODO command about this versioning)

medyagh · 2021-04-25T04:39:26Z

@sherbang @joshuamataaraya @btalbot @bpb54321 @dgoldssfo

do you mind trying the binary in this PR #11189
and verify if this fixes the problem for you ?

http://storage.googleapis.com/minikube-builds/11189/minikube-linux-amd64
http://storage.googleapis.com/minikube-builds/11189/minikube-darwin-amd64
http://storage.googleapis.com/minikube-builds/11189/minikube-windows-amd64.exe

mnebus · 2021-04-25T17:46:19Z

@medyagh - The 11189 darwin build resolved this issue for me.

dgoldssfo · 2021-04-25T17:49:17Z

We changed the apiVersion in our manifests to work around the problem, so I'm not in a good position to validate. Apologies.

joshuamataaraya · 2021-05-11T18:05:18Z

@medyagh This didn't work for me
minikube config set vm-driver hyperkit
minikube delete
minikube start
minikube addons enable ingress
❗ These changes will take effect upon a minikube delete and then a minikube start
🙄 "minikube" profile does not exist, trying anyways.
💀 Removed all traces of the "minikube" cluster.
😄 minikube v1.19.0 on Darwin 11.3.1
✨ Using the hyperkit driver based on user configuration
💿 Downloading VM boot image ...
🎉 minikube 1.20.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.20.0
💡 To disable this notice, run: 'minikube config set WantUpdateNotification false'

> minikube-v1.19.0-1618897865...: 65 B / 65 B [----------] 100.00% ? p/s 0s
> minikube-v1.19.0-1618897865...: 244.81 MiB / 244.81 MiB  100.00% 12.49 Mi

👍 Starting control plane node minikube in cluster minikube
🔥 Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
❗ This VM is having trouble accessing https://k8s.gcr.io
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
🔎 Verifying ingress addon...

❌ Exiting due to MK_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

minikube_addons_657d376187cd72746604141ceddc839ee4e6f05e_0.log

I've tried upgrading to the latest version and it doesn't work either 1.20.0 it shows exactly the same error

btalbot · 2021-05-11T19:19:17Z

For the past couple of versions, when there have been issues with the ingress, I've had to run this purge command to clean up state and allow startup to be successful.

minikube delete --all --purge

robrich · 2021-05-11T21:01:17Z

@joshuamataaraya Hit up https://github.com/kubernetes/minikube/releases and grab the 1.20 release. See the notes in 1.20 alpha about the Ingress Addon fix.

UnknownGnome · 2021-05-12T21:29:27Z

Same problem on 1.20

[ec2-user@ip-10-4-20-69~]$ minikube version
minikube version: v1.20.0
commit: c61663e942ec43b20e8e70839dcca52e44cd85ae
[ec2-user@ip-10-4-20-69~]$ minikube delete --all --purge
* Deleting "minikube" in docker ...
* Removing /home/ec2-user/.minikube/machines/minikube ...
* Removed all traces of the "minikube" cluster.
* Successfully deleted all profiles
* Successfully purged minikube directory located at - [/home/ec2-user/.minikube]
[ec2-user@ip-10-4-20-69~]$ minikube start
* minikube v1.20.0 on Amazon 2 (xen/amd64)
* Automatically selected the docker driver. Other choices: none, ssh
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Downloading Kubernetes v1.20.2 preload ...
    > gcr.io/k8s-minikube/kicbase...: 358.10 MiB / 358.10 MiB  100.00% 41.33 Mi
    > preloaded-images-k8s-v10-v1...: 491.71 MiB / 491.71 MiB  100.00% 73.97 Mi
* Creating docker container (CPUs=2, Memory=4000MB) ...
! This container is having trouble accessing https://k8s.gcr.io
* To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
* Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
  - Generating certificates and keys ...
  - Booting up control plane ...
  - Configuring RBAC rules ...
* Verifying Kubernetes components...
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[ec2-user@ip-10-4-20-69~]$ minikube addons enable ingress
  - Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
  - Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
  - Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
* Verifying ingress addon...

X Exiting due to MK_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

spowelljr added the kind/support Categorizes issue or PR as a support question. label Apr 19, 2021

spowelljr added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Apr 21, 2021

medyagh mentioned this issue Apr 24, 2021

minikube v1.19.0 issues with ingress-nginx validation webhook. ansible/awx-operator#205

Closed

medyagh added kind/bug Categorizes issue or PR as related to a bug. kind/regression Categorizes issue or PR as related to a regression from a prior release. addon/ingress top-10-issues Top 10 support issues and removed kind/support Categorizes issue or PR as a support question. labels Apr 24, 2021

govargo mentioned this issue Apr 24, 2021

[Ingress Addon] Fix bug which the networking.k8s.io/v1 ingress is always rejected #11189

Merged

medyagh closed this as completed in #11189 Apr 25, 2021

hcguersoy mentioned this issue May 10, 2021

Set up Ingress on Minikube with the NGINX Ingress Controller kubernetes/website#27621

Closed

jmazzitelli mentioned this issue Jun 7, 2021

[helm][operator] do not use deprecated Ingress kind - update to latest apiVersion kiali/kiali#3706

Closed

seizadi mentioned this issue Jul 1, 2021

kind nginx ingress validation webhook failed calling "validate.nginx.ingress.kubernetes.io": an error on the server ("") has prevented the request from succeeding kubernetes/ingress-nginx#7306

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ingress addon: `Error: server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name` #11121

Ingress addon: `Error: server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name` #11121

sherbang commented Apr 17, 2021

btalbot commented Apr 20, 2021 •

edited

Loading

robrich commented Apr 22, 2021 •

edited

Loading

balnbibarbi commented Apr 22, 2021

btalbot commented Apr 22, 2021

medyagh commented Apr 24, 2021 •

edited

Loading

medyagh commented Apr 24, 2021 •

edited

Loading

govargo commented Apr 24, 2021 •

edited

Loading

govargo commented Apr 24, 2021 •

edited

Loading

govargo commented Apr 24, 2021 •

edited

Loading

govargo commented Apr 24, 2021 •

edited

Loading

medyagh commented Apr 25, 2021 •

edited

Loading

mnebus commented Apr 25, 2021

dgoldssfo commented Apr 25, 2021

joshuamataaraya commented May 11, 2021

btalbot commented May 11, 2021

robrich commented May 11, 2021

UnknownGnome commented May 12, 2021

Ingress addon: Error: server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name #11121

Ingress addon: Error: server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name #11121

Comments

sherbang commented Apr 17, 2021

btalbot commented Apr 20, 2021 • edited Loading

robrich commented Apr 22, 2021 • edited Loading

balnbibarbi commented Apr 22, 2021

btalbot commented Apr 22, 2021

medyagh commented Apr 24, 2021 • edited Loading

medyagh commented Apr 24, 2021 • edited Loading

govargo commented Apr 24, 2021 • edited Loading

govargo commented Apr 24, 2021 • edited Loading

govargo commented Apr 24, 2021 • edited Loading

govargo commented Apr 24, 2021 • edited Loading

medyagh commented Apr 25, 2021 • edited Loading

mnebus commented Apr 25, 2021

dgoldssfo commented Apr 25, 2021

joshuamataaraya commented May 11, 2021

btalbot commented May 11, 2021

robrich commented May 11, 2021

UnknownGnome commented May 12, 2021

Ingress addon: `Error: server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name` #11121

Ingress addon: `Error: server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name` #11121

btalbot commented Apr 20, 2021 •

edited

Loading

robrich commented Apr 22, 2021 •

edited

Loading

medyagh commented Apr 24, 2021 •

edited

Loading

medyagh commented Apr 24, 2021 •

edited

Loading

govargo commented Apr 24, 2021 •

edited

Loading

govargo commented Apr 24, 2021 •

edited

Loading

govargo commented Apr 24, 2021 •

edited

Loading

govargo commented Apr 24, 2021 •

edited

Loading

medyagh commented Apr 25, 2021 •

edited

Loading