webhook admission controller error - no validation for extensions/v1beta1 or networking.k8s.io/v1 ingresses #7066

arno-pons · 2021-04-21T10:20:14Z

NGINX Ingress controller version:
v0.41.2

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: vanilla kubernetes 1.19 on centOS Azure vms
OS (e.g. from /etc/os-release): CentOS Linux 7
Kernel (e.g. uname -a): Linux lnx001617 3.10.0-1160.24.1.el7.x86_64 Basic structure #1 SMP Thu Apr 8 19:51:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Install tools: kubespray

What happened:

Here my ValidatingWebhookConfiguration rules:

    rules:
      - apiGroups: ["networking.k8s.io", "extensions"]
        apiVersions: ["*"]
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses

Try to deploy diffrent ingresses api with the following incorrect annotation:

    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Permissions-Policy: geolocation=(), microphone=(), camera=()";|

Webhook admission controller seems to validate configuration only for networking.k8s.io/v1beta1 ingress api version.

Try to deploy an extensions/v1beta1 ingress

Deployment result ok (validation failure was expected here):

# k apply -f extensions_v1beta1_ingress_invalid_annotations.yaml 
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/extensions-v1beta1-ingress-invalid-annotations created

Admission controller result:

ingress-nginx-private-controller-8d55f ingress-nginx-private-controller 2021-04-21T09:32:03.624734151Z E0421 09:32:03.624446       6 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but extensions/v1beta1, Kind=Ingress with name extensions-v1beta1-ingress-invalid-annotations in namespace fx00964-1"

Try to deploy a networking.k8s.io/v1beta1 ingress

Deployment result ko (expected behavior):

# k apply -f networking_v1beta1_ingress_invalid_annotations.yaml Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Error from server (BadRequest): error when creating "networking_v1beta1_ingress_invalid_annotations.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: 
-------------------------------------------------------------------------------
Error: exit status 1
2021/04/21 09:34:52 [emerg] 487#487: unknown directive "|" in /tmp/nginx-cfg949687373:5803
nginx: [emerg] unknown directive "|" in /tmp/nginx-cfg949687373:5803
nginx: configuration file /tmp/nginx-cfg949687373 test failed

-------------------------------------------------------------------------------

Try to deploy a networking.k8s.io/v1

Deployment result ok (validation failure was expected here):

# k apply -f networking_v1_ingress_invalid_annotations.yaml 
ingress.networking.k8s.io/networking-v1-ingress-invalid-annotations created

Admission controller result:

ingress-nginx-private-controller-zwz52 ingress-nginx-private-controller 2021-04-21T09:38:58.070252966Z E0421 09:38:58.070074       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name networking-v1-ingress-invalid-annotations in namespace fx00964-1"

What you expected to happen:

At least networking.k8s.io/v1beta1 and networking.k8s.io/v1 are supported by ingress admission controller webhook. Ideally, kubernetes n and n-1 versions valid ingresses should be supported by admission controller.

How to reproduce it:

Deploy an v0.41.2 ingress controller with an ValidatingWebhookConfiguration rules targeting networking.k8s.io/v1beta1, networking.k8s.io/v1 and extensions/v1beta1 object and deploy the following ingresses:

extensions/v1beta1 test ingress

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: extensions-v1beta1-ingress-invalid-annotations
  annotations:
    # Annotation used to define a private access
    kubernetes.io/ingress.class: "private"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Permissions-Policy: geolocation=(), microphone=(), camera=()";|
  labels:
    app: invalid-test
spec:
  rules:
  - host: "fake-host.com"
    http:
      paths:
      - path: /
        backend:
          serviceName: invalid-test-svc
          servicePort: http

networking.k8s.io/v1beta1 test ingress

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: networking-v1beta1-ingress-invalid-annotations
  annotations:
    # Annotation used to define a private access
    kubernetes.io/ingress.class: "private"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Permissions-Policy: geolocation=(), microphone=(), camera=()";|
  labels:
    app: invalid-test
spec:
  rules:
  - host: "fake-host.com"
    http:
      paths:
      - path: /
        backend:
          serviceName: invalid-test-svc
          servicePort: http

networking.k8s.io/v1 test ingress

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: networking-v1-ingress-invalid-annotations
  annotations:
    # Annotation used to define a private access
    kubernetes.io/ingress.class: "private"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Permissions-Policy: geolocation=(), microphone=(), camera=()";|
  labels:
    app: invalid-test
spec:
  rules:
  - host: "fake-host.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: invalid-test-svc
            port:
              name: http

Anything else we need to know:

I notice there are issues related to this admission controller errors:

that should be included in v0.41.2 version but i still have the problems as decribed above.

/kind bug

The text was updated successfully, but these errors were encountered:

longwuyuan · 2021-04-22T05:12:02Z

/remove-kind bug

Can you help by providing following information ;

Can you use the latest release of the controller
If you can not use the latest release of the controller, explain reason please
After/If you move to the latest release of the controller, Can you show the actual command and output for
- kubeclt get validatingwebhookconfigurations.admissionregistration.k8s.io -o yaml
- kubectl -n get all -A -o wide
- helm ls -A

longwuyuan · 2021-04-22T05:13:50Z

/triage needs-information

arno-pons · 2021-04-22T14:03:30Z

Hello,

I think I understand why this is not working.
We are using the same (unique) ValidatingWebhookConfiguration for a long time with the following configuration:

[...]
    service:
      name: ingress-private-validation-webhook
      namespace: ingress-controllers
      path: /extensions/v1beta1/ingress
      port: 443
 [...]
  rules:
  - apiGroups:
    - networking.k8s.io
    - extensions
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - ingresses
    scope: '*'

meaning that we are sending each validation request for each apiGroup to the same endpoint: /extensions/v1beta1/ingress.

I split this ValidatingWebhookConfiguration into three separated instances, using the following configuration:

apiGroup	Version	endpoint
extensions	v1beta1	/extensions/v1beta1/ingress
networking.k8s.io	v1beta1	/networking/v1beta1/ingress
networking.k8s.io	v1	/networking/v1/ingress

and it seems to work now.

Could you please confirm this configuration ?

Thanks and regards.

longwuyuan · 2021-04-22T14:10:12Z

Depends on version of kubernetes. Can you kindly close the issue. Thanks, ; Long

…

On Thu, 22 Apr, 2021, 7:33 PM arno-snop, ***@***.***> wrote: Hello, I think I understand why this is not working. We are using the same (unique) ValidatingWebhookConfiguration for a long time with the following configuration: [...] service: name: ingress-private-validation-webhook namespace: ingress-controllers path: /extensions/v1beta1/ingress port: 443 [...] rules: - apiGroups: - networking.k8s.io - extensions apiVersions: - '*' operations: - CREATE - UPDATE resources: - ingresses scope: '*' meaning that we are sending each validation request for each apiGroup to the same endpoint: /extensions/v1beta1/ingress. I split this ValidatingWebhookConfiguration into three separated instances, using the following configuration: apiGroup Version endpoint extensions v1beta1 /extensions/v1beta1/ingress networking.k8s.io v1beta1 /networking/v1beta1/ingress networking.k8s.io v1 /extensions/v1/ingress and it seems to work now. Could you please confirm this configuration ? Thanks and regards. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7066 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWXLM6LFFAEBMSMJEYDTKAUENANCNFSM43KBYW6A> .

arno-pons · 2021-04-22T14:21:00Z

I understand that the list of apiGroup depends on kubernetes version, but on nginx controller side, can you please confirm that the service path to use are the ones above ?

longwuyuan · 2021-04-22T14:53:30Z

on k8s v1.21 I am seeing this /networking/v1beta1/ingresses .
On v1.22 it will be v1.
What version is your k8s ?

arno-pons · 2021-04-23T06:23:09Z

I'm using k8s 1.19, but here you are talking about k8s ingress api support, and i'm aware that in 1.22, only networking.k8s.io/v1 will be supported.

What i need to understand is wath is the service path on ingress controller side to send admission request to (path configured in service part of ValidatingWebhookConfiguration object)?
Is the following table is correct ?

apiGroup	Version	ingress controller path
extensions	v1beta1	/extensions/v1beta1/ingress
networking.k8s.io	v1beta1	/networking/v1beta1/ingress
networking.k8s.io	v1	/networking/v1/ingress

longwuyuan · 2021-04-23T07:25:15Z

Based on a live working cluster's resource, I can confirm these 2 ;

networking.k8s.io	v1beta1	/networking/v1beta1/ingress
networking.k8s.io	v1	/networking/v1/ingress

I also think the 1st one is good.

They seem to be the api path.

I guess you can look at the code or a developer should confirm.
I should look at code too and confirm.
Still thinking that without specifying api version, the k8s v19.1 will default to networking.k8s.io/v1beta1 .

I think this is already documented.
Have you compared with k8s v1.21

arno-pons · 2021-04-23T07:45:45Z

I think there is something that I not really understand here.

When we first start to use the admission controller validating webhook on ingress-controller side, we follow this documentation (v0.25.1):

The way to make it work was to

enable validating-webhook feature in ingress-controller
deploy some certificates
and deploy this ValidatingWebhookConfiguration

We still deploy current version (v0.41.2) using this procedure.

So, if we don't deploy ValidatingWebhookConfiguration how k8s will target ingress-controller validating-webhook?
I can't find information in current documentation about this mechanism, so maybe I miss something here...

longwuyuan · 2021-04-23T08:08:16Z

That is 2 yrs old.
What happens if you remove all and just follow https://kubernetes.github.io/ingress-nginx/deploy/#provider-specific-steps

tvizor · 2021-06-08T21:12:07Z

@arno-snop

EKS: 1.19
ingress-controller: 0.44.0, 0.46.0, 0.47.0
Try the next configuration. This works with 0.47.0 for both networking.k8s.io/v1beta1 and extensions/v1beta1 Ingresses.

matchPolicy should be set to Equivalent to allow ingress resource to be converted before validation.
extensions should be removed from apiGroups.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1beta1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
      - v1beta1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission
        path: /networking/v1beta1/ingresses

k8s-triage-robot · 2021-09-06T22:04:57Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2021-10-06T22:53:35Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2021-11-05T23:09:34Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2021-11-05T23:09:53Z

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

arno-pons added the kind/bug Categorizes issue or PR as related to a bug. label Apr 21, 2021

k8s-ci-robot removed the kind/bug Categorizes issue or PR as related to a bug. label Apr 22, 2021

k8s-ci-robot added the triage/needs-information Indicates an issue needs more information in order to work on it. label Apr 22, 2021

govargo mentioned this issue Apr 24, 2021

Ingress addon: Error: server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name kubernetes/minikube#11121

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 6, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 6, 2021

k8s-ci-robot closed this as completed Nov 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

webhook admission controller error - no validation for extensions/v1beta1 or networking.k8s.io/v1 ingresses #7066

webhook admission controller error - no validation for extensions/v1beta1 or networking.k8s.io/v1 ingresses #7066

arno-pons commented Apr 21, 2021

longwuyuan commented Apr 22, 2021

longwuyuan commented Apr 22, 2021

arno-pons commented Apr 22, 2021 •

edited

Loading

longwuyuan commented Apr 22, 2021 via email

arno-pons commented Apr 22, 2021

longwuyuan commented Apr 22, 2021

arno-pons commented Apr 23, 2021 •

edited

Loading

longwuyuan commented Apr 23, 2021

arno-pons commented Apr 23, 2021

longwuyuan commented Apr 23, 2021

tvizor commented Jun 8, 2021 •

edited

Loading

k8s-triage-robot commented Sep 6, 2021

k8s-triage-robot commented Oct 6, 2021

k8s-triage-robot commented Nov 5, 2021

k8s-ci-robot commented Nov 5, 2021

webhook admission controller error - no validation for extensions/v1beta1 or networking.k8s.io/v1 ingresses #7066

webhook admission controller error - no validation for extensions/v1beta1 or networking.k8s.io/v1 ingresses #7066

Comments

arno-pons commented Apr 21, 2021

longwuyuan commented Apr 22, 2021

longwuyuan commented Apr 22, 2021

arno-pons commented Apr 22, 2021 • edited Loading

longwuyuan commented Apr 22, 2021 via email

arno-pons commented Apr 22, 2021

longwuyuan commented Apr 22, 2021

arno-pons commented Apr 23, 2021 • edited Loading

longwuyuan commented Apr 23, 2021

arno-pons commented Apr 23, 2021

longwuyuan commented Apr 23, 2021

tvizor commented Jun 8, 2021 • edited Loading

k8s-triage-robot commented Sep 6, 2021

k8s-triage-robot commented Oct 6, 2021

k8s-triage-robot commented Nov 5, 2021

k8s-ci-robot commented Nov 5, 2021

arno-pons commented Apr 22, 2021 •

edited

Loading

arno-pons commented Apr 23, 2021 •

edited

Loading

tvizor commented Jun 8, 2021 •

edited

Loading