Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Unable to apply v4.0 CRDs: Invalid value: "v1alpha1": must appear in spec.versions #7010

Open
kyrofa opened this issue Dec 17, 2024 · 7 comments
Open

[Bug]: Unable to apply v4.0 CRDs: Invalid value: "v1alpha1": must appear in spec.versions #7010

kyrofa opened this issue Dec 17, 2024 · 7 comments
Labels
bug An issue reporting a potential bug waiting for response Waiting for author's response

Comments

@kyrofa
Copy link

kyrofa commented Dec 17, 2024 •
edited
Loading

Version

3.7.2

What Kubernetes platforms are you running on?

DigitalOcean k8s

Steps to reproduce

I have the ingress controller installed via helm chart (v1.4.2 specifically, which contains v3.7.2 of the controller). I still need to manage the CRDs manually though (as documented in the installation instructions), so I have a standard practice of applying them before I upgrade the chart. I've been following the deprecation notices, so I don't have any v1alpha1 resources. Today I decided to upgrade the chart to v2.0.0 (v4.0.0 of the controller). The first step was to apply the CRDs, but I got this:

$ kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v4.0.0/deploy/crds.yaml
customresourcedefinition.apiextensions.k8s.io/dnsendpoints.externaldns.nginx.org unchanged
customresourcedefinition.apiextensions.k8s.io/policies.k8s.nginx.org configured
customresourcedefinition.apiextensions.k8s.io/virtualserverroutes.k8s.nginx.org unchanged
customresourcedefinition.apiextensions.k8s.io/virtualservers.k8s.nginx.org unchanged
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.16.5\"},\"name\":\"globalconfigurations.k8s.nginx.org\"},\"spec\":{\"group\":\"k8s.nginx.org\",\"names\":{\"kind\":\"GlobalConfiguration\",\"listKind\":\"GlobalConfigurationList\",\"plural\":\"globalconfigurations\",\"shortNames\":[\"gc\"],\"singular\":\"globalconfiguration\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"description\":\"GlobalConfiguration defines the GlobalConfiguration resource.\",\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object.\\nServers should convert recognized schemas to the latest internal value, and\\nmay reject unrecognized values.\\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents.\\nServers may infer this from the endpoint the client submits requests to.\\nCannot be updated.\\nIn CamelCase.\\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"spec\":{\"description\":\"GlobalConfigurationSpec is the spec of the GlobalConfiguration resource.\",\"properties\":{\"listeners\":{\"items\":{\"description\":\"Listener defines a listener.\",\"properties\":{\"ipv4\":{\"type\":\"string\"},\"ipv6\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"port\":{\"type\":\"integer\"},\"protocol\":{\"type\":\"string\"},\"ssl\":{\"type\":\"boolean\"}},\"type\":\"object\"},\"type\":\"array\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]}}\n"}},"spec":{"versions":[{"name":"v1","schema":{"openAPIV3Schema":{"description":"GlobalConfiguration defines the GlobalConfiguration resource.","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object.\nServers should convert recognized schemas to the latest internal value, and\nmay reject unrecognized values.\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents.\nServers may infer this from the endpoint the client submits requests to.\nCannot be updated.\nIn CamelCase.\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"GlobalConfigurationSpec is the spec of the GlobalConfiguration resource.","properties":{"listeners":{"items":{"description":"Listener defines a listener.","properties":{"ipv4":{"type":"string"},"ipv6":{"type":"string"},"name":{"type":"string"},"port":{"type":"integer"},"protocol":{"type":"string"},"ssl":{"type":"boolean"}},"type":"object"},"type":"array"}},"type":"object"}},"type":"object"}},"served":true,"storage":true}]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "globalconfigurations.k8s.nginx.org", Namespace: ""
for: "https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v4.0.0/deploy/crds.yaml": error when patching "https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v4.0.0/deploy/crds.yaml": CustomResourceDefinition.apiextensions.k8s.io "globalconfigurations.k8s.nginx.org" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.16.5\"},\"name\":\"transportservers.k8s.nginx.org\"},\"spec\":{\"group\":\"k8s.nginx.org\",\"names\":{\"kind\":\"TransportServer\",\"listKind\":\"TransportServerList\",\"plural\":\"transportservers\",\"shortNames\":[\"ts\"],\"singular\":\"transportserver\"},\"scope\":\"Namespaced\",\"versions\":[{\"additionalPrinterColumns\":[{\"description\":\"Current state of the TransportServer. If the resource has a valid status, it means it has been validated and accepted by the Ingress Controller.\",\"jsonPath\":\".status.state\",\"name\":\"State\",\"type\":\"string\"},{\"jsonPath\":\".status.reason\",\"name\":\"Reason\",\"type\":\"string\"},{\"jsonPath\":\".metadata.creationTimestamp\",\"name\":\"Age\",\"type\":\"date\"}],\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"description\":\"TransportServer defines the TransportServer resource.\",\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object.\\nServers should convert recognized schemas to the latest internal value, and\\nmay reject unrecognized values.\\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents.\\nServers may infer this from the endpoint the client submits requests to.\\nCannot be updated.\\nIn CamelCase.\\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"spec\":{\"description\":\"TransportServerSpec is the spec of the TransportServer resource.\",\"properties\":{\"action\":{\"description\":\"TransportServerAction defines an action.\",\"properties\":{\"pass\":{\"type\":\"string\"}},\"type\":\"object\"},\"host\":{\"type\":\"string\"},\"ingressClassName\":{\"type\":\"string\"},\"listener\":{\"description\":\"TransportServerListener defines a listener for a TransportServer.\",\"properties\":{\"name\":{\"type\":\"string\"},\"protocol\":{\"type\":\"string\"}},\"type\":\"object\"},\"serverSnippets\":{\"type\":\"string\"},\"sessionParameters\":{\"description\":\"SessionParameters defines session parameters.\",\"properties\":{\"timeout\":{\"type\":\"string\"}},\"type\":\"object\"},\"streamSnippets\":{\"type\":\"string\"},\"tls\":{\"description\":\"TransportServerTLS defines TransportServerTLS configuration for a TransportServer.\",\"properties\":{\"secret\":{\"type\":\"string\"}},\"type\":\"object\"},\"upstreamParameters\":{\"description\":\"UpstreamParameters defines parameters for an upstream.\",\"properties\":{\"connectTimeout\":{\"type\":\"string\"},\"nextUpstream\":{\"type\":\"boolean\"},\"nextUpstreamTimeout\":{\"type\":\"string\"},\"nextUpstreamTries\":{\"type\":\"integer\"},\"udpRequests\":{\"type\":\"integer\"},\"udpResponses\":{\"type\":\"integer\"}},\"type\":\"object\"},\"upstreams\":{\"items\":{\"description\":\"TransportServerUpstream defines an upstream.\",\"properties\":{\"backup\":{\"type\":\"string\"},\"backupPort\":{\"type\":\"integer\"},\"failTimeout\":{\"type\":\"string\"},\"healthCheck\":{\"description\":\"TransportServerHealthCheck defines the parameters for active Upstream HealthChecks.\",\"properties\":{\"enable\":{\"type\":\"boolean\"},\"fails\":{\"type\":\"integer\"},\"interval\":{\"type\":\"string\"},\"jitter\":{\"type\":\"string\"},\"match\":{\"description\":\"TransportServerMatch defines the parameters of a custom health check.\",\"properties\":{\"expect\":{\"type\":\"string\"},\"send\":{\"type\":\"string\"}},\"type\":\"object\"},\"passes\":{\"type\":\"integer\"},\"port\":{\"type\":\"integer\"},\"timeout\":{\"type\":\"string\"}},\"type\":\"object\"},\"loadBalancingMethod\":{\"type\":\"string\"},\"maxConns\":{\"type\":\"integer\"},\"maxFails\":{\"type\":\"integer\"},\"name\":{\"type\":\"string\"},\"port\":{\"type\":\"integer\"},\"service\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"}},\"type\":\"object\"},\"status\":{\"description\":\"TransportServerStatus defines the status for the TransportServer resource.\",\"properties\":{\"message\":{\"type\":\"string\"},\"reason\":{\"type\":\"string\"},\"state\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true,\"subresources\":{\"status\":{}}}]}}\n"}},"spec":{"versions":[{"additionalPrinterColumns":[{"description":"Current state of the TransportServer. If the resource has a valid status, it means it has been validated and accepted by the Ingress Controller.","jsonPath":".status.state","name":"State","type":"string"},{"jsonPath":".status.reason","name":"Reason","type":"string"},{"jsonPath":".metadata.creationTimestamp","name":"Age","type":"date"}],"name":"v1","schema":{"openAPIV3Schema":{"description":"TransportServer defines the TransportServer resource.","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object.\nServers should convert recognized schemas to the latest internal value, and\nmay reject unrecognized values.\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents.\nServers may infer this from the endpoint the client submits requests to.\nCannot be updated.\nIn CamelCase.\nMore info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"TransportServerSpec is the spec of the TransportServer resource.","properties":{"action":{"description":"TransportServerAction defines an action.","properties":{"pass":{"type":"string"}},"type":"object"},"host":{"type":"string"},"ingressClassName":{"type":"string"},"listener":{"description":"TransportServerListener defines a listener for a TransportServer.","properties":{"name":{"type":"string"},"protocol":{"type":"string"}},"type":"object"},"serverSnippets":{"type":"string"},"sessionParameters":{"description":"SessionParameters defines session parameters.","properties":{"timeout":{"type":"string"}},"type":"object"},"streamSnippets":{"type":"string"},"tls":{"description":"TransportServerTLS defines TransportServerTLS configuration for a TransportServer.","properties":{"secret":{"type":"string"}},"type":"object"},"upstreamParameters":{"description":"UpstreamParameters defines parameters for an upstream.","properties":{"connectTimeout":{"type":"string"},"nextUpstream":{"type":"boolean"},"nextUpstreamTimeout":{"type":"string"},"nextUpstreamTries":{"type":"integer"},"udpRequests":{"type":"integer"},"udpResponses":{"type":"integer"}},"type":"object"},"upstreams":{"items":{"description":"TransportServerUpstream defines an upstream.","properties":{"backup":{"type":"string"},"backupPort":{"type":"integer"},"failTimeout":{"type":"string"},"healthCheck":{"description":"TransportServerHealthCheck defines the parameters for active Upstream HealthChecks.","properties":{"enable":{"type":"boolean"},"fails":{"type":"integer"},"interval":{"type":"string"},"jitter":{"type":"string"},"match":{"description":"TransportServerMatch defines the parameters of a custom health check.","properties":{"expect":{"type":"string"},"send":{"type":"string"}},"type":"object"},"passes":{"type":"integer"},"port":{"type":"integer"},"timeout":{"type":"string"}},"type":"object"},"loadBalancingMethod":{"type":"string"},"maxConns":{"type":"integer"},"maxFails":{"type":"integer"},"name":{"type":"string"},"port":{"type":"integer"},"service":{"type":"string"}},"type":"object"},"type":"array"}},"type":"object"},"status":{"description":"TransportServerStatus defines the status for the TransportServer resource.","properties":{"message":{"type":"string"},"reason":{"type":"string"},"state":{"type":"string"}},"type":"object"}},"type":"object"}},"served":true,"storage":true,"subresources":{"status":{}}}]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "transportservers.k8s.nginx.org", Namespace: ""
for: "https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v4.0.0/deploy/crds.yaml": error when patching "https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v4.0.0/deploy/crds.yaml": CustomResourceDefinition.apiextensions.k8s.io "transportservers.k8s.nginx.org" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

I'm not sure what to make of it. Any help would be appreciated!
@kyrofa kyrofa added bug An issue reporting a potential bug needs triage An issue that needs to be triaged labels Dec 17, 2024
@github-actions GitHub Actions
Copy link

Hi @kyrofa thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

@vepatel
Copy link
Contributor

vepatel commented Dec 18, 2024 •
edited
Loading

Hi @kyrofa , i ran an upgrade from v3.7.2 to v4.0.0 and not observing the same behaviour:

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v3.7.2 ● λ helm install test-release .
walk.go:74: found symbolic link in path: /Users/ve.patel/nginx/kubernetes-ingress/charts/nginx-ingress/crds resolves to /Users/ve.patel/nginx/kubernetes-ingress/config/crd/bases. Contents of linked file included and used
NAME: test-release
LAST DEPLOYED: Wed Dec 18 09:57:17 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
NGINX Ingress Controller 3.7.2 has been installed.

For release notes for this version please see: https://docs.nginx.com/nginx-ingress-controller/releases/

Installation and upgrade instructions: https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v3.7.2 ● λ k get crds
NAME                                               CREATED AT
allowlistedv2workloads.auto.gke.io                 2024-12-17T13:53:11Z
allowlistedworkloads.auto.gke.io                   2024-12-17T13:53:09Z
apdoslogconfs.appprotectdos.f5.com                 2024-12-18T09:57:13Z
apdospolicies.appprotectdos.f5.com                 2024-12-18T09:57:14Z
aplogconfs.appprotect.f5.com                       2024-12-18T09:57:13Z
appolicies.appprotect.f5.com                       2024-12-18T09:57:13Z
apusersigs.appprotect.f5.com                       2024-12-18T09:57:13Z
audits.warden.gke.io                               2024-12-17T13:53:11Z
backendconfigs.cloud.google.com                    2024-12-17T13:53:24Z
capacityrequests.internal.autoscaling.gke.io       2024-12-17T13:53:03Z
clusternodemonitorings.monitoring.googleapis.com   2024-12-17T13:53:26Z
clusterpodmonitorings.monitoring.googleapis.com    2024-12-17T13:53:26Z
clusterrules.monitoring.googleapis.com             2024-12-17T13:53:27Z
computeclasses.cloud.google.com                    2024-12-17T13:53:04Z
dnsendpoints.externaldns.nginx.org                 2024-12-18T09:57:14Z
dosprotectedresources.appprotectdos.f5.com         2024-12-18T09:57:14Z
frontendconfigs.networking.gke.io                  2024-12-17T13:53:25Z
gcpdatasources.datalayer.gke.io                    2024-12-17T13:53:16Z
gcpresourceallowlists.node.gke.io                  2024-12-17T13:53:09Z
gkenetworkparamsets.networking.gke.io              2024-12-17T13:52:58Z
globalconfigurations.k8s.nginx.org                 2024-12-18T09:57:14Z
globalrules.monitoring.googleapis.com              2024-12-17T13:53:27Z
managedcertificates.networking.gke.io              2024-12-17T13:53:35Z
memberships.hub.gke.io                             2024-12-17T13:53:10Z
networks.networking.gke.io                         2024-12-17T13:52:58Z
nginxingresses.charts.nginx.org                    2024-12-17T15:59:02Z
operatorconfigs.monitoring.googleapis.com          2024-12-17T13:53:27Z
podmonitorings.monitoring.googleapis.com           2024-12-17T13:53:28Z
policies.k8s.nginx.org                             2024-12-18T09:57:14Z
provisioningrequests.autoscaling.x-k8s.io          2024-12-17T13:53:04Z
rules.monitoring.googleapis.com                    2024-12-17T13:53:28Z
serviceattachments.networking.gke.io               2024-12-17T13:53:26Z
servicefunctionchains.networking.gke.io            2024-12-17T13:53:44Z
servicenetworkendpointgroups.networking.gke.io     2024-12-17T13:53:25Z
trafficselectors.networking.gke.io                 2024-12-17T13:53:45Z
transportservers.k8s.nginx.org                     2024-12-18T09:57:15Z
updateinfos.nodemanagement.gke.io                  2024-12-17T13:53:48Z
virtualserverroutes.k8s.nginx.org                  2024-12-18T09:57:15Z
virtualservers.k8s.nginx.org                       2024-12-18T09:57:15Z
volumesnapshotclasses.snapshot.storage.k8s.io      2024-12-17T13:53:47Z
volumesnapshotcontents.snapshot.storage.k8s.io     2024-12-17T13:53:47Z
volumesnapshots.snapshot.storage.k8s.io            2024-12-17T13:53:48Z
workloadallowlists.auto.gke.io                     2024-12-17T13:53:12Z

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v3.7.2 ● λ k get po
NAME                                                     READY   STATUS    RESTARTS   AGE
test-release-nginx-ingress-controller-7dcd5fb67b-2s4jc   1/1     Running   0          18s

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v3.7.2 ● λ git checkout v4.0.0
Previous HEAD position was 561824f30 update go toolchain (#6859)
HEAD is now at 81bae7d03 Release 4.0.0 (#6986)

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v4.0.0 ● λ kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v4.0.0/deploy/crds.yaml

Warning: resource customresourcedefinitions/dnsendpoints.externaldns.nginx.org is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/dnsendpoints.externaldns.nginx.org configured
Warning: resource customresourcedefinitions/globalconfigurations.k8s.nginx.org is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/globalconfigurations.k8s.nginx.org configured
Warning: resource customresourcedefinitions/policies.k8s.nginx.org is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/policies.k8s.nginx.org configured
Warning: resource customresourcedefinitions/transportservers.k8s.nginx.org is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/transportservers.k8s.nginx.org configured
Warning: resource customresourcedefinitions/virtualserverroutes.k8s.nginx.org is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/virtualserverroutes.k8s.nginx.org configured
Warning: resource customresourcedefinitions/virtualservers.k8s.nginx.org is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/virtualservers.k8s.nginx.org configured

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v4.0.0 ● λ helm upgrade test-release .
walk.go:74: found symbolic link in path: /Users/ve.patel/nginx/kubernetes-ingress/charts/nginx-ingress/crds resolves to /Users/ve.patel/nginx/kubernetes-ingress/config/crd/bases. Contents of linked file included and used
Release "test-release" has been upgraded. Happy Helming!
NAME: test-release
LAST DEPLOYED: Wed Dec 18 09:58:28 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
NGINX Ingress Controller 4.0.0 has been installed.

For release notes for this version please see: https://docs.nginx.com/nginx-ingress-controller/releases/

Installation and upgrade instructions: https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/

If you are upgrading from a version of the chart that uses older Custom Resource Definitions (CRD) it is necessary to manually upgrade the CRDs as this is not managed by Helm.
To update to the latest version of the CRDs:
  $ kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v4.0.0/deploy/crds.yaml

More details on upgrading the CRDs: https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/#upgrading-the-crds

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v4.0.0 ● λ k get po
NAME                                                     READY   STATUS        RESTARTS   AGE
test-release-nginx-ingress-controller-6dccc95969-tkg7l   1/1     Running       0          12s
test-release-nginx-ingress-controller-7dcd5fb67b-2s4jc   1/1     Terminating   0          84s

can you please post outputs of kubectl describe crds gc and kubectl describe crds ts ?

@vepatel vepatel added waiting for response Waiting for author's response and removed needs triage An issue that needs to be triaged labels Dec 18, 2024
@kyrofa
Copy link
Author

kyrofa commented Dec 18, 2024

Of course! Maybe this is an odd artifact of updating the CRDs for several years before now? Here's what I've got:

$ kubectl describe crds globalconfigurations.k8s.nginx.org
Name:         globalconfigurations.k8s.nginx.org
Namespace:    
Labels:       <none>
Annotations:  controller-gen.kubebuilder.io/version: v0.16.5
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2022-11-03T03:29:30Z
  Generation:          4
  Resource Version:    309629787
  UID:                 258d5adb-2f60-47c3-8bf0-73dd6a722a13
Spec:
  Conversion:
    Strategy:  None
  Group:       k8s.nginx.org
  Names:
    Kind:       GlobalConfiguration
    List Kind:  GlobalConfigurationList
    Plural:     globalconfigurations
    Short Names:
      gc
    Singular:  globalconfiguration
  Scope:       Namespaced
  Versions:
    Name:  v1
    Schema:
      openAPIV3Schema:
        Description:  GlobalConfiguration defines the GlobalConfiguration resource.
        Properties:
          API Version:
            Description:  APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            Type:  string
          Kind:
            Description:  Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            Type:  string
          Metadata:
            Type:  object
          Spec:
            Description:  GlobalConfigurationSpec is the spec of the GlobalConfiguration resource.
            Properties:
              Listeners:
                Items:
                  Description:  Listener defines a listener.
                  Properties:
                    ipv4:
                      Type:  string
                    ipv6:
                      Type:  string
                    Name:
                      Type:  string
                    Port:
                      Type:  integer
                    Protocol:
                      Type:  string
                    Ssl:
                      Type:  boolean
                  Type:      object
                Type:        array
            Type:            object
        Type:                object
    Served:                  true
    Storage:                 true
    Name:                    v1alpha1
    Schema:
      openAPIV3Schema:
        Description:  GlobalConfiguration defines the GlobalConfiguration resource.
        Properties:
          API Version:
            Description:  APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            Type:  string
          Kind:
            Description:  Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            Type:  string
          Metadata:
            Type:  object
          Spec:
            Description:  GlobalConfigurationSpec is the spec of the GlobalConfiguration resource.
            Properties:
              Listeners:
                Items:
                  Description:  Listener defines a listener.
                  Properties:
                    Name:
                      Type:  string
                    Port:
                      Type:  integer
                    Protocol:
                      Type:  string
                    Ssl:
                      Type:  boolean
                  Type:      object
                Type:        array
            Type:            object
        Type:                object
    Served:                  true
    Storage:                 false
Status:
  Accepted Names:
    Kind:       GlobalConfiguration
    List Kind:  GlobalConfigurationList
    Plural:     globalconfigurations
    Short Names:
      gc
    Singular:  globalconfiguration
  Conditions:
    Last Transition Time:  2022-11-03T03:29:30Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2022-11-03T03:29:30Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1alpha1
    v1
Events:  <none>

$ kubectl describe crds transportservers.k8s.nginx.org
Name:         transportservers.k8s.nginx.org
Namespace:    
Labels:       <none>
Annotations:  controller-gen.kubebuilder.io/version: v0.16.5
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2022-11-03T03:29:31Z
  Generation:          3
  Resource Version:    309629789
  UID:                 49cae03b-191c-4243-9fea-3ed11de1e70e
Spec:
  Conversion:
    Strategy:  None
  Group:       k8s.nginx.org
  Names:
    Kind:       TransportServer
    List Kind:  TransportServerList
    Plural:     transportservers
    Short Names:
      ts
    Singular:  transportserver
  Scope:       Namespaced
  Versions:
    Additional Printer Columns:
      Description:  Current state of the TransportServer. If the resource has a valid status, it means it has been validated and accepted by the Ingress Controller.
      Json Path:    .status.state
      Name:         State
      Type:         string
      Json Path:    .status.reason
      Name:         Reason
      Type:         string
      Json Path:    .metadata.creationTimestamp
      Name:         Age
      Type:         date
    Name:           v1
    Schema:
      openAPIV3Schema:
        Description:  TransportServer defines the TransportServer resource.
        Properties:
          API Version:
            Description:  APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            Type:  string
          Kind:
            Description:  Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            Type:  string
          Metadata:
            Type:  object
          Spec:
            Description:  TransportServerSpec is the spec of the TransportServer resource.
            Properties:
              Action:
                Description:  TransportServerAction defines an action.
                Properties:
                  Pass:
                    Type:  string
                Type:      object
              Host:
                Type:  string
              Ingress Class Name:
                Type:  string
              Listener:
                Description:  TransportServerListener defines a listener for a TransportServer.
                Properties:
                  Name:
                    Type:  string
                  Protocol:
                    Type:  string
                Type:      object
              Server Snippets:
                Type:  string
              Session Parameters:
                Description:  SessionParameters defines session parameters.
                Properties:
                  Timeout:
                    Type:  string
                Type:      object
              Stream Snippets:
                Type:  string
              Tls:
                Description:  TransportServerTLS defines TransportServerTLS configuration for a TransportServer.
                Properties:
                  Secret:
                    Type:  string
                Type:      object
              Upstream Parameters:
                Description:  UpstreamParameters defines parameters for an upstream.
                Properties:
                  Connect Timeout:
                    Type:  string
                  Next Upstream:
                    Type:  boolean
                  Next Upstream Timeout:
                    Type:  string
                  Next Upstream Tries:
                    Type:  integer
                  Udp Requests:
                    Type:  integer
                  Udp Responses:
                    Type:  integer
                Type:      object
              Upstreams:
                Items:
                  Description:  TransportServerUpstream defines an upstream.
                  Properties:
                    Backup:
                      Type:  string
                    Backup Port:
                      Type:  integer
                    Fail Timeout:
                      Type:  string
                    Health Check:
                      Description:  TransportServerHealthCheck defines the parameters for active Upstream HealthChecks.
                      Properties:
                        Enable:
                          Type:  boolean
                        Fails:
                          Type:  integer
                        Interval:
                          Type:  string
                        Jitter:
                          Type:  string
                        Match:
                          Description:  TransportServerMatch defines the parameters of a custom health check.
                          Properties:
                            Expect:
                              Type:  string
                            Send:
                              Type:  string
                          Type:      object
                        Passes:
                          Type:  integer
                        Port:
                          Type:  integer
                        Timeout:
                          Type:  string
                      Type:      object
                    Load Balancing Method:
                      Type:  string
                    Max Conns:
                      Type:  integer
                    Max Fails:
                      Type:  integer
                    Name:
                      Type:  string
                    Port:
                      Type:  integer
                    Service:
                      Type:  string
                  Type:      object
                Type:        array
            Type:            object
          Status:
            Description:  TransportServerStatus defines the status for the TransportServer resource.
            Properties:
              Message:
                Type:  string
              Reason:
                Type:  string
              State:
                Type:  string
            Type:      object
        Type:          object
    Served:            true
    Storage:           true
    Subresources:
      Status:
    Additional Printer Columns:
      Description:  Current state of the TransportServer. If the resource has a valid status, it means it has been validated and accepted by the Ingress Controller.
      Json Path:    .status.state
      Name:         State
      Type:         string
      Json Path:    .status.reason
      Name:         Reason
      Type:         string
      Json Path:    .metadata.creationTimestamp
      Name:         Age
      Type:         date
    Name:           v1alpha1
    Schema:
      openAPIV3Schema:
        Description:  TransportServer defines the TransportServer resource.
        Properties:
          API Version:
            Description:  APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            Type:  string
          Kind:
            Description:  Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            Type:  string
          Metadata:
            Type:  object
          Spec:
            Description:  TransportServerSpec is the spec of the TransportServer resource.
            Properties:
              Action:
                Description:  TransportServerAction defines an action.
                Properties:
                  Pass:
                    Type:  string
                Type:      object
              Host:
                Type:  string
              Ingress Class Name:
                Type:  string
              Listener:
                Description:  TransportServerListener defines a listener for a TransportServer.
                Properties:
                  Name:
                    Type:  string
                  Protocol:
                    Type:  string
                Type:      object
              Server Snippets:
                Type:  string
              Session Parameters:
                Description:  SessionParameters defines session parameters.
                Properties:
                  Timeout:
                    Type:  string
                Type:      object
              Stream Snippets:
                Type:  string
              Tls:
                Description:  TransportServerTLS defines TransportServerTLS configuration for a TransportServer.
                Properties:
                  Secret:
                    Type:  string
                Type:      object
              Upstream Parameters:
                Description:  UpstreamParameters defines parameters for an upstream.
                Properties:
                  Connect Timeout:
                    Type:  string
                  Next Upstream:
                    Type:  boolean
                  Next Upstream Timeout:
                    Type:  string
                  Next Upstream Tries:
                    Type:  integer
                  Udp Requests:
                    Type:  integer
                  Udp Responses:
                    Type:  integer
                Type:      object
              Upstreams:
                Items:
                  Description:  TransportServerUpstream defines an upstream.
                  Properties:
                    Backup:
                      Type:  string
                    Backup Port:
                      Type:  integer
                    Fail Timeout:
                      Type:  string
                    Health Check:
                      Description:  TransportServerHealthCheck defines the parameters for active Upstream HealthChecks.
                      Properties:
                        Enable:
                          Type:  boolean
                        Fails:
                          Type:  integer
                        Interval:
                          Type:  string
                        Jitter:
                          Type:  string
                        Match:
                          Description:  TransportServerMatch defines the parameters of a custom health check.
                          Properties:
                            Expect:
                              Type:  string
                            Send:
                              Type:  string
                          Type:      object
                        Passes:
                          Type:  integer
                        Port:
                          Type:  integer
                        Timeout:
                          Type:  string
                      Type:      object
                    Load Balancing Method:
                      Type:  string
                    Max Conns:
                      Type:  integer
                    Max Fails:
                      Type:  integer
                    Name:
                      Type:  string
                    Port:
                      Type:  integer
                    Service:
                      Type:  string
                  Type:      object
                Type:        array
            Type:            object
          Status:
            Description:  TransportServerStatus defines the status for the TransportServer resource.
            Properties:
              Message:
                Type:  string
              Reason:
                Type:  string
              State:
                Type:  string
            Type:      object
        Type:          object
    Served:            true
    Storage:           false
    Subresources:
      Status:
Status:
  Accepted Names:
    Kind:       TransportServer
    List Kind:  TransportServerList
    Plural:     transportservers
    Short Names:
      ts
    Singular:  transportserver
  Conditions:
    Last Transition Time:  2022-11-03T03:29:31Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2022-11-03T03:29:31Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1alpha1
    v1
Events:  <none>

@vepatel
Copy link
Contributor

vepatel commented Dec 19, 2024 •
edited
Loading

yep those two you've posted seems different from 3.7.2 CRDs. see https://github.com/nginxinc/kubernetes-ingress/blob/v3.7.2/config/crd/bases/k8s.nginx.org_transportservers.yaml#L330 and https://github.com/nginxinc/kubernetes-ingress/blob/v3.7.2/config/crd/bases/k8s.nginx.org_globalconfigurations.yaml#L110

 Name:           v1alpha1
    Schema:
      openAPIV3Schema:
        Description:  TransportServer defines the TransportServer resource.
        Properties:
          API Version:
            Description:  APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            Type:  string
          Kind:
            Description:  Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            Type:  string
          Metadata:
            Type:  object
          Spec:
            Description:  TransportServerSpec is the spec of the TransportServer resource.
            Properties:
              Action:
                Description:  TransportServerAction defines an action.
                Properties:
                  Pass:
                    Type:  string
                Type:      object
              Host:
                Type:  string
              Ingress Class Name:
                Type:  string
              Listener:
                Description:  TransportServerListener defines a listener for a TransportServer.
                Properties:
                  Name:
                    Type:  string
                  Protocol:
                    Type:  string
                Type:      object
              Server Snippets:
                Type:  string
              Session Parameters:
                Description:  SessionParameters defines session parameters.
                Properties:
                  Timeout:
                    Type:  string
                Type:      object
              Stream Snippets:
                Type:  string
              Tls:
                Description:  TransportServerTLS defines TransportServerTLS configuration for a TransportServer.
                Properties:
                  Secret:
                    Type:  string
                Type:      object
              Upstream Parameters:
                Description:  UpstreamParameters defines parameters for an upstream.
                Properties:
                  Connect Timeout:
                    Type:  string
                  Next Upstream:
                    Type:  boolean
                  Next Upstream Timeout:
                    Type:  string
                  Next Upstream Tries:
                    Type:  integer
                  Udp Requests:
                    Type:  integer
                  Udp Responses:
                    Type:  integer
                Type:      object
              Upstreams:
                Items:
                  Description:  TransportServerUpstream defines an upstream.
                  Properties:
                    Backup:
                      Type:  string
                    Backup Port:
                      Type:  integer
                    Fail Timeout:
                      Type:  string
                    Health Check:
                      Description:  TransportServerHealthCheck defines the parameters for active Upstream HealthChecks.
                      Properties:
                        Enable:
                          Type:  boolean
                        Fails:
                          Type:  integer
                        Interval:
                          Type:  string
                        Jitter:
                          Type:  string
                        Match:
                          Description:  TransportServerMatch defines the parameters of a custom health check.
                          Properties:
                            Expect:
                              Type:  string
                            Send:
                              Type:  string
                          Type:      object
                        Passes:
                          Type:  integer
                        Port:
                          Type:  integer
                        Timeout:
                          Type:  string
                      Type:      object
                    Load Balancing Method:
                      Type:  string
                    Max Conns:
                      Type:  integer
                    Max Fails:
                      Type:  integer
                    Name:
                      Type:  string
                    Port:
                      Type:  integer
                    Service:
                      Type:  string
                  Type:      object
                Type:        array
            Type:            object
          Status:
            Description:  TransportServerStatus defines the status for the TransportServer resource.
            Properties:
              Message:
                Type:  string
              Reason:
                Type:  string
              State:
                Type:  string
            Type:      object
        Type:          object
    Served:            true
    Storage:           false
    Subresources:
      Status:
Status:
  Accepted Names:
    Kind:       TransportServer
    List Kind:  TransportServerList
    Plural:     transportservers
    Short Names:
      ts
    Singular:  transportserver
  Conditions:
    Last Transition Time:  2024-12-19T11:37:40Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2024-12-19T11:37:40Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1
Events:  <none>

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v3.7.2

in 3.7.2 storage is set to false for v1alpha1 however in your output v1alpha1 appears in Stored Versions.

@kyrofa
Copy link
Author

kyrofa commented Dec 19, 2024 •
edited
Loading

Okay interesting, I ran kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v3.7.2/deploy/crds.yaml without issue (I do this every update), are the docs wrong on this point? Is there some sort of migration that needs to happen here? I have never manually edited these, so something went sideways over time.

@vepatel
Copy link
Contributor

vepatel commented Dec 20, 2024 •
edited
Loading

Hi @kyrofa it does looks like an issue when coming from earlier releases, going from 3.3.2 to 3.4.0 (which added v1 for TS and GC), we're looking into this.
Thanks for all the info

@vepatel
Copy link
Contributor

vepatel commented Dec 20, 2024 •
edited
Loading

@kyrofa If you don't have any v1alpha1 resources, please run:

kubectl patch customresourcedefinitions transportservers.k8s.nginx.org --subresource='status' --type='merge' -p '{"status":{"storedVersions": ["v1"]}}'
and
kubectl patch customresourcedefinitions globalconfigurations.k8s.nginx.org --subresource='status' --type='merge' -p '{"status":{"storedVersions": ["v1"]}}'
then
kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v4.0.0/deploy/crds.yaml

  Stored Versions:
    v1alpha1
    v1
Events:  <none>


~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v4.0.0 ● λ kubectl patch customresourcedefinitions transportservers.k8s.nginx.org --subresource='status' --type='merge' -p '{"status":{"storedVersions": ["v1"]}}'
customresourcedefinition.apiextensions.k8s.io/transportservers.k8s.nginx.org patched

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v4.0.0 ● λ kubectl patch customresourcedefinitions globalconfigurations.k8s.nginx.org --subresource='status' --type='merge' -p '{"status":{"storedVersions": ["v1"]}}'
customresourcedefinition.apiextensions.k8s.io/globalconfigurations.k8s.nginx.org patched

~/nginx/kubernetes-ingress/charts/nginx-ingress on tags/v4.0.0 ● λ kaf crds/                                      
customresourcedefinition.apiextensions.k8s.io/aplogconfs.appprotect.f5.com configured
customresourcedefinition.apiextensions.k8s.io/appolicies.appprotect.f5.com configured
customresourcedefinition.apiextensions.k8s.io/apusersigs.appprotect.f5.com configured
customresourcedefinition.apiextensions.k8s.io/apdoslogconfs.appprotectdos.f5.com configured
customresourcedefinition.apiextensions.k8s.io/apdospolicies.appprotectdos.f5.com configured
customresourcedefinition.apiextensions.k8s.io/dosprotectedresources.appprotectdos.f5.com unchanged
customresourcedefinition.apiextensions.k8s.io/dnsendpoints.externaldns.nginx.org unchanged
customresourcedefinition.apiextensions.k8s.io/globalconfigurations.k8s.nginx.org configured
customresourcedefinition.apiextensions.k8s.io/policies.k8s.nginx.org unchanged
customresourcedefinition.apiextensions.k8s.io/transportservers.k8s.nginx.org configured
customresourcedefinition.apiextensions.k8s.io/virtualserverroutes.k8s.nginx.org unchanged
customresourcedefinition.apiextensions.k8s.io/virtualservers.k8s.nginx.org unchanged

this should fix the issue.
Also we'll update the docs to reflect the additional steps.

@vepatel vepatel mentioned this issue Dec 20, 2024
Merged
6 tasks
@nginx-bot nginx-bot mentioned this issue Dec 20, 2024
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug An issue reporting a potential bug waiting for response Waiting for author's response
Projects
NGINX Ingress Controller
Status: Todo ☑
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kyrofa @vepatel