system:serviceaccount:cert-manager:csi-snapshotter cannot create resource ... #127

tpoxa · 2019-12-08T20:30:19Z

~/csi-driver-host-path/deploy/kubernetes-1.16$ ./deploy-hostpath.sh

Installation failed after 5m tmeout

pod/csi-hostpath-snapshotter-0                0/1     CrashLoopBackOff   7          14m

Status:"Failure", Message:"customresourcedefinitions.apiextensions.k8s.io is forbidden: User \"system:serviceaccount:cert-manager:csi-snapshotter\" cannot create resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope", Reason:"Forbidden"

Linux Debian-911-stretch-64-minimal 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3 (2019-09-02) x86_64 GNU/Linux

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

The text was updated successfully, but these errors were encountered:

pohly · 2019-12-09T07:02:30Z

https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/deploy/kubernetes-1.16/hostpath/csi-hostpath-snapshotter.yaml#L43 install v1.2.0 of the external-snapshotter, which means that the deploy script should download the corresponding RBAC rules, which have: https://github.com/kubernetes-csi/external-snapshotter/blob/b3f591d85cce516e431c70e5337d5c67611ae2fe/deploy/kubernetes/rbac.yaml#L50-L52

"Should" - some of the recent changes to the script broke the content of CSI_SNAPSHOTTER_RBAC_YAML, leading to it not just containing the URL but also some extra garbage:

Using non-default RBAC rules for CSI_SNAPSHOTTER. Changes from https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml to https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml
CSI_RESIZER_RBAC_YAML=https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml
: https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml

INSTALL_CRD=false

# Some images are not affected by *_REGISTRY/*_TAG and IMAGE_* variables.
# The default is to update unless explicitly excluded.
update_image () {
    case  in socat) return 1;; esac
 are:

Everything between CSI_RESIZER_RBAC_YAML (inclusive) and are (exclusive) shouldn't be there.

The reason is a missing closing bracket, this fixes it:

diff --git a/deploy/util/deploy-hostpath.sh b/deploy/util/deploy-hostpath.sh
index 69d790de..bb5148a0 100755
--- a/deploy/util/deploy-hostpath.sh
+++ b/deploy/util/deploy-hostpath.sh
@@ -121,7 +121,7 @@ CSI_PROVISIONER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/exte
 CSI_ATTACHER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher false)/deploy/kubernetes/rbac.yaml"
 : ${CSI_ATTACHER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}
 CSI_SNAPSHOTTER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter false)/deploy/kubernetes/${SNAPSHOTTER_RBAC_RELATIVE_PATH}"
-: ${CSI_SNAPSHOTTER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter "${UPDATE_RBAC_RULES}")/deploy/kubernetes/${SNAPSHOTTER_RBAC_RELATIVE_PATH}
+: ${CSI_SNAPSHOTTER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter "${UPDATE_RBAC_RULES}")/deploy/kubernetes/${SNAPSHOTTER_RBAC_RELATIVE_PATH}}
 CSI_RESIZER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-resizer/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-resizer.yaml" csi-resizer false)/deploy/kubernetes/rbac.yaml"
 : ${CSI_RESIZER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-resizer/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-resizer.yaml" csi-resizer "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}

The more interesting question is: why was this not caught by the CI when merging #98?

The test job shows the same broken output (https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_csi-driver-host-path/98/pull-kubernetes-csi-csi-driver-host-path-1-16-on-kubernetes-1-16/1202409554004938755/build-log.txt).

Does the KinD cluster perhaps not have RBAC enforcement turned on?

/cc @xing-yang @ggriffiths

pohly · 2019-12-09T07:03:18Z

/cc @msau42

pohly · 2019-12-09T07:17:02Z

The other problem is that the wrong URL didn't lead to some obvious error and early aborting of the script. For wget, it is the --quiet option which suppresses the error reporting and we don't check the return code during the diff. But that's just for diagnostics.

The actual download happens with kubectl apply -f. My version of kubectl (v1.16.0-rc.1.19+4cb51f0d2d8392) seems to ignore the extra garbage:

$ kubectl apply -f "https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml
hello

fjahud72="
serviceaccount/csi-snapshotter unchanged
clusterrole.rbac.authorization.k8s.io/external-snapshotter-runner unchanged
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-role unchanged
role.rbac.authorization.k8s.io/external-snapshotter-leaderelection unchanged
rolebinding.rbac.authorization.k8s.io/external-snapshotter-leaderelection unchanged

@tpoxa: does your version perhaps handle this differently? Which version is that?

I bet there is a difference in kubectl and that also explains why it worked in the CI, because the CI run does install the RBAC rules:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml
CSI_RESIZER_RBAC_YAML=https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml
: https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml

INSTALL_CRD=false

# Some images are not affected by *_REGISTRY/*_TAG and IMAGE_* variables.
# The default is to update unless explicitly excluded.
update_image () {
    case  in socat) return 1;; esac

serviceaccount/csi-snapshotter created
clusterrole.rbac.authorization.k8s.io/external-snapshotter-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-role created
role.rbac.authorization.k8s.io/external-snapshotter-leaderelection created
rolebinding.rbac.authorization.k8s.io/external-snapshotter-leaderelection created

tpoxa · 2019-12-09T09:47:12Z

This branch did not make big difference for me.

maksym@Debian-911-stretch-64-minimal:~/csi-driver-host-path/deploy/kubernetes-1.16$ ./deploy-hostpath.sh
SNAPSHOTTER_RBAC_RELATIVE_PATH rbac.yaml
applying RBAC rules
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.4.0/deploy/kubernetes/rbac.yaml
serviceaccount/csi-provisioner unchanged
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner unchanged
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role unchanged
role.rbac.authorization.k8s.io/external-provisioner-cfg unchanged
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg unchanged
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-attacher/v2.0.0/deploy/kubernetes/rbac.yaml
serviceaccount/csi-attacher unchanged
clusterrole.rbac.authorization.k8s.io/external-attacher-runner unchanged
clusterrolebinding.rbac.authorization.k8s.io/csi-attacher-role unchanged
role.rbac.authorization.k8s.io/external-attacher-cfg unchanged
rolebinding.rbac.authorization.k8s.io/csi-attacher-role-cfg unchanged
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml
serviceaccount/csi-snapshotter created
clusterrole.rbac.authorization.k8s.io/external-snapshotter-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-role created
role.rbac.authorization.k8s.io/external-snapshotter-leaderelection created
rolebinding.rbac.authorization.k8s.io/external-snapshotter-leaderelection created
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml
serviceaccount/csi-resizer created
clusterrole.rbac.authorization.k8s.io/external-resizer-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-resizer-role created
role.rbac.authorization.k8s.io/external-resizer-cfg created
rolebinding.rbac.authorization.k8s.io/csi-resizer-role-cfg created
deploying hostpath components
   ./hostpath/csi-hostpath-attacher.yaml
        using           image: quay.io/k8scsi/csi-attacher:v2.0.0
service/csi-hostpath-attacher created
statefulset.apps/csi-hostpath-attacher created
   ./hostpath/csi-hostpath-driverinfo.yaml
csidriver.storage.k8s.io/hostpath.csi.k8s.io unchanged
   ./hostpath/csi-hostpath-plugin.yaml
        using           image: quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
        using           image: quay.io/k8scsi/hostpathplugin:v1.2.0
        using           image: quay.io/k8scsi/livenessprobe:v1.1.0
service/csi-hostpathplugin created
statefulset.apps/csi-hostpathplugin created
   ./hostpath/csi-hostpath-provisioner.yaml
        using           image: quay.io/k8scsi/csi-provisioner:v1.4.0
service/csi-hostpath-provisioner created
statefulset.apps/csi-hostpath-provisioner created
   ./hostpath/csi-hostpath-resizer.yaml
        using           image: quay.io/k8scsi/csi-resizer:v0.3.0
service/csi-hostpath-resizer created
statefulset.apps/csi-hostpath-resizer created
   ./hostpath/csi-hostpath-snapshotter.yaml
        using           image: quay.io/k8scsi/csi-snapshotter:v1.2.0
service/csi-hostpath-snapshotter created
statefulset.apps/csi-hostpath-snapshotter created
   ./hostpath/csi-hostpath-testing.yaml
        using           image: alpine/socat:1.0.3
service/hostpath-service created
statefulset.apps/csi-hostpath-socat created
09:14:26 waiting for hostpath deployment to complete, attempt #0
09:14:36 waiting for hostpath deployment to complete, attempt #1
09:14:47 waiting for hostpath deployment to complete, attempt #2
09:14:57 waiting for hostpath deployment to complete, attempt #3
09:15:07 waiting for hostpath deployment to complete, attempt #4

kubectl logs pod/csi-hostpath-snapshotter-0
I1209 08:17:39.976602       1 main.go:89] Version: v1.2.0-0-gb3f591d8
W1209 08:17:39.976658       1 main.go:92] --connection-timeout is deprecated and will have no effect
F1209 08:17:39.985822       1 create_crd.go:50] failed to create VolumeSnapshotResource: &v1beta1.CustomResourceDefinition{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1beta1.CustomResourceDefinitionSpec{Group:"", Version:"", Names:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, Scope:"", Validation:(*v1beta1.CustomResourceValidation)(nil), Subresources:(*v1beta1.CustomResourceSubresources)(nil), Versions:[]v1beta1.CustomResourceDefinitionVersion(nil), AdditionalPrinterColumns:[]v1beta1.CustomResourceColumnDefinition(nil), Conversion:(*v1beta1.CustomResourceConversion)(nil)}, Status:v1beta1.CustomResourceDefinitionStatus{Conditions:[]v1beta1.CustomResourceDefinitionCondition(nil), AcceptedNames:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, StoredVersions:[]string(nil)}}, err: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:"customresourcedefinitions.apiextensions.k8s.io is forbidden: User \"system:serviceaccount:cert-manager:csi-snapshotter\" cannot create resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc00039c3c0), Code:403}}
User \"system:serviceaccount:cert-manager:csi-snapshotter\" cannot create resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope", Reason:"Forbidden"

Maybe this is cert-manager related issue?

pohly · 2019-12-09T11:08:20Z

@tpoxa which version of kubectl are you using?

Can you check that you now get the expected RBAC rules? I have:

$ kubectl get -o yaml pods/csi-hostpath-snapshotter-0 | grep serviceAccount
  serviceAccount: csi-snapshotter
  serviceAccountName: csi-snapshotter

$ kubectl describe serviceaccount/csi-snapshotter
Name:                csi-snapshotter
Namespace:           default
Labels:              <none>
Annotations:         kubectl.kubernetes.io/last-applied-configuration:
                       {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"csi-snapshotter","namespace":"default"}}
Image pull secrets:  <none>
Mountable secrets:   csi-snapshotter-token-8894h
Tokens:              csi-snapshotter-token-8894h
Events:              <none>

$ kubectl describe clusterroles/external-snapshotter-runner
Name:         external-snapshotter-runner
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"external-snapshotter-runner"},"rule...
PolicyRule:
  Resources                                       Non-Resource URLs  Resource Names  Verbs
  ---------                                       -----------------  --------------  -----
  volumesnapshotcontents.snapshot.storage.k8s.io  []                 []              [create get list watch update delete]
  customresourcedefinitions.apiextensions.k8s.io  []                 []              [create list watch delete get update]
  persistentvolumeclaims                          []                 []              [get list watch update]
  volumesnapshots.snapshot.storage.k8s.io         []                 []              [get list watch update]
  persistentvolumes                               []                 []              [get list watch]
  volumesnapshotclasses.snapshot.storage.k8s.io   []                 []              [get list watch]
  storageclasses.storage.k8s.io                   []                 []              [get list watch]
  secrets                                         []                 []              [get list]
  events                                          []                 []              [list watch create update patch]
  volumesnapshots.snapshot.storage.k8s.io/status  []                 []              [update]

$ kubectl describe clusterrolebinding/csi-snapshotter-role
Name:         csi-snapshotter-role
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"csi-snapshotter-role"},"role...
Role:
  Kind:  ClusterRole
  Name:  external-snapshotter-runner
Subjects:
  Kind            Name             Namespace
  ----            ----             ---------
  ServiceAccount  csi-snapshotter  default

tpoxa · 2019-12-09T14:58:06Z

kubectl get -o yaml pods/csi-hostpath-snapshotter-0 | grep serviceAccount
  serviceAccount: csi-snapshotter
  serviceAccountName: csi-snapshotter



maksym@Debian-911-stretch-64-minimal:/root$ kubectl describe serviceaccount/csi-snapshotter
Name:                csi-snapshotter
Namespace:           cert-manager
Labels:              <none>
Annotations:         kubectl.kubernetes.io/last-applied-configuration:
                       {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"csi-snapshotter","namespace":"cert-manager"}}
Image pull secrets:  <none>
Mountable secrets:   csi-snapshotter-token-hh2ld
Tokens:              csi-snapshotter-token-hh2ld
Events:              <none>




maksym@Debian-911-stretch-64-minimal:/root$ kubectl describe clusterroles/external-snapshotter-runner
Name:         external-snapshotter-runner
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"external-snapshotter-runner"},"rule...
PolicyRule:
  Resources                                       Non-Resource URLs  Resource Names  Verbs
  ---------                                       -----------------  --------------  -----
  volumesnapshotcontents.snapshot.storage.k8s.io  []                 []              [create get list watch update delete]
  customresourcedefinitions.apiextensions.k8s.io  []                 []              [create list watch delete get update]
  persistentvolumeclaims                          []                 []              [get list watch update]
  volumesnapshots.snapshot.storage.k8s.io         []                 []              [get list watch update]
  persistentvolumes                               []                 []              [get list watch]
  volumesnapshotclasses.snapshot.storage.k8s.io   []                 []              [get list watch]
  storageclasses.storage.k8s.io                   []                 []              [get list watch]
  secrets                                         []                 []              [get list]
  events                                          []                 []              [list watch create update patch]
  volumesnapshots.snapshot.storage.k8s.io/status  []                 []              [update]



maksym@Debian-911-stretch-64-minimal:/root$ kubectl describe clusterrolebinding/csi-snapshotter-role
Name:         csi-snapshotter-role
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"csi-snapshotter-role"},"role...
Role:
  Kind:  ClusterRole
  Name:  external-snapshotter-runner
Subjects:
  Kind            Name             Namespace
  ----            ----             ---------
  ServiceAccount  csi-snapshotter  default

pohly · 2019-12-09T15:49:15Z

That looks sane to me. Sorry, I'm out of ideas.

xing-yang · 2019-12-09T15:52:14Z

@ggriffiths can you please take a look? thanks.

msau42 · 2019-12-09T15:59:55Z

Hold on. Running the 1.16 versions of the hostpath deployment against a 1.12 cluster probably is going to have a lot of problems

msau42 · 2019-12-09T16:02:04Z

Sorry ignore my last comment. I misread the kubectl output. What is the cluster version?

ggriffiths · 2019-12-09T17:52:06Z

I'm not sure why #98 didn't catch this. I think we tested as far back as
pull-kubernetes-csi-csi-driver-host-path-1-14-on-kubernetes-1-14, so if it's 1.16 hostpath on an older version, that might have issues.

We create the CRDs in prow.sh before the snapshotter is deployed.

Seems like the error is coming from here in the old snapshotter:
https://github.com/kubernetes-csi/external-snapshotter/blob/release-1.2/cmd/csi-snapshotter/create_crd.go#L71

pohly · 2019-12-09T17:59:30Z

Grant Griffiths <[email protected]> writes:

I'm not sure why #98 didn't catch this. I think we tested as far back as `pull-kubernetes-csi-csi-driver-host-path-1-14-on-kubernetes-1-14`, so if it's 1.16 hostpath on an older version, that might have issues.

It seems to depend on the kubectl version. A recent kubectl silently ignored the garbage after the URL.

tpoxa · 2019-12-09T18:38:41Z

@msau42

maksym@Debian-911-stretch-64-minimal:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

tpoxa · 2019-12-09T18:41:50Z

Its clean kubernetes installation. I only have cert-manager installed with Helm.
I see cert-manager mentioned in error message.

kubectl logs pod/csi-hostpath-snapshotter-0
I1209 18:38:03.988445       1 main.go:89] Version: v1.2.0-0-gb3f591d8
W1209 18:38:03.988506       1 main.go:92] --connection-timeout is deprecated and will have no effect
F1209 18:38:03.997233       1 create_crd.go:50] failed to create VolumeSnapshotResource: &v1beta1.CustomResourceDefinition{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1beta1.CustomResourceDefinitionSpec{Group:"", Version:"", Names:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, Scope:"", Validation:(*v1beta1.CustomResourceValidation)(nil), Subresources:(*v1beta1.CustomResourceSubresources)(nil), Versions:[]v1beta1.CustomResourceDefinitionVersion(nil), AdditionalPrinterColumns:[]v1beta1.CustomResourceColumnDefinition(nil), Conversion:(*v1beta1.CustomResourceConversion)(nil)}, Status:v1beta1.CustomResourceDefinitionStatus{Conditions:[]v1beta1.CustomResourceDefinitionCondition(nil), AcceptedNames:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, StoredVersions:[]string(nil)}}, err: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:"customresourcedefinitions.apiextensions.k8s.io is forbidden: User \"system:serviceaccount:cert-manager:csi-snapshotter\" cannot create resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc0004261e0), Code:403}}

Maybe its somehow related with Cert-manager finalizers or so... Sorry I am still quite new to k8s.

msau42 · 2019-12-24T02:22:01Z

I think the issue is the namespaces. The cluster rolebinding shows csi-snapshotter service account in default namespace:

$ kubectl describe clusterrolebinding/csi-snapshotter-role
Name:         csi-snapshotter-role
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"csi-snapshotter-role"},"role...
Role:
  Kind:  ClusterRole
  Name:  external-snapshotter-runner
Subjects:
  Kind            Name             Namespace
  ----            ----             ---------
  ServiceAccount  csi-snapshotter  default

But your error message is csi-snapshotter in cert-manager namespace.

User \"system:serviceaccount:cert-manager:csi-snapshotter\" cannot create resource \"customresourcedefinitions\"

fejta-bot · 2020-03-23T03:19:02Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2020-04-22T04:01:24Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2020-05-22T04:47:24Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2020-05-22T04:47:26Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

8fdf0f7 Merge pull request kubernetes-csi#128 from fengzixu/master 1c94220 fix: fix a bug of csi-sanity a4c41e6 Merge pull request kubernetes-csi#127 from pohly/fix-boilerplate dbd8967 verify-boilerplate.sh: fix path to script 9289fd1 Merge pull request kubernetes-csi#125 from sachinkumarsingh092/optional-spelling-boilerplate-checks ad29307 Make the spelling and boilerplate checks optional 5f06d02 Merge pull request kubernetes-csi#124 from sachinkumarsingh092/fix-spellcheck-boilerplate-tests 48186eb Fix spelling and boilerplate errors 71690af Merge pull request kubernetes-csi#122 from sachinkumarsingh092/include-spellcheck-boilerplate-tests 981be3f Adding spelling and boilerplate checks. git-subtree-dir: release-tools git-subtree-split: 8fdf0f7

fe1f284 Merge pull request kubernetes-csi#121 from kvaps/namespace-check 8fdf0f7 Merge pull request kubernetes-csi#128 from fengzixu/master 1c94220 fix: fix a bug of csi-sanity a4c41e6 Merge pull request kubernetes-csi#127 from pohly/fix-boilerplate ece0f50 check namespace for snapshot-controller dbd8967 verify-boilerplate.sh: fix path to script 9289fd1 Merge pull request kubernetes-csi#125 from sachinkumarsingh092/optional-spelling-boilerplate-checks ad29307 Make the spelling and boilerplate checks optional 5f06d02 Merge pull request kubernetes-csi#124 from sachinkumarsingh092/fix-spellcheck-boilerplate-tests 48186eb Fix spelling and boilerplate errors 71690af Merge pull request kubernetes-csi#122 from sachinkumarsingh092/include-spellcheck-boilerplate-tests 981be3f Adding spelling and boilerplate checks. 2bb7525 Merge pull request kubernetes-csi#117 from fengzixu/master 3b6d17b Merge pull request kubernetes-csi#118 from pohly/cloud-build-timeout 9318c6c cloud build: double the timeout, now 1 hour 4ab8b15 use the tag to replace commit of csi-test 5d74e45 change the csi-test import path to v4 7dcd0a9 upgrade csi-test to v4.0.2 86ff580 Merge pull request kubernetes-csi#116 from andyzhangx/export-image-name c3a9662 allow export image name and registry name git-subtree-dir: release-tools git-subtree-split: fe1f284

00bc64a9 remove travis.yml, Go 1.16 REVERT: 7bc70e5 Merge pull request kubernetes-csi#129 from pohly/squash-documentation REVERT: e0b02e7 README.md: document usage of --squash REVERT: 316cb95 Merge pull request kubernetes-csi#132 from yiyang5055/bugfix/boilerplate REVERT: 26e2ab1 fix: default boilerplate path REVERT: 1add8c1 Merge pull request kubernetes-csi#133 from pohly/kubernetes-1.20-tag REVERT: 3e811d6 prow.sh: fix "on-master" prow jobs REVERT: 1d60e77 Merge pull request kubernetes-csi#131 from pohly/kubernetes-1.20-tag REVERT: 9f10459 prow.sh: support building Kubernetes for a specific version REVERT: fe1f284 Merge pull request kubernetes-csi#121 from kvaps/namespace-check REVERT: 8fdf0f7 Merge pull request kubernetes-csi#128 from fengzixu/master REVERT: 1c94220 fix: fix a bug of csi-sanity REVERT: a4c41e6 Merge pull request kubernetes-csi#127 from pohly/fix-boilerplate REVERT: ece0f50 check namespace for snapshot-controller REVERT: dbd8967 verify-boilerplate.sh: fix path to script REVERT: 9289fd1 Merge pull request kubernetes-csi#125 from sachinkumarsingh092/optional-spelling-boilerplate-checks REVERT: ad29307 Make the spelling and boilerplate checks optional REVERT: 5f06d02 Merge pull request kubernetes-csi#124 from sachinkumarsingh092/fix-spellcheck-boilerplate-tests REVERT: 48186eb Fix spelling and boilerplate errors REVERT: 71690af Merge pull request kubernetes-csi#122 from sachinkumarsingh092/include-spellcheck-boilerplate-tests REVERT: 981be3f Adding spelling and boilerplate checks. REVERT: 2bb7525 Merge pull request kubernetes-csi#117 from fengzixu/master REVERT: 3b6d17b Merge pull request kubernetes-csi#118 from pohly/cloud-build-timeout REVERT: 9318c6c cloud build: double the timeout, now 1 hour REVERT: 4ab8b15 use the tag to replace commit of csi-test REVERT: 5d74e45 change the csi-test import path to v4 REVERT: 7dcd0a9 upgrade csi-test to v4.0.2 REVERT: 86ff580 Merge pull request kubernetes-csi#116 from andyzhangx/export-image-name REVERT: c3a9662 allow export image name and registry name REVERT: c6a88c6 Merge pull request kubernetes-csi#113 from xing-yang/install_snapshot_controller REVERT: 45ec4c6 Fix the install of snapshot CRDs and controller REVERT: 5d874cc Merge pull request kubernetes-csi#112 from xing-yang/cleanup REVERT: 79bbca7 Cleanup REVERT: d437673 Merge pull request kubernetes-csi#111 from xing-yang/update_snapshot_v1_rc REVERT: 57718f8 Update snapshot CRD version REVERT: 4aff857 Merge pull request kubernetes-csi#109 from pohly/alpha-test-defaults REVERT: 0427289 Merge pull request kubernetes-csi#110 from pohly/kind-0.9-bazel-build-workaround REVERT: 9a370ab prow.sh: work around "kind build node-image" failure REVERT: 522361e prow.sh: only run alpha tests for latest Kubernetes release REVERT: 22c0395 Merge pull request kubernetes-csi#108 from bnrjee/master REVERT: b5b447b Add go ldflags using LDFLAGS at the time of compilation REVERT: 16f4afb Merge pull request kubernetes-csi#107 from pohly/kind-update git-subtree-dir: release-tools git-subtree-split: 00bc64a9b7c2585676b1da301a7acae8797229a9

verify-boilerplate.sh: fix path to script

pohly mentioned this issue Dec 9, 2019

deploy: fix snapshotter RBAC creation #128

Merged

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 23, 2020

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 22, 2020

k8s-ci-robot closed this as completed May 22, 2020

TerryHowe pushed a commit to TerryHowe/csi-driver-host-path that referenced this issue Oct 17, 2024

Merge pull request kubernetes-csi#127 from pohly/fix-boilerplate

a4c41e6

verify-boilerplate.sh: fix path to script

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

system:serviceaccount:cert-manager:csi-snapshotter cannot create resource ... #127

system:serviceaccount:cert-manager:csi-snapshotter cannot create resource ... #127

tpoxa commented Dec 8, 2019 •

edited

Loading

pohly commented Dec 9, 2019

pohly commented Dec 9, 2019

pohly commented Dec 9, 2019

tpoxa commented Dec 9, 2019 •

edited

Loading

pohly commented Dec 9, 2019 •

edited

Loading

tpoxa commented Dec 9, 2019

pohly commented Dec 9, 2019

xing-yang commented Dec 9, 2019

msau42 commented Dec 9, 2019

msau42 commented Dec 9, 2019

ggriffiths commented Dec 9, 2019

pohly commented Dec 9, 2019 via email

tpoxa commented Dec 9, 2019

tpoxa commented Dec 9, 2019

msau42 commented Dec 24, 2019 •

edited

Loading

fejta-bot commented Mar 23, 2020

fejta-bot commented Apr 22, 2020

fejta-bot commented May 22, 2020

k8s-ci-robot commented May 22, 2020

system:serviceaccount:cert-manager:csi-snapshotter cannot create resource ... #127

system:serviceaccount:cert-manager:csi-snapshotter cannot create resource ... #127

Comments

tpoxa commented Dec 8, 2019 • edited Loading

pohly commented Dec 9, 2019

pohly commented Dec 9, 2019

pohly commented Dec 9, 2019

tpoxa commented Dec 9, 2019 • edited Loading

pohly commented Dec 9, 2019 • edited Loading

tpoxa commented Dec 9, 2019

pohly commented Dec 9, 2019

xing-yang commented Dec 9, 2019

msau42 commented Dec 9, 2019

msau42 commented Dec 9, 2019

ggriffiths commented Dec 9, 2019

pohly commented Dec 9, 2019 via email

tpoxa commented Dec 9, 2019

tpoxa commented Dec 9, 2019

msau42 commented Dec 24, 2019 • edited Loading

fejta-bot commented Mar 23, 2020

fejta-bot commented Apr 22, 2020

fejta-bot commented May 22, 2020

k8s-ci-robot commented May 22, 2020

tpoxa commented Dec 8, 2019 •

edited

Loading

tpoxa commented Dec 9, 2019 •

edited

Loading

pohly commented Dec 9, 2019 •

edited

Loading

msau42 commented Dec 24, 2019 •

edited

Loading