PVC stuck in pending state after provisioner retry exceeds threshold

[leakingtapan]

/kind bug

When CSI driver returns error on CreateVolume, the same operation will be retried by external-provisioner. However, after provisioner retry threshold is exceeded, the pvc is stuck in pending state:

>> kk get pvc
NAME            STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
voltype-claim   Pending                                      voltype-sc     6h17m

>> kk describe pvc
Name:          voltype-claim
Namespace:     default
StorageClass:  voltype-sc
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: com.amazon.aws.csi.ebs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
Events:
  Type       Reason                Age                     From                         Message
  ----       ------                ----                    ----                         -------
  Normal     ExternalProvisioning  79s (x1524 over 6h21m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "com.amazon.aws.csi.ebs" or manually created by system administrator
Mounted By:  <none>

provisioner log:

I1019 01:09:44.103840       1 controller.go:927] provision "default/voltype-claim" class "voltype-sc": started
I1019 01:09:44.116655       1 controller.go:124] GRPC call: /csi.v0.Identity/GetPluginCapabilities
I1019 01:09:44.116671       1 controller.go:125] GRPC request:
I1019 01:09:44.116828       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"voltype-claim", UID:"a8d939de-d324-11e8-81f1-0a75e9a76798", APIVersion:"v1", ResourceVersion:"539476", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/voltype-claim"
I1019 01:09:44.117364       1 controller.go:127] GRPC response: capabilities:<service:<type:CONTROLLER_SERVICE > > capabilities:<service:<type:ACCESSIBILITY_CONSTRAINTS > >
I1019 01:09:44.117398       1 controller.go:128] GRPC error: <nil>
I1019 01:09:44.117408       1 controller.go:124] GRPC call: /csi.v0.Controller/ControllerGetCapabilities
I1019 01:09:44.117412       1 controller.go:125] GRPC request:
I1019 01:09:44.117752       1 controller.go:127] GRPC response: capabilities:<rpc:<type:CREATE_DELETE_VOLUME > > capabilities:<rpc:<type:PUBLISH_UNPUBLISH_VOLUME > >
I1019 01:09:44.117797       1 controller.go:128] GRPC error: <nil>
I1019 01:09:44.117812       1 controller.go:124] GRPC call: /csi.v0.Identity/GetPluginInfo
I1019 01:09:44.117818       1 controller.go:125] GRPC request:
I1019 01:09:44.118081       1 controller.go:127] GRPC response: name:"com.amazon.aws.csi.ebs" vendor_version:"0.0.1"
I1019 01:09:44.118106       1 controller.go:128] GRPC error: <nil>
I1019 01:09:44.124193       1 controller.go:432] CreateVolumeRequest {Name:pvc-a8d939de-d324-11e8-81f1-0a75e9a76798 CapacityRange:required_bytes:4294967296  VolumeCapabilities:[mount:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[iopsPerGB:100 type:io1] ControllerCreateSecrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:requisite:<segments:<key:"com.amazon.aws.csi.ebs/zone" value:"us-east-1c" > > requisite:<segments:<key:"com.amazon.aws.csi.ebs/zone" value:"us-east-1a" > > requisite:<segments:<key:"com.amazon.aws.csi.ebs/zone" value:"us-east-1b" > >  XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1019 01:09:44.124261       1 controller.go:124] GRPC call: /csi.v0.Controller/CreateVolume
I1019 01:09:44.124267       1 controller.go:125] GRPC request: name:"pvc-a8d939de-d324-11e8-81f1-0a75e9a76798" capacity_range:<required_bytes:4294967296 > volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"iopsPerGB" value:"100" > parameters:<key:"type" value:"io1" > accessibility_requirements:<requisite:<segments:<key:"com.amazon.aws.csi.ebs/zone" value:"us-east-1c" > > requisite:<segments:<key:"com.amazon.aws.csi.ebs/zone" value:"us-east-1a" > > requisite:<segments:<key:"com.amazon.aws.csi.ebs/zone" value:"us-east-1b" > > >
I1019 01:09:44.542455       1 controller.go:127] GRPC response:
I1019 01:09:44.542491       1 controller.go:128] GRPC error: rpc error: code = Internal desc = Could not create volume "pvc-a8d939de-d324-11e8-81f1-0a75e9a76798": could not create volume in EC2: InvalidParameterValue: Iops to volume size ratio of 100.000000 is too high; maximum is 50
        status code: 400, request id: a3283dcc-40a6-43ea-9eb6-7ea44387605a
E1019 01:09:44.542552       1 controller.go:689] Giving up syncing claim "default/voltype-claim" because failures 15 >= threshold 15
E1019 01:09:44.542613       1 controller.go:701] error syncing claim "default/voltype-claim": failed to provision volume with StorageClass "voltype-sc": rpc error: code = Internal desc = Could not create volume "pvc-a8d939de-d324-11e8-81f1-0a75e9a76798": could not create volume in EC2: InvalidParameterValue: Iops to volume size ratio of 100.000000 is too high; maximum is 50
        status code: 400, request id: a3283dcc-40a6-43ea-9eb6-7ea44387605a
I1019 01:09:44.542651       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"voltype-claim", UID:"a8d939de-d324-11e8-81f1-0a75e9a76798", APIVersion:"v1", ResourceVersion:"539476", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "voltype-sc": rpc error: code = Internal desc = Could not create volume "pvc-a8d939de-d324-11e8-81f1-0a75e9a76798": could not create volume in EC2: InvalidParameterValue: Iops to volume size ratio of 100.000000 is too high; maximum is 50
        status code: 400, request id: a3283dcc-40a6-43ea-9eb6-7ea44387605a

[humblec]

@leakingtapan this is expected. External provisioner fail after threshold attempts. In that stage, user has to delete the PVC and recreate it.

[humblec]

/assign @humblec

[k8s-ci-robot]

@humblec: GitHub didn't allow me to assign the following users: humblec.

Note that only kubernetes-csi members and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @humblec

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[humblec]

@jsafrane @msau42 can you confirm about our plans on sticking on current design ?

[msau42]

I don't think the external provisioner should give up. It can retry with backoff, but I think it should be forever reconciling until the PVC is deleted.

[jsafrane]

Filled #216 that re-tries to provision a volume forever, until user explicitly asks for fixed retry count.

[jsafrane]

#216 is merged
/close

[k8s-ci-robot]

@jsafrane: Closing this issue.

In response to this:

#216 is merged
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.