Handle recovery from resize failure

[gnufied]

Currently when resize fails of a volume it keeps retrying indefinitely. This leads to two problems:

1. Unnecessary API usage for an action that is not going to succeed (for example when you are out of quota on cloud provider or out of bricks on glusterfs).

2. Sometimes user may want to retry volume expansion with lower value. For example - my current PVC is of 12GB and I tried to expand it to 40GB. But that failed. Now I want to retry volume expansion with 20GB except I can’t.

/sig storage

cc @bswartz @saad-ali

[bswartz]

I honestly don't see (1) as a problem. This is how everything in kubernetes works.

(2) Is the case that interests me. As long as the actual size doesn't change, it should be legal to "cancel" the resize request by changing the spec size back to the old value. Or as you say, retry a resize with a smaller increment.

I think we also want to think about future compatibility with a volume-shrink feature. It's inevitable that users will ask for it, and it's inevitable that at least a subset of implementers will want to implement it. Assuming we allow the feature to happen, we'll want the interface to work straightforwardly.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gnufied]

/reopen
/remove-lifecycle-rotten
/lifecycle frozen

[k8s-ci-robot]

@gnufied: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle-rotten
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[andyzhangx]

@gnufied do you know how to cancel volume resize? e.g. current size is 4TB, and resize to 6TB failed, how can I change back to 4TB?

[gnufied]

@andyzhangx You can use - https://docs.openshift.com/container-platform/4.5/storage/expanding-persistent-volumes.html#expanding-recovering-from-failure_expanding-persistent-volumes

[FengJunLiu]

@gnufied https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1790-recover-resize-failure#recovery-from-volume-expansion-failure
This KEP describes the drawback of not being able to restore the capacity of PVC to its original capacity in the event of PVC expansion failure. Is there a plan to address this drawback in the future? thanks