Bug 2047732: [IBM]Volume is not deleted after destroy cluster #5962

sameshai · 2022-06-03T12:27:04Z

Description

The existing GCP destroy disk logic will continue to execute listdisks() until there are zero pending items in the list i.e until all disks are deleted successfully. We found this logic as redundant as there is no point in calling list disks as we are in uninstallation phase. There is no chance of new disk coming into picture.

Hence we have come-up with combination of both the flows. Note that we will be running infinitely until all the items in the list are deleted successfully. The difference is we are avoiding unnecessary listDisks() calls like GCP and making sure in the same first call to destroy disk we will wait for all disks to go away. If still something is pending it will be retried from the pending list.

1.) We will list all disks using pagination and filter it based on tags.
2.) The list of disks in #1 will be added to pendingItem list.
3.) Disk deletion will be invoked for all the disks and if any of the call fails it will return error and retried later with pendingItem list logic.
4.) Once all disk deletion is invoked we will wait for the all the disk deletion calls to get completed. Ideally first few disk deletion will take time , rest of others would have been already deleted.
5.) The pendingItem list will be empty only if all disk deletion would have been deleted successfully.
6.) If there is any issue in disk deletion for around 20 mins user will have to take action as suggested in the error for respective disks.

time="2022-06-10T12:20:22Z" level=debug msg="Failed to delete disk name=pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8, id=r018-ef565808-3491-4262-a6d5-27428e27ec0b.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-ef565808-3491-4262-a6d5-27428e27ec0b: Delete volume failed. Volume can be deleted only when its status is available or failed."

openshift-ci · 2022-06-03T12:27:38Z

Hi @sameshai. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/destroy/ibmcloud/disk.go

pkg/destroy/ibmcloud/ibmcloud.go

pkg/destroy/ibmcloud/disk.go

r4f4 · 2022-06-07T12:12:02Z

You need to fix the title to start with "Bug XXXX" so it's properly linked with bugzilla.

go.mod

r4f4 · 2022-06-07T12:51:45Z

/ok-to-test

r4f4 · 2022-06-07T15:03:31Z

/bugzilla-refresh

r4f4 · 2022-06-07T15:39:11Z

/bugzilla refresh

openshift-ci · 2022-06-07T15:39:13Z

@r4f4: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

r4f4 · 2022-06-08T07:37:37Z

/hold

jsafrane · 2022-06-10T11:32:57Z

/bugzilla refresh

openshift-ci · 2022-06-10T11:32:59Z

@jsafrane: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jsafrane · 2022-06-10T11:33:31Z

missing double colon :-)
/retitle Bug 2047732: [IBM]Volume is not deleted after destroy cluster

openshift-ci · 2022-06-10T11:33:39Z

@sameshai: This pull request references Bugzilla bug 2047732, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.11.0) matches configured target release for branch (4.11.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @chao007

Details

In response to this:

Bug 2047732: [IBM]Volume is not deleted after destroy cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sameshai · 2022-06-10T12:42:28Z

Uninstall logs for verification

time="2022-06-10T12:20:21Z" level=info msg="Listing disks"
time="2022-06-10T12:20:22Z" level=debug msg="Found disk: pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea"
time="2022-06-10T12:20:22Z" level=debug msg="Found disk: pvc-0fe97d2d-873c-4c94-a8d6-41370be32599"
time="2022-06-10T12:20:22Z" level=debug msg="Found disk: pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8"
time="2022-06-10T12:20:22Z" level=debug msg="Found disk: pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c"
time="2022-06-10T12:20:22Z" level=debug msg="Found disk: pvc-8644d499-9154-41f9-b82f-7a3eb69690f6"
time="2022-06-10T12:20:22Z" level=debug msg="Found disk: pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36"
time="2022-06-10T12:20:22Z" level=info msg="Deleting disk pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8"
time="2022-06-10T12:20:22Z" level=debug msg="Failed to delete disk name=pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8, id=r018-ef565808-3491-4262-a6d5-27428e27ec0b.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-ef565808-3491-4262-a6d5-27428e27ec0b: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:22Z" level=info msg="Deleting disk pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c"
time="2022-06-10T12:20:23Z" level=debug msg="Failed to delete disk name=pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c, id=r018-5703dafa-8620-4983-9b25-2bfc850076f5.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-5703dafa-8620-4983-9b25-2bfc850076f5: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:23Z" level=info msg="Deleting disk pvc-8644d499-9154-41f9-b82f-7a3eb69690f6"
time="2022-06-10T12:20:24Z" level=debug msg="Failed to delete disk name=pvc-8644d499-9154-41f9-b82f-7a3eb69690f6, id=r018-9eaf5f2b-3c07-437f-96d0-e6b159c47435.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-9eaf5f2b-3c07-437f-96d0-e6b159c47435: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:24Z" level=info msg="Deleting disk pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36"
time="2022-06-10T12:20:25Z" level=debug msg="Failed to delete disk name=pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36, id=r018-437aac0e-31b1-4d94-b5de-c690a6a1cac5.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-437aac0e-31b1-4d94-b5de-c690a6a1cac5: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:25Z" level=info msg="Deleting disk pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea"
time="2022-06-10T12:20:26Z" level=debug msg="Failed to delete disk name=pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea, id=r018-75590acc-a404-478f-9816-81bbcd278bbc.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-75590acc-a404-478f-9816-81bbcd278bbc: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:26Z" level=info msg="Deleting disk pvc-0fe97d2d-873c-4c94-a8d6-41370be32599"
time="2022-06-10T12:20:27Z" level=debug msg="Failed to delete disk name=pvc-0fe97d2d-873c-4c94-a8d6-41370be32599, id=r018-60c5a890-fd52-49a6-bc92-ca9ceb427863.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-60c5a890-fd52-49a6-bc92-ca9ceb427863: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:27Z" level=debug msg="Disks: 6 items pending"
time="2022-06-10T12:20:31Z" level=info msg="Listing disks"
time="2022-06-10T12:20:32Z" level=debug msg="Found disk: pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea"
time="2022-06-10T12:20:32Z" level=debug msg="Found disk: pvc-0fe97d2d-873c-4c94-a8d6-41370be32599"
time="2022-06-10T12:20:32Z" level=debug msg="Found disk: pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8"
time="2022-06-10T12:20:32Z" level=debug msg="Found disk: pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c"
time="2022-06-10T12:20:32Z" level=debug msg="Found disk: pvc-8644d499-9154-41f9-b82f-7a3eb69690f6"
time="2022-06-10T12:20:32Z" level=debug msg="Found disk: pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36"
time="2022-06-10T12:20:32Z" level=info msg="Deleting disk pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea"
time="2022-06-10T12:20:32Z" level=debug msg="Failed to delete disk name=pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea, id=r018-75590acc-a404-478f-9816-81bbcd278bbc.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-75590acc-a404-478f-9816-81bbcd278bbc: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:32Z" level=info msg="Deleting disk pvc-0fe97d2d-873c-4c94-a8d6-41370be32599"
time="2022-06-10T12:20:33Z" level=debug msg="Failed to delete disk name=pvc-0fe97d2d-873c-4c94-a8d6-41370be32599, id=r018-60c5a890-fd52-49a6-bc92-ca9ceb427863.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-60c5a890-fd52-49a6-bc92-ca9ceb427863: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:33Z" level=info msg="Deleting disk pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8"
time="2022-06-10T12:20:34Z" level=debug msg="Failed to delete disk name=pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8, id=r018-ef565808-3491-4262-a6d5-27428e27ec0b.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-ef565808-3491-4262-a6d5-27428e27ec0b: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:34Z" level=info msg="Deleting disk pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c"
time="2022-06-10T12:20:34Z" level=debug msg="Failed to delete disk name=pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c, id=r018-5703dafa-8620-4983-9b25-2bfc850076f5.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-5703dafa-8620-4983-9b25-2bfc850076f5: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:34Z" level=info msg="Deleting disk pvc-8644d499-9154-41f9-b82f-7a3eb69690f6"
time="2022-06-10T12:20:35Z" level=debug msg="Failed to delete disk name=pvc-8644d499-9154-41f9-b82f-7a3eb69690f6, id=r018-9eaf5f2b-3c07-437f-96d0-e6b159c47435.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-9eaf5f2b-3c07-437f-96d0-e6b159c47435: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:35Z" level=info msg="Deleting disk pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36"
time="2022-06-10T12:20:36Z" level=debug msg="Failed to delete disk name=pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36, id=r018-437aac0e-31b1-4d94-b5de-c690a6a1cac5.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-437aac0e-31b1-4d94-b5de-c690a6a1cac5: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:36Z" level=debug msg="Disks: 6 items pending"
time="2022-06-10T12:20:41Z" level=info msg="Listing disks"
time="2022-06-10T12:20:41Z" level=debug msg="Found disk: pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea"
time="2022-06-10T12:20:41Z" level=debug msg="Found disk: pvc-0fe97d2d-873c-4c94-a8d6-41370be32599"
time="2022-06-10T12:20:41Z" level=debug msg="Found disk: pvc-8644d499-9154-41f9-b82f-7a3eb69690f6"
time="2022-06-10T12:20:41Z" level=debug msg="Found disk: pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36"
time="2022-06-10T12:20:41Z" level=info msg="Deleting disk pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea"
time="2022-06-10T12:20:42Z" level=debug msg="Failed to delete disk name=pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea, id=r018-75590acc-a404-478f-9816-81bbcd278bbc.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-75590acc-a404-478f-9816-81bbcd278bbc: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:42Z" level=info msg="Deleting disk pvc-0fe97d2d-873c-4c94-a8d6-41370be32599"
time="2022-06-10T12:20:43Z" level=debug msg="Failed to delete disk name=pvc-0fe97d2d-873c-4c94-a8d6-41370be32599, id=r018-60c5a890-fd52-49a6-bc92-ca9ceb427863.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-60c5a890-fd52-49a6-bc92-ca9ceb427863: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:43Z" level=info msg="Deleting disk pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8"
time="2022-06-10T12:20:44Z" level=debug msg="Failed to delete disk name=pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8, id=r018-ef565808-3491-4262-a6d5-27428e27ec0b.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-ef565808-3491-4262-a6d5-27428e27ec0b: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:44Z" level=info msg="Deleting disk pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c"
time="2022-06-10T12:20:45Z" level=debug msg="Failed to delete disk name=pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c, id=r018-5703dafa-8620-4983-9b25-2bfc850076f5.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-5703dafa-8620-4983-9b25-2bfc850076f5: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:45Z" level=info msg="Deleting disk pvc-8644d499-9154-41f9-b82f-7a3eb69690f6"
time="2022-06-10T12:20:45Z" level=debug msg="Failed to delete disk name=pvc-8644d499-9154-41f9-b82f-7a3eb69690f6, id=r018-9eaf5f2b-3c07-437f-96d0-e6b159c47435.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-9eaf5f2b-3c07-437f-96d0-e6b159c47435: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:45Z" level=info msg="Deleting disk pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36"
time="2022-06-10T12:20:46Z" level=debug msg="Failed to delete disk name=pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36, id=r018-437aac0e-31b1-4d94-b5de-c690a6a1cac5.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-437aac0e-31b1-4d94-b5de-c690a6a1cac5: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:46Z" level=debug msg="Disks: 6 items pending"
time="2022-06-10T12:20:51Z" level=info msg="Listing disks"
time="2022-06-10T12:20:52Z" level=info msg="Deleting disk pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea"
time="2022-06-10T12:20:53Z" level=debug msg="Failed to delete disk name=pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea, id=r018-75590acc-a404-478f-9816-81bbcd278bbc.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-75590acc-a404-478f-9816-81bbcd278bbc: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:53Z" level=info msg="Deleting disk pvc-0fe97d2d-873c-4c94-a8d6-41370be32599"
time="2022-06-10T12:20:53Z" level=debug msg="Failed to delete disk name=pvc-0fe97d2d-873c-4c94-a8d6-41370be32599, id=r018-60c5a890-fd52-49a6-bc92-ca9ceb427863.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-60c5a890-fd52-49a6-bc92-ca9ceb427863: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:53Z" level=info msg="Deleting disk pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8"
time="2022-06-10T12:20:53Z" level=info msg="Deleted disk pvc-0d111d56-cf74-4045-8df0-fc439b1b4cb8"
time="2022-06-10T12:20:53Z" level=info msg="Deleting disk pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c"
time="2022-06-10T12:20:54Z" level=debug msg="Failed to delete disk name=pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c, id=r018-5703dafa-8620-4983-9b25-2bfc850076f5.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-5703dafa-8620-4983-9b25-2bfc850076f5: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:54Z" level=info msg="Deleting disk pvc-8644d499-9154-41f9-b82f-7a3eb69690f6"
time="2022-06-10T12:20:56Z" level=debug msg="Failed to delete disk name=pvc-8644d499-9154-41f9-b82f-7a3eb69690f6, id=r018-9eaf5f2b-3c07-437f-96d0-e6b159c47435.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-9eaf5f2b-3c07-437f-96d0-e6b159c47435: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:56Z" level=info msg="Deleting disk pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36"
time="2022-06-10T12:20:57Z" level=debug msg="Failed to delete disk name=pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36, id=r018-437aac0e-31b1-4d94-b5de-c690a6a1cac5.If this error continues to persist for more than 20 minutes then please try to manually cleanup the volume using - ibmcloud is vol r018-437aac0e-31b1-4d94-b5de-c690a6a1cac5: Delete volume failed. Volume can be deleted only when its status is available or failed."
time="2022-06-10T12:20:57Z" level=debug msg="Disks: 5 items pending"
time="2022-06-10T12:21:01Z" level=info msg="Listing disks"
time="2022-06-10T12:21:02Z" level=info msg="Deleting disk pvc-8644d499-9154-41f9-b82f-7a3eb69690f6"
time="2022-06-10T12:21:02Z" level=info msg="Deleted disk pvc-8644d499-9154-41f9-b82f-7a3eb69690f6"
time="2022-06-10T12:21:02Z" level=info msg="Deleting disk pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36"
time="2022-06-10T12:21:02Z" level=info msg="Deleted disk pvc-b182ddcf-e5b4-47cd-8bda-fc9f976e1e36"
time="2022-06-10T12:21:02Z" level=info msg="Deleting disk pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea"
time="2022-06-10T12:21:02Z" level=info msg="Deleted disk pvc-9f4e35b2-724d-440a-bfdd-4f2be06874ea"
time="2022-06-10T12:21:02Z" level=info msg="Deleting disk pvc-0fe97d2d-873c-4c94-a8d6-41370be32599"
time="2022-06-10T12:21:02Z" level=info msg="Deleted disk pvc-0fe97d2d-873c-4c94-a8d6-41370be32599"
time="2022-06-10T12:21:02Z" level=info msg="Deleting disk pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c"
time="2022-06-10T12:21:02Z" level=info msg="Deleted disk pvc-6f956a92-4a4d-49ff-8307-a4d120a4c42c"

jstuever

The PR look functionally ok. However, I suspect there is some unnecessary code here. I see this borrows heavily from the GCP destroy. One significant difference is that GCP destroy runs in an infinite loop until everything is deleted whereas this code runs only one time with loops on individual items. I see it is using the pendingItems() functionality, which I'm not sure makes (as much) sense without that primary loop. However, this PR didn't add this pattern, so I don't feel we should block on this.

Please add a description to the PR.

arahamad-zz

please check these comments and take action accordingly

pkg/destroy/ibmcloud/disk.go

pkg/destroy/ibmcloud/ibmcloud.go

pkg/destroy/ibmcloud/disk.go

openshift-ci · 2022-06-17T06:55:49Z

@sameshai: This pull request references Bugzilla bug 2047732, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.11.0) matches configured target release for branch (4.11.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @chao007

Details

In response to this:

Bug 2047732: [IBM]Volume is not deleted after destroy cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-06-17T07:12:30Z

@sameshai: This pull request references Bugzilla bug 2047732, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.11.0) matches configured target release for branch (4.11.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @chao007

Details

In response to this:

Bug 2047732: [IBM]Volume is not deleted after destroy cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sameshai · 2022-06-17T07:13:25Z

The PR look functionally ok. However, I suspect there is some unnecessary code here. I see this borrows heavily from the GCP destroy. One significant difference is that GCP destroy runs in an infinite loop until everything is deleted whereas this code runs only one time with loops on individual items. I see it is using the pendingItems() functionality, which I'm not sure makes (as much) sense without that primary loop. However, this PR didn't add this pattern, so I don't feel we should block on this.

Please add a description to the PR.

@jstuever thanks for the quick review. Apologies missed the PR description. Please refer the logic description we are doing like GCP but avoiding redundant list disk calls.

#5962 (comment)

MayXuQQ · 2022-06-21T03:02:10Z

@chao007 pre-merge build

MayXuQQ · 2022-06-21T15:45:33Z

pre-merge test OK,
after add label such as 'kubernetes-io-cluster-zhsunibm-j68d5:owned' on volume, destroy the cluster, just remains the subnet resource.
/label qe-approved
/bugzilla cc-qa

openshift-ci · 2022-06-21T15:45:46Z

@MayXuQQ: This pull request references Bugzilla bug 2047732, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.11.0) matches configured target release for branch (4.11.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @chao007

Details

In response to this:

pre-merge test OK,
after add label such as 'kubernetes-io-cluster-zhsunibm-j68d5:owned' on volume, destroy the cluster, just remains the subnet resource.
/label qe-approved
/bugzilla cc-qa

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Fix for Bug 2047732 - [IBM]Volume is not deleted after destroy cluster Fix for Bug 2047732 - [IBM]Volume is not deleted after destroy cluster Bug 2047732 - [IBM]Volume is not deleted after destroy cluster Bug 2047732 - [IBM]Volume is not deleted after destroy cluster Bug 2047732 - [IBM]Volume is not deleted after destroy cluster Bug 2047732 - [IBM]Volume is not deleted after destroy cluster Bug 2047732 - [IBM]Volume is not deleted after destroy cluster Bug:2047732 - [IBM]Volume is not deleted after destroy cluster Bug:2047732 - [IBM]Volume is not deleted after destroy cluster Bug:2047732 - [IBM]Volume is not deleted after destroy cluster Bug:2047732 - [IBM]Volume is not deleted after destroy cluster Bug:2047732 - [IBM]Volume is not deleted after destroy cluster Bug:2047732 - [IBM]Volume is not deleted after destroy cluster Bug:2047732 - [IBM]Volume is not deleted after destroy cluster Bug:2047732 - [IBM]Volume is not deleted after destroy cluster

sameshai · 2022-06-22T10:51:30Z

@r4f4 can we go ahead with merge

arahamad-zz

lgtm, only I see issue at https://github.com/openshift/installer/pull/5962/files#diff-c98fece70f92d74f8db64396e4256d51b5ff7c1861479479360410ec388556e3R23 line where we are returning which will cause issue as any page in that re-try fail then we are marking entire list failed and this loop will keep going on.

sameshai · 2022-06-22T11:30:11Z

lgtm, only I see issue at https://github.com/openshift/installer/pull/5962/files#diff-c98fece70f92d74f8db64396e4256d51b5ff7c1861479479360410ec388556e3R23 line where we are returning which will cause issue as any page in that re-try fail then we are marking entire list failed and this loop will keep going on.

@arahamad I have already mentioned in the slack channel discussion.So are following GCP flow as of now.

https://coreos.slack.com/archives/C01U40AM37F/p1655735741598449?thread_ts=1655372841.635669&cid=C01U40AM37F

patrickdillon · 2022-06-22T17:50:06Z

/approve

openshift-ci · 2022-06-22T17:51:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: arahamad, GunaKKIBM, patrickdillon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [patrickdillon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

patrickdillon · 2022-06-22T18:02:16Z

/lgtm

MayXuQQ · 2022-06-23T06:12:29Z

/hold cancel

openshift-ci-robot · 2022-06-23T06:42:49Z

/retest-required

Remaining retests: 2 against base HEAD 7cdf85d and 8 for PR HEAD cc59a99 in total

openshift-ci · 2022-06-23T09:31:35Z

@sameshai: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ibmcloud	`cc59a99`	link	false	`/test e2e-ibmcloud`
ci/prow/e2e-gcp	`cc59a99`	link	false	`/test e2e-gcp`
ci/prow/e2e-metal-ipi	`cc59a99`	link	false	`/test e2e-metal-ipi`
ci/prow/e2e-crc	`cc59a99`	link	false	`/test e2e-crc`
ci/prow/e2e-ovirt	`cc59a99`	link	false	`/test e2e-ovirt`
ci/prow/e2e-vsphere	`cc59a99`	link	false	`/test e2e-vsphere`
ci/prow/okd-e2e-aws	`cc59a99`	link	false	`/test okd-e2e-aws`
ci/prow/e2e-azure	`cc59a99`	link	false	`/test e2e-azure`
ci/prow/e2e-openstack	`cc59a99`	link	false	`/test e2e-openstack`
ci/prow/e2e-libvirt	`cc59a99`	link	false	`/test e2e-libvirt`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sameshai · 2022-06-23T10:03:38Z

The PR look functionally ok. However, I suspect there is some unnecessary code here. I see this borrows heavily from the GCP destroy. One significant difference is that GCP destroy runs in an infinite loop until everything is deleted whereas this code runs only one time with loops on individual items. I see it is using the pendingItems() functionality, which I'm not sure makes (as much) sense without that primary loop. However, this PR didn't add this pattern, so I don't feel we should block on this.
Please add a description to the PR.

@jstuever thanks for the quick review. Apologies missed the PR description. Please refer the logic description we are doing like GCP but avoiding redundant list disk calls.

#5962 (comment)

@jstuever there is no duplicate code and it is similar to GCP code.

openshift-ci · 2022-06-23T10:48:49Z

@sameshai: Some pull requests linked via external trackers have merged:

The following pull requests linked via external trackers have not merged:

openshift/ibm-vpc-block-csi-driver-operator#40 is open

These pull request must merge or be unlinked from the Bugzilla bug in order for it to move to the next state. Once unlinked, request a bug refresh with /bugzilla refresh.

Bugzilla bug 2047732 has not been moved to the MODIFIED state.

Details

In response to this:

Bug 2047732: [IBM]Volume is not deleted after destroy cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 3, 2022

openshift-ci bot requested review from jstuever and rna-afk June 3, 2022 12:29

GunaKKIBM reviewed Jun 6, 2022

View reviewed changes

pkg/destroy/ibmcloud/disk.go Outdated Show resolved Hide resolved

GunaKKIBM reviewed Jun 6, 2022

View reviewed changes

pkg/destroy/ibmcloud/ibmcloud.go Outdated Show resolved Hide resolved

GunaKKIBM reviewed Jun 6, 2022

View reviewed changes

pkg/destroy/ibmcloud/ibmcloud.go Outdated Show resolved Hide resolved

GunaKKIBM reviewed Jun 6, 2022

View reviewed changes

pkg/destroy/ibmcloud/disk.go Outdated Show resolved Hide resolved

r4f4 reviewed Jun 7, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

sameshai changed the title ~~Fix for Bug 2047732 - [IBM]Volume is not deleted after destroy cluster~~ Bug 2047732 - [IBM]Volume is not deleted after destroy cluster Jun 7, 2022

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 7, 2022

sameshai changed the title ~~Bug 2047732 - [IBM]Volume is not deleted after destroy cluster~~ Bug 2047732 - [IBM]Volume is not deleted after destroy cluster -- [DO-NOT-MERGE] under test Jun 8, 2022

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 8, 2022

sameshai changed the title ~~Bug 2047732 - [IBM]Volume is not deleted after destroy cluster -- [DO-NOT-MERGE] under test~~ Bug 2047732 - [IBM]Volume is not deleted after destroy cluster Jun 10, 2022

openshift-ci bot changed the title ~~Bug 2047732 - [IBM]Volume is not deleted after destroy cluster~~ Bug 2047732: [IBM]Volume is not deleted after destroy cluster Jun 10, 2022

openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Jun 10, 2022

openshift-ci bot requested a review from chao007 June 10, 2022 11:33

GunaKKIBM approved these changes Jun 16, 2022

View reviewed changes

jstuever reviewed Jun 17, 2022

View reviewed changes

arahamad-zz reviewed Jun 17, 2022

View reviewed changes

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Jun 21, 2022

sameshai force-pushed the samdev branch from d5e47d5 to 8329f47 Compare June 21, 2022 16:15

sameshai force-pushed the samdev branch from 8329f47 to cc59a99 Compare June 22, 2022 10:05

arahamad-zz approved these changes Jun 22, 2022

View reviewed changes

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 22, 2022

openshift-ci bot assigned patrickdillon Jun 22, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 22, 2022

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 23, 2022

openshift-ci bot merged commit 02a04a2 into openshift:master Jun 23, 2022

openshift-ci bot mentioned this pull request Jun 29, 2022

Bug 2047732: [IBM]Volume is not deleted after destroy cluster openshift/ibm-vpc-block-csi-driver-operator#40

Merged

Bug 2047732: [IBM]Volume is not deleted after destroy cluster #5962

Bug 2047732: [IBM]Volume is not deleted after destroy cluster #5962

Uh oh!

Conversation

sameshai commented Jun 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Jun 3, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

r4f4 commented Jun 7, 2022

Uh oh!

Uh oh!

r4f4 commented Jun 7, 2022

Uh oh!

r4f4 commented Jun 7, 2022

Uh oh!

r4f4 commented Jun 7, 2022

Uh oh!

openshift-ci bot commented Jun 7, 2022

Uh oh!

r4f4 commented Jun 8, 2022

Uh oh!

jsafrane commented Jun 10, 2022

Uh oh!

openshift-ci bot commented Jun 10, 2022

Uh oh!

jsafrane commented Jun 10, 2022

Uh oh!

openshift-ci bot commented Jun 10, 2022

Uh oh!

sameshai commented Jun 10, 2022

Uh oh!

jstuever left a comment

Choose a reason for hiding this comment

Uh oh!

arahamad-zz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci bot commented Jun 17, 2022

Uh oh!

openshift-ci bot commented Jun 17, 2022

Uh oh!

sameshai commented Jun 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MayXuQQ commented Jun 21, 2022

Uh oh!

MayXuQQ commented Jun 21, 2022

Uh oh!

openshift-ci bot commented Jun 21, 2022

Uh oh!

sameshai commented Jun 22, 2022

Uh oh!

arahamad-zz left a comment

Choose a reason for hiding this comment

Uh oh!

sameshai commented Jun 22, 2022

Uh oh!

patrickdillon commented Jun 22, 2022

Uh oh!

openshift-ci bot commented Jun 22, 2022

Uh oh!

patrickdillon commented Jun 22, 2022

Uh oh!

MayXuQQ commented Jun 23, 2022

Uh oh!

openshift-ci-robot commented Jun 23, 2022

Uh oh!

openshift-ci bot commented Jun 23, 2022

Uh oh!

sameshai commented Jun 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

sameshai commented Jun 3, 2022 •

edited

Loading

sameshai commented Jun 17, 2022 •

edited

Loading

sameshai commented Jun 23, 2022 •

edited

Loading