Skip to content

Conversation

@hardys
Copy link

@hardys hardys commented Nov 19, 2020

For some reason the virsh --remove-all-storage isn't actually removing
the storage, this may be because we moved to a non-default pool?

This can quickly use up space as each leaked directory is about 2.2G
so lets clean it up until we can find a more robust way to handle
this in the installer.

@hardys hardys force-pushed the image_cleanup branch 2 times, most recently from f3d0f30 to 456af93 Compare November 19, 2020 09:36
For some reason the virsh --remove-all-storage isn't actually removing
the storage, this may be because we moved to a non-default pool?

This can quickly use up space as each leaked directory is about 2.2G
so lets clean it up until we can find a more robust way to handle
this in the installer.
@cybertron
Copy link
Contributor

Interestingly, I no longer see this happening in my environments. On the node I already cleaned up, I have (with a cluster actively running):

$ ls -lah /var/lib/libvirt/openshift-images/
total 8.0K
drwxr-xr-x. 2 root root 4.0K Nov 18 17:50 .
drwxr-xr-x. 10 root root 4.0K Nov 18 15:03 ..
$

On another node that I didn't clean up I just have a bunch of empty directories there. I wonder if there was a bug that affected deployments during a certain window of time that left a bunch of cruft, but has since been fixed.

@hardys
Copy link
Author

hardys commented Nov 23, 2020

Interestingly, I no longer see this happening in my environments. On the node I already cleaned up, I have (with a cluster actively running):

I think it's a corner case where if the installer fails during the terraform apply phase, it then doesn't clean up properly in the destroy cluster step - if your deployment succeeded then I'd expect the existing cleanup to work, but if you somehow cause the deployment of masters to fail I suspect it won't?

@cybertron
Copy link
Contributor

/lgtm

Fair enough. It shouldn't hurt anything to clean these up in any case.

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 25, 2020
@andfasano
Copy link
Member

/approve

@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andfasano

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 27, 2020
@openshift-merge-robot openshift-merge-robot merged commit 3f9e768 into openshift-metal3:master Nov 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants