:bug: Power off nodes upon deletion by honza · Pull Request #1176 · metal3-io/baremetal-operator

honza · 2022-10-20T12:48:29Z

This is a continuation of #816 which in turn tries to fix #410.

Co-authored-by: Sandhya Dasu sadasu@redhat.com @sadasu

elfosardo · 2022-10-20T13:05:58Z

/test-centos-integration-main
/test-ubuntu-integration-main

elfosardo · 2022-10-20T13:07:25Z

/lgtm

furkatgofurov7 · 2022-10-20T13:42:30Z

Heads up: centOS failure is unrelated to these changes, we are facing issues with CI.

xref: https://kubernetes.slack.com/archives/CHD49TLE7/p1666273231196429

Cellebyte · 2022-11-08T11:18:03Z

what is the progress here?

elfosardo · 2022-11-29T14:30:36Z

/test-centos-integration-main
/test-ubuntu-integration-main

elfosardo · 2022-11-29T14:31:29Z

/lgtm

dtantsur · 2022-12-05T09:50:45Z

/lgtm cancel

One issue inline, otherwise looking good.

zaneb

Refamiliarising myself with how all of this works 😅

zaneb · 2023-03-23T02:59:45Z

+
+		if err != nil {
+			if info.host.Status.ErrorCount < maxPowerOffRetryCount {
+				return actionError{errors.Wrap(err, "failed to power off")}


I'm not seeing anything addressing my earlier comment about an infinite error loop in cases where the node is missing from ironic and we don't have credentials to re-register it, which we handle in deprovisioning here and also now need to handle in deleting.

I wonder if this would all be made simpler by putting the new code into a separate actionPowerOffBeforeDeleting() method, so that the state machine code can easily distinguish between errors coming from the power off vs. the delete.

zaneb · 2023-03-23T03:05:33Z

+		}
+	}
+
+	info.host.Status.ErrorCount = 0


As mentioned in the last part of https://github.com/metal3-io/baremetal-operator/pull/1176/files#r1053444986, iff the error count wasn't already 0 we'll want to return actionUpdate wrapping actionContinue on line 533.

zaneb · 2023-03-23T03:19:59Z

+				}
+				return result
+			}
+		}


We should probably also set Status.PoweredOn to false so that we are reporting what we've actually done.

honza · 2023-03-23T19:05:56Z

This might be overkill but the suggested changes led me to create a new step in the state machine. When a delete is requested, instead of StateDeleting, we now enter StatePoweringOffBeforeDelete. This separates the two operations nicely, and allows us to handle errors in the usual way (i.e. as was said above regarding skipping to delete when too many errors happen, or we don't have host creds).

When we clear ErrorCount, we do it as part of the actionComplete result, and thus don't need an explicit update (as is idiomatic in the existing code).

elfosardo · 2023-04-03T11:59:34Z

/test-centos-integration-main
/test-ubuntu-integration-main

elfosardo · 2023-04-03T11:59:49Z

/test-centos-e2e-integration-main

dtantsur · 2023-05-25T12:32:20Z


+func (hsm *hostStateMachine) handlePoweringOffBeforeDelete(info *reconcileInfo) actionResult {
+	actResult := hsm.Reconciler.actionPowerOffBeforeDeleting(hsm.Provisioner, info)
+	skipToDelete := func() actionResult {


nit: most of this function is repeating handleDeprovisioning. It would be great to refactor them.

dtantsur · 2023-05-25T12:32:56Z

/test-centos-e2e-integration-main
/test-ubuntu-integration-main
/lgtm

honza · 2023-05-31T23:13:31Z

/assign @zaneb

We introduce a new step in the state machine where the node goes through a power off stage before it's deleted. We attempt to power it off 3 times before giving up, and proceeding to the delete.

elfosardo · 2023-06-28T14:27:55Z

/lgtm

dtantsur · 2023-07-06T16:11:38Z

/test-centos-e2e-integration-main
/test-ubuntu-integration-main

dtantsur · 2023-07-06T16:12:39Z

/approve

metal3-io-bot · 2023-07-06T16:12:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dtantsur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dtantsur]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

metal3-io-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 20, 2022

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 20, 2022

zhouhao3 reviewed Oct 26, 2022

View reviewed changes

Comment thread pkg/provisioner/ironic/delete_test.go Outdated

honza force-pushed the bmh-delete-2022-10-20 branch from 72efc2a to 8b41131 Compare November 29, 2022 14:11

metal3-io-bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 29, 2022

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Nov 29, 2022

dtantsur reviewed Dec 5, 2022

View reviewed changes

Comment thread pkg/provisioner/ironic/ironic.go Outdated

metal3-io-bot removed the lgtm Indicates that a PR is ready to be merged. label Dec 5, 2022

zaneb reviewed Dec 5, 2022

View reviewed changes

Comment thread controllers/metal3.io/baremetalhost_controller.go Outdated

Comment thread pkg/provisioner/ironic/ironic.go Outdated

honza force-pushed the bmh-delete-2022-10-20 branch from 8b41131 to 2f4593e Compare December 19, 2022 19:42

metal3-io-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 19, 2022

zaneb reviewed Dec 19, 2022

View reviewed changes

Comment thread pkg/provisioner/ironic/ironic.go Outdated

honza force-pushed the bmh-delete-2022-10-20 branch from 2f4593e to 277a166 Compare December 19, 2022 20:31

zaneb reviewed Dec 20, 2022

View reviewed changes

Comment thread controllers/metal3.io/baremetalhost_controller.go Outdated

Comment thread controllers/metal3.io/baremetalhost_controller.go Outdated

Comment thread controllers/metal3.io/baremetalhost_controller.go Outdated

Comment thread controllers/metal3.io/baremetalhost_controller.go Outdated

honza force-pushed the bmh-delete-2022-10-20 branch from 277a166 to 77fdf84 Compare March 17, 2023 18:30

zaneb reviewed Mar 23, 2023

View reviewed changes

honza force-pushed the bmh-delete-2022-10-20 branch from 77fdf84 to b61cb84 Compare March 23, 2023 19:01

metal3-io-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 23, 2023

honza force-pushed the bmh-delete-2022-10-20 branch from b61cb84 to 1630c13 Compare March 29, 2023 17:41

dtantsur reviewed Apr 20, 2023

View reviewed changes

Comment thread apis/metal3.io/v1alpha1/baremetalhost_types.go Outdated

dtantsur reviewed Apr 20, 2023

View reviewed changes

Comment thread controllers/metal3.io/host_state_machine.go Outdated

honza force-pushed the bmh-delete-2022-10-20 branch from 1630c13 to 2f38157 Compare April 20, 2023 12:23

dtantsur reviewed May 25, 2023

View reviewed changes

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label May 25, 2023

metal3-io-bot assigned zaneb May 31, 2023

metal3-io-bot added the needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. label Jun 8, 2023

Power off nodes upon deletion

6f65d8e

We introduce a new step in the state machine where the node goes through a power off stage before it's deleted. We attempt to power it off 3 times before giving up, and proceeding to the delete.

honza force-pushed the bmh-delete-2022-10-20 branch from 2f38157 to 6f65d8e Compare June 9, 2023 20:00

metal3-io-bot removed lgtm Indicates that a PR is ready to be merged. needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 9, 2023

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Jun 28, 2023

metal3-io-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 6, 2023

metal3-io-bot merged commit 7654cd7 into metal3-io:main Jul 6, 2023

honza mentioned this pull request Jul 14, 2023

🐛 Proceed to power off after deprovisioning #1312

Merged

Conversation

honza commented Oct 20, 2022

Uh oh!

elfosardo commented Oct 20, 2022

Uh oh!

elfosardo commented Oct 20, 2022

Uh oh!

furkatgofurov7 commented Oct 20, 2022

Uh oh!

Uh oh!

Cellebyte commented Nov 8, 2022

Uh oh!

elfosardo commented Nov 29, 2022

Uh oh!

elfosardo commented Nov 29, 2022

Uh oh!

Uh oh!

dtantsur commented Dec 5, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zaneb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zaneb Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

zaneb Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

zaneb Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

honza commented Mar 23, 2023

Uh oh!

elfosardo commented Apr 3, 2023

Uh oh!

elfosardo commented Apr 3, 2023

Uh oh!

Uh oh!

Uh oh!

dtantsur May 25, 2023

Choose a reason for hiding this comment

Uh oh!

dtantsur commented May 25, 2023

Uh oh!

honza commented May 31, 2023

Uh oh!

elfosardo commented Jun 28, 2023

Uh oh!

dtantsur commented Jul 6, 2023

Uh oh!

dtantsur commented Jul 6, 2023

Uh oh!

metal3-io-bot commented Jul 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants