Do BM nodes get re-provisioned with custom remediation set to reboot?

[Privatecoder]

Hello,

I have another question: How can we prevent BM nodes from being re-provisioned when they become unhealthy with MachineHealthChecks in place?

When cloud-nodes have pvc's through Hetzners csi, bm-nodes don't, so if one uses local storage, everything will be lost if a node gets re-provisioned after becoming unhealthy.

It would be better to manually re-provision the node if necessary. So:

With three BM nodes running a HA-deployment (with local storage) and MaxUnhealthy set to 2 along with a custom HetznerBareMetalRemediationTemplate set to strategy.type: "Reboot" the HealthCheckController will try to Reboot a BM node only if two or fewer BM nodes are unhealthy.

But: What happens after strategy.retryLimit has been reached? Will the controller then proceed to re-initialize the BM node or just mark it as unhealthy forever?

Thank you!

[Privatecoder]

I did read https://github.com/syself/cluster-api-provider-hetzner/blob/main/docs/topics/advanced-caph.md#machine-health-checks-with-custom-remediation-template however I am wondering if the strategy set to reboot will cause the node to be re-provisioned once retryLimit has been reached and if so, if this can be prevented by a setting that will always prevent auto-re-provisioning, forcing the cluster-operator to manually trigger it.

[batistein]

1. You cannot prevent re-provisioning even if you have not set up MachineHealthChecks. A MachineDeployment behaves similarly to a Kubernetes Deplyoment, where each pod is replaceable. In general, the cluster-api project sees nodes as Caddles, not Pet. Therefore, I would not recommend using local storage.
   There are more complex strategies for provisioning storage on baremetal nodes, but first you need to split the OS and storage disks so that no data can be lost, and then you need to hook into the Capi's lifecycle management to ensure that storage replication and balancing work properly. This is not something we could provide in general....
2. After the retryLimit, the node is restored as usual.