Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions docs/enhancements/backup-restore-support.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
---
title: backup-restore-support
authors:
- "@danielerez"
creation-date: 2024-08-08
last-updated: 2024-08-08
---

# Backup/Restore Support

## Summary

The Assisted Installer should support backup/restore and disaster recovery scenarios, either using OADP (OpenShift API for Data Protection) for ACM (Advanced Cluster Management), or, using Kube-API.
I.e. the assisted-service should be resilient in such scenarios which, for this context and effort, means that restored/moved spoke clusters should keep the same state and behave the same on the new hub cluster.

## Motivation

Provide resiliency in the assisted-service for safe backup/restore flows, allowing spoke clusters to be used without any restriction after DR scenarios or moving between hubs.

### Goals
Comment thread
carbonin marked this conversation as resolved.
Resolved

Spoke clusters (with/without BMHs) should have a consistent state and function properly, including the ability to add/remove nodes, after these scenarios:
* Spoke clusters restored on a passive hub with ACM/OADP functionality.
* Spoke clusters moved between hubs using Kube-API.

### Non-Goals

* Improve import day2 cluster solution by getting full host data (e.g. using an agent on existing nodes).

## Proposal

### User Stories

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All stories are "as an ACM user". Why do we care about how resources are backed up and restored? OADP, ArgoCD, or typing each CRD by hand should be the same, right?

I think the important variations are:

  1. Are they using BMHs? This affects installed hosts getting rebooted with our ISO (bad) and uninstalled hosts getting rebooted with our ISO (good, so we can use them in the new hub). If they aren't using BMHs, they need to manually boot uninstalled hosts if the hub changed.
  2. What is the cluster state? Installation not started, installing, or installed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good points. I'll change to something like 'Infrastructure Admin' to make it more generic.
I'll also add the details for these variations.


#### Story 1

As an Infrastructure Admin, I want to backup an installed spoke cluster from an active hub, including the associated BMHs (e.g. by using ACM/Kube-API/GitOps/etc).
Then, I want to safely restore the cluster on a passive hub, including the BMHs that should keep the same state. I.e. the restored cluster should behave exactly as in the active hub with no restrictions (support day2 operations, etc).
Note: if unbound hosts are included, they should still be automatically reprovisioned using a new discovery ISO.

#### Story 2

As an Infrastructure Admin, I want to backup an installed spoke cluster that doesn't have associated BMHs. Then, I want to safely restore the cluster on a passive hub.
Note: if unbound hosts are included, they should be manually booted using a new discovery ISO.

#### Story 3

As an Infrastructure Admin, I want to restore a spoke cluster in non-installed status (draft/in-progress/etc). The cluster on target hub should, ideally, keep the same status. Or, at least, fail gracefully and let the user retry the installation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: If we back up and restore a spoke cluster where the installation hasn't started yet and we don't have the full inventory, will it impact our ability to start the installation? will we get a validation failure?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the user restores all resources, the agents will be in "disconnected" state. Once the hosts boot with a new ISO, the agents will populate the inventory, and installation can proceed.

Unbound hosts should be treated as previous stories (depends on BMHs).

#### Story 4

As an Infrastructure Admin, I want to safely move installed spoke clusters between hubs.
Concerns are similar to previous stories.

### Implementation Details/Notes/Constraints

#### Solution for BMHs reconcile (due to a missing status)

According to BMO documentation, the suggested approach for moving BareMetalHost from one cluster to another is using the `baremetalhost.metal3.io/status` [annotation](https://github.com/metal3-io/metal3-docs/blob/6a656b3eb195c1b09ba35fcad4d011c6cb02dbc2/docs/user-guide/src/bmo/status_annotation.md).
I.e. Extract the current status of the BMH, store it in an annotation, create the BMH on the new hub:
```
# Save the status in json format to a file
kubectl get bmh <name-of-bmh> -o jsonpath="{.status}" > status.json
# Save the BMH and apply the status annotation to the saved BMH.
kubectl -n metal3 annotate bmh <name-of-bmh> \
baremetalhost.metal3.io/status="$(cat status.json)" \
--dry-run=client -o yaml > bmh.yaml
Comment thread
carbonin marked this conversation as resolved.
Resolved
# Apply the saved BMH on the target hub
kubectl --kubeconfig <target-hub-kubeconfig> apply -f bmh.yaml
```
This flow ensures that the BMH on the new hub has status (BMO applies the status using the annotation). Hence, as the BMH should be in state 'Provisioned', the BMH reconcile is avoided.
I.e. prevent destruction of restored spoke clusters.

##### Suggested solution for automating the flow (make it transparent to the user):

In assisted-service -> `bmh_agent_controller`:
* If a BMH is in 'Provisioned' state (and, doesn't have `status`+`paused` annotations):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the non-converged flow the BMH moves to Provisioned directly after the ISO is booted IIRC

I know this is in open questions, but I figured it would be better to mention it here.

I'm not actually sure if this is a problem, but we should be sure to test both converged and non-converged if we intend this to work with both. Otherwise I think it would be okay to only support new features in converged flow, but I'm curious what @avishayt thinks about that one.

* Extract BMH's status and add `baremetalhost.metal3.io/status` to the object metadata.
* Add also, add the `baremetalhost.metal3.io/paused` annotation:
* This is required since once the annotation is added, BMO updates the BMH's status and delete the annotation.
* The [paused](https://github.com/metal3-io/baremetal-operator/blob/ac05af9fb9a0b444c4441a758c731e6f518c21df/docs/api.md#pausing-reconciliation) annotation, is similar to the `detached` annotation we're already using - but, it's also pausing the reconcile loop. Thus, keeping the status annotation intact.
* If a BMH is missing 'Status' (and, has `status`+`paused` annotation):
* To restore the original Status:
* Remove the `baremetalhost.metal3.io/paused` annotation
* Status is restored, but `baremetalhost.metal3.io/status` annotation is removed.
* The `status` and `paused` annotation will be restored (see previous bullet).

#### Solution for deleted Agent CRs

1. Don't delete the Agent CR when an associated host doesn't exits on DB.
Comment thread
carbonin marked this conversation as resolved.
Resolved
* Remove relevant [code](https://github.com/openshift/assisted-service/blob/5bbf696fc0cc5a220fd8a0d1e408ab0bcc889e3f/internal/controller/controllers/agent_controller.go#L143-L151) from AgentController.
Comment thread
carbonin marked this conversation as resolved.
Resolved

2. Restore the Host record in DB according to properties from the Agent CR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nick had a good idea before which was to run the assisted-installer-agent to recollect the discovery information. Do you think it's worth pursuing?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this was already called out as a non goal for this proposal.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see 🤦‍♀️ thanks!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say that the information in the agent status shouldn't be relied on after the host is installed anyway. Some things won't change, but other things might. Better to use the OpenShift node info.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean by your comment @avishayt.

Are you suggesting @danielerez not use the agent status for populating the database or replying to @CrystalChun's original comment?

FWIW I think it's fine to populate the DB on the restore side using the agent status as that's what would have been in the DB on the backup side.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm saying that restoring status is generally frowned upon, but it's also not worth doing because we finished installing the host. So what's the point? If the user wants to see the hardware inventory, they should use the k8s node API.

* Seems we can infer more than the mandatory properties:
* host.ID <-- agent.Name
* host.Role <-- agent.Role
* host.InfraEnvID <-- agent.metadata.annotations['infraenvs.agent-install.openshift.io']
* host.Kind <-- 'Host' or 'AddToExistingClusterHost' if cluster is already installed
* host.InstallerArgs <-- agent.InstallerArgs
* host.RequestedHostname <-- agent.Hostname
* The entire inventory exists in Agent.Status. But, need to ensure it exists also post restore. If not, we can consider adding an annotation for it (i.e something as `status` in BMH).
* Note: some of the inventory data can also be restored from existing annotations.
Comment thread
carbonin marked this conversation as resolved.
Resolved
E.g.:
* inventory.agent-install.openshift.io/cpu-architecture: x86_64
* inventory.agent-install.openshift.io/cpu-virtenabled: "true"
* inventory.agent-install.openshift.io/host-isvirtual: "true"
* inventory.agent-install.openshift.io/host-manufacturer: RedHat
* inventory.agent-install.openshift.io/host-productname: KVM
* inventory.agent-install.openshift.io/storage-hasnonrotationaldisk: "false"

### Risks and Mitigations

* In general, these changes might affect flows on spoke cluster (before and after restore), need to ensure no regressions on all scenarios.
* Adding the 'paused' annotation means a stale BMH status, need to check if it's crucial.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also what if we want to deprovision using BMH - will this pause annotation affect that?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have the detached annotation which our controller removes when we need to deprovision (unbind or delete). I imagine we'll do the same for paused. Especially if it fully replaces detached.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet yeah just want to make sure it's called out that's something in our scope too!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do have some people complaining about the BMH status not being accurate.

Do we want to make this entire procedure (pausing BMH and saving status annotation) opt-in? Maybe it would be good to put this behind a feature flag in any case to start?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do have some people complaining about the BMH status not being accurate.

Do we want to make this entire procedure (pausing BMH and saving status annotation) opt-in? Maybe it would be good to put this behind a feature flag in any case to start?

Good point. As an alternative, it could be opt-out perhaps? I mean, disabling this logic when required.
I.e. Not sure how crucial it is to keep the BMH status updated once already provisioned:/

@avishayt @gamli75 @oourfali WDYT?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as we have some flag I'm fine.

I'm not as concerned with it being enabled or not.

* Keeping the Agent CR and restoring values into Host in DB could affect current flows (e.g. BindHost, UnbindHost, etc).
* Restoring Hosts will result in partial records in DB. E.g. missing Inventory - could be challenging to tackle (I.e. probably need to add annotations to Agent).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we'd be missing inventory, but do we know of any other gaps here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet, would need to investigate.


### Open Questions

* Any specific concerns for the ConvergedFlow/Non-ConvergedFlow?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it was the converged flow, would we be concerned about how the node is registered in Ironic? Like if the state of the node in Ironic somehow wasn't preserved after restoring and we can't deprovision (or scale down) the host

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think everything they need should be in the BMH. So as long as that is restored correctly it should be fine. 🙏

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope so 😅 I just recalled times where the BMH would either be stuck "deprovisioning" or "inspecting" when ironic was reporting "clean wait" or something else

* How seamlessly should we support restoring of 'in-progress' clusters (as the focus is on 'installed' clusters)?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this I think we just need a clear answer as to what will happen along with the ability to recover.

DR is a necessarily uncontrolled scenario and some things will be in an unstable state when it happens. As long as nothing gets "stuck" on the restore side we should be fine.

Also something to think about is that the most recent backup might not reflect the current state of the world.
Imagine a scenario where the backup has an unbound agent, but that agent had actually been installed since the backup was taken. Also the opposite might be true. Obviously there's not much we can do to make this "correct" in the restore, but it's worth thinking about.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably ask @racedo about the requirements here. My opinion is that we don't need to continue with the installations that are in progress. Ideally, the cluster and agent resources should either indicate that an error occurred or allow the user to retry. I'm guessing that if we do no work here then the agents will show up as disconnected, the cluster will show up as waiting for the agents, and installation will start once agents reboot with the new ISO. This is also acceptable IMO.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good points, I'll add some notes about it.

* Need to consider conflicts when restoring from stale a backup. E.g. a backup with an unbound agent, that has already been installed since the backup was taken.
* We could just ensure that such clusters have some indication that an error has occurred. I.e. so the user could retry the installation.

### UI Impact

* No UI changes are required.

* The ACM UI should display the correct status for spoke clusters, etc.

### Test Plan

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add testing day 2 operations after the restore


* All existing E2E tests should pass on restored/moved spoke clusters.
* Specifically, should test day2 flows on clusters.
* Should be tested both with ACM/OADP Backup/Restore feature and with the ZTP flow.

## Current Alternatives

* Using ACM/OADP, the current workaround for including BMHs safely:
* Forcing BMHs in restore by adding an annotation: `cluster.open-cluster-management.io/backup: cluster-activation`
* Include BMHs original status during the restore (to avoid reprovisioning of the BMHs).
I.e. add 'restoreStatus' in the Restore CR
```
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Restore
...
spec:
restoreStatus:
includedResources:
- BareMetalHost
```

* For moving a spoke cluster between hubs:
* Current workaround is to use [import an existing cluster](../hive-integration/import-installed-cluster.md) flow.