Skip to content

enhancements/etcd/disaster-recovery-with-ceo: add backup proposal #240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hexfusion wants to merge 2 commits into from
Closed

enhancements/etcd/disaster-recovery-with-ceo: add backup proposal #240

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
177eb96
enhancements/etcd/disaster-recovery-with-ceo: add backup proposal
hexfusion Mar 5, 2020
3b39abe
enhancements/etcd/disaster-recovery-with-ceo: add user stories
hexfusion Mar 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions enhancements/etcd/disaster-recovery-with-ceo.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,38 @@ This procedure enables to recover from a situation where the control plane certi

## Proposal

### Backup

1. On all masters create a new backup revision `/etc/kubernetes/static-pod-manifests/backup-N`.
2. On all masters write `/etc/kubernetes/static-pod-manifests/backup-N/backup.env` file containing 3 environmental
variables `CREATED`, `ETCD_REVISION` and `APISERVER_REVISION`.
Copy link
Contributor Author

@hexfusion hexfusion Jun 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to be extended now as we are managing more static pods resources. I think we should also include OCP version.

Copy link
Contributor

@retroflexer retroflexer Jun 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree.

2. On all masters copy directory `/etc/kubernetes/static-pod-manifests/etcd-pod-${ETCD_REVISION}` to
`/etc/kubernetes/static-pod-manifests/backup-N/etcd-pod`.
3. On all masters take an etcd snapshot `etcdctl snapshot save
/etc/kubernetes/static-pod-manifests/backup-N/etcd-data/backup.db`.
4. On all masters copy directory `/etc/kubernetes/static-pod-manifests/kube-apiserver-pod-${APISERVER_REVISION}`
to `/etc/kubernetes/static-pod-manifests/backup-N/kube-apiserver-pod`.
5. On all masters replace directory `/etc/kubernetes/static-pod-manifests/backups` with a copy of
`/etc/kubernetes/static-pod-manifests/backup-N` directory.
Copy link
Contributor

@retroflexer retroflexer Mar 5, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On 5, would a symoblic link of /etc/kubernetes/static-pod-manifests/backups to /etc/kubernetes/static-pod-manifests/backup-N be better, perhaps?

Copy link
Contributor Author

@hexfusion hexfusion Jun 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably, the size of the etcd snapshot would not make a copy reasonable. Managing the link would be simpler.


## User Stories [optional]

### Security

Your clusters backup data is as secure as your cluster. If someone were to root the system they would have direct
access to all data.

### Availability

Your data is as resilient as your cluster. We make N copies of your data so in the case of failure you dont have to
worry about your last backup location.

### Recovery Automation

If the cluster were to lose quorum and every master is seeded with data required to restore. Automation of recovery
tasks becomes easier.


## Implementation Plan

1. Make changes to the scripts as needed
Expand Down