Documentation: enumerate self-hosted etcd operator failure scenarios

[philips]

Currently we can bringup self-hosted etcd operator setups but we don't document failure or recovery scenarios even though we handle many of them.

• master cluster down and power-on
• master cluster API server failure
• disk loss failure of entire master cluster
• pod checkpoint checkpoints bad versions

cc @jbeda @justinsb @luxas

[philips]

@justinsb what others am I missing?

[philips]

xref kubernetes-retired/bootkube#432

[justinsb]

That's a good start. A few more off-the-top-of-my-head:

• power-off recovery during or soon after etcd2 -> etcd3 upgrade (or any major failure), where "bootstrap" version is older than local version

• etcd running but not responding to queries (e.g. disk full?)

• apiserver runs but does not make progress after etcd operation

• k-c-m runs but does not make progress after etcd operation

• kube-scheduler runs but does not make progress after etcd operation

• kubelet upgrade runs but does not make progress after etcd operation

(These are the standard gotchas of self-hosting, but I guess there's a particularly likely scenario for etcd upgrades)

• understanding recovery semantics in terms of data loss from catastrophic failure scenario, and giving users a choice as to whether they prefer downtime or data-loss, or at least define what choice has been made

• recovery from SSL key expiry

[philips]

There are recovery tools being built in bootkube now: https://github.com/kubernetes-incubator/bootkube#recover-a-downed-cluster

[philips]

Quick notes on places to start on writing these docs:

• master cluster down and power-on: uses pod checkpointing and network checkpointing
• master cluster API server failure: if non-HA you need to recover the load balancer or pods via static pods
• disk loss failure of entire master cluster: you need to recover from backups, see bootkube recovery
• pod checkpoint checkpoints bad versions: you need to manually fix the static manifests checkpointed in /etc/kubernetes/inactive-manifests

[xiang90]

@philips

I agree it is a good idea to write docs about handling the failure cases for self hosted etcd. The items you listed are good starts! I believe the doc should focus on the difference between self hosted and external etcd, and highlights the potential risks self hosted etcd might introduce and how we solve them.

@justinsb

The a lot of items you listed are not really specific to self hosted etcd in my opinion.

etcd running but not responding to queries (e.g. disk full?)
apiserver runs but does not make progress after etcd operation
k-c-m runs but does not make progress after etcd operation
kube-scheduler runs but does not make progress after etcd operation
kubelet upgrade runs but does not make progress after etcd operation
recovery from SSL key expiry

If you manually operator etcd, you might have these issues too. They are not introduced by self hosted etcd.

power-off recovery during or soon after etcd2 -> etcd3 upgrade (or any major failure), where "bootstrap" version is older than local version

understanding recovery semantics in terms of data loss from catastrophic failure scenario, and giving users a choice as to whether they prefer downtime or data-loss, or at least define what choice has been made

These two are relevant. We will cover it when writing the doc. But I would suggest you to give self hosted etcd a try if you are interested. So we can discuss in more depth.

[zbwright]

@radhikapc is working on etcd docs now. assigning to her, @xiang90

[Quentin-M]

Hi @radhikapc, any update on that one?

[xiang90]

@Quentin-M hongchao or I need to write something similar to https://github.com/coreos/etcd/blob/master/Documentation/op-guide/failures.md. Then @radhikapc can start to help cleaning things up. we will get started after finishing up the TLS thing.

[justinsb]

Where was this moved to?