decouple creation progress into phases

[xetys]

Currently, hetzner-kube cluster create starts the complete creation progress from start to the end. So if something breaks for some reason in the end, the user must start the progress and wait until all the previous steps will happen again.

That's why we need phases. With phases, the creation progress (and in future some more features) is split up into single parts, which can be run solely.

This gives at least two benefits:

1. as mentioned above, the user can repeat any task, without repeating the whole process
2. it highly supports custom administrative tasks, e.g. if hetzner-kube is used for cluster migration, and we need to repeat only certain tasks after custom modification of the cluster (this is where my personal motivation comes from)
3. it allows us to customize these phases for the future, by giving them more options than which are used in the general creation progress

with this in mind, we can use hetzner-kube as a tool for complex administrative tasks in a mature cluster, and not just initial creation.

[xetys]

Design Proposal

For this, a new Phase interface is introduced, which takes care of a single phase to run (and to decide, if it should run, depending on the cluster options). The basic phases for a cluster creation in the first version should be ProvisionNodesPhase, NetworkSetupPhase, EtcdSetupPhase, InstallMastersPhase, SetupHighAvailabilityPhase and InstallWorkersPhase.

These phases can be used both from cluster create and from distinct sub-commands.

Furthermore, a helper class PhaseChain is introduced, which helps to run several phases at once, while unifying error handling and more.

Commands structure

With this enhancement, the cluster create command doesn't change its observable behavior, but uses the phases system to solve its function. The enhancement introduces the cluster phase sub-command, which lists all the above phases and allows them to run them, given the state that the nodes are already created.

hetzner-kube cluster phase

should list all available phases

hetzner-kube cluster phase network-setup my-cluster

should reconfigure wireguard in my-cluster

[mavimo]

I really like this approach, that go in the same direction of kubeadm init phase IMHO we should consider two more points:

1. hetzner-kube cluster phase network-setup {CLUSTER_NAME} sounds "strange" since I suppose to put the cluster name before the action we should do (but I know is not easy, and I can't find any other tool doing something similar :D )
2. Commands should be idempotent so if a phase is already executed on a cluster no action should be done, this will allow to preserve the cluster status, and an optional flag "overwrite" should be added to re-execute the task

@xetys WDYT?

[xetys]

1. is hard to solve, as of the way of cobra is working. AFAIK.
2. I'm not agreeing fully on this one, as something could go wrong and you really want to do that again. During a recent migration, I used hetzner-kube cluster phase install-masters quite often. About the point with overwriting data, I decided to allow to give concrete phases additional flags. So the etcd phase has a --keep-data flag, which installs etcd without killing /var/lib/etcd. You can use this for upgrading etcd, if you installed version 3.2 using hetzner-kube 0.3.X, and use the flag to install etcd 3.3, you will end up with an upgrade, but not losing any data. Same with the install-masters phase and its --keep-all-certs option, which doesn't overwrite certificates. In general, I suggest to use flags to decide which kind of data to overwrite or to keep. In fact, this is how etcd management, phases & fixes #268 is doing this

[mavimo]

@xetys thx on your time to read what I wrote:

1. I agree on complexity and skip it sounds good to me
2. The doubt I have it that add custom flag for each kind of behaviour on each step should be really hard for user since they need exactly to know what happen inside the step in order to use the appropriate flag, use "idempotency" by default allow user to change behaviour only when they need (and know that something is going wrong so they can check how to solve issue using appropriate flag)

I know that's only my PoV so if you're not convinced NP, pleas skip my suggestion I also understand your PoV ;)