tf/common: write an installer-dedicated kubelet.env

[lucab]

This updates all platforms to write the initial kubelet.env under
an installer-dedicated directory /etc/kubernetes/installer/, in
order to differentiate between the file dropped by terraform/ignition
and the one written by a bootstrap agent.

It is a prerequisite for OST-23.

[squat]

@lucab do you need to update the kubelet.service units to load this new envfile [0]? Otherwise, I think this change will make no difference in booted clusters.

[0] https://github.com/coreos/tectonic-installer/blob/master/modules/aws/ignition/resources/services/kubelet.service#L5

[lucab]

@squat no, that will be done by a dedicate bootstrapper. See the TODO line in this PR and the linked Jira ticket.

[squat]

@lucab got it. looks good

[lucab]

@s-urbaniak would prefer to hold this for one more iteration, in order to try avoiding introducing more assets duplication.

[alexsomesan]

Looks really cool. I like the approach!
Spoke to @s-urbaniak off-line about keeping this open for further testing.

[s-urbaniak]

retest this

[s-urbaniak]

ok to test

[Quentin-M]

Except for the bootkube master node, this file must not exist so that kubelet.service is blocked until another service queries an available API server for the version to use (e.g. the bootstrapping API server).

The process of launching bootkube+writing the env file on bootkube master, or querying the bootstraping API server+writing the env file is generic to all platforms. We worked a lot on simplifying the node states, the amount of configuration/files/services needed on each of them and the bootstrapping flow. I'd advocate for continuing to keep this as simple as possible and unifying the ignition files for all platforms.

Adding an extra file /etc/kubernetes/installer/kubelet.env on every nodes, that is then moved/copied/used conditionally does not sound too appealing. I'd rather have the actual env file always written by ignition - but overwrote by the kubelet service if an API server is available to give us the data. Worse case scenario, if we consider that no bootkube master election is available, there could be a time-out on the process of retrieving the version to use. The fact that all kubelet services will fail to retrieve the version to use on first cluster boot is acceptable as we already know that a cluster we are just bootstrapping will already have the appropriate kubelet version. For new nodes, the query will go through and the file will be updated before kubelet starts.

However, note that the interaction with the node agent is sensitive too and would have to be taken into consideration.

/cc @yifan-gu (who worked on the initial solution with me)

[s-urbaniak]

ok to test

[lucab]

The module/ignition refactoring is now happening in #1743. I'll keep rebasing this until the other lands.

[lucab]

Most of the pre-requisite work here landed in #1743. Closing this, I'll keep torcx changes in #1710.