diff --git a/Documentation/dev/node-bootstrap-flow.md b/Documentation/dev/node-bootstrap-flow.md index aa364f7a2a..4500ddeaa4 100644 --- a/Documentation/dev/node-bootstrap-flow.md +++ b/Documentation/dev/node-bootstrap-flow.md @@ -7,32 +7,26 @@ This is a development document which describes the bootstrapping flow for Contai When a cluster node is being bootstrapped from scratch, it goes through several phases in the following order: 1. first-boot OS configuration, via ignition (systemd units, node configuration, etc) -1. provisioning of additional assets (k8s manifests, TLS material), via either of: +2. provisioning of additional assets (k8s manifests, TLS material), via either of: * pushing from terraform file/remote-exec (SSH) * pulling from private cloud stores (S3 buckets) -1. system-wide updates via `k8s-node-bootstrap.service`, which includes: - * determining current kubernetes cluster version (when joining an existing cluster) - * triggering a ContainerLinux update, via update-engine (optional) - * downloading and deploying proper docker addon version, via tectonic-torcx - * writing the `kubelet.env` file -1. if needed, a node reboot is triggered to apply systemd-wide changes and to clean container runtime datadir -1. `kubelet.service` picks up the `kubelet.env` file and actually starts the kubelet as a rkt-fly service. +3. if needed, a node reboot is triggered to apply systemd-wide changes and to clean container runtime datadir +4. `kubelet.service` picks up the `kubelet.env` file and actually starts the kubelet as a rkt-fly service. Additionally, only on one of the master nodes the following kubernetes bootstrapping happens: 1. `bootkube.service` is started after `kubelet.service` start -1. a static bootstrapping control-plane is deployed -1. a fully self-hosted control-plane starts and takes over the previous one -1. `bootkube.service` is completed with success -1. `tectonic.service` is started -1. a self-hosted tectonic control-plane is deployed -1. `tectonic.service` is completed with success +2. a static bootstrapping control-plane is deployed +3. a fully self-hosted control-plane starts and takes over the previous one +4. `bootkube.service` is completed with success +5. `tectonic.service` is started +6. a self-hosted tectonic control-plane is deployed +7. `tectonic.service` is completed with success ## Systemd units -The following systemd units are deployed to a node by tectonic-installer and take part in the bootstrapping process: +The following systemd unit is deployed to a node by tectonic-installer and take part in the bootstrapping process: -* `k8s-node-bootstrap.service` ensures node and assets freshness. It is automatically started on boot, can crash-loop, and it runs only during bootstrap * `kubelet.service` is the main kubelet daemon. It is automatically started on boot, it is crash-looping until `kubelet.env` is populated, and it runs on each boot Additionally, only on one of the master nodes the following kubernetes bootstrapping happens: @@ -42,35 +36,12 @@ Additionally, only on one of the master nodes the following kubernetes bootstrap * `tectonic.service` deploys tectonic control-plane. It is started only after `bootkube.service` _has completed_. It is a oneshot unit and cannot crash, and it runs only during bootstrap * `bootkube.path` waits for tectonic assets/scripts to exist on disk and triggers `tectonic.service` -`k8s-node-bootstrap` runs [tectonic-torcx][tectonic-torcx] as a containerized service, thus relying on a container runtime being already on the node. -It currently assumes that Docker is available and working. In case of version changes, a cleanup of the Docker datadir `/var/lib/docker` is scheduled before rebooting. - [tectonic-torcx]: https://github.com/coreos/tectonic-torcx ## Service ordering Service ordering is enforced via systemd dependencies. This is the rationale for the settings, with relevant snippets: -### `k8s-node-bootstrap.service` - -``` -ConditionPathExists=!/etc/kubernetes/kubelet.env -Before=kubelet.service -Restart=on-failure -ExecStartPre=[...] -ExecStart=/usr/bin/echo "node components bootstrapped" -WantedBy=multi-user.target kubelet.service -``` - -This service is enabled by default and can crash-loop until success. -Main logic happens in `Pre`, before the unit is marked as started, to block further services (a synchronous reboot can happen here). - -In particular, this blocks kubelet from starting by: - * a `WantedBy=` and `Before=` - * writing the actual `kubelet.env` file on success. - -It is skipped on further boots, as the condition-path exists. - ### `kubelet.service` ``` @@ -81,7 +52,6 @@ WantedBy=multi-user.target ``` This service is enabled by default and can crash-loop until success. -On first boot, it is initially blocked by `k8s-node-bootstrap.service`. It crash-loop until the `kubelet.env` file exists. It is started on every boot. @@ -135,7 +105,6 @@ This is a visual simplified representation of the overall bootstrapping flow. Legend: * TF -> terraform provisioner * IGN -> ignition - * knb.s -> k8s-node-bootstrap.service * k.s -> kubelet.service * b.p -> bootkube.path * b.s -> bootkube.service @@ -144,28 +113,28 @@ Legend: * rm.p -> rm-assets.path * rm.s -> rm-assets.service -.---------------------------------------------------------------------------------------------------------------------------------------+ -| | -| Provision cloud/userdata +----------+ Provision files | -| ,----------------------------------------------o| TF |o-----------------.------------------------.-----------------+ | -| | +----------+ | | | | -| | v v v | -| | +----------+ +-----+ +-------+ +------+ | -| | .--->| (reboot) |----. | b.p | | t.p | | rm.p | | -| | | +----------+ | +-----+ +-------+ +------+ | -| V | | o o o | -| +-------+ | v Before +------------+ Before | Trigger Trigger | Trigger | | -| | IGN | | *---------->| k.s |o--------. | | | | -| +-------+ o ^ +------------+ | v v v | -| | +----------+ | ^ | | +-----+ Before +-------+ Before +-----+ | -| '------>| knb.s |o--------------' | v '--->| b.s |o--------------->| t.s |--------> |rm.s | | -| Enable +----------+ '------' +-----+ +-------+ +-----+ | -| ^ | | -| | v | -| '----' o o | -| | | | -| * First boot | * Each boot | * First boot | -| * All nodes | * All nodes | * Bootkube master | -| | | | -'----------------------------------------------o----------------------------o-----------------------------------------------------------+ +.--------------------------------------------------------------------------------------------------------------------------------+ +| | +| Provision cloud/userdata +----------+ Provision files | +| ,---------------------------------------o| TF |o-----------------.------------------------.-----------------+ | +| | +----------+ | | | | +| | v v v | +| | +-----+ +-------+ +------+ | +| | | b.p | | t.p | | rm.p | | +| | +-----+ +-------+ +------+ | +| V o o o | +| +-------+ Before +------------+ Before | Trigger Trigger | Trigger | | +| | IGN | .--------------->| k.s |o--------. | | | | +| +-------+ | +------------+ | v v v | +| | | ^ | | +-----+ Before +-------+ Before +-----+ | +| '----------------------' | v '--->| b.s |o--------------->| t.s |--------> |rm.s | | +| Enable '------' +-----+ +-------+ +-----+ | +| | +| | +| o o | +| | | | +| | * Each boot | * First boot | +| | * All nodes | * Bootkube master | +| | | | +'---------------------------------------o----------------------------o-----------------------------------------------------------+ ``` diff --git a/Documentation/variables/config.md b/Documentation/variables/config.md index 96a8223931..e96d3bdfdf 100644 --- a/Documentation/variables/config.md +++ b/Documentation/variables/config.md @@ -9,7 +9,6 @@ This document gives an overview of variables used in all platforms of the Tecton | tectonic_admin_email | (internal) The e-mail address used to: 1. login as the admin user to the Tectonic Console. 2. generate DNS zones for some providers.

Note: This field MUST be in all lower-case e-mail address format and set manually prior to creating the cluster. | string | - | | tectonic_admin_password | (internal) The admin user password to login to the Tectonic Console.

Note: This field MUST be set manually prior to creating the cluster. Backslashes and double quotes must also be escaped. | string | - | | tectonic_base_domain | The base DNS domain of the cluster. It must NOT contain a trailing period. Some DNS providers will automatically add this if necessary.

Example: `openstack.dev.coreos.systems`.

Note: This field MUST be set manually prior to creating the cluster. This applies only to cloud platforms.

[Azure-specific NOTE] To use Azure-provided DNS, `tectonic_base_domain` should be set to `""` If using DNS records, ensure that `tectonic_base_domain` is set to a properly configured external DNS zone. Instructions for configuring delegated domains for Azure DNS can be found here: https://docs.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns | string | - | -| tectonic_bootstrap_upgrade_cl | (internal) Whether to trigger a ContainerLinux upgrade on node bootstrap. | string | `true` | | tectonic_ca_cert | (optional) The content of the PEM-encoded CA certificate, used to generate Tectonic Console's server certificate. If left blank, a CA certificate will be automatically generated. | string | `` | | tectonic_ca_key | (optional) The content of the PEM-encoded CA key, used to generate Tectonic Console's server certificate. This field is mandatory if `tectonic_ca_cert` is set. | string | `` | | tectonic_ca_key_alg | (optional) The algorithm used to generate tectonic_ca_key. The default value is currently recommended. This field is mandatory if `tectonic_ca_cert` is set. | string | `RSA` | diff --git a/config.tf b/config.tf index 872b38ee0f..96587038c7 100644 --- a/config.tf +++ b/config.tf @@ -323,12 +323,6 @@ variable "tectonic_networking" { EOF } -variable "tectonic_bootstrap_upgrade_cl" { - type = "string" - default = "true" - description = "(internal) Whether to trigger a ContainerLinux upgrade on node bootstrap." -} - variable "tectonic_kubelet_debug_config" { type = "string" default = "" diff --git a/modules/ignition/assets.tf b/modules/ignition/assets.tf index 008d4eba66..71cea0a88f 100644 --- a/modules/ignition/assets.tf +++ b/modules/ignition/assets.tf @@ -49,23 +49,6 @@ data "ignition_systemd_unit" "kubelet" { content = "${data.template_file.kubelet.rendered}" } -data "template_file" "k8s_node_bootstrap" { - template = "${file("${path.module}/resources/services/k8s-node-bootstrap.service")}" - - vars { - bootstrap_upgrade_cl = "${var.bootstrap_upgrade_cl}" - tectonic_torcx_image = "${var.container_images["tectonic_torcx"]}" - torcx_skip_setup = "false" - torcx_store_url = "${var.torcx_store_url}" - } -} - -data "ignition_systemd_unit" "k8s_node_bootstrap" { - name = "k8s-node-bootstrap.service" - enabled = true - content = "${data.template_file.k8s_node_bootstrap.rendered}" -} - data "ignition_systemd_unit" "rm_assets" { name = "rm-assets.service" enabled = true diff --git a/modules/ignition/outputs.import b/modules/ignition/outputs.import index 7bd99a6f17..c50348b840 100644 --- a/modules/ignition/outputs.import +++ b/modules/ignition/outputs.import @@ -24,10 +24,6 @@ variable "ign_installer_kubelet_env_id" { type = "string" } -variable "ign_k8s_node_bootstrap_service_id" { - type = "string" -} - variable "ign_update_ca_certificates_dropin_id" { type = "string" } diff --git a/modules/ignition/outputs.tf b/modules/ignition/outputs.tf index a74a032442..3a48a5a611 100644 --- a/modules/ignition/outputs.tf +++ b/modules/ignition/outputs.tf @@ -32,7 +32,6 @@ output "ignition_systemd_id_list" { "${data.ignition_systemd_unit.docker_dropin.id}", "${data.ignition_systemd_unit.kubelet.id}", "${data.ignition_systemd_unit.locksmithd.id}", - "${data.ignition_systemd_unit.k8s_node_bootstrap.id}", "${data.ignition_systemd_unit.update_ca_certificates_dropin.id}", "${data.ignition_systemd_unit.rm_assets.id}", ] diff --git a/modules/ignition/resources/services/k8s-node-bootstrap.service b/modules/ignition/resources/services/k8s-node-bootstrap.service deleted file mode 100644 index 84c13e757b..0000000000 --- a/modules/ignition/resources/services/k8s-node-bootstrap.service +++ /dev/null @@ -1,43 +0,0 @@ -[Unit] -Description=Bootstrap Kubernetes Node Components -ConditionPathExists=!/etc/kubernetes/kubelet.env -Wants=update-ca-certificates.service -After=update-ca-certificates.service -Before=kubelet.service - -[Service] -Type=simple -RemainAfterExit=true -Restart=on-failure -RestartSec=10 -TimeoutStartSec=1h -ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes -ExecStartPre=/usr/bin/docker run --rm \ - --env http_proxy \ - --env HTTP_PROXY \ - --env https_proxy \ - --env HTTPS_PROXY \ - --env no_proxy \ - --env NO_PROXY \ - --tmpfs /tmp \ - -v /usr/share:/usr/share:ro \ - -v /usr/lib/os-release:/usr/lib/os-release:ro \ - -v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro \ - -v /var/lib/torcx:/var/lib/torcx \ - -v /var/run/dbus:/var/run/dbus \ - -v /run/metadata:/run/metadata:ro \ - -v /run/torcx:/run/torcx:ro \ - -v /run/systemd:/run/systemd \ - -v /etc/coreos:/etc/coreos:ro \ - -v /etc/torcx:/etc/torcx \ - -v /etc/kubernetes:/etc/kubernetes \ - ${tectonic_torcx_image} \ - /tectonic-torcx-bootstrap \ - --upgrade-os=${bootstrap_upgrade_cl} \ - --torcx-manifest-url="${torcx_store_url}" \ - --torcx-skip-setup=${torcx_skip_setup} \ - --verbose=debug -ExecStart=/usr/bin/echo "node components bootstrapped" - -[Install] -WantedBy=multi-user.target kubelet.service diff --git a/modules/ignition/variables.tf b/modules/ignition/variables.tf index 174d1254c3..ef29ce0134 100644 --- a/modules/ignition/variables.tf +++ b/modules/ignition/variables.tf @@ -45,12 +45,6 @@ variable "cloud_provider_config" { default = "" } -variable "bootstrap_upgrade_cl" { - type = "string" - description = "(optional) Whether to trigger a ContainerLinux OS upgrade during the bootstrap process." - default = "true" -} - variable "torcx_store_url" { type = "string" description = "(optional) URL template for torcx store. Leave empty to use the default CoreOS endpoint." diff --git a/steps/assets/aws/main.tf b/steps/assets/aws/main.tf index b9104b6625..5e64dde133 100644 --- a/steps/assets/aws/main.tf +++ b/steps/assets/aws/main.tf @@ -21,7 +21,6 @@ module assets_base { ingress_kind = "haproxy-router" tectonic_base_domain = "${var.tectonic_base_domain}" - tectonic_bootstrap_upgrade_cl = "${var.tectonic_bootstrap_upgrade_cl}" tectonic_cluster_name = "${var.tectonic_cluster_name}" tectonic_container_images = "${var.tectonic_container_images}" tectonic_image_re = "${var.tectonic_image_re}" diff --git a/steps/assets/base/ignition-bootstrap.tf b/steps/assets/base/ignition-bootstrap.tf index 276f2bb4df..68676b7612 100644 --- a/steps/assets/base/ignition-bootstrap.tf +++ b/steps/assets/base/ignition-bootstrap.tf @@ -1,7 +1,6 @@ module "ignition_bootstrap" { source = "../../../modules/ignition" - bootstrap_upgrade_cl = "${var.tectonic_bootstrap_upgrade_cl}" cloud_provider = "${var.cloud_provider}" container_images = "${var.tectonic_container_images}" etcd_ca_cert_pem = "${local.etcd_ca_cert_pem}" diff --git a/steps/assets/libvirt/main.tf b/steps/assets/libvirt/main.tf index 4447b62f95..c267f23974 100644 --- a/steps/assets/libvirt/main.tf +++ b/steps/assets/libvirt/main.tf @@ -9,7 +9,6 @@ module assets_base { ingress_kind = "haproxy-router" tectonic_base_domain = "${var.tectonic_base_domain}" - tectonic_bootstrap_upgrade_cl = "${var.tectonic_bootstrap_upgrade_cl}" tectonic_cluster_name = "${var.tectonic_cluster_name}" tectonic_container_images = "${var.tectonic_container_images}" tectonic_image_re = "${var.tectonic_image_re}"