Add support for Container Linux Config snippets

[dghubble]

Introduce the ability for users to supply Container Linux Config "snippets" for controllers or workers. Snippets are valid Container Linux Configs which are additively merged into the base Container Linux Configs that Typhoon uses to provision controllers and workers. Configs are validated, merged into a single config for use in instance user-data. Validation errors are shown with line numbers during terraform plan.

This is a major feature and has been long awaited.

• Allows users to add custom systemd units, network configs, files, raid arrays, or other customizations related to on-host disk provisioning that is supported by Container Linux Configs (at Ignition v2.0.0).
• Reduces user temptation to add a Terraform variable for every option. This helps keep Typhoon modules readable and maintainable over the long run.
• See the Container Linux Config spec and examples

Examples

Suppose a user would like to enable the BBR congestion control algorithm (arbitrary example customization). That requires setting a /etc/sysctl.d/bbr.conf file on hosts and would previously have necessitated making a tiny fork of Typhoon with the change. Now:

# bbr-conf
storage:
  files:
    - path: /etc/sysctl.d/bbr.conf
      filesystem: root
      contents:
        inline: |
          net.core.default_qdisc=fq
          net.ipv4.tcp_congestion_control=bbr

module "digital-ocean-nemo" {
  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.9.3"

  providers = {
    digitalocean = "digitalocean.default"
    local = "local.default"
    null = "null.default"
    template = "template.default"
    tls = "tls.default"
  }

  region   = "nyc3"
  dns_zone = "digital-ocean.example.com"

  cluster_name     = "nemo"
  image            = "coreos-stable"
  controller_count = 1
  controller_type  = "s-2vcpu-2gb"
  worker_count     = 2
  worker_type      = "s-1vcpu-1gb"
  ssh_fingerprints = ["d7:9d:79:ae:56:32:73:79:95:88:e3:a2:ab:5d:45:e7"]

  # output assets dir
  asset_dir = "/home/user/.secrets/clusters/nemo"

  # Container Linux Config custom snippets (NEW!)
  controller_clc_snippets = [
    "${file("./bbr-conf")}",
  ]
  worker_clc_snippets = [
    "${file("./bbr-conf")}"
  ]
}

Users may supply a list of valid Container Linux Config snippets to declare host disk provisioning details like systemd units, files, network configuration, etc.

Caveats

• Requires terraform-provider-ct v0.2.1 plugin
• There are limits to what additive merging (technically append) will allow. You cannot remove bits of the base config used by Typhoon. This reaffirms Typhoon's focus on minimalism.
• Typhoon cannot make guarantees about different end-user customizations. We'll continue to design, build, and test clusters as usual, but will not entertain reports concerning end-user's customizations (e.g. "I customized X and now Y is broken").

Partially addresses #140

[cjose3]

Hi @dghubble, I would like run kubelet with enable-controller-attach-detach = false.
can I use worker_clc_snippets|controller_clc_snippets to update the the systemd unit of kubelet.service?

I would like add another option to ExecStart:

ExecStart=/usr/lib/coreos/kubelet-wrapper \
          --allow-privileged \
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins \
->        --enable-controller-attach-detach=false

Is it possible using this snippets?

I think that maybe could be I good idea add the option --config to to config kubelet via file

Like this:

ExecStart=/usr/lib/coreos/kubelet-wrapper \
          --allow-privileged \
          --anonymous-auth=false \
          --client-ca-file=/etc/kubernetes/ca.crt \
          --cluster_dns=${k8s_dns_service_ip} \
          --cluster_domain=${cluster_domain_suffix} \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --exit-on-lock-contention \
          --hostname-override=$${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0} \
          --kubeconfig=/etc/kubernetes/kubeconfig \
          --lock-file=/var/run/lock/kubelet.lock \
          --network-plugin=cni \
          --node-labels=node-role.kubernetes.io/node \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --volume-plugin-dir=/var/lib/kubelet/volumeplugins \
->        --config=/etc/kubernetes/kubeletconfig.yml       

And create this file /etc/kubernetes/kubeletconfig.yml using ct

storage:
  files:
    - path: /etc/kubernetes/kubeletconfig.yml
      filesystem: root
      contents:
        inline: |
          ${kubelet_config_file}

And kubelet_config_file will be a new variable to typhoon.

Please let me know if it is possible and I will be the pr.
Regards!

[valer-cara]

👍 @cjose3 - also encountered the need to customize the kubelet-wrapper params while trying to mount localvolumes via the local storage provisioner.

kubelet will fail to initialize local volumes located in a sane path on the system (eg: /mnt) because that path is not bound within the kubelet's container.

MountVolume.NewMounter initialization failed for volume "local-pv-7610af06" : path "/mnt/localstore" does not exist

I'd need to add another line to RKT_RUN_ARGS here, i'm guessing like:

--volume mnt,kind=host,source=/mnt
--mount volume=mnt,target=/mnt

Will fork for now, but I wish it was easier.

---

Side note, i found that out the hard way, stracing kubelet 🤦‍♀️ 😿

...
newfstatat(AT_FDCWD, "/mnt/localstore", 0xc001a046b8, 0) = -1 ENOENT 
...

ip-10-0-61-212 mnt # ls -l /proc/$(pgrep kubelet)/cwd
lrwxrwxrwx. 1 root root 0 Nov 29 20:55 /proc/971/cwd -> /var/lib/rkt/pods/run/33fc0b9d-4a38-45db-be49-e79fa051f009/stage1/rootfs/opt/stage2/hyperkube/rootfs

AT_FDCWD - "stat relative to the cwd of the process"