Skip to content
This repository was archived by the owner on Jul 19, 2024. It is now read-only.

Commit 683f0d1

Browse files
displaguedisplague
committed
Merge branch 'feature/update-operator'
2 parents 8a02287 + ec47b88 commit 683f0d1

16 files changed

+284
-45
lines changed

‎README.md

+23-5
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Kubernetes Terraform installer for Linode Instances
22

3-
This Terraform module creates a Kubernetes v1.13 Cluster on Linode Cloud infrastructure using the ContainerLinux operating system. The cluster is designed to take advantage of the Linode regional private network, and is equiped with Linode specific cluster enhancements.
3+
This Terraform module creates a Kubernetes v1.14.0 Cluster on Linode Cloud infrastructure using the ContainerLinux operating system. The cluster is designed to take advantage of the Linode regional private network, and is equiped with Linode specific cluster enhancements.
44

55
Cluster size and instance types are configurable through Terraform variables.
66

@@ -20,8 +20,8 @@ Before running the project you'll have to create an access token for Terraform t
2020
Using the token and your access key, create the `LINODE_TOKEN` environment variable:
2121

2222
```bash
23-
read -sp "Linode Token: " LINODE_TOKEN # Enter your Linode Token (it will be hidden)
24-
export LINODE_TOKEN
23+
read -sp "Linode Token: " TF_VAR_linode_token # Enter your Linode Token (it will be hidden)
24+
export TF_VAR_linode_token
2525
```
2626

2727
This variable will need to be supplied to every Terraform `apply`, `plan`, and `destroy` command using `-var linode_token=$LINODE_TOKEN` unless a `terraform.tfvars` file is created with this secret token.
@@ -74,8 +74,14 @@ This will do the following:
7474
* installs a Calico network between Linode Instances
7575
* runs kubeadm init on the master server and configures kubectl
7676
* joins the nodes in the cluster using the kubeadm token obtained from the master
77-
* installs Linode add-ons: CSI (LinodeBlock Storage Volumes), CCM (Linode NodeBalancers), External-DNS (Linode Domains)
78-
* installs cluster add-ons: Kubernetes dashboard, metrics server and Heapster
77+
* installs Linode add-ons:
78+
* [CSI](https://github.com/linode/linode-blockstorage-csi-driver/) (LinodeBlock Storage Volumes)
79+
* [CCM](https://github.com/linode/linode-cloud-controller-manager) (Linode NodeBalancers)
80+
* [External-DNS](https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/linode.md) (Linode Domains)
81+
* installs cluster add-ons:
82+
* Kubernetes dashboard
83+
* metrics server
84+
* [Container Linux Update Operator](https://github.com/coreos/container-linux-update-operator)
7985
* copies the kubectl admin config file for local `kubectl` use via the public IP of the API server
8086

8187
A full list of the supported variables are available in the [Terraform Module Registry](https://registry.terraform.io/modules/linode/k8s/linode/?tab=inputs).
@@ -161,6 +167,18 @@ As configured in this Terraform module, any service or ingress with a specific a
161167

162168
[Learn more at the External-DNS Github project.](https://github.com/kubernetes-incubator/external-dns)
163169

170+
### [**Container Linux Update Operator**](https://github.com/coreos/container-linux-update-operator/)
171+
172+
The Update Operator deploys an agent to all of the nodes (include the master) which will schedule Container Linux reboots when an update has been prepared. The Update Operator prevents multiple nodes from rebooting at the same time. Cordone and drain commands are sent to the nodes before rebooting. **System update reboots are paused by default** to prevent new clusters from rebooting in the first five minutes of their life-cycle which could have an adverse effect on the Terraform provisioning process.
173+
174+
Set the `update_agent_reboot_paused` variable using the `-var` argument, `TF_VAR_update_agent_reboot_paused` environment variable, or by creating a `update_agent.tfvars` file with the following contents:
175+
176+
```
177+
update_agent_reboot_paused = "false"
178+
```
179+
180+
In practice, rebooted nodes will be unavailable for a minute or two once the reboot has started. Take advantage of the Linode Block Storage CSI driver so Persistent Volumes can be rescheduled with workloads to the available nodes.
181+
164182
## Development
165183

166184
To make changes to this project, verify that you have the prerequisites and then clone the repo. Instead of using the Terraform `module` syntax, and being confined by the variables that are provided, you'll be able to make any changes necessary.

‎main.tf

+18-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
provider "linode" {
22
token = "${var.linode_token}"
3-
version = "1.4.0"
3+
version = "1.5.0"
44
}
55

66
provider "external" {
@@ -64,3 +64,20 @@ resource "null_resource" "local_kubectl" {
6464
on_failure = "continue"
6565
}
6666
}
67+
68+
resource "null_resource" "update-agent" {
69+
depends_on = ["module.masters", "module.nodes"]
70+
71+
triggers {
72+
cluster_ips = "${"${module.masters.k8s_master_public_ip} ${join(" ", module.nodes.nodes_public_ip)}"}"
73+
}
74+
75+
provisioner "remote-exec" {
76+
connection {
77+
host = "${module.masters.k8s_master_public_ip}"
78+
user = "core"
79+
}
80+
81+
inline = ["/opt/bin/kubectl annotate node --all --overwrite container-linux-update.v1.coreos.com/reboot-paused=${var.update_agent_reboot_paused}"]
82+
}
83+
}

‎modules/instances/main.tf

+15-4
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,20 @@ resource "linode_instance" "instance" {
2929
}
3030
}
3131

32+
provisioner "remote-exec" {
33+
inline = [
34+
"mkdir -p /home/core/init/",
35+
]
36+
37+
connection {
38+
user = "core"
39+
timeout = "300s"
40+
}
41+
}
42+
3243
provisioner "file" {
3344
source = "${path.module}/scripts/"
34-
destination = "/tmp"
45+
destination = "/home/core/init/"
3546

3647
connection {
3748
user = "core"
@@ -42,9 +53,9 @@ resource "linode_instance" "instance" {
4253
provisioner "remote-exec" {
4354
inline = [
4455
"set -e",
45-
"chmod +x /tmp/start.sh && sudo /tmp/start.sh",
46-
"chmod +x /tmp/linode-network.sh && sudo /tmp/linode-network.sh ${self.private_ip_address} ${self.label}",
47-
"chmod +x /tmp/kubeadm-install.sh && sudo /tmp/kubeadm-install.sh ${var.k8s_version} ${var.cni_version} ${var.crictl_version} ${self.label} ${var.use_public ? self.ip_address : self.private_ip_address} ${var.k8s_feature_gates}",
56+
"chmod +x /home/core/init/start.sh && sudo /home/core/init/start.sh",
57+
"chmod +x /home/core/init/linode-network.sh && sudo /home/core/init/linode-network.sh ${self.private_ip_address} ${self.label}",
58+
"chmod +x /home/core/init/kubeadm-install.sh && sudo /home/core/init/kubeadm-install.sh ${var.k8s_version} ${var.cni_version} ${var.crictl_version} ${self.label} ${var.use_public ? self.ip_address : self.private_ip_address} ${var.k8s_feature_gates}",
4859
]
4960

5061
connection {

‎modules/instances/outputs.tf

+9-6
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,19 @@
11
// todo: ha, return nb address
22
output "public_ip_address" {
3-
depends_on = ["linode_instance.instance.0"]
4-
value = "${element(concat(linode_instance.instance.*.ip_address, list("")), 0)}"
3+
depends_on = ["linode_instance.instance.0"]
4+
description = "Public IP Address of the first instance in the group"
5+
value = "${element(concat(linode_instance.instance.*.ip_address, list("")), 0)}"
56
}
67

78
// todo: this doesnt make sense in ha -- return all?
89
output "private_ip_address" {
9-
depends_on = ["linode_instance.instance.0"]
10-
value = "${element(concat(linode_instance.instance.*.private_ip_address, list("")), 0)}"
10+
description = "Private IP Address of the first instance in the group"
11+
depends_on = ["linode_instance.instance.0"]
12+
value = "${element(concat(linode_instance.instance.*.private_ip_address, list("")), 0)}"
1113
}
1214

1315
output "nodes_public_ip" {
14-
depends_on = ["linode_instance.instance.*"]
15-
value = "${concat(linode_instance.instance.*.ip_address)}"
16+
depends_on = ["linode_instance.instance.*"]
17+
description = "Public IP Address of the instance(s)"
18+
value = "${concat(linode_instance.instance.*.ip_address)}"
1619
}

‎modules/instances/scripts/end.sh

-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,3 @@
11
#!/usr/bin/env bash
22
set -e
33

4-
# TODO: https://github.com/coreos/container-linux-update-operator
5-
# systemctl start update-engine
6-

‎modules/instances/scripts/linode-addons.sh

+3-3
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,14 @@ LINODE_TOKEN="$2"
88
sed -i -E \
99
-e 's/\$\(LINODE_REGION\)/'$LINODE_REGION'/g' \
1010
-e 's/\$\(LINODE_TOKEN\)/'$LINODE_TOKEN'/g' \
11-
/tmp/linode-token.yaml
11+
/home/core/init/linode-token.yaml
1212

1313
# TODO swap these for helm charts
1414
for yaml in \
1515
linode-token.yaml \
1616
ccm-linode.yaml \
1717
csi-linode.yaml \
1818
external-dns.yaml \
19-
; do kubectl apply -f /tmp/${yaml}; done
19+
; do kubectl apply -f /home/core/init/${yaml}; done
2020

21-
rm /tmp/linode-token.yaml
21+
rm /home/core/init/linode-token.yaml

‎modules/instances/scripts/monitoring-install.sh

+3-3
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ set -e
44

55
# TODO swap these for helm charts
66

7-
kubectl apply -f /tmp/dashboard-rbac.yaml
8-
kubectl apply -f /tmp/dashboard.yaml
7+
kubectl apply -f /home/core/init/dashboard-rbac.yaml
8+
kubectl apply -f /home/core/init/dashboard.yaml
99

10-
kubectl apply -f /tmp/metrics-server.yaml
10+
kubectl apply -f /home/core/init/metrics-server.yaml

‎modules/instances/scripts/start.sh

+5-1
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,8 @@ set -e
33

44
for mod in ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr nf_conntrack_ipv4; do echo $mod | sudo tee /etc/modules-load.d/$mod.conf; done
55

6-
systemctl stop update-engine
6+
# Enable the update-engine, but disable the locksmith which it requires
7+
sudo systemctl unmask update-engine.service || true
8+
sudo systemctl start update-engine.service || true
9+
sudo systemctl stop locksmithd.service || true
10+
sudo systemctl mask locksmithd.service || true

‎modules/instances/scripts/update-operator.sh

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
# TODO swap these for helm charts
6+
7+
kubectl apply -f /home/core/init/update-operator.yaml

‎modules/masters/main.tf

+20-6
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,21 @@ module "master_instance" {
1919
resource "null_resource" "masters_provisioner" {
2020
depends_on = ["module.master_instance"]
2121

22+
provisioner "remote-exec" {
23+
inline = [
24+
"mkdir -p /home/core/init/",
25+
]
26+
27+
connection {
28+
user = "core"
29+
timeout = "300s"
30+
host = "${module.master_instance.public_ip_address}"
31+
}
32+
}
33+
2234
provisioner "file" {
2335
source = "${path.module}/manifests/"
24-
destination = "/tmp"
36+
destination = "/home/core/init/"
2537

2638
connection {
2739
user = "core"
@@ -34,13 +46,15 @@ resource "null_resource" "masters_provisioner" {
3446
# TODO advertise on public adress
3547
inline = [
3648
"set -e",
37-
"chmod +x /tmp/kubeadm-init.sh && sudo /tmp/kubeadm-init.sh ${var.cluster_name} ${var.k8s_version} ${module.master_instance.public_ip_address} ${module.master_instance.private_ip_address} ${var.k8s_feature_gates}",
49+
"chmod +x /home/core/init/kubeadm-init.sh && sudo /home/core/init/kubeadm-init.sh ${var.cluster_name} ${var.k8s_version} ${module.master_instance.public_ip_address} ${module.master_instance.private_ip_address} ${var.k8s_feature_gates}",
3850
"mkdir -p $HOME/.kube && sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown core $HOME/.kube/config",
3951
"export PATH=$${PATH}:/opt/bin",
40-
"kubectl apply -f /tmp/calico.yaml",
41-
"chmod +x /tmp/linode-addons.sh && /tmp/linode-addons.sh ${var.region} ${var.linode_token}",
42-
"chmod +x /tmp/monitoring-install.sh && /tmp/monitoring-install.sh",
43-
"chmod +x /tmp/end.sh && sudo /tmp/end.sh",
52+
"kubectl apply -f /home/core/init/calico.yaml",
53+
"chmod +x /home/core/init/linode-addons.sh && /home/core/init/linode-addons.sh ${var.region} ${var.linode_token}",
54+
"chmod +x /home/core/init/monitoring-install.sh && /home/core/init/monitoring-install.sh",
55+
"chmod +x /home/core/init/update-operator.sh && /home/core/init/update-operator.sh",
56+
"kubectl annotate node $${HOSTNAME} --overwrite container-linux-update.v1.coreos.com/reboot-paused=true",
57+
"chmod +x /home/core/init/end.sh && sudo /home/core/init/end.sh",
4458
]
4559

4660
connection {

‎modules/masters/manifests/update-operator.yaml

+153
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# xref: https://raw.githubusercontent.com/coreos/container-linux-update-operator/master/examples/deploy/00-namespace.yaml
2+
apiVersion: v1
3+
kind: Namespace
4+
metadata:
5+
name: reboot-coordinator
6+
---
7+
# xref: https://raw.githubusercontent.com/coreos/container-linux-update-operator/master/examples/deploy/update-operator.yaml
8+
apiVersion: extensions/v1beta1
9+
kind: Deployment
10+
metadata:
11+
name: container-linux-update-operator
12+
namespace: reboot-coordinator
13+
spec:
14+
replicas: 1
15+
template:
16+
metadata:
17+
labels:
18+
app: container-linux-update-operator
19+
spec:
20+
containers:
21+
- name: update-operator
22+
image: quay.io/coreos/container-linux-update-operator:v0.7.0
23+
command:
24+
- "/bin/update-operator"
25+
env:
26+
- name: POD_NAMESPACE
27+
valueFrom:
28+
fieldRef:
29+
fieldPath: metadata.namespace
30+
tolerations:
31+
- key: node-role.kubernetes.io/master
32+
operator: Exists
33+
effect: NoSchedule
34+
---
35+
# xref: https://raw.githubusercontent.com/coreos/container-linux-update-operator/master/examples/deploy/update-agent.yaml
36+
apiVersion: extensions/v1beta1
37+
kind: DaemonSet
38+
metadata:
39+
name: container-linux-update-agent
40+
namespace: reboot-coordinator
41+
spec:
42+
updateStrategy:
43+
type: RollingUpdate
44+
rollingUpdate:
45+
maxUnavailable: 1
46+
template:
47+
metadata:
48+
labels:
49+
app: container-linux-update-agent
50+
spec:
51+
containers:
52+
- name: update-agent
53+
image: quay.io/coreos/container-linux-update-operator:v0.7.0
54+
command:
55+
- "/bin/update-agent"
56+
volumeMounts:
57+
- mountPath: /var/run/dbus
58+
name: var-run-dbus
59+
- mountPath: /etc/coreos
60+
name: etc-coreos
61+
- mountPath: /usr/share/coreos
62+
name: usr-share-coreos
63+
- mountPath: /etc/os-release
64+
name: etc-os-release
65+
env:
66+
# read by update-agent as the node name to manage reboots for
67+
- name: UPDATE_AGENT_NODE
68+
valueFrom:
69+
fieldRef:
70+
fieldPath: spec.nodeName
71+
- name: POD_NAMESPACE
72+
valueFrom:
73+
fieldRef:
74+
fieldPath: metadata.namespace
75+
tolerations:
76+
- key: node-role.kubernetes.io/master
77+
operator: Exists
78+
effect: NoSchedule
79+
volumes:
80+
- name: var-run-dbus
81+
hostPath:
82+
path: /var/run/dbus
83+
- name: etc-coreos
84+
hostPath:
85+
path: /etc/coreos
86+
- name: usr-share-coreos
87+
hostPath:
88+
path: /usr/share/coreos
89+
- name: etc-os-release
90+
hostPath:
91+
path: /etc/os-release
92+
---
93+
# xref: https://raw.githubusercontent.com/coreos/container-linux-update-operator/master/examples/deploy/rbac/cluster-role.yaml
94+
apiVersion: rbac.authorization.k8s.io/v1beta1
95+
kind: ClusterRole
96+
metadata:
97+
name: reboot-coordinator
98+
rules:
99+
- apiGroups:
100+
- ""
101+
resources:
102+
- nodes
103+
verbs:
104+
- get
105+
- list
106+
- watch
107+
- update
108+
- apiGroups:
109+
- ""
110+
resources:
111+
- configmaps
112+
verbs:
113+
- create
114+
- get
115+
- update
116+
- list
117+
- watch
118+
- apiGroups:
119+
- ""
120+
resources:
121+
- events
122+
verbs:
123+
- create
124+
- watch
125+
- apiGroups:
126+
- ""
127+
resources:
128+
- pods
129+
verbs:
130+
- get
131+
- list
132+
- delete
133+
- apiGroups:
134+
- "extensions"
135+
resources:
136+
- daemonsets
137+
verbs:
138+
- get
139+
---
140+
# xref: https://raw.githubusercontent.com/coreos/container-linux-update-operator/master/examples/deploy/rbac/cluster-role-binding.yaml
141+
kind: ClusterRoleBinding
142+
apiVersion: rbac.authorization.k8s.io/v1beta1
143+
metadata:
144+
name: reboot-coordinator
145+
roleRef:
146+
apiGroup: rbac.authorization.k8s.io
147+
kind: ClusterRole
148+
name: reboot-coordinator
149+
subjects:
150+
- kind: ServiceAccount
151+
namespace: reboot-coordinator
152+
name: default
153+

0 commit comments

Comments
 (0)