Skip to content

Update kubadm ha installation for 1.12 #10264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 20, 2018
Merged

Update kubadm ha installation for 1.12 #10264

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1496d5f
Update kubadm ha installation for 1.12
chuckha Sep 10, 2018
c3ca07a
update stable version
chuckha Sep 17, 2018
b103c02
Update stacked control plane for v1.12 (#2)
detiber Sep 18, 2018
73fce39
use v1alpha3
chuckha Sep 19, 2018
c7af8a9
more v1alpha3 (#4)
detiber Sep 19, 2018
ded081e
updates
chuckha Sep 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 72 additions & 61 deletions content/en/docs/setup/independent/high-availability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ and control plane nodes are co-located.
- With an external etcd cluster. This approach requires more infrastructure. The
control plane nodes and etcd members are separated.

Your clusters must run Kubernetes version 1.11 or later. You should also be aware that
Your clusters must run Kubernetes version 1.12 or later. You should also be aware that
setting up HA clusters with kubeadm is still experimental. You might encounter issues
with upgrading your clusters, for example. We encourage you to try either approach,
and provide feedback.
Expand Down Expand Up @@ -139,15 +139,18 @@ different configuration.

### Bootstrap the first stacked control plane node

{{< note >}}
**Note**: Optionally replace `stable` with a different version of kubernetes such as `v1.12.0`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/kubernetes such as/Kubernetes, for example

{{< /note >}}

1. Create a `kubeadm-config.yaml` template file:

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.x
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this alpha3?

Also I'd just put the v1.12.0 in there explicitly and remove the meta comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, this is still WIP. rc1 is out so i'll finish this off today.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please see above discussions regarding exact version vs stable & meta comment. There will have to be a comment regardless, but this is the easier path imo.

apiServerCertSANs:
- "LOAD_BALANCER_DNS"
api:
controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
etcd:
local:
extraArgs:
Expand All @@ -166,9 +169,6 @@ different configuration.
# This CIDR is a Calico default. Substitute or remove for your CNI provider.
podSubnet: "192.168.0.0/16"

1. Replace `x` in `kubernetesVersion: v1.11.x` with the latest available version.
For example: `kubernetesVersion: v1.11.1`

1. Replace the following variables in the template with the appropriate
values for your cluster:

Expand Down Expand Up @@ -224,13 +224,12 @@ done

1. Create a second, different `kubeadm-config.yaml` template file:

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.x
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- "LOAD_BALANCER_DNS"
api:
controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
etcd:
local:
extraArgs:
Expand All @@ -250,9 +249,6 @@ done
# This CIDR is a calico default. Substitute or remove for your CNI provider.
podSubnet: "192.168.0.0/16"

1. Replace `x` in `kubernetesVersion: v1.11.x` with the latest available version.
For example: `kubernetesVersion: v1.11.1`

1. Replace the following variables in the template with the appropriate values for your cluster:

- `LOAD_BALANCER_DNS`
Expand Down Expand Up @@ -296,7 +292,7 @@ done
export CP1_IP=10.0.0.8
export CP1_HOSTNAME=cp1

export KUBECONFIG=/etc/kubernetes/admin.conf
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP1_HOSTNAME} https://${CP1_IP}:2380
kubeadm alpha phase etcd local --config kubeadm-config.yaml
```
Expand All @@ -317,13 +313,12 @@ done

1. Create a third, different `kubeadm-config.yaml` template file:

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.x
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- "LOAD_BALANCER_DNS"
api:
controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
etcd:
local:
extraArgs:
Expand All @@ -343,9 +338,6 @@ done
# This CIDR is a calico default. Substitute or remove for your CNI provider.
podSubnet: "192.168.0.0/16"

1. Replace `x` in `kubernetesVersion: v1.11.x` with the latest available version.
For example: `kubernetesVersion: v1.11.1`

1. Replace the following variables in the template with the appropriate values for your cluster:

- `LOAD_BALANCER_DNS`
Expand Down Expand Up @@ -391,7 +383,7 @@ done
export CP2_IP=10.0.0.9
export CP2_HOSTNAME=cp2

export KUBECONFIG=/etc/kubernetes/admin.conf
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP2_HOSTNAME} https://${CP2_IP}:2380
kubeadm alpha phase etcd local --config kubeadm-config.yaml
```
Expand All @@ -411,39 +403,48 @@ done
- Follow [these instructions](/docs/setup/independent/setup-ha-etcd-with-kubeadm/)
to set up the etcd cluster.

### Copy required files to other control plane nodes

The following certificates were created when you created the cluster. Copy them
to your other control plane nodes:

- `/etc/kubernetes/pki/etcd/ca.crt`
- `/etc/kubernetes/pki/apiserver-etcd-client.crt`
- `/etc/kubernetes/pki/apiserver-etcd-client.key`
#### Copy required files from an etcd node to all control plane nodes

In the following example, replace `USER` and `CONTROL_PLANE_HOSTS` values with values
for your environment.

```sh
# Make a list of required etcd certificate files
cat << EOF > etcd-pki-files.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw I'll submit a PR later, but we could bash-fu several of these commands together.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is intentionally made simple to understand. I value readability to more over fewer complex commands in public documentation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but if your bash-fu is more readable then by all means, submit away

/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-etcd-client.key
EOF

# create the archive
tar -czf etcd-pki.tar.gz -T etcd-pki-files.txt

# copy the archive to the control plane nodes
USER=ubuntu
CONTROL_PLANE_HOSTS="10.0.0.7 10.0.0.8 10.0.0.9"
for host in $CONTROL_PLANE_HOSTS; do
scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:
scp /etc/kubernetes/pki/apiserver-etcd-client.crt "${USER}"@$host:
scp /etc/kubernetes/pki/apiserver-etcd-client.key "${USER}"@$host:
scp etcd-pki.tar.gz "${USER}"@$host:
done
```

### Set up the first control plane node

1. Create a `kubeadm-config.yaml` template file:
1. Extract the etcd certificates

tar -xzf etcd-pki.tar.gz -C /etc/kubernetes/pki --strip-components=3

1. Create a `kubeadm-config.yaml`:

{{< note >}}
**Note**: Optionally replace `stable` with a different version of kubernetes such as `v1.11.3`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/kubernetes such as/Kubernetes, for example

{{< /note >}}

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.x
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- "LOAD_BALANCER_DNS"
api:
controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
etcd:
external:
endpoints:
Expand All @@ -457,9 +458,6 @@ done
# This CIDR is a calico default. Substitute or remove for your CNI provider.
podSubnet: "192.168.0.0/16"

1. Replace `x` in `kubernetesVersion: v1.11.x` with the latest available version.
For example: `kubernetesVersion: v1.11.1`

1. Replace the following variables in the template with the appropriate values for your cluster:

- `LOAD_BALANCER_DNS`
Expand All @@ -469,11 +467,12 @@ done
- `ETCD_2_IP`

1. Run `kubeadm init --config kubeadm-config.yaml`
1. Copy the output join commamnd.

### Copy required files to the correct locations

The following certificates and other required files were created when you ran `kubeadm init`.
Copy these files to your other control plane nodes:
The following pki files were created during the `kubeadm init` step and must be shared with
all other control plane nodes.

- `/etc/kubernetes/pki/ca.crt`
- `/etc/kubernetes/pki/ca.key`
Expand All @@ -486,24 +485,34 @@ In the following example, replace the list of
`CONTROL_PLANE_IPS` values with the IP addresses of the other control plane nodes.

```sh
# make a list of required kubernetes certificate files
cat << EOF > certificate_files.txt
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
EOF

# create the archive
tar -czf control-plane-certificates.tar.gz -T certificate_files.txt

USER=ubuntu # customizable
CONTROL_PLANE_IPS="10.0.0.7 10.0.0.8"
for host in ${CONTROL_PLANE_IPS}; do
scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:
scp /etc/kubernetes/pki/ca.key "${USER}"@$host:
scp /etc/kubernetes/pki/sa.key "${USER}"@$host:
scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:
scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:
scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:
scp control-plane-certificates.tar.gz "${USER}"@$host:
done
```

{{< note >}}
**Note**: Remember that your config may differ from this example.
{{< /note >}}

### Set up the other control plane nodes

1. Extract the required certificates

mkdir -p /etc/kubernetes/pki
tar -xzf etcd-pki.tar.gz -C /etc/kubernetes/pki --strip-components 3
tar -xzf control-plane-certificates.tar.gz -C /etc/kubernetes/pki --strip-components 3

1. Verify the location of the copied files.
Your `/etc/kubernetes` directory should look like this:

Expand All @@ -517,8 +526,10 @@ done
- `/etc/kubernetes/pki/sa.pub`
- `/etc/kubernetes/pki/etcd/ca.crt`

1. Run `kubeadm init --config kubeadm-config.yaml` on each control plane node, where
`kubeadm-config.yaml` is the file you already created.
1. Run the copied `kubeadm join` command from above. Add the flag "--experimental-control-plane".
The final command will look something like this:

kubeadm join ha.k8s.example.com:6443 --token 5ynki1.3erp9i3yo7gqg1nv --discovery-token-ca-cert-hash sha256:a00055bd8c710a9906a3d91b87ea02976334e1247936ac061d867a0f014ecd81 --experimental-control-plane

## Common tasks after bootstrapping control plane

Expand Down