Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubeadm - Ha upgrade updates #10340

Merged
merged 4 commits into from
Sep 24, 2018
Merged

kubeadm - Ha upgrade updates #10340

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
56bb16e
Update HA upgrade docs
detiber Sep 19, 2018
31dd3d5
Adds external etcd HA upgrade guide
chuckha Sep 21, 2018
8f96f42
copyedit
Sep 24, 2018
5902241
more edits
Sep 24, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 81 additions & 63 deletions content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,45 +10,45 @@ content_template: templates/task

{{% capture overview %}}

This guide is for upgrading `kubeadm` HA clusters from version 1.11 to 1.12. The term "`kubeadm` HA clusters" refers to clusters of more than one control plane node created with `kubeadm`. To set up an HA cluster for Kubernetes version 1.11 `kubeadm` requires additional manual steps. See [Creating HA clusters with kubeadm](/docs/setup/independent/high-availability/) for instructions on how to do this. The upgrade procedure described here targets clusters created following those very instructions.
This page explains how to upgrade a highly available (HA) Kubernetes cluster created with `kubeadm` from version 1.11.x to version 1.12.x. In addition to upgrading, you must also follow the instructions in [Creating HA clusters with kubeadm](/docs/setup/independent/high-availability/).

{{% /capture %}}

{{% capture prerequisites %}}

Before proceeding:

- You need to have a functional `kubeadm` HA cluster running version 1.11 or higher in order to use the process described here.
- You need to have a `kubeadm` HA cluster running version 1.11 or higher.
- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md) carefully.
- Note that `kubeadm upgrade` will not touch any of your workloads, only Kubernetes-internal components. As a best-practice you should back up anything important to you. For example, any application-level state, such as a database and application might depend on (like MySQL or MongoDB) should be backed up beforehand.
- Read [Upgrading/downgrading kubeadm clusters between v1.11 to v1.12](/docs/tasks/administer-cluster/kubeadm-upgrade-1-12/) to learn about the relevant prerequisites.
- Make sure to back up any important components, such as app-level state stored in a database. `kubeadm upgrade` does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice.
- Check the prerequisites for [Upgrading/downgrading kubeadm clusters between v1.11 to v1.12](/docs/tasks/administer-cluster/kubeadm-upgrade-1-12/).

{{< note >}}
**Note**: All commands on any control plane or etcd node should be
run as root.
{{< /note >}}

{{% /capture %}}

{{% capture steps %}}

## Preparation for both methods

{{< note >}}
**Note**: All commands in this guide on any control plane or etcd node should be
run as root.
{{< /note >}}
## Prepare for both methods

Some preparation is needed prior to starting the upgrade. First upgrade `kubeadm` to the version that matches the version of Kubernetes that you are upgrading to:
Upgrade `kubeadm` to the version that matches the version of Kubernetes that you are upgrading to:

```shell
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm && \
apt-mark hold kubeadm
```

Run this command for checking prerequisites and determining the versions you will receive:
Check prerequisites and determine the upgrade versions:

```shell
kubeadm upgrade plan
```

If the prerequisites are met you'll get a summary of the software versions kubeadm will upgrade to, like this:
You should see something like the following:

Upgrade to the latest stable version:

Expand All @@ -62,91 +62,109 @@ If the prerequisites are met you'll get a summary of the software versions kubea

## Stacked control plane nodes

### Upgrading the first control plane node

The following procedure must be applied on a single control plane node.
### Upgrade the first control plane node

Before initiating the upgrade with `kubeadm` `configmap/kubeadm-config` needs to be modified for the current control plane node.
Modify `configmap/kubeadm-config` for this control plane node:

```shell
kubectl get configmap -n kube-system kubeadm-config -o yaml >/tmp/kubeadm-config-cm.yaml
```

Open the file in an editor and replace the following values:

- api.advertiseAddress
- This should be set to the local node's IP address
- etcd.local.extraArgs.advertise-client-urls
- This should be updated for the local node's IP address
- etcd.local.extraArgs.initial-advertise-peer-urls
- This should be updated for the local node's IP address
- etcd.local.extraArgs.listen-client-urls
- This should be updated for the local node's IP address
- etcd.local.extraArgs.listen-peer-urls
- This should be updated for the local node's IP address
- etcd.local.extraArgs.initial-cluster
- This should be updated to include the hostname and IP address pairs for each control plane node in the cluster, for example:
- `api.advertiseAddress`

This should be set to the local node's IP address.

- `etcd.local.extraArgs.advertise-client-urls`

This should be updated to the local node's IP address.

- `etcd.local.extraArgs.initial-advertise-peer-urls`

This should be updated to the local node's IP address.

- `etcd.local.extraArgs.listen-client-urls`

This should be updated to the local node's IP address.

- `etcd.local.extraArgs.listen-peer-urls`

This should be updated to the local node's IP address.

- `etcd.local.extraArgs.initial-cluster`

This should be updated to include the hostname and IP address pairs for each control plane node in the cluster. For example:

"ip-172-31-92-42=https://172.31.92.42:2380,ip-172-31-89-186=https://172.31.89.186:2380,ip-172-31-90-42=https://172.31.90.42:2380"

An additional argument (`initial-cluster-state: existing`) also needs to be added to etcd.local.extraArgs.
You must also pass an additional argument (`initial-cluster-state: existing`) to etcd.local.extraArgs.

```shell
kubectl apply -f /tmp/kubeadm-config-cm.yaml --force
```

Now the upgrade process can start. Use the target version determined in the preparation step and run the following command (press “y” when prompted):
Start the upgrade:

```shell
kubeadm upgrade apply v<YOUR-CHOSEN-VERSION-HERE>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would 1.12-stable work here?

```

If the operation was successful you’ll get a message like this:
You should see something like the following:

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.0". Enjoy!

### Upgrading subsequent control plane nodes
The `kubeadm-config` ConfigMap is now updated from `v1alpha2` version to `v1alpha3`.

### Upgrading additional control plane nodes

After upgrading the first control plane node, the `kubeadm-config` config map will be updated from `v1alpha2` version to `v1alpha3`, which requires different modifications than were needed for the first control plane node.
Each additional control plane node requires modifications that are different from the first control plane node. Run:

```shell
kubectl get configmap -n kube-system kubeadm-config -o yaml >/tmp/kubeadm-config-cm.yaml
```

Open the file in an editor and replace the following values under ClusterConfiguration:
Open the file in an editor and replace the following values for `ClusterConfiguration`:

- `etcd.local.extraArgs.advertise-client-urls`

This should be updated to the local node's IP address.

- `etcd.local.extraArgs.initial-advertise-peer-urls`

This should be updated to the local node's IP address.

- `etcd.local.extraArgs.listen-client-urls`

This should be updated to the local node's IP address.

- `etcd.local.extraArgs.listen-peer-urls`

- etcd.local.extraArgs.advertise-client-urls
- This should be updated for the local node's IP address
- etcd.local.extraArgs.initial-advertise-peer-urls
- This should be updated for the local node's IP address
- etcd.local.extraArgs.listen-client-urls
- This should be updated for the local node's IP address
- etcd.local.extraArgs.listen-peer-urls
- This should be updated for the local node's IP address
This should be updated to the local node's IP address.

Modify the ClusterStatus to add an additional mapping for the current host under apiEndpoints.
You must also modify the `ClusterStatus` to add a mapping for the current host under apiEndpoints.

Add an annotation for the cri-socket to the current node, for example to use docker:

```shell
kubectl annotate node <hostname> kubeadm.alpha.kubernetes.io/cri-socket=/var/run/dockershim.sock
kubectl annotate node <nodename> kubeadm.alpha.kubernetes.io/cri-socket=/var/run/dockershim.sock
```

Now the upgrade process can start. Use the target version determined in the preparation step and run the following command (press “y” when prompted):
Start the upgrade:

```shell
kubeadm upgrade apply v<YOUR-CHOSEN-VERSION-HERE>
```

If the operation was successful you’ll get a message like this:
You should see something like the following:

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.0". Enjoy!

## External etcd

### Upgrade each control plane

Get a copy of the kubeadm config used to create this cluster. The config should be the same for every node. The config must exist on every control plane node before the upgrade begins!
Get a copy of the kubeadm config used to create this cluster. The config should be the same for every node. The config must exist on every control plane node before the upgrade begins.

```
# on each control plane node
Expand All @@ -161,27 +179,27 @@ kubeadm upgrade apply v1.12.0 --config /tmp/kubeadm-config.yaml

### Upgrade etcd

Kubernetes v1.11 to v1.12 only changed the patch version of etcd from v3.2.18 to v3.2.24. The upgrade will be a rolling upgrade with no downtime since it is ok to run both of these versions in the same cluster.
Kubernetes v1.11 to v1.12 only changed the patch version of etcd from v3.2.18 to v3.2.24. This is a rolling upgrade with no downtime, because you can run both versions in the same cluster.

On the first host, as root, modify the etcd manifest with this command:
On the first host, modify the etcd manifest:

```shell
sed -i 's/3.2.18/3.2.24/' /etc/kubernetes/manifests/etcd.yaml
```

Wait for the etcd process to reconnect. There will be error warnings in the other member's logs. This is expected.
Wait for the etcd process to reconnect. There will be error warnings in the other etcd node logs. This is expected.

Repeat the process on the other etcd hosts, replacing the version and waiting for the process to come back.
Repeat this step on the other etcd hosts.

## Post control plane upgrade steps
## Next steps

### Manually upgrade your CNI provider

Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow. Check the [addons](/docs/concepts/cluster-administration/addons/) page to find your CNI provider and see if there are additional upgrade steps necessary.
Your Container Network Interface (CNI) provider might have its own upgrade instructions to follow. Check the [addons](/docs/concepts/cluster-administration/addons/) page to find your CNI provider and see whether you need to tak additional upgrade steps.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/tak/take/


### Update kubelet and kubectl packages

At this point all the static pod manifests in your cluster, for example API Server, Controller Manager, Scheduler, Kube Proxy have been upgraded, however the base software, for example `kubelet`, `kubectl` installed on your nodes’ OS are still of the old version. For upgrading the base software packages we will upgrade them and restart services on all nodes one by one:
Upgrade the kubelet and kubectl by running the following on each node:

```shell
# use your distro's package manager, e.g. 'apt-get' on Debian-based systems
Expand All @@ -193,30 +211,30 @@ apt-mark hold kubelet kubectl && \
systemctl restart kubelet
```

In this example a _deb_-based system is assumed and `apt-get` is used for installing the upgraded software. On _rpm_-based systems it will be `yum install <PACKAGE>=<NEW-K8S-VERSION>` for all packages.
In this example a _deb_-based system is assumed and `apt-get` is used for installing the upgraded software. On rpm-based systems the command is `yum install <PACKAGE>=<NEW-K8S-VERSION>` for all packages.

Now the new version of the `kubelet` should be running on the host. Verify this using the following command on the respective host:
Verify that the new version of the kubelet is running:

```shell
systemctl status kubelet
```

Verify that the upgraded node is available again by executing the following from wherever you run `kubectl` commands:
Verify that the upgraded node is available again by running the following command from wherever you run `kubectl`:

```shell
kubectl get nodes
```

If the `STATUS` column of the above command shows `Ready` for the upgraded host, you can continue (you may have to repeat this for a couple of time before the node gets `Ready`).
If the `STATUS` column shows `Ready` for the upgraded host, you can continue. You might need to repeat the command until the node shows `Ready`.

## If something goes wrong

If the upgrade fails the situation afterwards depends on the phase in which things went wrong:
If the upgrade fails, see whether one of the following scenarios applies:

1. If `/tmp/kubeadm upgrade apply` failed to upgrade the cluster it will try to perform a rollback. Hence if that happened on the first master, chances are pretty good that the cluster is still intact.
- If `/tmp/kubeadm upgrade apply` failed to upgrade the cluster, it will try to perform a rollback. If this is the case on the first master, the cluster is probably still intact.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to replace any /tmp/kubeadm references with just kubeadm, since we are upgrading kubeadm using the package manager again.


You can run `/tmp/kubeadm upgrade apply` again as it is idempotent and should eventually make sure the actual state is the desired state you are declaring. You can use `/tmp/kubeadm upgrade apply` to change a running cluster with `x.x.x --> x.x.x` with `--force`, which can be used to recover from a bad state.
You can run `/tmp/kubeadm upgrade apply` again, because it is idempotent and should eventually make sure the actual state is the desired state you are declaring. You can run `/tmp/kubeadm upgrade apply` to change a running cluster with `x.x.x --> x.x.x` with `--force` to recover from a bad state.

2. If `/tmp/kubeadm upgrade apply` on one of the secondary masters failed you still have a working, upgraded cluster, but with the secondary masters in a somewhat undefined condition. You will have to find out what went wrong and join the secondaries manually.
- If `/tmp/kubeadm upgrade apply` on one of the secondary masters failed, the cluster is upgraded and working, but the secondary masters are in an undefined state. You need to investigate further and join the secondaries manually.

{{% /capture %}}