Unable to upgrade to v1.21.5+k3s1

[AlexLast]

Environmental Info:
K3s Version: v1.21.5+k3s1

Node(s) CPU architecture, OS, and Version:
Linux 5.4.0-1044-raspi #48-Ubuntu SMP PREEMPT Thu Sep 9 15:24:01 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:
1 master node, 1 node

Describe the bug:
Unable to upgrade from v1.21.4+k3s1 to v1.21.5+k3s1, k3s fails to start. Get the following output when trying to upgrade the master node:

$ curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.21.5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.21.5+k3s1/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.21.5+k3s1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xe" for details.

Output from systemctl status k3s:

Oct 05 19:06:39 rpi1 systemd[1]: Starting Lightweight Kubernetes...
Oct 05 19:06:39 rpi1 sh[2767]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Oct 05 19:06:39 rpi1 sh[2768]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Oct 05 19:06:39 rpi1 k3s[2779]: time="2021-10-05T19:06:39Z" level=fatal msg="exec format error"
Oct 05 19:06:39 rpi1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 05 19:06:39 rpi1 systemd[1]: k3s.service: Failed with result 'exit-code'.

Steps To Reproduce:

curl -sfL https://get.k3s.io | sh -

Expected behavior:
To be able to upgrade

Actual behavior:
Unable to upgrade

Additional context / logs:
Couldn't find anyway around the issue, had to downgrade back to v1.21.4+k3s1 - which worked as expected.

Backporting

• Needs backporting to older releases

[brandond]

Which model of Rasperry Pi is this on? The only thing I can think of that might have caused this is the bump of golang from 1.16.6 to 1.16.8, which mirrors upstream's change.

[AlexLast]

Hi @brandond, it's a Raspberry Pi 4 Model B Rev 1.2

[brandond]

Hmm. Our QA team tests each release on Raspberry Pi before it goes out so I'm a bit stumped. Can you reproduce this on demand?

[AlexLast]

Can you reproduce this on demand?

Yes, I downgraded and attempted the upgrade again to ensure it wasn't some transient issue and still hit the same

[dweomer]

Oct 05 19:06:39 rpi1 sh[2767]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Oct 05 19:06:39 rpi1 sh[2768]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
This is an issue with the unit file that the install script sets up. Not sure where that nm-cloud-setup stuff came from.

[dweomer]

Oct 05 19:06:39 rpi1 sh[2767]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Oct 05 19:06:39 rpi1 sh[2768]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
This is an issue with the unit file that the install script sets up. Not sure where that nm-cloud-setup stuff came from.

gah, missed the exec format error. what's weird is this runs for me on k3os for both arm and arm64

[brandond]

I wonder if it has anything to do with the kernel ABI change in the most recent update from Ubuntu
https://ubuntu.com/security/notices/USN-5091-2

[dweomer]

This version works for me with k3OS overlays on top of Alpine 3.14.2 on both arm and arm64. I will try a raspbian install later today or tomorrow (but raspbian overlays did also work as of a couple of weeks ago).

[AlexLast]

Thanks @dweomer, just to confirm I'm running Ubuntu 20.04.3 LTS on these Raspberry Pi's

[mortenlj]

I have the same issue on an amd64 machine running in Oracle Cloud, with same Ubuntu version. Ubuntu is installed in the minimal variation, and doesn't have network-manager installed at all (which I presume is the source of the nm-cloud-setup service).

[brandond]

The nm-cloud-setup bit is a red herring. The check in the systemd unit ensures that k3s won't start if it IS running. If it is not present or is disabled, startup will continue as normal. Look further along in your logs for whatever's going wrong.

[mortenlj]

The nm-cloud-setup bit is a red herring. The check in the systemd unit ensures that k3s won't start if it IS running. If it is not present or is disabled, startup will continue as normal. Look further along in your logs for whatever's going wrong.

Thanks. Looking further in the logs a line that was less obviously a problem turned out to be the actual problem 🤦.

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.