Bootstrap Data Migration Error

[kralicky]

Environmental Info:
K3s Version:

k3s version v1.22.2+k3s-6b4d75d2 (6b4d75d)
go version go1.16.8

Node(s) CPU architecture, OS, and Version:

Linux opni-single-node-hpc 5.4.0-1047-kvm #49-Ubuntu SMP Mon Sep 20 21:27:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

Single node, default storage configuration

Describe the bug:

After upgrading k3s from commit 12e675e2cc3431d25630f63535f66ad4bb22ae85 to 6b4d75d2451ef26e1061e81f186f25f26b5a7e94, startup failed with the following error messages:

Oct 14 17:44:31 opni-single-node-hpc systemd[1]: Starting Lightweight Kubernetes...
Oct 14 17:44:31 opni-single-node-hpc sh[64905]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Oct 14 17:44:31 opni-single-node-hpc sh[64906]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=debug msg="Asset dir /var/lib/rancher/k3s/data/157e7133c019a632df3b8ec2becdfe8c22ce2a3ca3d89ba9de425b84c5914d8d"
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=debug msg="Running /var/lib/rancher/k3s/data/157e7133c019a632df3b8ec2becdfe8c22ce2a3ca3d89ba9de425b84c5914d8d/bin/k3s-server [/usr/local/bin/k3s server --debug]"
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=info msg="Starting k3s v1.22.2+k3s-6b4d75d2 (6b4d75d2)"
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=info msg="Database tables and indexes are up to date"
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=info msg="Kine available at unix://kine.sock"
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=info msg="Reconciling bootstrap data between datastore and disk"
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=info msg="Migrating bootstrap data to new format"
Oct 14 17:44:31 opni-single-node-hpc k3s[64909]: time="2021-10-14T17:44:31Z" level=fatal msg="starting kubernetes: preparing server: EOF"
Oct 14 17:44:31 opni-single-node-hpc systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 17:44:31 opni-single-node-hpc systemd[1]: k3s.service: Failed with result 'exit-code'.
Oct 14 17:44:31 opni-single-node-hpc systemd[1]: Failed to start Lightweight Kubernetes.

Steps To Reproduce:

• Installed K3s:
  $ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=12e675e2cc3431d25630f63535f66ad4bb22ae85 sh -
• Upgraded K3s:
  $ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=6b4d75d2451ef26e1061e81f186f25f26b5a7e94 sh -

Expected behavior:

K3s upgrades to the new version successfully

Actual behavior:

K3s fails to start with error messages.

Additional context / logs:

• Server args used: '--write-kubeconfig-mode=644' '--tls-san=<server's public ip>' '--kube-scheduler-arg=v=2' '--log=/var/log/k3s.log' '--alsologtostderr'
• Reverting to the previous version allows k3s to start normally again.
• Starting with --debug does not yield any additional information

Backporting

• Needs backporting to older releases

[rancher-max]

I have validated bootstrap functionality works on master-branch using commitid 8271d98a766b060463bc73ef66c5085b5797b4cc, however I believe this issue may be more related to the log file argument. We have other issues to resolve that, so I am going to close this one out.