v0.17.5 can't connect to a single host config

[nwneisen]

Summary

v0.17.5 can no longer connect to a single host node. v0.17.4 works fine.

Details

I just installed v0.17.5 and created a new config using k0sctl init and modified the hosts section to be a single host that pointed to a VM on my local machine. The apply command kept hanging up when I tried to apply the config.

nick:~/code/homelab: k0sctl apply -c homelab.yaml -d
DEBU[0000] Loaded configuration:
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: homelab-cluster
spec:
  hosts:
  - ssh:
      address: 10.252.198.229
      user: ubuntu
      port: 22
      keyPath: id_rsa
    role: single
  k0s:
    version: v1.28.7+k0s.0
    versionChannel: stable
    dynamicConfig: false
    config: {} 

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.17.5 Copyright 2023, k0sctl authors.
Anonymized telemetry of usage will be sent to the authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
DEBU[0000] Preparing phase 'Set k0s version'            
DEBU[0000] Preparing phase 'Connect to hosts'           
INFO[0000] ==> Running phase: Connect to hosts 
DEBU[0000] [ssh] 10.252.198.229:22: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0000] using SSH_AUTH_SOCK=/run/user/1000/keyring/ssh 
DEBU[0000] [ssh] 10.252.198.229:22: using all keys (2) from ssh agent because a keypath was not explicitly given 
DEBU[0000] [ssh] 10.252.198.229:22: using an unencrypted private key from id_rsa 
DEBU[0000] [SSH] 10.252.198.229: failed to connect: ssh dial: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 
DEBU[0005] retrying, attempt 1 - last error: not connected: client connect: ssh dial: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 
DEBU[0005] [ssh] 10.252.198.229:22: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0005] using SSH_AUTH_SOCK=/run/user/1000/keyring/ssh 
DEBU[0005] [ssh] 10.252.198.229:22: using all keys (2) from ssh agent because a keypath was not explicitly given 
DEBU[0005] [SSH] 10.252.198.229: failed to connect: ssh dial: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 

I reverted my k0sctl version to v0.17.4 and it worked fine.

nick:~/code/homelab: k0sctl apply -c homelab.yaml 

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.17.4 Copyright 2023, k0sctl authors.
Anonymized telemetry of usage will be sent to the authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
INFO ==> Running phase: Connect to hosts 
INFO [ssh] 10.252.198.229:22: connected           
INFO ==> Running phase: Detect host operating systems 
INFO [ssh] 10.252.198.229:22: is running Ubuntu 22.04.4 LTS 
INFO ==> Running phase: Acquire exclusive host lock 
INFO ==> Running phase: Prepare hosts    
INFO ==> Running phase: Gather host facts 
INFO [ssh] 10.252.198.229:22: using homelab as hostname 
INFO [ssh] 10.252.198.229:22: discovered ens3 as private interface 
INFO ==> Running phase: Validate hosts   
INFO ==> Running phase: Gather k0s facts 
INFO [ssh] 10.252.198.229:22: found existing configuration 
INFO [ssh] 10.252.198.229:22: is running k0s single version v1.28.7+k0s.0 
INFO ==> Running phase: Validate facts   
INFO [ssh] 10.252.198.229:22: validating configuration 
INFO ==> Running phase: Release exclusive host lock 
INFO ==> Running phase: Disconnect from hosts 
INFO ==> Finished in 2s                  
INFO k0s cluster version v1.28.7+k0s.0 is now installed 
INFO Tip: To access the cluster you can now fetch the admin kubeconfig using: 
INFO      k0sctl kubeconfig              
A new version v0.17.5 of k0sctl is available: https://github.com/k0sproject/k0sctl/releases/tag/v0.17.5

Environment

Host: Pop OS
VM manager: multipass
VM distro: Ubuntu 22.04

[twz123]

Just to make sure, can you try to invoke k0sctl v0.17.5 via env -u SSH_AUTH_SOCK k0sctl ... and see if it makes a difference?

[kke]

Probably has something to do with k0sproject/rig#162

[nwneisen]

Just to make sure, can you try to invoke k0sctl v0.17.5 via env -u SSH_AUTH_SOCK k0sctl ... and see if it makes a difference?

This is also with things switched over to a lima VM that works with v0.17.4

nick@amd-t14:~/Downloads: env -u SSH_AUTH_SOCK ./k0sctl-linux-x64 apply -c ~/code/homelab/
charts/       .git/         homelab.yaml  values.yaml   
Chart.yaml    .helmignore   Makefile      
nick@amd-t14:~/Downloads: env -u SSH_AUTH_SOCK ./k0sctl-linux-x64 apply -c ~/code/homelab/homelab.yaml --debug
DEBU[0000] Loaded configuration:
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: homelab-cluster
spec:
  hosts:
  # - ssh:
  #     address: 192.168.1.60
  #     user: nick
  #     port: 22
  #     keyPath: /home/nick/.ssh/id_rsa
  #   role: single
  - ssh:
      address: 127.0.0.1
      user: nick
      port: 60022
      keyPath: /home/nick/.lima/_config/user
    role: single
  k0s:
    version: v1.28.7+k0s.0
    versionChannel: stable
    dynamicConfig: false
    config: {} 

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.17.4 Copyright 2023, k0sctl authors.
Anonymized telemetry of usage will be sent to the authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
DEBU[0000] Preparing phase 'Set k0s version'            
DEBU[0000] Preparing phase 'Connect to hosts'           
INFO[0000] ==> Running phase: Connect to hosts 
DEBU[0000] [ssh] 127.0.0.1:60022: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0000] [ssh] 127.0.0.1:60022: using an unencrypted private key from /home/nick/.lima/_config/user 
DEBU[0000] [SSH] 127.0.0.1: failed to connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0005] retrying, attempt 1 - last error: not connected: client connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0005] [ssh] 127.0.0.1:60022: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0005] [SSH] 127.0.0.1: failed to connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0010] retrying, attempt 2 - last error: not connected: client connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0010] [ssh] 127.0.0.1:60022: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0010] [SSH] 127.0.0.1: failed to connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0015] retrying, attempt 3 - last error: not connected: client connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0015] [ssh] 127.0.0.1:60022: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0015] [SSH] 127.0.0.1: failed to connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 

[twz123]

Your last log snippets show network errors ("connection refused"), not SSH handshake failures ("ssh: handshake failed: ssh: unable to authenticate"), btw.

[jadolg]

What is the disposition of your keys? I see from the logs there are 2 keys loaded in the ssh agent and one key got loaded from the default configuration:

DEBU[0000] [ssh] 10.252.198.229:22: using all keys (2) from ssh agent because a keypath was not explicitly given 
DEBU[0000] [ssh] 10.252.198.229:22: using an unencrypted private key from id_rsa 

Is this correct?
If so, which of these should have access to the machine?
I want to try to reproduce it on a test.

[nwneisen]

Just to make sure, can you try to invoke k0sctl v0.17.5 via env -u SSH_AUTH_SOCK k0sctl ... and see if it makes a difference?

Running with env -u SSH_AUTH_SOCK is successful

nick@amd-t14:~/code/homelab (master): env -u SSH_AUTH_SOCK ~/Downloads/k0sctl-linux-x64 apply -c ~/code/homelab/homelab.yaml  -d
DEBU[0000] Loaded configuration:
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: homelab-cluster
spec:
  hosts:
  # - ssh:
  #     address: 192.168.1.2
  #     user: root
  #     port: 22
  #     keyPath: ~/.ssh/id_ed25519
  #   role: single
  - ssh:
      address: 10.252.198.12
      user: ubuntu
      port: 22
      keyPath: ~/id_rsa
    role: single
  # - ssh:
  #     address: 127.0.0.1
  #     user: nick
  #     port: 60022
  #     keyPath: /home/nick/.ssh/id_ed25519
  #   role: single
  k0s:
    version: v1.28.7+k0s.0
    versionChannel: stable
    dynamicConfig: false
    config: {} 

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.17.5 Copyright 2023, k0sctl authors.
Anonymized telemetry of usage will be sent to the authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
DEBU[0000] Preparing phase 'Set k0s version'            
DEBU[0000] Preparing phase 'Connect to hosts'           
INFO[0000] ==> Running phase: Connect to hosts 
DEBU[0000] [ssh] 10.252.198.12:22: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0000] [ssh] 10.252.198.12:22: using an unencrypted private key from /home/nick/id_rsa 
DEBU[0000] [ssh] 10.252.198.12:22: executing `uname | grep -q Linux` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `cat /etc/os-release || cat /usr/lib/os-release` 
DEBU[0000] [ssh] 10.252.198.12:22: PRETTY_NAME="Ubuntu 22.04.4 LTS" 
DEBU[0000] [ssh] 10.252.198.12:22: NAME="Ubuntu"        
DEBU[0000] [ssh] 10.252.198.12:22: VERSION_ID="22.04"   
DEBU[0000] [ssh] 10.252.198.12:22: VERSION="22.04.4 LTS (Jammy Jellyfish)" 
DEBU[0000] [ssh] 10.252.198.12:22: VERSION_CODENAME=jammy 
DEBU[0000] [ssh] 10.252.198.12:22: ID=ubuntu            
DEBU[0000] [ssh] 10.252.198.12:22: ID_LIKE=debian       
DEBU[0000] [ssh] 10.252.198.12:22: HOME_URL="https://www.ubuntu.com/" 
DEBU[0000] [ssh] 10.252.198.12:22: SUPPORT_URL="https://help.ubuntu.com/" 
DEBU[0000] [ssh] 10.252.198.12:22: BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 
DEBU[0000] [ssh] 10.252.198.12:22: PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 
DEBU[0000] [ssh] 10.252.198.12:22: UBUNTU_CODENAME=jammy 
DEBU[0000] [ssh] 10.252.198.12:22: executing `[ "$(id -u)" = 0 ]` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -n true` 
INFO[0000] [ssh] 10.252.198.12:22: connected            
DEBU[0000] Preparing phase 'Detect host operating systems' 
INFO[0000] ==> Running phase: Detect host operating systems 
INFO[0000] [ssh] 10.252.198.12:22: is running Ubuntu 22.04.4 LTS 
DEBU[0000] Preparing phase 'Acquire exclusive host lock' 
INFO[0000] ==> Running phase: Acquire exclusive host lock 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- test -d /run/lock` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `mktemp`   
DEBU[0000] [ssh] 10.252.198.12:22: /tmp/tmp.xpThjmAuvI  
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- cat > "/tmp/tmp.xpThjmAuvI"` 
DEBU[0000] [ssh] 10.252.198.12:22: writing 70 bytes to command stdin: 331f512cab4ba61b7d1883acdcf9606bc8a833b996544906657a073a67b19257-43696 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- mv -n "/tmp/tmp.xpThjmAuvI" "/run/lock/k0sctl"` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `test -f "/tmp/tmp.xpThjmAuvI"` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- rm -f "/tmp/tmp.xpThjmAuvI"` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- test -d /run/lock` 
DEBU[0000] [ssh] 10.252.198.12:22: started periodic update of lock file /run/lock/k0sctl timestamp 
DEBU[0000] Preparing phase 'Prepare hosts'              
INFO[0000] ==> Running phase: Prepare hosts    
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- command -v -- "curl" 2> /dev/null` 
DEBU[0000] [ssh] 10.252.198.12:22: /usr/bin/curl        
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- command -v -- "hostname" 2> /dev/null` 
DEBU[0001] [ssh] 10.252.198.12:22: /usr/bin/hostname    
DEBU[0001] [ssh] 10.252.198.12:22: executing `grep 'container=docker' /proc/1/environ 2> /dev/null` 
DEBU[0001] Preparing phase 'Gather host facts'          
INFO[0001] ==> Running phase: Gather host facts 
DEBU[0001] [ssh] 10.252.198.12:22: executing `uname -m` 
DEBU[0001] [ssh] 10.252.198.12:22: x86_64               
DEBU[0001] [ssh] 10.252.198.12:22: executing `cat /etc/machine-id || cat /var/lib/dbus/machine-id` 
DEBU[0001] [ssh] 10.252.198.12:22: cc23a39e4ffb40329de3dcb699bd92bd 
DEBU[0001] [ssh] 10.252.198.12:22: executing `hostname 2> /dev/null` 
DEBU[0001] [ssh] 10.252.198.12:22: homelab              
INFO[0001] [ssh] 10.252.198.12:22: using homelab as hostname 
DEBU[0001] [ssh] 10.252.198.12:22: executing `PATH=/usr/local/sbin:/usr/sbin:/sbin:$PATH; (ip route list scope global | grep -E "\b(172|10|192\.168)\.") || (ip route list | grep -m1 default)` 
DEBU[0001] [ssh] 10.252.198.12:22: default via 10.252.198.1 dev ens3 proto dhcp src 10.252.198.12 metric 100 
INFO[0001] [ssh] 10.252.198.12:22: discovered ens3 as private interface 
DEBU[0001] [ssh] 10.252.198.12:22: executing `PATH=/usr/local/sbin:/usr/sbin:/sbin:$PATH ip -o addr show dev ens3 scope global` 
DEBU[0001] [ssh] 10.252.198.12:22: 2: ens3    inet 10.252.198.12/24 metric 100 brd 10.252.198.255 scope global ens3\       valid_lft forever preferred_lft forever 
DEBU[0001] Preparing phase 'Validate hosts'             
INFO[0001] ==> Running phase: Validate hosts   
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- true` 
DEBU[0001] Preparing phase 'Gather k0s facts'           
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- command -v -- "k0s" 2> /dev/null` 
DEBU[0001] [ssh] 10.252.198.12:22: /usr/local/bin/k0s   
INFO[0001] ==> Running phase: Gather k0s facts 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- /usr/local/bin/k0s version` 
DEBU[0001] [ssh] 10.252.198.12:22: v1.28.7+k0s.0        
DEBU[0001] [ssh] 10.252.198.12:22: has k0s binary version v1.28.7+k0s.0 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- test -e /etc/k0s/k0s.yaml 2> /dev/null` 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- cat -- /etc/k0s/k0s.yaml 2> /dev/null` 
INFO[0001] [ssh] 10.252.198.12:22: found existing configuration 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- stat /run/systemd/system` 
DEBU[0001] [ssh] 10.252.198.12:22: File: /run/systemd/system 
DEBU[0001] [ssh] 10.252.198.12:22: Size: 80             Blocks: 0          IO Block: 4096   directory 
DEBU[0001] [ssh] 10.252.198.12:22: Device: 19h/25d      Inode: 284         Links: 3 
DEBU[0001] [ssh] 10.252.198.12:22: Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root) 
DEBU[0001] [ssh] 10.252.198.12:22: Access: 2024-04-14 17:48:33.102862615 -0600 
DEBU[0001] [ssh] 10.252.198.12:22: Modify: 2024-04-14 17:48:33.078862666 -0600 
DEBU[0001] [ssh] 10.252.198.12:22: Change: 2024-04-14 17:48:33.078862666 -0600 
DEBU[0001] [ssh] 10.252.198.12:22: Birth: -             
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- systemctl show -p FragmentPath k0scontroller.service 2> /dev/null | cut -d"=" -f2` 
DEBU[0001] [ssh] 10.252.198.12:22: /etc/systemd/system/k0scontroller.service 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- /usr/local/bin/k0s status -o json` 
DEBU[0001] [ssh] 10.252.198.12:22: {                    
DEBU[0001] [ssh] 10.252.198.12:22: "Version": "v1.28.7+k0s.0", 
DEBU[0001] [ssh] 10.252.198.12:22: "Pid": 3836,         
DEBU[0001] [ssh] 10.252.198.12:22: "PPid": 0,           
DEBU[0001] [ssh] 10.252.198.12:22: "Role": "controller", 
DEBU[0001] [ssh] 10.252.198.12:22: "SysInit": "",       
DEBU[0001] [ssh] 10.252.198.12:22: "StubFile": "",      
DEBU[0001] [ssh] 10.252.198.12:22: "Output": "",        
DEBU[0001] [ssh] 10.252.198.12:22: "Workloads": true,   
DEBU[0001] [ssh] 10.252.198.12:22: "SingleNode": true,  
DEBU[0001] [ssh] 10.252.198.12:22: "Args": [            
DEBU[0001] [ssh] 10.252.198.12:22: "/usr/local/bin/k0s", 
DEBU[0001] [ssh] 10.252.198.12:22: "controller",        
DEBU[0001] [ssh] 10.252.198.12:22: "--config=/etc/k0s/k0s.yaml", 
DEBU[0001] [ssh] 10.252.198.12:22: "--data-dir=/var/lib/k0s", 
DEBU[0001] [ssh] 10.252.198.12:22: "--single=true"      
DEBU[0001] [ssh] 10.252.198.12:22: ],                   
DEBU[0001] [ssh] 10.252.198.12:22: "WorkerToAPIConnectionStatus": { 
DEBU[0001] [ssh] 10.252.198.12:22: "Message": "",       
DEBU[0001] [ssh] 10.252.198.12:22: "Success": true      
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "ClusterConfig": {   
DEBU[0001] [ssh] 10.252.198.12:22: "metadata": {        
DEBU[0001] [ssh] 10.252.198.12:22: "name": "k0s",       
DEBU[0001] [ssh] 10.252.198.12:22: "creationTimestamp": null 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "kind": "ClusterConfig", 
DEBU[0001] [ssh] 10.252.198.12:22: "apiVersion": "k0s.k0sproject.io/v1beta1", 
DEBU[0001] [ssh] 10.252.198.12:22: "spec": {            
DEBU[0001] [ssh] 10.252.198.12:22: "api": {             
DEBU[0001] [ssh] 10.252.198.12:22: "address": "10.252.198.12", 
DEBU[0001] [ssh] 10.252.198.12:22: "k0sApiPort": 9443,  
DEBU[0001] [ssh] 10.252.198.12:22: "port": 6443,        
DEBU[0001] [ssh] 10.252.198.12:22: "sans": [            
DEBU[0001] [ssh] 10.252.198.12:22: "10.252.198.12",     
DEBU[0001] [ssh] 10.252.198.12:22: "fe80::5054:ff:fe83:7af3", 
DEBU[0001] [ssh] 10.252.198.12:22: "127.0.0.1"          
DEBU[0001] [ssh] 10.252.198.12:22: ]                    
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "controllerManager": {}, 
DEBU[0001] [ssh] 10.252.198.12:22: "scheduler": {},     
DEBU[0001] [ssh] 10.252.198.12:22: "storage": {         
DEBU[0001] [ssh] 10.252.198.12:22: "etcd": {            
DEBU[0001] [ssh] 10.252.198.12:22: "externalCluster": null, 
DEBU[0001] [ssh] 10.252.198.12:22: "peerAddress": "10.252.198.12" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "kine": {            
DEBU[0001] [ssh] 10.252.198.12:22: "dataSource": "sqlite:/var/lib/k0s/db/state.db?mode=rwc\u0026_journal=WAL\u0026cache=shared" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "type": "etcd"       
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "network": {         
DEBU[0001] [ssh] 10.252.198.12:22: "calico": null,      
DEBU[0001] [ssh] 10.252.198.12:22: "dualStack": {},     
DEBU[0001] [ssh] 10.252.198.12:22: "kubeProxy": {       
DEBU[0001] [ssh] 10.252.198.12:22: "mode": "iptables",  
DEBU[0001] [ssh] 10.252.198.12:22: "metricsBindAddress": "0.0.0.0:10249", 
DEBU[0001] [ssh] 10.252.198.12:22: "iptables": {        
DEBU[0001] [ssh] 10.252.198.12:22: "syncPeriod": "0s",  
DEBU[0001] [ssh] 10.252.198.12:22: "minSyncPeriod": "0s" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "ipvs": {            
DEBU[0001] [ssh] 10.252.198.12:22: "syncPeriod": "0s",  
DEBU[0001] [ssh] 10.252.198.12:22: "minSyncPeriod": "0s", 
DEBU[0001] [ssh] 10.252.198.12:22: "tcpTimeout": "0s",  
DEBU[0001] [ssh] 10.252.198.12:22: "tcpFinTimeout": "0s", 
DEBU[0001] [ssh] 10.252.198.12:22: "udpTimeout": "0s"   
DEBU[0001] [ssh] 10.252.198.12:22: }                    
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "kuberouter": {      
DEBU[0001] [ssh] 10.252.198.12:22: "autoMTU": true,     
DEBU[0001] [ssh] 10.252.198.12:22: "mtu": 0,            
DEBU[0001] [ssh] 10.252.198.12:22: "metricsPort": 8080, 
DEBU[0001] [ssh] 10.252.198.12:22: "hairpin": "Enabled", 
DEBU[0001] [ssh] 10.252.198.12:22: "ipMasq": false,     
DEBU[0001] [ssh] 10.252.198.12:22: "peerRouterASNs": "", 
DEBU[0001] [ssh] 10.252.198.12:22: "peerRouterIPs": ""  
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "nodeLocalLoadBalancing": { 
DEBU[0001] [ssh] 10.252.198.12:22: "type": "EnvoyProxy", 
DEBU[0001] [ssh] 10.252.198.12:22: "envoyProxy": {      
DEBU[0001] [ssh] 10.252.198.12:22: "image": {           
DEBU[0001] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/envoy-distroless", 
DEBU[0001] [ssh] 10.252.198.12:22: "version": "v1.24.12" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "apiServerBindPort": 7443, 
DEBU[0001] [ssh] 10.252.198.12:22: "konnectivityServerBindPort": 7132 
DEBU[0001] [ssh] 10.252.198.12:22: }                    
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "podCIDR": "10.244.0.0/16", 
DEBU[0001] [ssh] 10.252.198.12:22: "provider": "kuberouter", 
DEBU[0001] [ssh] 10.252.198.12:22: "serviceCIDR": "10.96.0.0/12", 
DEBU[0001] [ssh] 10.252.198.12:22: "clusterDomain": "cluster.local" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "telemetry": {       
DEBU[0001] [ssh] 10.252.198.12:22: "enabled": true      
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "installConfig": {   
DEBU[0001] [ssh] 10.252.198.12:22: "users": {           
DEBU[0001] [ssh] 10.252.198.12:22: "etcdUser": "etcd",  
DEBU[0001] [ssh] 10.252.198.12:22: "kineUser": "kube-apiserver", 
DEBU[0001] [ssh] 10.252.198.12:22: "konnectivityUser": "konnectivity-server", 
DEBU[0001] [ssh] 10.252.198.12:22: "kubeAPIserverUser": "kube-apiserver", 
DEBU[0001] [ssh] 10.252.198.12:22: "kubeSchedulerUser": "kube-scheduler" 
DEBU[0001] [ssh] 10.252.198.12:22: }                    
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "images": {          
DEBU[0001] [ssh] 10.252.198.12:22: "konnectivity": {    
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/apiserver-network-proxy-agent", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v0.1.4"  
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "pushgateway": {     
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/pushgateway-ttl", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "1.4.0-k0s.0" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "metricsserver": {   
DEBU[0002] [ssh] 10.252.198.12:22: "image": "registry.k8s.io/metrics-server/metrics-server", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v0.6.4"  
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "kubeproxy": {       
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/kube-proxy", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v1.28.7" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "coredns": {         
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/coredns", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "1.11.1"  
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "pause": {           
DEBU[0002] [ssh] 10.252.198.12:22: "image": "registry.k8s.io/pause", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "3.8"     
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "calico": {          
DEBU[0002] [ssh] 10.252.198.12:22: "cni": {             
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/calico-cni", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v3.26.1-1" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "node": {            
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/calico-node", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v3.26.1-1" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "kubecontrollers": { 
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/calico-kube-controllers", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v3.26.1-1" 
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "kuberouter": {      
DEBU[0002] [ssh] 10.252.198.12:22: "cni": {             
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/kube-router", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v1.6.0-iptables1.8.9-1" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "cniInstaller": {    
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/cni-node", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "1.1.1-k0s.1" 
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "default_pull_policy": "IfNotPresent" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "extensions": {      
DEBU[0002] [ssh] 10.252.198.12:22: "storage": {         
DEBU[0002] [ssh] 10.252.198.12:22: "type": "external_storage", 
DEBU[0002] [ssh] 10.252.198.12:22: "create_default_storage_class": false 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "helm": {            
DEBU[0002] [ssh] 10.252.198.12:22: "concurrencyLevel": 5, 
DEBU[0002] [ssh] 10.252.198.12:22: "repositories": null, 
DEBU[0002] [ssh] 10.252.198.12:22: "charts": null       
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "konnectivity": {    
DEBU[0002] [ssh] 10.252.198.12:22: "adminPort": 8133,   
DEBU[0002] [ssh] 10.252.198.12:22: "agentPort": 8132    
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "K0sVars": {         
DEBU[0002] [ssh] 10.252.198.12:22: "AdminKubeConfigPath": "/var/lib/k0s/pki/admin.conf", 
DEBU[0002] [ssh] 10.252.198.12:22: "BinDir": "/var/lib/k0s/bin", 
DEBU[0002] [ssh] 10.252.198.12:22: "CertRootDir": "/var/lib/k0s/pki", 
DEBU[0002] [ssh] 10.252.198.12:22: "DataDir": "/var/lib/k0s", 
DEBU[0002] [ssh] 10.252.198.12:22: "EtcdCertDir": "/var/lib/k0s/pki/etcd", 
DEBU[0002] [ssh] 10.252.198.12:22: "EtcdDataDir": "/var/lib/k0s/etcd", 
DEBU[0002] [ssh] 10.252.198.12:22: "KineSocketPath": "/run/k0s/kine/kine.sock:2379", 
DEBU[0002] [ssh] 10.252.198.12:22: "KonnectivitySocketDir": "/run/k0s/konnectivity-server", 
DEBU[0002] [ssh] 10.252.198.12:22: "KubeletAuthConfigPath": "/var/lib/k0s/kubelet.conf", 
DEBU[0002] [ssh] 10.252.198.12:22: "KubeletVolumePluginDir": "/usr/libexec/k0s/kubelet-plugins/volume/exec", 
DEBU[0002] [ssh] 10.252.198.12:22: "ManifestsDir": "/var/lib/k0s/manifests", 
DEBU[0002] [ssh] 10.252.198.12:22: "RunDir": "/run/k0s", 
DEBU[0002] [ssh] 10.252.198.12:22: "KonnectivityKubeConfigPath": "/var/lib/k0s/pki/konnectivity.conf", 
DEBU[0002] [ssh] 10.252.198.12:22: "OCIBundleDir": "/var/lib/k0s/images", 
DEBU[0002] [ssh] 10.252.198.12:22: "DefaultStorageType": "kine", 
DEBU[0002] [ssh] 10.252.198.12:22: "RuntimeConfigPath": "/run/k0s/k0s.yaml", 
DEBU[0002] [ssh] 10.252.198.12:22: "StatusSocketPath": "/run/k0s/status.sock", 
DEBU[0002] [ssh] 10.252.198.12:22: "StartupConfigPath": "/etc/k0s/k0s.yaml", 
DEBU[0002] [ssh] 10.252.198.12:22: "EnableDynamicConfig": false, 
DEBU[0002] [ssh] 10.252.198.12:22: "HelmHome": "/var/lib/k0s/helmhome", 
DEBU[0002] [ssh] 10.252.198.12:22: "HelmRepositoryCache": "/var/lib/k0s/helmhome/cache", 
DEBU[0002] [ssh] 10.252.198.12:22: "HelmRepositoryConfig": "/var/lib/k0s/helmhome/repositories.yaml" 
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: }                    
INFO[0002] [ssh] 10.252.198.12:22: is running k0s single version v1.28.7+k0s.0 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- test -e /var/lib/k0s/pki/admin.conf 2> /dev/null` 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- env "KUBECONFIG=/var/lib/k0s/pki/admin.conf" /usr/local/bin/k0s kubectl get -n kube-system namespace kube-system -o template={{.metadata.uid}}` 
DEBU[0002] [ssh] 10.252.198.12:22: f6d0f3ba-9a40-4731-aeea-179b84dc8e4b 
DEBU[0002] Preparing phase 'Validate facts'             
INFO[0002] ==> Running phase: Validate facts   
DEBU[0002] Preparing phase 'Download k0s binaries to local host' 
DEBU[0002] Preparing phase 'Upload k0s binaries to hosts' 
DEBU[0002] Preparing phase 'Download k0s on hosts'      
DEBU[0002] [ssh] 10.252.198.12:22: k0s version on target host is already v1.28.7+k0s.0 
DEBU[0002] Preparing phase 'Upload files to hosts'      
DEBU[0002] Preparing phase 'Install k0s binaries on hosts' 
DEBU[0002] Preparing phase 'Prepare ARM nodes'          
DEBU[0002] Preparing phase 'Configure k0s'              
DEBU[0002] using existing k0s config                    
DEBU[0002] [ssh] 10.252.198.12:22: executing `mktemp`   
DEBU[0002] [ssh] 10.252.198.12:22: /tmp/tmp.WAjV6pMRJA  
DEBU[0002] [ssh] 10.252.198.12:22: executing `mktemp 2> /dev/null` 
DEBU[0002] [ssh] 10.252.198.12:22: /tmp/tmp.TTSimMYnR4  
DEBU[0002] [ssh] 10.252.198.12:22: executing `cat > /tmp/tmp.TTSimMYnR4` 
DEBU[0002] [ssh] 10.252.198.12:22: writing 1654 bytes to command stdin 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- install -D -m 0600 -- /tmp/tmp.TTSimMYnR4 /tmp/tmp.WAjV6pMRJA` 
INFO[0002] [ssh] 10.252.198.12:22: validating configuration 
DEBU[0002] [ssh] 10.252.198.12:22: comparing k0s version v1.28.7+k0s.0 with >= v1.23.1+k0s.0 
DEBU[0002] [ssh] 10.252.198.12:22: comparison result true 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- /usr/local/bin/k0s config validate --config "/tmp/tmp.WAjV6pMRJA"` 
DEBU[0002] [ssh] 10.252.198.12:22: configuration will not change 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- rm -f -- /tmp/tmp.WAjV6pMRJA 2> /dev/null` 
DEBU[0002] Preparing phase 'Restore cluster state'      
DEBU[0002] Preparing phase 'Run Before Apply Hooks'     
DEBU[0002] Preparing phase 'Initialize the k0s cluster' 
DEBU[0002] Preparing phase 'Install controllers'        
DEBU[0002] Preparing phase 'Install workers'            
DEBU[0002] Preparing phase 'Upgrade controllers'        
DEBU[0002] UpgradeControllers phase prep starting       
DEBU[0002] 1 controllers in total                       
DEBU[0002] UpgradeControllers phase prepared, 0 controllers needs upgrade 
DEBU[0002] Preparing phase 'Upgrade workers'            
DEBU[0002] 0 workers in total                           
DEBU[0002] UpgradeWorkers phase prepared, 0 workers needs upgrade 
DEBU[0002] Preparing phase 'Reset workers'              
DEBU[0002] 0 workers in total                           
DEBU[0002] ResetWorkers phase prepared, 0 workers will be reset 
DEBU[0002] Preparing phase 'Reset controllers'          
DEBU[0002] 1 controllers in total                       
DEBU[0002] ResetControllers phase prepared, 0 controllers will be reset 
DEBU[0002] Preparing phase 'Run After Apply Hooks'      
DEBU[0002] Preparing phase 'Release exclusive host lock' 
INFO[0002] ==> Running phase: Release exclusive host lock 
DEBU[0002] [ssh] 10.252.198.12:22: stopped lock cycle, removing file 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- rm -f -- /run/lock/k0sctl 2> /dev/null` 
DEBU[0002] Preparing phase 'Disconnect from hosts'      
INFO[0002] ==> Running phase: Disconnect from hosts 
INFO[0003] ==> Finished in 2s                  
INFO[0003] k0s cluster version v1.28.7+k0s.0 is now installed 
INFO[0003] Tip: To access the cluster you can now fetch the admin kubeconfig using: 
INFO[0003]      k0sctl kubeconfig    

What is the disposition of your keys?

I copy the multipass key from /var/snap/multipass/common/data/multipassd/ssh-keys/id_rsa to my home directory and change the ownership to be my logged in user. This is the same setup I have used for previous versions of k0sctl. I am explicitly setting the path in my config file.

[cgroschupp]

@nwneisen can you try it with my fix, inside rig k0sproject/rig#264