ssh driver --ssh-ip-address seems to be ignored

[horihel]

Steps to reproduce the issue:

1. prepare VM listening on 10.0.0.10, allowing root login with ssh key
2. minikube start --driver=ssh --ssh-ip-address=10.0.0.10 --ssh-key='id_rsa' --alsologtostderr

Full output of failed command:

I0223 09:18:23.815988   11444 out.go:229] Setting OutFile to fd 76 ...
I0223 09:18:23.816966   11444 out.go:281] isatty.IsTerminal(76) = true
I0223 09:18:23.818920   11444 out.go:242] Setting ErrFile to fd 76...
I0223 09:18:23.824778   11444 out.go:281] isatty.IsTerminal(76) = true
I0223 09:18:23.843751   11444 out.go:236] Setting JSON to false
I0223 09:18:23.856498   11444 start.go:106] hostinfo: {"hostname":"hellap-18","uptime":6142,"bootTime":1614062161,"procs":313,"os":"windows","platform":"Microsoft Windows 10 Pro","platformFamily":"Standalone Workstation","platformVersion":"10.0.19042 Build 19042","kernelVersion":"10.0.19042 Build 19042","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"76c9fbb9-3105-4327-84cc-0aa28cf6cf2b"}
W0223 09:18:23.857939   11444 start.go:114] gopshost.Virtualization returned error: not implemented yet
I0223 09:18:23.863446   11444 out.go:119] 😄  minikube v1.17.1 auf Microsoft Windows 10 Pro 10.0.19042 Build 19042
😄  minikube v1.17.1 auf Microsoft Windows 10 Pro 10.0.19042 Build 19042
I0223 09:18:23.869254   11444 notify.go:126] Checking for updates...
W0223 09:18:23.869811   11444 start.go:631] api.Load failed for minikube: filestore "minikube": Docker machine "minikube" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
I0223 09:18:23.871142   11444 driver.go:315] Setting default libvirt URI to qemu:///system
W0223 09:18:23.871673   11444 start.go:631] api.Load failed for minikube: filestore "minikube": Docker machine "minikube" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
I0223 09:18:23.876776   11444 out.go:119] ✨  Using the ssh driver based on existing profile
✨  Using the ssh driver based on existing profile
I0223 09:18:23.885842   11444 start.go:279] selected driver: ssh
I0223 09:18:23.887009   11444 start.go:702] validating driver "ssh" against &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e Memory:10240 CPUs:4 DiskSize:64000 VMDriver: Driver:ssh HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] Network: MultiNodeRequested:false}
I0223 09:18:23.889055   11444 start.go:713] status for ssh: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc:}
I0223 09:18:24.961633   11444 start_flags.go:390] config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e Memory:10240 CPUs:4 DiskSize:64000 VMDriver: Driver:ssh HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] Network: MultiNodeRequested:false}    
I0223 09:18:24.965938   11444 out.go:119] 👍  Starting control plane node minikube in cluster minikube
👍  Starting control plane node minikube in cluster minikube
I0223 09:18:24.980460   11444 preload.go:97] Checking if preload exists for k8s version v1.20.2 and runtime docker
I0223 09:18:24.982367   11444 preload.go:105] Found local preload: C:\Users\helmle\.minikube\cache\preloaded-tarball\preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4
I0223 09:18:24.982367   11444 cache.go:54] Caching tarball of preloaded images
I0223 09:18:24.983341   11444 preload.go:131] Found C:\Users\helmle\.minikube\cache\preloaded-tarball\preloaded-images-k8s-v8-v1.20.2-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0223 09:18:24.984429   11444 cache.go:57] Finished verifying existence of preloaded tar for  v1.20.2 on docker
I0223 09:18:24.985880   11444 profile.go:148] Saving config to C:\Users\helmle\.minikube\profiles\minikube\config.json ...
I0223 09:18:24.989480   11444 cache.go:185] Successfully downloaded all kic artifacts
I0223 09:18:24.990057   11444 start.go:313] acquiring machines lock for minikube: {Name:mk8434f663cb29d24c669c2b134192ed9f8514db Clock:{} Delay:500ms Timeout:13m0s Cancel:<nil>}
I0223 09:18:24.990596   11444 start.go:317] acquired machines lock for "minikube" in 18.8µs
I0223 09:18:24.992265   11444 start.go:89] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e Memory:10240 CPUs:4 DiskSize:64000 VMDriver: Driver:ssh HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] Network: MultiNodeRequested:false} &{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}
I0223 09:18:25.022450   11444 start.go:126] createHost starting for "" (driver="ssh")
I0223 09:18:25.041384   11444 ssh_runner.go:149] Run: systemctl --version
I0223 09:18:25.043070   11444 retry.go:31] will retry after 276.165072ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set      
I0223 09:18:25.320332   11444 retry.go:31] will retry after 540.190908ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set
I0223 09:18:25.862704   11444 retry.go:31] will retry after 655.06503ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set
I0223 09:18:26.518184   11444 retry.go:31] will retry after 791.196345ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set
I0223 09:18:27.310110   11444 start.go:129] duration metric: createHost completed in 2.2869021s
I0223 09:18:27.310159   11444 start.go:80] releasing machines lock for "minikube", held for 2.3178933s
W0223 09:18:27.310834   11444 start.go:377] error starting host: config: please provide an IP address
I0223 09:18:27.319577   11444 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
W0223 09:18:27.451358   11444 cli_runner.go:149] docker container inspect minikube --format={{.State.Status}} returned with exit code 1
I0223 09:18:27.451448   11444 delete.go:46] couldn't inspect container "minikube" before deleting: unknown state "minikube": docker container inspect minikube --format={{.State.Status}}: exit status 1
stdout:


stderr:
error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/containers/minikube/json: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
I0223 09:18:27.491270   11444 cli_runner.go:111] Run: podman container inspect minikube --format={{.State.Status}}
I0223 09:18:27.491270   11444 delete.go:46] couldn't inspect container "minikube" before deleting: unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exec: "podman": executable file not found in %PATH%
stdout:

stderr:
W0223 09:18:27.503677   11444 start.go:382] delete host: Docker machine "minikube" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
W0223 09:18:27.504308   11444 out.go:181] 🤦  StartHost failed, but will try again: config: please provide an IP address
🤦  StartHost failed, but will try again: config: please provide an IP address
I0223 09:18:27.506173   11444 start.go:392] Will try again in 5 seconds ...
I0223 09:18:32.506835   11444 start.go:313] acquiring machines lock for minikube: {Name:mk8434f663cb29d24c669c2b134192ed9f8514db Clock:{} Delay:500ms Timeout:13m0s Cancel:<nil>}
I0223 09:18:32.506835   11444 start.go:317] acquired machines lock for "minikube" in 0s
I0223 09:18:32.508029   11444 start.go:89] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e Memory:10240 CPUs:4 DiskSize:64000 VMDriver: Driver:ssh HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] Network: MultiNodeRequested:false} &{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}
I0223 09:18:32.535085   11444 start.go:126] createHost starting for "" (driver="ssh")
I0223 09:18:32.536062   11444 start.go:129] duration metric: createHost completed in 0s
I0223 09:18:32.537067   11444 start.go:80] releasing machines lock for "minikube", held for 29.038ms
W0223 09:18:32.538228   11444 out.go:181] 😿  Failed to start ssh bare metal machine. Running "minikube delete" may fix it: config: please provide an IP address
😿  Failed to start ssh bare metal machine. Running "minikube delete" may fix it: config: please provide an IP address
I0223 09:18:32.542661   11444 out.go:119]

W0223 09:18:32.555904   11444 out.go:181] ❌  Exiting due to GUEST_PROVISION: Failed to start host: config: please provide an IP address
❌  Exiting due to GUEST_PROVISION: Failed to start host: config: please provide an IP address
W0223 09:18:32.557670   11444 out.go:181]

W0223 09:18:32.570796   11444 out.go:181] 😿  If the above advice does not help, please let us know: 
😿  If the above advice does not help, please let us know:
W0223 09:18:32.572479   11444 out.go:181] 👉  https://github.com/kubernetes/minikube/issues/new/choose
👉  https://github.com/kubernetes/minikube/issues/new/choose
I0223 09:18:32.575542   11444 out.go:119]

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

no logs yet, startup fails

notice that config mentions "SSHIPAddress:" being empty, although the command line specifies --ssh-ip-address=10.0.0.10. AFAICS it doesn't even try to connect.

Removing the --ssh-key doesn't make a difference.

[afbjorklund]

Looks like a bug, was there a previous attempt to start without the parameter ? Not sure how much it updates "running" profiles

✨ Using the ssh driver based on existing profile

i.e. you might have to delete the old profile, and create a new one. if it saved the initial attempt as the config, it will continue that.

Hopefully the new minikube has better error now... (#10309)

[afbjorklund]

The old version (1.17.1) was less helpful:

$ minikube start --driver=ssh
😄  minikube v1.17.1 on Ubuntu 20.04
✨  Using the ssh driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🤦  StartHost failed, but will try again: config: please provide an IP address
😿  Failed to start ssh bare metal machine. Running "minikube delete" may fix it: config: please provide an IP address

❌  Exiting due to GUEST_PROVISION: Failed to start host: config: please provide an IP address

😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new/choose

$ minikube start --driver=ssh --ssh-ip-address=whatever
😄  minikube v1.17.1 on Ubuntu 20.04
✨  Using the ssh driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🤦  StartHost failed, but will try again: config: please provide an IP address
😿  Failed to start ssh bare metal machine. Running "minikube delete" may fix it: config: please provide an IP address

❌  Exiting due to GUEST_PROVISION: Failed to start host: config: please provide an IP address

😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new/choose

[horihel]

You're right - manually deleting "profile" and "machine" in %USERPROFILE%.minikube solved the problem, because I had a typo in my very first try (which failed then of course).

So to me it looks like minikube does not update the SSH IP at all. And if there's an unreachable IP recorded, minikube delete will also fail, forcing you to manually edit/delete in ~/.minikube

Further testing with manually installed VMs (with changing IPs due to a twitchy DHCP) showed, that minikube will always prefer its recorded IP over the one provided via cmdline, meaning that if your remote VM changed IP during a reboot, you can't reach it any more.

[afbjorklund]

Unfortunately the current workaround is to edit the profile/machine configuration directly, just like you have discovered.

This shortcoming is shared with the VM, that it cannot be modified after creation. I would recommend DNS, for the IP.

i.e. add some alias to /etc/hosts

We can probably improve this...

[daehyeok]

@afbjorklund how about add --overwrite flag into start command to change config values in config files.

minikube/cmd/minikube/cmd/start_flags.go

Lines 433 to 435 in 1fe7615

	// updateExistingConfigFromFlags will update the existing config from the flags - used on a second start
	// skipping updating existing docker env , docker opt, InsecureRegistry, registryMirror, extra-config, apiserver-ips
	func updateExistingConfigFromFlags(cmd *cobra.Command, existing *config.ClusterConfig) config.ClusterConfig { //nolint to suppress cyclomatic complexity 45 of func `updateExistingConfigFromFlags` is high (> 30)

currently it looks overwrite flags only when first start fail.(but not cover all fail cases)

[afbjorklund]

Right, that could work.

[daehyeok]

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[medyagh]

@horihel do u mind checking if this issue has been fixed by this PR ? #10899
with latets version of minikube?

[medyagh]

/triage needs-information

[spowelljr]

I tried and the SSHIPAddress was populated for me SSHIPAddress:10.0.0.10, so I'm going to close this issue.