Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retry logic to minikub SSH connectivity check #5834

Closed
cvila84 opened this issue Nov 5, 2019 · 5 comments · Fixed by #5848
Closed

Add retry logic to minikub SSH connectivity check #5834

cvila84 opened this issue Nov 5, 2019 · 5 comments · Fixed by #5848
Assignees
Labels
area/networking networking issues co/virtualbox good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. kind/feature Categorizes issue or PR as related to a new feature. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@cvila84
Copy link

cvila84 commented Nov 5, 2019

The exact command to reproduce the issue:
minikube start

The full output of the command that failed:

  • minikube v1.5.2 on Microsoft Windows 7 Enterprise Service Pack 1 6.1.7601 Build 7601
    • MINIKUBE_VERSION=v1.5.2
  • Automatically selected the 'virtualbox' driver
  • Downloading VM boot image ...

    minikube-v1.5.1.iso.sha256: 65 B / 65 B [--------------] 100.00% ? p/s 0s
    minikube-v1.5.1.iso: 143.76 MiB / 143.76 MiB [-] 100.00% 5.96 MiB p/s 24s

  • Creating virtualbox VM (CPUs=4, Memory=4096MB, Disk=20000MB) ...
  • Found network options:
    • HTTP_PROXY=http://10.43.216.8:8080
    • HTTPS_PROXY=http://10.43.216.8:8080
    • NO_PROXY=localhost,valmante,minikube,.gemalto.com,127.0.0.1,10.10.154.37,192.168.99.100
    • http_proxy=http://10.43.216.8:8080
    • https_proxy=http://10.43.216.8:8080
    • no_proxy=localhost,valmante,minikube,.gemalto.com,127.0.0.1,10.10.154.37,192.168.99.100
      X minikube is unable to connect to the VM: dial tcp 192.168.99.100:22: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

This is likely due to one of two reasons:

  • VPN or firewall interference
  • virtualbox network configuration issue

Suggested workarounds:

  • Disable your local VPN or firewall software
  • Configure your local VPN or firewall to allow access to 192.168.99.100
  • Restart or reinstall virtualbox
  • Use an alternative --vm-driver
    panic: exit status 74

The operating system version: Windows 7

Hello, it seems the trySSH method in validateNetwork in start.go seems a little bit aggressive when testing connectivity to VM. Immediately after the failure, I tried minikube ssh but it was not responding. When I tried 10s later, everything was working well

user@FSP4KC2 ~
$ ssh -l docker 192.168.99.100
^C

[wait 10 seconds]

cvila@FSP4KC2 ~
$ ssh -l docker 192.168.99.100
Warning: Permanently added '192.168.99.100' (ECDSA) to the list of known hosts.
[email protected]'s password:
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ exit
logout
Connection to 192.168.99.100 closed.
@medyagh
Copy link
Member

medyagh commented Nov 6, 2019

@cvila84 thanks for creating the issue,
couple querstions

1- do you use a VPN as the minikube is suggesting ?

2- do you mind clarifying what do you mean the trySSH is aggressive when testing connectivity to VM ? do you mean if trySSH waits longer it would work ?

@medyagh medyagh added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. area/networking networking issues labels Nov 6, 2019
@cvila84
Copy link
Author

cvila84 commented Nov 6, 2019

Hello,

  1. I have no VPN, nor firewall active on host-only network interface.
  2. Yes, this is my point. It seems we need some time between VM creation and SSH availability. trySSH is made too soon, because as explained in the issue, if I run minikube ssh 10 seconds after minikube start is failing, it works

@medyagh
Copy link
Member

medyagh commented Nov 6, 2019

thank you @cvila84 for updating !

I agree with you, we could add a retry loigc to

	conn, err := net.Dial("tcp", sshAddr)

and retry up to 10 seconds

conn, err := net.Dial("tcp", sshAddr)

this will improve rare cases that minikbue prematurely perevents to continue

@medyagh medyagh changed the title minikube start often fails on SSH connectivity check Add retry logic to minikub SSH connectivity check Nov 6, 2019
@medyagh medyagh added kind/feature Categorizes issue or PR as related to a new feature. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. and removed priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Nov 6, 2019
@medyagh
Copy link
Member

medyagh commented Nov 6, 2019

I will be happy to review a pr that fixes, basicly we need a retry.X on dial

@hypnoglow
Copy link
Contributor

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/networking networking issues co/virtualbox good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. kind/feature Categorizes issue or PR as related to a new feature. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants