TestVersionUpgrade on none: error: failed to create listener... bind: address already in use #4418

medyagh · 2019-06-03T16:56:42Z

the TestVersionUpgrade randomly fails sometimes in the integration tests.
with error

error: failed to create listener: failed to listen on 0.0.0.0:8443: listen tcp 0.0.0.0:8443: bind: address already in use

--- FAIL: TestVersionUpgrade (128.05s)
    util.go:130: Error running command: start --vm-driver=none  --v=10 --logtostderr --bootstrapper=kubeadm --kubernetes-version=v1.14.2 --alsologtostderr --v=2 . Output: * minikube v1.1.0 on linux (amd64)
        * minikube will upgrade the local cluster from Kubernetes 1.10.13 to 1.14.2
        * Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
        * Restarting existing none VM for "minikube" ...
        * Waiting for SSH access ...
        * Configuring environment for Kubernetes v1.14.2 on Docker 18.06.1-ce
        * Pulling images ...
        * Relaunching Kubernetes v1.14.2 using kubeadm ... 
        * Problems detected in "kube-apiserver":
          - error: failed to create listener: failed to listen on 0.0.0.0:8443: listen tcp 0.0.0.0:8443: bind: address already in use
FAIL

The text was updated successfully, but these errors were encountered:

medyagh · 2019-06-03T22:28:39Z

and this is the random fail I got for OS X Virtualbox integration test:


12:30:06 --- FAIL: TestVersionUpgrade (121.48s)
12:30:06     util.go:130: Error running command: start --vm-driver=virtualbox  --v=10 --logtostderr --bootstrapper=kubeadm --kubernetes-version=v1.10.13 --alsologtostderr --v=2 . Output: * minikube v1.1.0 on darwin (amd64)
12:30:06         2019/06/03 12:28:06 No matching credentials were found, falling back on anonymous
12:30:06         2019/06/03 12:28:06 No matching credentials were found, falling back on anonymous
12:30:06         2019/06/03 12:28:06 No matching credentials were found, falling back on anonymous
12:30:06         2019/06/03 12:28:06 No matching credentials were found, falling back on anonymous
12:30:06         2019/06/03 12:28:06 No matching credentials were found, falling back on anonymous
12:30:06         * Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
12:30:06         2019/06/03 12:28:06 No matching credentials were found, falling back on anonymous
12:30:06         2019/06/03 12:28:06 No matching credentials were found, falling back on anonymous
12:30:06         2019/06/03 12:28:06 No matching credentials were found, falling back on anonymous
12:30:06         * Configuring environment for Kubernetes v1.10.13 on Docker 18.09.6
12:30:06         * Pulling images ...
12:30:06         * Unable to pull images, which may be OK: pull command is not supported by kubeadm v1.10.13
12:30:06         * Launching Kubernetes ... 
12:30:06 FAIL

medyagh · 2019-06-05T17:07:23Z

here is one failure on linux kvm seems like might need retry logic for downloading binaries

06:50:51 === RUN   TestVersionUpgrade
06:50:51  13:50:51 | Run: [/home/jenkins/workspace/Linux_Integration_Tests_KVM/out/minikube-linux-amd64 delete]
06:50:51  13:50:51 | > ! "minikube" cluster does not exist
06:50:51  13:50:51 | > ! "minikube" profile does not exist
06:50:51  13:50:51 | Run: [/home/jenkins/workspace/Linux_Integration_Tests_KVM/out/minikube-linux-amd64 status --format={{.Host}} --v=10 --logtostderr ]
06:50:51  13:50:51 | ! I0605 13:50:51.706315    8499 notify.go:128] Checking for updates...
06:50:51 --- FAIL: TestVersionUpgrade (0.41s)
06:50:51     version_upgrade_test.go:43: Failed to get latest release binary: Get https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64: proxyconnect tcp: dial tcp [::1]:41253: connect: connection refused
06:50:51 FAIL

medyagh · 2019-06-14T23:59:21Z

another error on linux KVM

--- FAIL: TestVersionUpgrade (304.15s)
    util.go:248: got state: "", expected "Running"

medyagh · 2019-06-19T22:38:59Z

A diff error on
Linux-None

--- FAIL: TestVersionUpgrade (151.89s)
    util.go:130: Error running command: start --vm-driver=none  --v=10 --logtostderr --bootstrapper=kubeadm --kubernetes-version=v1.15.0-beta.1 --alsologtostderr --v=2 . Output: * minikube v1.1.1 on linux (amd64)
        * minikube will upgrade the local cluster from Kubernetes 1.10.13 to 1.15.0-beta.1
        * Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
        * Restarting existing none VM for "minikube" ...
        * Waiting for SSH access ...
        * Configuring environment for Kubernetes v1.15.0-beta.1 on Docker 18.06.1-ce
        * Downloading kubelet v1.15.0-beta.1
        * Downloading kubeadm v1.15.0-beta.1
        * Pulling images ...
        * Relaunching Kubernetes v1.15.0-beta.1 using kubeadm ... 
        * Problems detected in "kube-addon-manager":
          - error: unable to recognize "STDIN": Get https://localhost:8443/api?timeout=32s: dial tcp [::1]:8443: connect: connection refused
          - error: error when retrieving current configuration of:
FAIL

medyagh · 2019-06-19T22:53:57Z


15:33:15  22:33:15 | Run: [/home/jenkins/workspace/Linux_Integration_Tests_KVM/out/minikube-linux-amd64 delete]
15:33:16  22:33:16 | > * Deleting "minikube" from kvm2 ...
15:33:16  22:33:16 | ! X Failed to delete cluster: destroying running domain: virError(Code=55, Domain=10, Message='Requested operation is not valid: domain is not running')

tstromberg · 2019-06-20T00:12:48Z

I feel like this has increased frequency recently, perhaps related to v1.15.0?

medyagh · 2019-06-20T00:45:20Z

I don't know about 1.15.0 yet, but it has been feeling for the following reasons mainly:

Minikube not stopping for real and and fail to bind port on next stard.
Minikube stop not setting the "" current context after stopping
the Kvm2 domain error when deleting ( I wonder if I need to upgrade the kvm2 driver on the integraiton machines )

medyagh · 2019-06-26T03:00:18Z

Issue still there for restarting on none:


 02:41:05 | ! W0626 02:41:05.603055   15919 logs.go:88] Found kube-apiserver problem: error: failed to create listener: failed to listen on 0.0.0.0:8443: listen tcp 0.0.0.0:8443: bind: address already in use
 02:41:05 | ! W0626 02:41:05.603121   15919 exit.go:100] Error restarting cluster: addon phase: running command: sudo kubeadm init phase addon all --config /var/lib/kubeadm.yaml: exit status 1
 02:41:05 | ! X Error restarting cluster: addon phase: running command: sudo kubeadm init phase addon all --config /var/lib/kubeadm.yaml: exit status 1
 02:41:05 | ! * Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
 02:41:05 | !   - https://github.com/kubernetes/minikube/issues/new
 02:41:05 | > * Problems detected in "kube-apiserver":
 02:41:05 | >   - error: failed to create listener: failed to listen on 0.0.0.0:8443: listen tcp 0.0.0.0:8443: bind: address already in use
--- FAIL: TestVersionUpgrade (144.33s)
    minikube_runner.go:127: Error running command: start --vm-driver=none  --v=10 --logtostderr --bootstrapper=kubeadm --kubernetes-version=1.15.0 --alsologtostderr --v=2 . Output: * minikube v1.2.0 on linux (amd64)
        * minikube will upgrade the local cluster from Kubernetes 1.10.13 to 1.15.0
        * Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
        * Restarting existing none VM for "minikube" ...
        * Waiting for SSH access ...
        * Configuring environment for Kubernetes v1.15.0 on Docker 18.06.1-ce
        * Pulling images ...
        * Relaunching Kubernetes v1.15.0 using kubeadm ... 
        * Problems detected in "kube-apiserver":
          - error: failed to create listener: failed to listen on 0.0.0.0:8443: listen tcp 0.0.0.0:8443: bind: address already in use```

medyagh · 2019-07-25T20:20:10Z

Please feel free to re-open this issue if seen again !

medyagh · 2019-07-26T09:55:43Z

unfortunately this still happens, for some reason the systemctl wont release the port somttimes ! we need to add some proc kill in the code to release the port

tstromberg · 2019-09-16T20:12:08Z

Related and/or dupe: #5332

mansurali901 · 2022-10-03T08:53:46Z

One of the reason is sometimes systemctl holds the port for little longer so when docker tries to bind the port it gets failed so the best solution is to stop minikube cluster then stop docker hold for atleast 30 seconds and then start docker first and then start your minikube cluster it will solver the problem

medyagh added the kind/bug Categorizes issue or PR as related to a bug. label Jun 3, 2019

sharifelgamal added kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. and removed kind/bug Categorizes issue or PR as related to a bug. labels Jun 3, 2019

sharifelgamal added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jun 6, 2019

fenglixa mentioned this issue Jun 10, 2019

Detect status before enable/disable addon #4424

Merged

medyagh added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jun 11, 2019

This was referenced Jun 12, 2019

adding retriable http get #4474

Merged

Fix busybox version in integration tests #4480

Merged

Upgrade CNI config version to 0.3.0 #4410

Merged

medyagh mentioned this issue Jun 19, 2019

kvm2 v1.2.0: Failed to delete cluster: Requested operation is not valid: domain is not running #4319

Closed

medyagh changed the title ~~Fix TestVersionUpgrade integration test~~ None: Error Restarting Jun 26, 2019

medyagh changed the title ~~None: Error Restarting~~ None driver: error restarting: TestVersionUpgrade integration test Jun 26, 2019

tstromberg added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jul 12, 2019

tstromberg changed the title ~~None driver: error restarting: TestVersionUpgrade integration test~~ TestVersionUpgrade on none: error: failed to create listener... bind: address already in use Jul 15, 2019

tstromberg added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Jul 15, 2019

tstromberg mentioned this issue Jul 15, 2019

Allow users to skip pod health checks with --wait=false #4744

Merged

tstromberg added this to the v1.3.0-candidate milestone Jul 15, 2019

This was referenced Jul 25, 2019

none: waiting for apiserver: timed out waiting for the condition (kubelet port conflict) #4500

Closed

more reliable stop for none driver #4871

Merged

sharifelgamal closed this as completed in #4871 Jul 25, 2019

medyagh reopened this Jul 26, 2019

medyagh mentioned this issue Jul 26, 2019

CI: kill hung proc with sudo and release port #4882

Merged

tstromberg closed this as completed Aug 6, 2019

tstromberg modified the milestones: v1.3.0, v1.4.0 Candidate Aug 6, 2019

tstromberg added this to the v1.5.0-candidate milestone Sep 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TestVersionUpgrade on none: error: failed to create listener... bind: address already in use #4418

TestVersionUpgrade on none: error: failed to create listener... bind: address already in use #4418

medyagh commented Jun 3, 2019 •

edited

Loading

medyagh commented Jun 3, 2019 •

edited

Loading

medyagh commented Jun 5, 2019

medyagh commented Jun 14, 2019

medyagh commented Jun 19, 2019

medyagh commented Jun 19, 2019

tstromberg commented Jun 20, 2019

medyagh commented Jun 20, 2019 •

edited

Loading

medyagh commented Jun 26, 2019

medyagh commented Jul 25, 2019

medyagh commented Jul 26, 2019

tstromberg commented Sep 16, 2019

mansurali901 commented Oct 3, 2022

TestVersionUpgrade on none: error: failed to create listener... bind: address already in use #4418

TestVersionUpgrade on none: error: failed to create listener... bind: address already in use #4418

Comments

medyagh commented Jun 3, 2019 • edited Loading

medyagh commented Jun 3, 2019 • edited Loading

medyagh commented Jun 5, 2019

medyagh commented Jun 14, 2019

medyagh commented Jun 19, 2019

medyagh commented Jun 19, 2019

tstromberg commented Jun 20, 2019

medyagh commented Jun 20, 2019 • edited Loading

medyagh commented Jun 26, 2019

medyagh commented Jul 25, 2019

medyagh commented Jul 26, 2019

tstromberg commented Sep 16, 2019

mansurali901 commented Oct 3, 2022

medyagh commented Jun 3, 2019 •

edited

Loading

medyagh commented Jun 3, 2019 •

edited

Loading

medyagh commented Jun 20, 2019 •

edited

Loading