Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make sure worker nodes actually join control plane on restart #9476

Merged
merged 4 commits into from
Oct 16, 2020

Conversation

sharifelgamal
Copy link
Collaborator

This should fix #9456

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 15, 2020
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 15, 2020
} else {
return errors.Wrapf(err, "cmd failed: %s\n%+v\n", joinCmd, out.Output())
}
return errors.Wrapf(err, "cmd failed: %s\n%+v\n", joinCmd, out.Output())
Copy link
Member

@medyagh medyagh Oct 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do the same check for the error message like before but this time retry and also let user know we are retrying

already exists in the cluster. You must delete the existing Node or change the name of this new joining Node

if that error is there, we will klog.Info (so the IDE users who usually dont like to wait without any logs, know we are retrying

if rr.Output() has that ^^ error, {
klog.Info("retrying to re-register worker node to the controlplane.... ")
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, RunCmd already includes the rr.Output in the error message,
return errors.Wrapf(err, "join cmd") is enough

} else {
return errors.Wrapf(err, "cmd failed: %s\n%+v\n", joinCmd, out.Output())
}
return errors.Wrapf(err, "cmd failed: %s\n%+v\n", joinCmd, out.Output())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, RunCmd already includes the rr.Output in the error message,
return errors.Wrapf(err, "join cmd") is enough

Copy link
Member

@medyagh medyagh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets also remove the manual time.Sleep in our test
minikube should wait enough for the kubelet thing itself so users dont have to do that.
https://github.com/medyagh/minikube/blob/aad43a2e43317540ef6167c17715a4dd449bcbaa/test/integration/multinode_test.go#L231

@sharifelgamal
Copy link
Collaborator Author

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Oct 16, 2020
@kubernetes kubernetes deleted a comment from minikube-pr-bot Oct 16, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: 61.6s 61.7s 60.1s
Average time for minikube: 61.2s

Times for Minikube (PR 9476): 60.3s 62.0s 59.8s
Average time for Minikube (PR 9476): 60.7s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9476) |
+--------------------------------+----------+--------------------+
| * minikube v1.14.0 on Debian   | 0.1s     | 0.0s               |
|                           9.11 |          |                    |
| * Using the kvm2 driver based  | 0.0s     | 0.0s               |
| on user configuration          |          |                    |
| * Starting control plane node  | 0.0s     | 0.0s               |
| minikube in cluster minikube   |          |                    |
| * Creating kvm2 VM (CPUs=2,    | 35.8s    | 35.7s              |
| Memory=3700MB, Disk=20000MB)   |          |                    |
| ...                            |          |                    |
| * Preparing Kubernetes v1.19.2 | 23.1s    | 22.9s              |
| on Docker 19.03.12 ...         |          |                    |
| * Verifying Kubernetes         | 1.6s     | 1.6s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.1s     | 0.3s               |
| default-storageclass,          |          |                    |
| storage-provisioner            |          |                    |
|                                | 0.0s     | 0.0s               |
| * Want kubectl v1.19.2? Try    |          |                    |
| 'minikube kubectl -- get pods  |          |                    |
| -A'                            |          |                    |
| * Done! kubectl is now         |          |                    |
| configured to use "minikube"   |          |                    |
| by default                     |          |                    |
+--------------------------------+----------+--------------------+

docker Driver
Times for minikube: 29.9s 29.2s 28.9s
Average time for minikube: 29.3s

Times for Minikube (PR 9476): 28.5s 31.1s 31.5s
Average time for Minikube (PR 9476): 30.4s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9476) |
+--------------------------------+----------+--------------------+
| * minikube v1.14.0 on Debian   | 0.2s     | 0.2s               |
|                           9.11 |          |                    |
| * Using the docker driver      | 0.1s     | 0.1s               |
| based on user configuration    |          |                    |
| * Starting control plane node  | 0.1s     | 0.1s               |
| minikube in cluster minikube   |          |                    |
| * Creating docker container    | 9.2s     | 9.0s               |
| (CPUs=2, Memory=3700MB) ...    |          |                    |
| * Preparing Kubernetes v1.19.2 | 18.6s    | 19.9s              |
| on Docker 19.03.8 ...          |          |                    |
| * Verifying Kubernetes         | 1.1s     | 1.1s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.1s     | 0.1s               |
| storage-provisioner,           |          |                    |
| default-storageclass           |          |                    |
|                                | 0.0s     | 0.0s               |
| * Want kubectl v1.19.2? Try    |          |                    |
| 'minikube kubectl -- get pods  |          |                    |
| -A'                            |          |                    |
| * Done! kubectl is now         |          |                    |
| configured to use "minikube"   |          |                    |
| by default                     |          |                    |
+--------------------------------+----------+--------------------+

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [medyagh,sharifelgamal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@codecov-io
Copy link

Codecov Report

Merging #9476 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #9476   +/-   ##
=======================================
  Coverage   29.05%   29.05%           
=======================================
  Files         171      171           
  Lines       10443    10443           
=======================================
  Hits         3034     3034           
  Misses       6987     6987           
  Partials      422      422           

@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: 62.0s 63.9s 60.2s
Average time for minikube: 62.0s

Times for Minikube (PR 9476): 59.6s 60.7s 61.3s
Average time for Minikube (PR 9476): 60.5s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9476) |
+--------------------------------+----------+--------------------+
| * minikube v1.14.0 on Debian   | 0.1s     | 0.0s               |
|                           9.11 |          |                    |
| * Using the kvm2 driver based  | 0.0s     | 0.0s               |
| on user configuration          |          |                    |
| * Starting control plane node  | 0.0s     | 0.0s               |
| minikube in cluster minikube   |          |                    |
| * Creating kvm2 VM (CPUs=2,    | 37.0s    | 35.7s              |
| Memory=3700MB, Disk=20000MB)   |          |                    |
| ...                            |          |                    |
| * Preparing Kubernetes v1.19.2 | 22.8s    | 23.1s              |
| on Docker 19.03.12 ...         |          |                    |
| * Verifying Kubernetes         | 1.6s     | 1.5s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.8s     | 0.1s               |
| default-storageclass,          |          |                    |
| storage-provisioner            |          |                    |
|                                | 0.0s     | 0.0s               |
| * Want kubectl v1.19.2? Try    |          |                    |
| 'minikube kubectl -- get pods  |          |                    |
| -A'                            |          |                    |
| * Done! kubectl is now         |          |                    |
| configured to use "minikube"   |          |                    |
| by default                     |          |                    |
+--------------------------------+----------+--------------------+

docker Driver
Times for minikube: 29.4s 28.6s 27.5s
Average time for minikube: 28.5s

Times for Minikube (PR 9476): 29.4s 29.9s 28.9s
Average time for Minikube (PR 9476): 29.4s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9476) |
+--------------------------------+----------+--------------------+
| * minikube v1.14.0 on Debian   | 0.2s     | 0.2s               |
|                           9.11 |          |                    |
| * Using the docker driver      | 0.1s     | 0.1s               |
| based on user configuration    |          |                    |
| * Starting control plane node  | 0.1s     | 0.1s               |
| minikube in cluster minikube   |          |                    |
| * Creating docker container    | 9.2s     | 8.8s               |
| (CPUs=2, Memory=3700MB) ...    |          |                    |
| * Preparing Kubernetes v1.19.2 | 17.9s    | 19.2s              |
| on Docker 19.03.8 ...          |          |                    |
| * Verifying Kubernetes         | 0.9s     | 1.0s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.1s     | 0.1s               |
| storage-provisioner,           |          |                    |
| default-storageclass           |          |                    |
|                                | 0.0s     | 0.0s               |
| * Want kubectl v1.19.2? Try    |          |                    |
| 'minikube kubectl -- get pods  |          |                    |
| -A'                            |          |                    |
| * Done! kubectl is now         |          |                    |
| configured to use "minikube"   |          |                    |
| by default                     |          |                    |
+--------------------------------+----------+--------------------+

@sharifelgamal sharifelgamal merged commit a282354 into kubernetes:master Oct 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

FailedTest: TestMultiNode/serial/RestartMultiNode kubelet not running after restart
5 participants