Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

restart: validate configs with new hostname, add logging #8022

Merged
merged 7 commits into from
May 7, 2020

Conversation

tstromberg
Copy link
Contributor

@tstromberg tstromberg commented May 6, 2020

Fixes an important root cause issue:

  • The kubeconfig endpoint check still referenced IP addresses after we had migrated to using DNS names, causing us to unnecessarily remove files out every stop/start cycle.

Fixes issues which made this difficult to debug:

kubeadm:

  • replaced mis-use of reset terminology with reconfigure
  • separately log when a file has incorrect endpoint information which needs removal

kverify:

  • apiServerHealthz now returns errors
  • apiServerHealthz now logs the response body, and includes it with errors

May fix #7704 #7921

This will also make #7895 much easier to debug, but may not fix the underlying issue.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 6, 2020
@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label May 6, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tstromberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 6, 2020
@tstromberg
Copy link
Contributor Author

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label May 6, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
docker Driver

@kubernetes kubernetes deleted a comment from minikube-pr-bot May 6, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [67.534931004 66.800264081 67.23989479100001]
Average time for minikube: 67.19169662533334

Times for Minikube (PR 8022): [65.43820656700001 65.48533936100002 65.50595167399999]
Average time for Minikube (PR 8022): 65.47649920066668

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8022) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.2 on   |  0.064803 |           0.067240 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.023883 |           0.021713 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.005946 |           0.003789 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 40.897165 |          41.304137 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.1 | 23.968936 |          21.874025 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.500286 |           1.362447 |
| components...                  |           |                    |
| * Enabled addons:              |  0.635559 |           0.756147 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.090782 |           0.080114 |
| configured to use "minikube"   |           |                    |
|                                |  0.004338 |           0.006888 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [26.591965455 26.071949288 25.504208657]
Average time for minikube: 26.056041133333334

Times for Minikube (PR 8022): [26.311212225000006 27.346009181999996 26.412358615]
Average time for Minikube (PR 8022): 26.689860007333333

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 8022) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.2 on           |  0.075176 |           0.075682 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002376 |           0.002600 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.058384 |           0.060507 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.596480 |           7.539181 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.1         |  0.145236 |           0.144789 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.052807 |          17.659135 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.847045 |           1.131427 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.210266 |           0.003059 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.065099 |           0.068530 |
| configured to use "minikube"           |           |                    |
|                                        |  0.003173 |           0.004950 |
+----------------------------------------+-----------+--------------------+

@tstromberg tstromberg changed the title Fix cluster reconfiguration issues, improve behavior reconfigure: validate config using name rather than IP, add logging galore May 6, 2020
@tstromberg tstromberg changed the title reconfigure: validate config using name rather than IP, add logging galore restart: validate config using name rather than IP, add logging galore May 6, 2020
@tstromberg tstromberg changed the title restart: validate config using name rather than IP, add logging galore restart: validate configs with new hostname, add logging May 6, 2020
Copy link
Member

@medyagh medyagh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with required comment with links for integration test failures.

@medyagh medyagh merged commit 269b938 into kubernetes:master May 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TestStartStop/group/containerd: busybox does not appear after restart
4 participants