Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wip: Fix StartStopTest/group/containerd flake … #7969

Closed
wants to merge 4 commits into from

Conversation

priyawadhwa
Copy link

This test was failing regularly with the following error showing up on all pods:

FailedCreatePodSandbox

after doing some digging, it seems that this error is always CNI related. I found these docs on the k8s website about running containerd in minikube:

https://kubernetes.io/docs/setup/learning-environment/minikube/

which says the flags --network-plugin=cni and --enable-default-cni should be set with running containerd. I tried passing those flags in, and a flag for pod subnet.

This test was then failing because the kubernetes dashboard addon would never come up.

The dashboard addon wasn't being enabled because of this flow:

minikube stop
minikube addons enable dashboard
minikube start

but on start, dashboard was skipped because it seemed like it had already been enabled.

I refactored addons code to always enable addons on minikube start, and to only check if they've already been enabled on minikube addons enable/disable xxx

Should fix #7704

Priya Wadhwa added 4 commits April 30, 2020 14:54
This test was failing regularly with the following error showing up on all pods:

```
FailedCreatePodSandbox
```

after doing some digging, it seems that this error is always CNI related. I found these docs on the k8s website about running containerd in minikube:

https://kubernetes.io/docs/setup/learning-environment/minikube/

which says the flags `--network-plugin=cni` and `--enable-default-cni` should be set with running containerd. I tried passing those flags in, and ran this test locally 3x. It passed everytime, so I'm hoping this fixes the flake.
Also, the dashboard addon wasn't being enabled because of this flow:

```
minikube stop
minikube addons enable dashboard
minikube start
```

but on start, dashboard was skipped because it seemed like it had already been enabled.
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 1, 2020
@priyawadhwa
Copy link
Author

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 1, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: priyawadhwa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 1, 2020
@codecov-io
Copy link

Codecov Report

Merging #7969 into master will decrease coverage by 0.12%.
The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7969      +/-   ##
==========================================
- Coverage   36.06%   35.94%   -0.13%     
==========================================
  Files         144      144              
  Lines        9191     9204      +13     
==========================================
- Hits         3315     3308       -7     
- Misses       5475     5497      +22     
+ Partials      401      399       -2     
Impacted Files Coverage Δ
cmd/minikube/cmd/config/disable.go 15.38% <0.00%> (-4.62%) ⬇️
cmd/minikube/cmd/config/enable.go 15.38% <0.00%> (-4.62%) ⬇️
pkg/addons/addons.go 39.89% <0.00%> (-4.09%) ⬇️
cmd/minikube/cmd/node_add.go 17.85% <0.00%> (-1.38%) ⬇️
cmd/minikube/cmd/start.go 15.61% <0.00%> (-0.12%) ⬇️
cmd/minikube/cmd/tunnel.go 5.26% <0.00%> (+0.13%) ⬆️

@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [65.8907328 65.82264305900001 63.743056302]
Average time for minikube: 65.15214405366666

Times for Minikube (PR 7969): [63.594053806 66.225509132 68.282022962]
Average time for Minikube (PR 7969): 66.03386196666666

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7969) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.2 on   |  0.062409 |           0.064055 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.023321 |           0.023195 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.003673 |           0.003929 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 40.918610 |          40.243512 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.1 | 21.761112 |          23.311281 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  2.064856 |           1.368132 |
| components...                  |           |                    |
| * Enabled addons:              |  0.229271 |           0.933529 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.083959 |           0.081026 |
| configured to use "minikube"   |           |                    |
|                                |  0.004935 |           0.005202 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [26.781936987000005 28.161053797 26.084910303999994]
Average time for minikube: 27.009300362666664

Times for Minikube (PR 7969): [26.27470847 27.049298984 28.288583654999996]
Average time for Minikube (PR 7969): 27.20419703633333

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7969) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.2 on           |  0.079073 |           0.079417 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002737 |           0.002492 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.063670 |           0.061900 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.783768 |           7.689030 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.1         |  0.125244 |           0.150383 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.343198 |          17.960837 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  1.483054 |           1.132106 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.062195 |           0.053751 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.062970 |           0.070308 |
| configured to use "minikube"           |           |                    |
|                                        |  0.003392 |           0.003972 |
+----------------------------------------+-----------+--------------------+

@priyawadhwa priyawadhwa changed the title Fix StartStopTest/group/containerd flake … wip: Fix StartStopTest/group/containerd flake … May 1, 2020
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 1, 2020
@priyawadhwa priyawadhwa deleted the busybox branch May 12, 2020 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TestStartStop/group/containerd: busybox does not appear after restart
4 participants