Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an integration test for ambassador addon, fix filename and bump operator version #8372

Closed

Conversation

concaf
Copy link
Contributor

@concaf concaf commented Jun 4, 2020

This PR adds an integration test for the ambassador addon as discussed in #8161. Also bumps the operator version.

CC: @priyawadhwa @medyagh

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 4, 2020
@k8s-ci-robot
Copy link
Contributor

Hi @concaf. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: concaf
To complete the pull request process, please assign sharifelgamal
You can assign the PR to them by writing /assign @sharifelgamal in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jun 4, 2020
@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@codecov-commenter
Copy link

codecov-commenter commented Jun 4, 2020

Codecov Report

Merging #8372 into master will increase coverage by 0.00%.
The diff coverage is 0.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #8372   +/-   ##
=======================================
  Coverage   31.88%   31.89%           
=======================================
  Files         165      165           
  Lines       10822    10821    -1     
=======================================
  Hits         3451     3451           
+ Misses       6957     6956    -1     
  Partials      414      414           
Impacted Files Coverage Δ
pkg/addons/addons.go 41.42% <0.00%> (+0.19%) ⬆️

Copy link
Member

@medyagh medyagh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please see comments above

@medyagh
Copy link
Member

medyagh commented Jun 5, 2020

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 5, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
docker Driver

@priyawadhwa
Copy link

Hey @concaf thank you for opening this PR. Are you still working on this? If so could you please:

  1. Address review comments
  2. Fix the integration tests you wrote; TestAddons is failing across multiple testing environments

@concaf
Copy link
Contributor Author

concaf commented Jul 9, 2020

Hey @priyawadhwa, I'm winding up a few things right now, I'll be fixing this next week. Sorry.

@priyawadhwa
Copy link

No worries, totally understand. Just checking in :)

@concaf concaf force-pushed the concaf/addons/ambassador-test branch from 3d6b3b3 to 4531b40 Compare July 21, 2020 08:11
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 21, 2020
concaf added a commit to concaf/minikube that referenced this pull request Jul 21, 2020
Whenever a `kubectl apply` fails while enabling an addon, it is
retried with exponential backoff. The command (type `*exec.Cmd`)
that this retry function runs in created outside the function -
which means that it is reused on every retry. This is a problem
because `exec.Cmd` (https://godoc.org/github.com/pkg/exec#Cmd)
states that "... Cmd cannot be reused after calling its Run or
Start methods."

This retry is a common case due to, say, a CRD and its resource
being present in the same YAML file of the addon which causes
a race condition where the resource is created before its CRD is
created in the cluster - this race is fixed by subsequent retries.
I've noticed this in the dashboard and the ambassador addon.

Due to the above mentioned bug, minikube throws errors like
`exec: already started` in every retry and the retry is never
successful, manifests are never deployed and addon creation errors
out.

Fix kubernetes#8138
Fix kubernetes#8119
Fix a few CI errors in kubernetes#8372
concaf added a commit to concaf/minikube that referenced this pull request Jul 21, 2020
Whenever a `kubectl apply` fails while enabling an addon, it is
retried with exponential backoff. The command (type `*exec.Cmd`)
that this retry function runs in created outside the function -
which means that it is reused on every retry. This is a problem
because `exec.Cmd` (https://godoc.org/github.com/pkg/exec#Cmd)
states that "... Cmd cannot be reused after calling its Run or
Start methods."

This retry is a common case due to, say, a CRD and its resource
being present in the same YAML file of the addon which causes
a race condition where the resource is created before its CRD is
created in the cluster - this race is fixed by subsequent retries.
I've noticed this in the dashboard and the ambassador addon.

Due to the above mentioned bug, minikube throws errors like
`exec: already started` in every retry and the retry is never
successful, manifests are never deployed and addon creation errors
out.

Related to kubernetes#8138 kubernetes#8119 kubernetes#8372
@concaf
Copy link
Contributor Author

concaf commented Jul 21, 2020

I've sent #8792 which fixes intermittent failures while deploying ambassador (and dashboard) addon with the none driver. Keeping an eye out for the rest of the tests in CI right now.

@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [64.55344182900001 61.330627352 58.160757620999995]
Average time for minikube: 61.34827560066666

Times for Minikube (PR 8372): [65.38053608800001 62.40060027799999 62.480269337]
Average time for Minikube (PR 8372): 63.42046856766666

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8372) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.1 on Debian   |  0.063324 |           0.055707 |
|                           9.11 |           |                    |
| * Using the kvm2 driver based  |  0.018978 |           0.018615 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.002906 |           0.004646 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 37.832591 |          40.607911 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.3 | 21.493100 |          21.190309 |
| on Docker 19.03.12 ...         |           |                    |
| * Verifying Kubernetes         |  1.350878 |           1.328768 |
| components...                  |           |                    |
| * Enabled addons:              |  0.516784 |           0.131337 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.065907 |           0.078058 |
| configured to use "minikube"   |           |                    |
|                                |  0.003807 |           0.005117 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [26.735441337999998 26.9512948 25.033801834000002]
Average time for minikube: 26.240179324

Times for Minikube (PR 8372): [25.551426993 27.567862065 25.657753418]
Average time for Minikube (PR 8372): 26.259014158666663

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8372) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.1 on Debian   |  0.092700 |           0.071918 |
|                           9.11 |           |                    |
| * Using the docker driver      |  0.129597 |           0.099349 |
| based on existing profile      |           |                    |
| * Starting control plane node  |  0.054702 |           0.054976 |
| minikube in cluster minikube   |           |                    |
| * Creating docker container    |  8.190352 |           8.229213 |
| (CPUs=2, Memory=3700MB) ...    |           |                    |
| * Preparing Kubernetes v1.18.3 | 16.819596 |          16.910293 |
| on Docker 19.03.2 ...          |           |                    |
| * Verifying Kubernetes         |  0.873027 |           0.807281 |
| components...                  |           |                    |
| * Enabled addons:              |  0.014538 |           0.010911 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.061237 |           0.068670 |
| configured to use "minikube"   |           |                    |
|                                |  0.004432 |           0.006402 |
+--------------------------------+-----------+--------------------+

@concaf concaf force-pushed the concaf/addons/ambassador-test branch from c3723f2 to 18f3842 Compare July 22, 2020 08:59
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 22, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: [67.35508839900001 63.79077186000001 60.644929236]
Average time for minikube: 63.93026316500001

Times for Minikube (PR 8372): [64.37840650199999 58.90594394 64.32788087299998]
Average time for Minikube (PR 8372): 62.537410438333325

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8372) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.1 on Debian   |  0.053124 |           0.053493 |
|                           9.11 |           |                    |
| * Using the kvm2 driver based  |  0.017396 |           0.021675 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.018952 |           0.004859 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 40.767633 |          40.078684 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.3 | 21.164212 |          20.527508 |
| on Docker 19.03.12 ...         |           |                    |
| * Verifying Kubernetes         |  1.436491 |           1.308920 |
| components...                  |           |                    |
| * Enabled addons:              |  0.382835 |           0.466004 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.085747 |           0.072863 |
| configured to use "minikube"   |           |                    |
|                                |  0.003875 |           0.003404 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [27.974557487000002 26.63131945 25.551172287000004]
Average time for minikube: 26.719016408

Times for Minikube (PR 8372): [26.628006410999998 25.561492643000005 26.446466915]
Average time for Minikube (PR 8372): 26.211988656333332

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 8372) |
+--------------------------------+-----------+--------------------+
| * minikube v1.12.1 on Debian   |  0.064022 |           0.063300 |
|                           9.11 |           |                    |
| * Using the docker driver      |  0.099372 |           0.100491 |
| based on existing profile      |           |                    |
| * Starting control plane node  |  0.056091 |           0.054504 |
| minikube in cluster minikube   |           |                    |
| * Creating docker container    |  8.385937 |           7.977089 |
| (CPUs=2, Memory=3700MB) ...    |           |                    |
| * Preparing Kubernetes v1.18.3 | 17.061787 |          16.967105 |
| on Docker 19.03.2 ...          |           |                    |
| * Verifying Kubernetes         |  0.885314 |           0.821899 |
| components...                  |           |                    |
| * Enabled addons:              |  0.100049 |           0.163843 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.062135 |           0.060366 |
| configured to use "minikube"   |           |                    |
|                                |  0.004309 |           0.003392 |
+--------------------------------+-----------+--------------------+

@concaf concaf changed the title Add an integration test for ambassador addon Add an integration test for ambassador addon + fix filename Jul 22, 2020
@concaf concaf changed the title Add an integration test for ambassador addon + fix filename Add an integration test for ambassador addon, fix filename and bump operator version Jul 22, 2020
This commit does the following:
- Update Ambassador operator to v1.2.8
- Add integration test for `ambassador` addon
- Fix a previously incorrect filename
@concaf concaf force-pushed the concaf/addons/ambassador-test branch from 18f3842 to 6f353ea Compare July 22, 2020 10:29
@concaf
Copy link
Contributor Author

concaf commented Jul 22, 2020

Hey @priyawadhwa @medyagh, I've addressed the review comments and fixed the tests. I don't see any failures on "none_Linux" and "KVM_Linux". Failures on "Docker_Linux" and "VirtualBox_Linux" look unrelated. Waiting on other CI results but I think this is ready for another shot ;)

@concaf
Copy link
Contributor Author

concaf commented Jul 28, 2020

Gentle nudge 😉

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 1, 2020
@tstromberg
Copy link
Contributor

Fixed merge conflict.

/ok-to-test

@TravisBuddy
Copy link

Travis tests have failed

Hey @concaf,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

make test
which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.6.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.3/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/...
gofmt -s -w pkg/minikube/assets/assets.go
which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
/home/travis/gopath/bin/go-bindata
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.6.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.3/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/translate/translations.go -pkg translate translations/...
gofmt -s -w pkg/minikube/translate/translations.go
MINIKUBE_LDFLAGS="-X k8s.io/minikube/pkg/version.version=v1.12.3 -X k8s.io/minikube/pkg/version.isoVersion=v1.12.2 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="dc650450e8b9ad1c73a470118b27e799f1c70496" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v2" ./test.sh
= make lint =============================================================
golangci/golangci-lint info checking GitHub for tag 'v1.30.0'
golangci/golangci-lint info found version: 1.30.0 for v1.30.0/linux/amd64
golangci/golangci-lint info installed out/linters/golangci-lint
test/integration/addons_test.go:45: File is not `goimports`-ed (goimports)
  if !NoneDriver() { // none doesn't support ingress
Makefile:408: recipe for target 'lint-ci' failed
make[1]: *** [lint-ci] Error 1
= go mod ================================================================
ok
= boilerplate ===========================================================
ok
Makefile:292: recipe for target 'test' failed
make: *** [test] Error 4
TravisBuddy Request Identifier: 88d7aca0-ecab-11ea-831e-55c1b07d6d51

@minikube-pr-bot
Copy link

kvm2 Driver
Times for minikube: 98.8s 100.9s 100.4s
Average time for minikube: 100.1s

Times for Minikube (PR 8372): 95.9s 96.4s 92.1s
Average time for Minikube (PR 8372): 94.8s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 8372) |
+--------------------------------+----------+--------------------+
| * minikube v1.12.3 on Debian   | 0.1s     | 0.1s               |
|                           9.11 |          |                    |
| * Using the kvm2 driver based  | 0.0s     | 0.0s               |
| on user configuration          |          |                    |
| * Starting control plane node  | 0.0s     | 0.0s               |
| minikube in cluster minikube   |          |                    |
| * Creating kvm2 VM (CPUs=2,    | 20.5s    | 40.2s              |
| Memory=3700MB, Disk=20000MB)   |          |                    |
| ...                            |          |                    |
| * Preparing Kubernetes v1.19.0 | 77.2s    | 52.4s              |
| on Docker 19.03.12 ...         |          |                    |
| * Verifying Kubernetes         | 1.7s     | 1.6s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.3s     | 0.4s               |
| default-storageclass,          |          |                    |
| storage-provisioner            |          |                    |
| * Done! kubectl is now         | 0.2s     | 0.1s               |
| configured to use "minikube"   |          |                    |
|                                | 0.0s     | 0.0s               |
+--------------------------------+----------+--------------------+

docker Driver
Times for minikube: 58.5s 57.6s 59.0s
Average time for minikube: 58.4s

Times for Minikube (PR 8372): 54.4s 52.1s 52.8s
Average time for Minikube (PR 8372): 53.1s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 8372) |
+--------------------------------+----------+--------------------+
| * minikube v1.12.3 on Debian   | 0.1s     | 0.1s               |
|                           9.11 |          |                    |
| * Using the docker driver      | 0.0s     | 0.0s               |
| based on user configuration    |          |                    |
|                                | 51.2s    | 9.3s               |
|                                |          |                    |
| * Starting control plane node  |          |                    |
| minikube in cluster minikube   |          |                    |
| * Creating docker container    |          |                    |
| (CPUs=2, Memory=3700MB) ...    |          |                    |
| * Preparing Kubernetes v1.19.0 |          |                    |
| on Docker 19.03.8 ...          |          |                    |
| * Verifying Kubernetes         |          |                    |
| components...                  |          |                    |
| * Enabled addons:              |          |                    |
| default-storageclass,          |          |                    |
| storage-provisioner            |          |                    |
| * Done! kubectl is now         |          |                    |
| configured to use "minikube"   |          |                    |
|                                |          |                    |
+--------------------------------+----------+--------------------+

@medyagh
Copy link
Member

medyagh commented Sep 23, 2020

@concaf sorry for the long delay in the PR review, do you mind pulling master one more time ? feel free to ping me on slack if got delayed again

@k8s-ci-robot
Copy link
Contributor

@concaf: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 23, 2020
@medyagh
Copy link
Member

medyagh commented Nov 11, 2020

@concaf would you mind please rebasing the PR, otherwise it looks good

@medyagh
Copy link
Member

medyagh commented Nov 25, 2020

@concaf do u mind please to solve merge conflicts and try one more time?

@tstromberg
Copy link
Contributor

Please re-open once merge conflicts are addressed. Thanks!

@tstromberg tstromberg closed this Dec 9, 2020
@concaf
Copy link
Contributor Author

concaf commented Jan 18, 2021

Apologies for the delay - I've lost my dev setup to rebase and test this and I don't have the bandwidth to fix this right now.
CC: @kflynn @khussey do you folks have someone to do a quick rebase and test this - should not take long.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants