Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use kernel 5.10 for minikube.iso. #12707

Merged

Conversation

eiffel-fl
Copy link
Contributor

Hi.

With this PR, I jumped the kernel version in minikube.iso to 5.10.
Jumping the actual version of the kernel is something needed by the community (e.g. #10501 or #8556).
The following shows the result:

francis@machine:~/Codes/kinvolk/minikube-git$ minikube start --driver=kvm2 --iso-url=file://$(pwd)/out/minikube.iso                        
😄  minikube v1.23.2 sur Debian 11.1
✨  Utilisation du pilote kvm2 basé sur la configuration de l'utilisateur
👍  Démarrage du noeud de plan de contrôle minikube dans le cluster minikube
🔥  Création de VM kvm2 (CPUs=2, Mémoire=2048MB, Disque=20000MB)...
🐳  Préparation de Kubernetes v1.22.2 sur Docker 20.10.8...
    ▪ Génération des certificats et des clés
    ▪ Démarrage du plan de contrôle ...
    ▪ Configuration des règles RBAC ...
🔎  Vérification des composants Kubernetes...
    ▪ Utilisation de l'image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Modules activés: storage-provisioner, default-storageclass
🏄  Terminé ! kubectl est maintenant configuré pour utiliser "minikube" cluster et espace de noms "default" par défaut.
francis@machine:~/Codes/kinvolk/minikube-git$ minikube ssh                                                 francis/upstream/use-kernel-5.10 % u=
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ uname -a
Linux minikube 5.10.57 #4 SMP Wed Oct 13 15:51:53 UTC 2021 x86_64 GNU/Linux
$ docker run -t --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete 
Digest: sha256:37a0b92b08d4919615c3ee023f7ddb068d12b8387475d64c622ac30f45c29c51
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

However, I would like to test it more because there are a lot of parameters and I cannot ensure everything work fine.
Also, #8187 (I gathered inspiration from this once, particularly for CONFIG_PCI) was removed after some tests failed to pass.

I understand this PR goes maybe in an opposite direction to this minikube enhancement process.
So, I am open to discussion in the hope to find a solution which suits everyone.

Best regards.

@k8s-ci-robot
Copy link
Contributor

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


  • If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
  • If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
  • If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
  • Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Oct 13, 2021
@k8s-ci-robot
Copy link
Contributor

Welcome @eiffel-fl!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 13, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @eiffel-fl. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Oct 13, 2021
@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@afbjorklund
Copy link
Collaborator

afbjorklund commented Oct 13, 2021

I understand this PR goes maybe in an opposite direction to this minikube enhancement process.

They are parallel tracks, it is possible to upgrade software in Buildroot without affect Ubuntu* versions and vice-versa

We have talked about upgrading the kernel before, but had to rollback when issues were found later during testing...

You can find default BR kernel versions here: https://github.com/buildroot/buildroot/blob/2021.02.x/linux/linux.hash

* currently the KIC base uses Ubuntu 20.04 LTS, which normally ships with a Linux 5.4 kernel (or 5.11 in HWE)

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Oct 13, 2021
@eiffel-fl
Copy link
Contributor Author

Thank you for your quick answer.

How can I use the Ubuntu based image you are talking?
Because I got the following with the default image:

minikube start --driver=kvm2
...
Téléchargement de l'image de démarrage de la VM...
    > minikube-v1.23.1.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
    > minikube-v1.23.1.iso: 225.22 MiB / 225.22 MiB  100.00% 332.18 MiB p/s 900
...
minikube ssh 'uname -a'
Linux minikube 4.19.202 #1 SMP Thu Sep 16 17:44:03 UTC 2021 x86_64 GNU/Linux

@afbjorklund
Copy link
Collaborator

afbjorklund commented Oct 13, 2021

That proposal you are talking about was just that, replacing the custom ISO with a "standard" one.

The closest thing to being actually available is the Vagrant file, as described in PR #12185

The idea was always to make a Linux 5 version too of the ISO, after it was updated for ARM...


@eiffel-fl
Copy link
Contributor Author

Hi.

I am not familiar with vagrant, I just know it by name and know it can be used to run VM.
I will take a look to it then! Thank you for the pointers.

Best regards.

@kakkoyun
Copy link
Contributor

kakkoyun commented Dec 1, 2021

@eiffel-fl @afbjorklund Hey, it would be amazing to have this. Any progress?

@k8s-ci-robot k8s-ci-robot removed the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 1, 2021
@k8s-ci-robot
Copy link
Contributor

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


  • If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
  • If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
  • If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
  • Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Dec 1, 2021
@eiffel-fl
Copy link
Contributor Author

Hi.

@kakkoyun This is not merged for the moment, but I am using this image (plus some options) as my daily minikube image.
If you want to use it, you can clone https://github.com/kinvolk/minikube.git and jump on branch francis/upstream/use-kernel-5.10 (there are others branches based on 5.10 kernel with specific CONFIG_ enabled).
Then run the following to build the iso:

make buildroot-image
make out/minikube.iso

Please be patient because this process involves buildroot which fetches all the software sources present in the image and compiles them on your machine, so it can take time (half a day depending on your machine, I would advise you to run it during the night or weekend).
The final instruction executed by the Makefile should be:

mv ./out/buildroot/output/images/rootfs.iso9660 ./out/minikube.iso

You should normally end with the image in out/minikube.iso, just use it like this:

minikube start --driver kvm --iso-url=file://$(pwd)/out/minikube.iso
# To test if kernel 5.10 is used, test the following:
minikube ssh 'uname -r'
5.10.57

If 5.10.57 is printed, you are all good and using kernel 5.10 as minikube kernel!
Please, do not use this in production, as I cannot guarantee it is bugless and OK regarding security.

Best regards.

@eiffel-fl
Copy link
Contributor Author

@k8s-ci-robot Please, add back the "cncf-cla: yes" flag because I already signed this document.

@kakkoyun
Copy link
Contributor

kakkoyun commented Dec 6, 2021

Hey @eiffel-fl, I'm trying to build an image on kinvolk@686450f and it's failing with the following error:

Failed to generate BTF for vmlinux
Try to disable CONFIG_DEBUG_INFO_BTF
make[3]: *** [vmlinux] Error 1
Makefile:1179: recipe for target 'vmlinux' failed
make[3]: Leaving directory '/mnt/out/buildroot/output/build/linux-5.10.57'
make[2]: *** [/mnt/out/buildroot/output/build/linux-5.10.57/.stamp_built] Error 2
package/pkg-generic.mk:247: recipe for target '/mnt/out/buildroot/output/build/linux-5.10.57/.stamp_built' failed
make[2]: Leaving directory '/mnt/out/buildroot'
make[1]: *** [minikube_iso] Error 2
Makefile:286: recipe for target 'minikube_iso' failed
make[1]: Leaving directory '/mnt'
Makefile:326: recipe for target 'out/minikube.iso' failed
make: *** [out/minikube.iso] Error 2
make: *** [Makefile:328: out/minikube.iso] Error 2

Do you have any pointers? I'd appreciate it.
Also, do you happen to have a prebuilt version of this image in somewhere public? It would be even more awesome to validate if things actually work for me? Happy to invest further to push this forward.

@eiffel-fl
Copy link
Contributor Author

Hi @kakkoyun!

Your error is "normal", you want to compile with CONFIG_DEBUG_INFO_BTF but you need pahole and buildroot does not know about pahole.
In branch alban/kernel-5.10-options, I added more commits to have a "BPF ready" kernel.
Sorry, I should have shared with you this branch sooner but I shared the "generic" one.

Regarding the image, sadly, I do not have it published "ready to use" somewhere...
As it is a big file, I do not really know where I can publish this, but I will discuss this internally!

If you have problem testing this, do not hesitate to ask for help.

Best regards.

@eiffel-fl
Copy link
Contributor Author

@kakkoyun I uploaded the image here:
https://github.com/eiffel-fl/minikube/releases/tag/v42-prerelease

@kakkoyun
Copy link
Contributor

kakkoyun commented Dec 6, 2021

@eiffel-fl Amazing! Thanks for the help. I'll check it out.

And I'm looking forward to having a completely BPF enabled image in the main branch. We don't have any alternatives to minikube to test eBPF based programs.

Let me know if I can help in any way.

cc @afbjorklund

@eiffel-fl
Copy link
Contributor Author

Hi @kakkoyun.

I think we are on the same wave length regarding having a "cutting-edge" BPF ready kernel in default minikube image.
Nonetheless, this does not depend on me and I would accept if minikube maintainers do not want such a feature to be included.
Also, we can still maintain such a feature somewhere.

When you would have tried the ISO, I would be happy to add a line like this to my commit message:

Tested-by: kakkoyun_real_name <[email protected]>

Best regards.

@kakkoyun
Copy link
Contributor

kakkoyun commented Dec 7, 2021

The ISO works perfectly. I was able to load the BTF enabled eBPF program and made it work with minikube. I will continue further tests and let you know if anything emerges. Thanks again @eiffel-fl and @alban. Your work is much appreciated.

@eiffel-fl
Copy link
Contributor Author

You are welcome :D
I needed to do this modification, better to publish it so it can empower the community!

@sharifelgamal
Copy link
Collaborator

Ah, looks like our bot doesn't have access to push to this PR, which is totally fine.

Could you run

sed -i 's/ISO_VERSION ?= .*/ISO_VERSION ?= v1.26.0-1654280115-12707/' Makefile
sed -i 's|isoBucket := .*|isoBucket := "minikube-builds/iso/12707"|' pkg/minikube/download/iso.go

and push up the changes? That way we can run our test suites with your ISOs.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 7, 2022
@eiffel-fl
Copy link
Contributor Author

/retest-required

Fixes: d92f42a ("fix the minikube.iso make target")
Signed-off-by: Francis Laniel <[email protected]>
This is needed by sysdig 0.27.1.
It is based on the following patch:
https://lore.kernel.org/buildroot/[email protected]/T/#u

Signed-off-by: Francis Laniel <[email protected]>
This package is used in place of sysdig 0.23.1 which comes with buildroot.
Indeed, sysdig 0.23.1 does not compile with recent kernel.

It is based on the following patch:
https://lore.kernel.org/buildroot/[email protected]/T/#u

Signed-off-by: Francis Laniel <[email protected]>
This commit jumps kernel version used in minikube.iso to 5.10.
To do so, the following kernel configuration options were added to
linux_*_defconfig:
- CONFIG_TMPFS to be able to use mount -t tmpfs in init.
- CONFIG_PCI to have network.
- CONFIG_BRIDGE_NETFILTER to have /proc/sys/net/bridge/bridge-nf-call-iptables
which is needed by kubeadm.

CONFIG_* relaed to vbox were added to linux_x86_64_defconfig and as consequence
vbox related packages were removed since vbox modules are available in
upstream kernel.

To compile falco module, CONFIG_FTRACE_* were added to linux_aarch64_defconfig.

Signed-off-by: Francis Laniel <[email protected]>
CHANGELOG file is created by a jenkins related script which I do not run when
building the image myself outside of CI

Signed-off-by: Francis Laniel <[email protected]>
It now uses the output-* directory corresponding to the target architecture
under out/buildroot.

Fixes: d1829f2 ("small fixes")
Signed-off-by: Francis Laniel <[email protected]>
@eiffel-fl eiffel-fl force-pushed the francis/upstream/use-kernel-5.10 branch from 158e15a to c0a0f45 Compare June 7, 2022 09:36
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 7, 2022
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12707) |
+----------------+----------+---------------------+
| minikube start | 52.3s    | 52.1s               |
| enable ingress | 27.3s    | 28.4s               |
+----------------+----------+---------------------+

Times for minikube (PR 12707) start: 53.6s 51.7s 52.3s 51.9s 51.3s
Times for minikube start: 54.2s 52.7s 50.9s 51.9s 51.9s

Times for minikube ingress: 26.1s 26.1s 29.2s 29.2s 26.1s
Times for minikube (PR 12707) ingress: 29.1s 29.1s 29.6s 25.2s 29.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12707) |
+----------------+----------+---------------------+
| minikube start | 25.5s    | 24.6s               |
| enable ingress | 23.8s    | 23.1s               |
+----------------+----------+---------------------+

Times for minikube start: 28.9s 23.7s 25.4s 26.0s 23.7s
Times for minikube (PR 12707) start: 24.3s 23.9s 25.0s 24.9s 24.9s

Times for minikube ingress: 26.4s 23.9s 23.9s 22.5s 22.4s
Times for minikube (PR 12707) ingress: 23.0s 22.9s 22.4s 24.0s 23.4s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 12707) |
+----------------+----------+---------------------+
| minikube start | 30.0s    | 31.8s               |
| enable ingress | 23.5s    | 22.0s               |
+----------------+----------+---------------------+

Times for minikube start: 34.0s 29.6s 28.9s 29.0s 28.5s
Times for minikube (PR 12707) start: 32.1s 28.7s 40.5s 28.5s 29.3s

Times for minikube ingress: 18.9s 21.9s 21.9s 32.4s 22.4s
Times for minikube (PR 12707) ingress: 18.9s 22.4s 22.9s 22.4s 23.5s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Windows TestMinikubeProfile (gopogh) n/a
Docker_Windows TestNoKubernetes/serial/Stop (gopogh) 0.00 (chart)
KVM_Linux_containerd TestStoppedBinaryUpgrade/Upgrade (gopogh) 7.83 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/Stop (gopogh) 16.36 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/AddonExistsAfterStop (gopogh) 16.97 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/EnableAddonAfterStop (gopogh) 16.97 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/SecondStart (gopogh) 16.97 (chart)
Docker_Cloud_Shell TestStartStop/group/cloud-shell/serial/UserAppExistsAfterStop (gopogh) 17.47 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/DeployApp (gopogh) 30.58 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/SecondStart (gopogh) 30.58 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/UserAppExistsAfterStop (gopogh) 31.15 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/FirstStart (gopogh) 32.26 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/DeployApp (gopogh) 33.06 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/FirstStart (gopogh) 33.06 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh) 33.61 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/SecondStart (gopogh) 34.68 (chart)
Docker_Linux TestFunctional/serial/ComponentHealth (gopogh) 36.88 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/DeployApp (gopogh) 49.59 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/SecondStart (gopogh) 50.00 (chart)
Docker_Linux TestNetworkPlugins/group/false/DNS (gopogh) 50.00 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/FirstStart (gopogh) 50.41 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/UserAppExistsAfterStop (gopogh) 50.41 (chart)
Docker_Windows TestAddons/Setup (gopogh) 51.85 (chart)
Docker_Windows TestErrorSpam/setup (gopogh) 51.85 (chart)
Docker_Windows TestFunctional/parallel/CertSync (gopogh) 51.85 (chart)
Docker_Windows TestFunctional/parallel/CpCmd (gopogh) 51.85 (chart)
Docker_Windows TestFunctional/parallel/DockerEnv/powershell (gopogh) 51.85 (chart)
Docker_Windows TestFunctional/parallel/FileSync (gopogh) 51.85 (chart)
Docker_Windows TestFunctional/parallel/ImageCommands/ImageBuild (gopogh) 51.85 (chart)
Docker_Windows TestFunctional/parallel/ImageCommands/ImageListJson (gopogh) 51.85 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@eiffel-fl
Copy link
Contributor Author

It seems some tests are not passing, is there a way for me to run the test locally it would be easier to debug them?

@sharifelgamal
Copy link
Collaborator

Truthfully, a lot of our tests are flaky (and some just outright fail), it's something we're working on but is definitely an issue. This test run actually looks quite good comparatively. Since your change only affects the VM ISO, the docker driver test failures are entirely unrelated. I apologize for the confusion this causes.

If you'd still like to run the tests locally, you can run

env TEST_ARGS='-minikube-start-args=--driver=kvm' make integration 

and you can supply any go test argument in that test arg env var, e.g. -test.run TestFunctional.

With the change I asked to make above, your ISO is the default ISO to run minikube against, so that's easy. We're actually going to keep it that way when we merge.

If you're comfortable having this merged, (and so is @afbjorklund), we'll do so today.

@eiffel-fl
Copy link
Contributor Author

eiffel-fl commented Jun 8, 2022

Truthfully, a lot of our tests are flaky (and some just outright fail), it's something we're working on but is definitely an issue. This test run actually looks quite good comparatively. Since your change only affects the VM ISO, the docker driver test failures are entirely unrelated. I apologize for the confusion this causes.

No problem, I also have flaky test in my CI so I understand totally and I thank you for your help :D.

I will run the tests related to --driver=kvm2 during this night and I will come back with the results and try to understand them to see if this contribution does not break too much things :D (for now I see a lot of messages regarding client.crt).

@sharifelgamal
Copy link
Collaborator

sharifelgamal commented Jun 8, 2022

If you're talking about

05:54:14 E0608 12:54:15.341885    7971 cert_rotation.go:168] key failed with : open /home/jenkins/minikube-integration/[...]/client.crt: no such file or directory

then don't worry about it. It has something to with cert rotation in our tests and it's completely harmless, just annoying since it clogs up the logs.

Copy link
Collaborator

@sharifelgamal sharifelgamal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks good to me. Thanks again for sticking with this!

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eiffel-fl, sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 8, 2022
@eiffel-fl
Copy link
Contributor Author

Hi!

Thank you for the approval!

Just for information, here are the failed tests when I ran them on my machine:
francis@pwmachine:~/Codes/kinvolk/minikube-git$ grep 'FAIL:' out/testout_c0a0f4542.txt
--- FAIL: TestAddons (43.66s)
    --- FAIL: TestAddons/Setup (42.73s)
--- FAIL: TestErrorSpam (45.71s)
    --- FAIL: TestErrorSpam/setup (30.69s)
--- FAIL: TestFunctional (139.45s)
    --- FAIL: TestFunctional/serial (87.83s)
        --- FAIL: TestFunctional/serial/CopySyncFile (0.00s)
        --- FAIL: TestFunctional/serial/StartWithProxy (47.83s)
        --- FAIL: TestFunctional/serial/MinikubeKubectlCmdDirectly (0.95s)
        --- FAIL: TestFunctional/serial/LogsCmd (0.71s)
        --- FAIL: TestFunctional/serial/LogsFileCmd (0.70s)
    --- FAIL: TestFunctional/parallel (0.00s)
        --- FAIL: TestFunctional/parallel/ConfigCmd (0.24s)
        --- FAIL: TestFunctional/parallel/MountCmd (3.35s)
            --- FAIL: TestFunctional/parallel/MountCmd/any-port (1.84s)
            --- FAIL: TestFunctional/parallel/MountCmd/specific-port (1.50s)
        --- FAIL: TestFunctional/parallel/FileSync (1.26s)
        --- FAIL: TestFunctional/parallel/PersistentVolumeClaim (6.26s)
        --- FAIL: TestFunctional/parallel/CertSync (2.26s)
        --- FAIL: TestFunctional/parallel/CpCmd (0.54s)
        --- FAIL: TestFunctional/parallel/UpdateContextCmd (0.00s)
            --- FAIL: TestFunctional/parallel/UpdateContextCmd/no_changes (0.07s)
            --- FAIL: TestFunctional/parallel/UpdateContextCmd/no_clusters (0.07s)
            --- FAIL: TestFunctional/parallel/UpdateContextCmd/no_minikube_cluster (0.06s)
        --- FAIL: TestFunctional/parallel/MySQL (1.37s)
        --- FAIL: TestFunctional/parallel/ImageCommands (12.94s)
            --- FAIL: TestFunctional/parallel/ImageCommands/ImageBuild (0.45s)
--- FAIL: TestIngressAddonLegacy (667.98s)
    --- FAIL: TestIngressAddonLegacy/serial (581.08s)
        --- FAIL: TestIngressAddonLegacy/serial/ValidateIngressAddons (554.34s)
--- FAIL: TestErrorJSONOutput (0.17s)
--- FAIL: TestMultiNode (491.63s)
    --- FAIL: TestMultiNode/serial (488.87s)
        --- FAIL: TestMultiNode/serial/DeployApp2Nodes (1.74s)
        --- FAIL: TestMultiNode/serial/PingHostFrom2Pods (1.34s)
        --- FAIL: TestMultiNode/serial/CopyFile (6.01s)
--- FAIL: TestSkaffold (40.97s)
--- FAIL: TestKVMDriverInstallOrUpdate (0.00s)
--- FAIL: TestCertExpiration (257.26s)
--- FAIL: TestStartStop (2631.35s)
    --- FAIL: TestStartStop/group (0.00s)
        --- FAIL: TestStartStop/group/no-preload (495.62s)
            --- FAIL: TestStartStop/group/no-preload/serial (495.48s)
                --- FAIL: TestStartStop/group/no-preload/serial/DeployApp (2.12s)
        --- FAIL: TestStartStop/group/embed-certs (450.46s)
            --- FAIL: TestStartStop/group/embed-certs/serial (450.29s)
                --- FAIL: TestStartStop/group/embed-certs/serial/DeployApp (2.31s)
        --- FAIL: TestStartStop/group/old-k8s-version (568.11s)
            --- FAIL: TestStartStop/group/old-k8s-version/serial (567.96s)
                --- FAIL: TestStartStop/group/old-k8s-version/serial/DeployApp (2.08s)
        --- FAIL: TestStartStop/group/default-k8s-different-port (459.77s)
            --- FAIL: TestStartStop/group/default-k8s-different-port/serial (459.61s)
                --- FAIL: TestStartStop/group/default-k8s-different-port/serial/DeployApp (2.13s)
--- FAIL: TestNetworkPlugins (2685.92s)
    --- FAIL: TestNetworkPlugins/group (0.00s)
        --- FAIL: TestNetworkPlugins/group/auto (1851.77s)
            --- FAIL: TestNetworkPlugins/group/auto/NetCatPod (1800.28s)
        --- FAIL: TestNetworkPlugins/group/custom-flannel (0.18s)
            --- FAIL: TestNetworkPlugins/group/custom-flannel/Start (0.00s)
        --- FAIL: TestNetworkPlugins/group/false (0.18s)
            --- FAIL: TestNetworkPlugins/group/false/Start (0.00s)
        --- FAIL: TestNetworkPlugins/group/enable-default-cni (0.18s)
            --- FAIL: TestNetworkPlugins/group/enable-default-cni/Start (0.00s)
        --- FAIL: TestNetworkPlugins/group/flannel (0.20s)
            --- FAIL: TestNetworkPlugins/group/flannel/Start (0.00s)
        --- FAIL: TestNetworkPlugins/group/bridge (0.18s)
            --- FAIL: TestNetworkPlugins/group/bridge/Start (0.00s)
        --- FAIL: TestNetworkPlugins/group/kubenet (0.18s)
            --- FAIL: TestNetworkPlugins/group/kubenet/Start (0.00s)
        --- FAIL: TestNetworkPlugins/group/kindnet (1874.34s)
            --- FAIL: TestNetworkPlugins/group/kindnet/NetCatPod (1800.31s)
        --- FAIL: TestNetworkPlugins/group/calico (1906.93s)
            --- FAIL: TestNetworkPlugins/group/calico/NetCatPod (1800.33s)
        --- FAIL: TestNetworkPlugins/group/cilium (1929.92s)
            --- FAIL: TestNetworkPlugins/group/cilium/NetCatPod (1800.71s)

Best regards.

@sharifelgamal sharifelgamal merged commit 34882bf into kubernetes:master Jun 9, 2022
@eiffel-fl
Copy link
Contributor Author

Thank you a lot for the merge :D

@kakkoyun
Copy link
Contributor

kakkoyun commented Jun 9, 2022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants