-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use kernel 5.10 for minikube.iso. #12707
Use kernel 5.10 for minikube.iso. #12707
Conversation
Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA. It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Welcome @eiffel-fl! |
Hi @eiffel-fl. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Can one of the admins verify this patch? |
They are parallel tracks, it is possible to upgrade software in Buildroot without affect Ubuntu* versions and vice-versa We have talked about upgrading the kernel before, but had to rollback when issues were found later during testing... You can find default BR kernel versions here: https://github.com/buildroot/buildroot/blob/2021.02.x/linux/linux.hash * currently the KIC base uses Ubuntu 20.04 LTS, which normally ships with a Linux 5.4 kernel (or 5.11 in HWE) |
Thank you for your quick answer. How can I use the Ubuntu based image you are talking?
|
That proposal you are talking about was just that, replacing the custom ISO with a "standard" one. The closest thing to being actually available is the Vagrant file, as described in PR #12185 The idea was always to make a Linux 5 version too of the ISO, after it was updated for ARM... |
Hi. I am not familiar with Best regards. |
@eiffel-fl @afbjorklund Hey, it would be amazing to have this. Any progress? |
Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA. It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Hi. @kakkoyun This is not merged for the moment, but I am using this image (plus some options) as my daily minikube image. make buildroot-image
make out/minikube.iso Please be patient because this process involves buildroot which fetches all the software sources present in the image and compiles them on your machine, so it can take time (half a day depending on your machine, I would advise you to run it during the night or weekend). mv ./out/buildroot/output/images/rootfs.iso9660 ./out/minikube.iso You should normally end with the image in minikube start --driver kvm --iso-url=file://$(pwd)/out/minikube.iso
# To test if kernel 5.10 is used, test the following:
minikube ssh 'uname -r'
5.10.57 If Best regards. |
@k8s-ci-robot Please, add back the "cncf-cla: yes" flag because I already signed this document. |
Hey @eiffel-fl, I'm trying to build an image on kinvolk@686450f and it's failing with the following error:
Do you have any pointers? I'd appreciate it. |
Hi @kakkoyun! Your error is "normal", you want to compile with Regarding the image, sadly, I do not have it published "ready to use" somewhere... If you have problem testing this, do not hesitate to ask for help. Best regards. |
@kakkoyun I uploaded the image here: |
@eiffel-fl Amazing! Thanks for the help. I'll check it out. And I'm looking forward to having a completely BPF enabled image in the main branch. We don't have any alternatives to Let me know if I can help in any way. cc @afbjorklund |
Hi @kakkoyun. I think we are on the same wave length regarding having a "cutting-edge" BPF ready kernel in default When you would have tried the ISO, I would be happy to add a line like this to my commit message:
Best regards. |
The ISO works perfectly. I was able to load the BTF enabled eBPF program and made it work with |
You are welcome :D |
Ah, looks like our bot doesn't have access to push to this PR, which is totally fine. Could you run
and push up the changes? That way we can run our test suites with your ISOs. |
/retest-required |
Fixes: d92f42a ("fix the minikube.iso make target") Signed-off-by: Francis Laniel <[email protected]>
This is needed by sysdig 0.27.1. It is based on the following patch: https://lore.kernel.org/buildroot/[email protected]/T/#u Signed-off-by: Francis Laniel <[email protected]>
This package is used in place of sysdig 0.23.1 which comes with buildroot. Indeed, sysdig 0.23.1 does not compile with recent kernel. It is based on the following patch: https://lore.kernel.org/buildroot/[email protected]/T/#u Signed-off-by: Francis Laniel <[email protected]>
This commit jumps kernel version used in minikube.iso to 5.10. To do so, the following kernel configuration options were added to linux_*_defconfig: - CONFIG_TMPFS to be able to use mount -t tmpfs in init. - CONFIG_PCI to have network. - CONFIG_BRIDGE_NETFILTER to have /proc/sys/net/bridge/bridge-nf-call-iptables which is needed by kubeadm. CONFIG_* relaed to vbox were added to linux_x86_64_defconfig and as consequence vbox related packages were removed since vbox modules are available in upstream kernel. To compile falco module, CONFIG_FTRACE_* were added to linux_aarch64_defconfig. Signed-off-by: Francis Laniel <[email protected]>
CHANGELOG file is created by a jenkins related script which I do not run when building the image myself outside of CI Signed-off-by: Francis Laniel <[email protected]>
It now uses the output-* directory corresponding to the target architecture under out/buildroot. Fixes: d1829f2 ("small fixes") Signed-off-by: Francis Laniel <[email protected]>
Signed-off-by: Francis Laniel <[email protected]>
158e15a
to
c0a0f45
Compare
kvm2 driver with docker runtime
Times for minikube (PR 12707) start: 53.6s 51.7s 52.3s 51.9s 51.3s Times for minikube ingress: 26.1s 26.1s 29.2s 29.2s 26.1s docker driver with docker runtime
Times for minikube start: 28.9s 23.7s 25.4s 26.0s 23.7s Times for minikube ingress: 26.4s 23.9s 23.9s 22.5s 22.4s docker driver with containerd runtime
Times for minikube start: 34.0s 29.6s 28.9s 29.0s 28.5s Times for minikube ingress: 18.9s 21.9s 21.9s 32.4s 22.4s |
These are the flake rates of all failed tests.
Too many tests failed - See test logs for more details. To see the flake rates of all tests by environment, click here. |
It seems some tests are not passing, is there a way for me to run the test locally it would be easier to debug them? |
Truthfully, a lot of our tests are flaky (and some just outright fail), it's something we're working on but is definitely an issue. This test run actually looks quite good comparatively. Since your change only affects the VM ISO, the docker driver test failures are entirely unrelated. I apologize for the confusion this causes. If you'd still like to run the tests locally, you can run
and you can supply any go test argument in that test arg env var, e.g. With the change I asked to make above, your ISO is the default ISO to run minikube against, so that's easy. We're actually going to keep it that way when we merge. If you're comfortable having this merged, (and so is @afbjorklund), we'll do so today. |
No problem, I also have flaky test in my CI so I understand totally and I thank you for your help :D. I will run the tests related to |
If you're talking about
then don't worry about it. It has something to with cert rotation in our tests and it's completely harmless, just annoying since it clogs up the logs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change looks good to me. Thanks again for sticking with this!
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: eiffel-fl, sharifelgamal The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Hi! Thank you for the approval! Just for information, here are the failed tests when I ran them on my machine:francis@pwmachine:~/Codes/kinvolk/minikube-git$ grep 'FAIL:' out/testout_c0a0f4542.txt
--- FAIL: TestAddons (43.66s)
--- FAIL: TestAddons/Setup (42.73s)
--- FAIL: TestErrorSpam (45.71s)
--- FAIL: TestErrorSpam/setup (30.69s)
--- FAIL: TestFunctional (139.45s)
--- FAIL: TestFunctional/serial (87.83s)
--- FAIL: TestFunctional/serial/CopySyncFile (0.00s)
--- FAIL: TestFunctional/serial/StartWithProxy (47.83s)
--- FAIL: TestFunctional/serial/MinikubeKubectlCmdDirectly (0.95s)
--- FAIL: TestFunctional/serial/LogsCmd (0.71s)
--- FAIL: TestFunctional/serial/LogsFileCmd (0.70s)
--- FAIL: TestFunctional/parallel (0.00s)
--- FAIL: TestFunctional/parallel/ConfigCmd (0.24s)
--- FAIL: TestFunctional/parallel/MountCmd (3.35s)
--- FAIL: TestFunctional/parallel/MountCmd/any-port (1.84s)
--- FAIL: TestFunctional/parallel/MountCmd/specific-port (1.50s)
--- FAIL: TestFunctional/parallel/FileSync (1.26s)
--- FAIL: TestFunctional/parallel/PersistentVolumeClaim (6.26s)
--- FAIL: TestFunctional/parallel/CertSync (2.26s)
--- FAIL: TestFunctional/parallel/CpCmd (0.54s)
--- FAIL: TestFunctional/parallel/UpdateContextCmd (0.00s)
--- FAIL: TestFunctional/parallel/UpdateContextCmd/no_changes (0.07s)
--- FAIL: TestFunctional/parallel/UpdateContextCmd/no_clusters (0.07s)
--- FAIL: TestFunctional/parallel/UpdateContextCmd/no_minikube_cluster (0.06s)
--- FAIL: TestFunctional/parallel/MySQL (1.37s)
--- FAIL: TestFunctional/parallel/ImageCommands (12.94s)
--- FAIL: TestFunctional/parallel/ImageCommands/ImageBuild (0.45s)
--- FAIL: TestIngressAddonLegacy (667.98s)
--- FAIL: TestIngressAddonLegacy/serial (581.08s)
--- FAIL: TestIngressAddonLegacy/serial/ValidateIngressAddons (554.34s)
--- FAIL: TestErrorJSONOutput (0.17s)
--- FAIL: TestMultiNode (491.63s)
--- FAIL: TestMultiNode/serial (488.87s)
--- FAIL: TestMultiNode/serial/DeployApp2Nodes (1.74s)
--- FAIL: TestMultiNode/serial/PingHostFrom2Pods (1.34s)
--- FAIL: TestMultiNode/serial/CopyFile (6.01s)
--- FAIL: TestSkaffold (40.97s)
--- FAIL: TestKVMDriverInstallOrUpdate (0.00s)
--- FAIL: TestCertExpiration (257.26s)
--- FAIL: TestStartStop (2631.35s)
--- FAIL: TestStartStop/group (0.00s)
--- FAIL: TestStartStop/group/no-preload (495.62s)
--- FAIL: TestStartStop/group/no-preload/serial (495.48s)
--- FAIL: TestStartStop/group/no-preload/serial/DeployApp (2.12s)
--- FAIL: TestStartStop/group/embed-certs (450.46s)
--- FAIL: TestStartStop/group/embed-certs/serial (450.29s)
--- FAIL: TestStartStop/group/embed-certs/serial/DeployApp (2.31s)
--- FAIL: TestStartStop/group/old-k8s-version (568.11s)
--- FAIL: TestStartStop/group/old-k8s-version/serial (567.96s)
--- FAIL: TestStartStop/group/old-k8s-version/serial/DeployApp (2.08s)
--- FAIL: TestStartStop/group/default-k8s-different-port (459.77s)
--- FAIL: TestStartStop/group/default-k8s-different-port/serial (459.61s)
--- FAIL: TestStartStop/group/default-k8s-different-port/serial/DeployApp (2.13s)
--- FAIL: TestNetworkPlugins (2685.92s)
--- FAIL: TestNetworkPlugins/group (0.00s)
--- FAIL: TestNetworkPlugins/group/auto (1851.77s)
--- FAIL: TestNetworkPlugins/group/auto/NetCatPod (1800.28s)
--- FAIL: TestNetworkPlugins/group/custom-flannel (0.18s)
--- FAIL: TestNetworkPlugins/group/custom-flannel/Start (0.00s)
--- FAIL: TestNetworkPlugins/group/false (0.18s)
--- FAIL: TestNetworkPlugins/group/false/Start (0.00s)
--- FAIL: TestNetworkPlugins/group/enable-default-cni (0.18s)
--- FAIL: TestNetworkPlugins/group/enable-default-cni/Start (0.00s)
--- FAIL: TestNetworkPlugins/group/flannel (0.20s)
--- FAIL: TestNetworkPlugins/group/flannel/Start (0.00s)
--- FAIL: TestNetworkPlugins/group/bridge (0.18s)
--- FAIL: TestNetworkPlugins/group/bridge/Start (0.00s)
--- FAIL: TestNetworkPlugins/group/kubenet (0.18s)
--- FAIL: TestNetworkPlugins/group/kubenet/Start (0.00s)
--- FAIL: TestNetworkPlugins/group/kindnet (1874.34s)
--- FAIL: TestNetworkPlugins/group/kindnet/NetCatPod (1800.31s)
--- FAIL: TestNetworkPlugins/group/calico (1906.93s)
--- FAIL: TestNetworkPlugins/group/calico/NetCatPod (1800.33s)
--- FAIL: TestNetworkPlugins/group/cilium (1929.92s)
--- FAIL: TestNetworkPlugins/group/cilium/NetCatPod (1800.71s) Best regards. |
Thank you a lot for the merge :D |
Hi.
With this PR, I jumped the kernel version in
minikube.iso
to 5.10.Jumping the actual version of the kernel is something needed by the community (e.g. #10501 or #8556).
The following shows the result:
However, I would like to test it more because there are a lot of parameters and I cannot ensure everything work fine.
Also, #8187 (I gathered inspiration from this once, particularly for
CONFIG_PCI
) was removed after some tests failed to pass.I understand this PR goes maybe in an opposite direction to this
minikube
enhancement process.So, I am open to discussion in the hope to find a solution which suits everyone.
Best regards.