Following the documentation for minikube deployment doesn't work #1941
Comments
|
Thanks for filing the issue Tom! Just to add to Tom's issue. I think the bug is in the falco-driver-loader script in the if statement here: falco/scripts/falco-driver-loader Line 104 in a5d3663 Falco uses the /etc/VERSION file to detect it is running in minikube. Unfortunately, it never gets to that check in the if statement above because I think minikube instances also have the os-release file as well, so it pops out of that if statement without detecting it's running in minikube. The old sysdig-probe-loader script dealt with this by breaking out the /etc/VERSION check into a separate if statement, which did work, but I'm not sure of the other implications of that. Also, I think minikube requires ebpf to be enabled, but it doesn't look like that is documented in the docs. |
|
Minikube ships its own Falco pre-built driver (see kubernetes/minikube#6560) since it is impossible to build the driver on the fly for Minikube because it doesn't provide a compiler or kernel-headers. Unfortunately, the last update in Minikube was two years ago 👇 https://github.com/kubernetes/minikube/tree/master/deploy/iso/minikube-iso/package/falco-module The driver version they ship should work up to Falco 0.30.x, but not with 0.31.1. |
|
This does not affect only minikube. I have it on Ubuntu 18.04 Downgrading to 0.31.0 restores functionality |
|
@leogr would it be preferable to just align all minikube installs to the ebpf probe and have the init script download that, as I have done in my workaround? I am unsure as to what the "most supported" method of setting up falco is wrt ebpf/kernel driver though. |
|
@ChaosInTheCRD ebpf and kmod have function parity, they are also almost equivalent in performance. Specifically for Minikube, upgrading the driver version directly in their repo would be preferable, so everything will work seamlessly (driver incompatibility does not happen on each Falco release, it's a rare occurrence). If I find some spare time, I'll try to investigate more and eventually open a PR in Minikube to fix that. However, your solution is a valid alternative. Indeed, after looking at again I realized that Btw, I hope we will fix both cases. Having Minikube works both with the kmod and ebpf is for sure the best option. |
Your issue is slightly different. You should be able to install the new driver version by running: falco-driver-loader --cleanThen falco-driver-loaderLet me know if that works. Perhaps we will have to improve the documentation regarding that. |
|
no, it didn't fix the issue but made me understand the problem. so kernel modules unloading/loading issue. I did a reboot to confirm and new version is working after. |
|
Oh yeah, there are cases where This issue happened with Falco 0.31.1 because we introduced a mechanism to detect incompatible driver versions, and this specific Falco version requires a new driver version. Combining these two factors produced the issue, which does not usually happen. |
|
We also experience this issue: We build the driver through driverkit-builder, load the module successfully and start Falco. When using falco 0.31.1 it croaks with the above message. When using Falco 0.31.0 it works fine. This happens on nodes where it replaces the old Falco instances (0.29.1) but also on clean nodes that did not have any modules loaded before this new version (0.31.1) started. Unloading modules cannot be the cause there. Version 0.31.0 does not show this problem. |
Note that 0.31.1 has a newer driver (ie. kernel module) version than 0.31.0.
AFAIK, the problem arises when an old driver is already installed and loaded. Basically, when one previously installed an older version and then installs the 0.31.1, the old driver remains up and running and The workaround is to uninstall the old driver manually before installing the 0.31.1. PS Falco 0.32.0 (not yet released) will come with a fix that forces the driver uninstallation when upgrading to a newer version. |
|
/milestone 0.32.0 |
|
/remove-milestone 0.32.0 |
Hey @jasondellaluce This issue should be actually fixed by 0.32.0 I know @alacuku had the same issue and is testing. Has 0.32.0 worked for you on minikube? Please let us know :) |
|
Hi @leogr, here are my findings on Falco and Minikube. The issue is still present even with There are two options in order to use the latest version of Falco with Minikube:
|
|
Seeing this still with 0.32.0 and Ubuntu 20.04.2 LTS on |
Hey @jrabbit |
|
Meanwhile, I've opened a PR in minikube to bump Falco to 0.32.0 👉 kubernetes/minikube#14329 |
So we're on a full 1.19 K8s cluster so it might be a different bug but we're having trouble getting the script or manually to unload the falco kernel mod. Strangely it seemed like falco kept spawning binaries (which then access the module, blocking rmmod) when the pods were unscheduled? (Maybe this is a k8s quirk i'm not used to?). Let me validate if the nodes that succeeded were ones that were freshly spun up (and thus wouldn't have falco kernel mods to remove) and get back to you w/ that. e: So the nodes that do have working pods aren't new, the others are in restart loops with errors in dkms and |
Not sure what is going on. The module is installed on the host, so it is still present after pods get unscheduled. The bug was that 0.31.1 was not able to upgrade the module. The 0.32.0 fixed the issue.
For pods in the restart loop, I guess for some reason the driver is not found on our DBG and the falco-driver-loader script can't build it on the fly. Could you provide some logs? Anyway, I think yours is a different problem. It would be better to open a dedicated issue,. |
|
Got falco working according to @ChaosInTheCRD workaround but had to also delete |
|
cc @alacuku |
@alacuku Thank you! 🙏 Please put |
Describe the bug
Following the documentation for creating a "Falco Learning Environment" unfortunately does not work. Upon using helm to deploy the Falco and getting this error in the pod logs:
Workaround
I went on to try a handful of other virtual machine drivers for minikube to no avail. I resorted to the Kubernetes slack where I got help from @terylt.
As it turns out, it seems that there is a script that runs at startup to try and install the kernel module / ebpf probe necessary to get Falco running in the relevant environment. The bash script seems to do some guesswork to determine what operating sytem it is in, then tries to decide the correct approach to install the kernel module / ebpf probe. This does not currently work correctly for minikube.
to get the script to pass through the logic linked here (and hence correctly determine that it is in a minikube vm), the daemonset must be modified like so, after a
helm templateor akubectl editafter deployment:containers: - name: falco ... volumeMounts: ... - mountPath: /host/etc/VERSION name: etc-fs readOnly: true ... volumes: ... - name: etc-fs hostPath: path: /etc/VERSION ...The daemonset also needs to have eBPF enabled, as otherwise it continues to fail. This can either be done by setting the env var on the falco pod in the manifest:
env: - name: FALCO_BPF_PROBEor by enabling eBPF in the helm values file:
once these two steps have been taken, the pod should turn to a
READYstate:How to reproduce it
Follow the documentation for creating a learning environment with minikube
Expected behaviour
I feel others thoughts might be mixed on this... but to me it doesn't seem unreasonable to request the user to specify the environment they are deploying to (e.g. GKE, minikube, kind etc.) in the form of an env var or a command-line argument. This way, there is no need for a script to exist or be maintained when inevitably, situations arise that break the mechanisms in which the script tries to decipher the environment it is in.
Environment
0.31.1
Minikube (v1.25.0 - commit: 3edf4801f38f3916c9ff96af4284df905a347c86)
Helm on Kubernetes
The text was updated successfully, but these errors were encountered: