Cannot run falco in Minikube

[anxolerd]

Describe the bug

Falco fails to run in minikube. I have followed the guide https://falco.org/blog/minikube-falco-kernel-module/ and expected to have falco up and running in my minikube VM. However, while helm install command worked fine (at least without any errors), daemonset pod is failing to use prebuilt kernel module.

How to reproduce it

Follow steps from guide: https://falco.org/blog/minikube-falco-kernel-module/

1. Create minikube minikube start. In my case kvm2 driver was used
2. Create separate namespace kubectl create ns falco
3. Add helm repository and install helm --namespace falco install falco falcosecurity/falco
4. Observe logs from falco pod: kubectl -n falco logs -l app=falco -f

Logs say the following:

➜ kubectl -n falco logs -l app=falco -f
* Trying to load a system falco driver, if present
* Trying to find locally a prebuilt falco module for kernel 4.19.107, if present
* Trying to download prebuilt module from https://dl.bintray.com/falcosecurity/driver/96bd9bc560f67742738eb7255aeb4d03046b8045/falco_buildroot_4.19.107_1.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco module and loading it or getting in touch with the Falco community
Tue Jun 23 12:34:46 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Tue Jun 23 12:34:46 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Tue Jun 23 12:34:47 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue Jun 23 12:34:47 2020: Unable to load the driver. Exiting.
Tue Jun 23 12:34:47 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

Pay attention at lines

* Trying to download prebuilt module from https://dl.bintray.com/falcosecurity/driver/96bd9bc560f67742738eb7255aeb4d03046b8045/falco_buildroot_4.19.107_1.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco module and loading it or getting in touch with the Falco community

and

Tue Jun 23 12:34:47 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

Expected behaviour

Pod successfuly ran. Logs are similar to one in blogpost. Module is downloaded and loaded.

Screenshots

Environment

• Falco version:

0.23.0

• Cloud provider or hardware configuration:

minikube version: v1.11.0
commit: 57e2f55f47effe9ce396cea42a1e0eb4f611ebbd```

- OS:
```$ cat /etc/os-release
NAME=Buildroot
VERSION=2019.02.10
ID=buildroot
VERSION_ID=2019.02.10
PRETTY_NAME="Buildroot 2019.02.10"

• Kernel:

$ uname -a
Linux minikube 4.19.107 #1 SMP Thu May 28 15:07:17 PDT 2020 x86_64 GNU/Linux

• Installation method:
  Kubernetes, helm, https://falco.org/blog/minikube-falco-kernel-module/

[leodido]

Hey @afbjorklund, I saw your commits (here) in the minikube repo (thanks, btw) to make it support Falco 0.23 and its driver.

Any hints why minikube v1.11 is not able to find a "system" Falco driver?

[afbjorklund]

Any hints why minikube v1.11 is not able to find a "system" Falco driver?

Not really, seemed to work here:

 minikube version
minikube version: v1.11.0
commit: 57e2f55f47effe9ce396cea42a1e0eb4f611ebbd
 minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ sudo modprobe falco-probe
$ lsmod
Module                  Size  Used by
falco_probe           630784  0
...
$ more /etc/os-release 
NAME=Buildroot
VERSION=2019.02.10
ID=buildroot
VERSION_ID=2019.02.10
PRETTY_NAME="Buildroot 2019.02.10"
$ uname -r
4.19.107
$ ls /lib/modules/4.19.107/extra/falco-probe.ko
/lib/modules/4.19.107/extra/falco-probe.ko

[afbjorklund]

Probably due to ab722c3 and 762ef01 ?

https://github.com/falcosecurity/falco/blob/0.23.0/CHANGELOG.md#major-changes

BREAKING CHANGE: the falco-driver-loader script now references falco-probe.o and falco-probe.ko as falco.o and falco.ko [#1158]

But for some reason, the "driver" and "install_driver" targets were not updated ?

[afbjorklund]

I think I understand what happened.

Minikube v1.11.0 is still using falco-probe 0.21.0, and then it was called "falco-probe.ko"

Minikube v1.12.0 (out next week or so) will use falco 0.23.0, and now it is called "falco.ko".

Maybe I should rename the package ?

[anxolerd]

@afbjorklund, thanks!

Shell snippet from #1276 (comment) helped to get out of CrashLoopBackoff. I'll proceed evaluating falco then)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Issues labeled "cncf", "roadmap" and "help wanted" will not be automatically closed. Please refer to a maintainer to get such label added if you think this should be kept open.

[leogr]

AFAIK it should work now. Could anybody confirm that, please?

[afbjorklund]

minikube version: v1.12.3
FALCO_MODULE_VERSION = 0.23.0
FALCO_MODULE_SYSDIG_VERSION = 96bd9bc560f67742738eb7255aeb4d03046b8045

name: falco
version: 1.2.2
appVersion: 0.24.0

* Setting up /usr/src links from host
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to dkms install falco module
* Running dkms build failed, couldn't find /var/lib/dkms/falco/85c88952b018fdbce2464222c3303229f5bfcfad/build/make.log
* Trying to load a system falco driver, if present
* Success: falco module found and loaded with modprobe

[afbjorklund]

Should update for 1.13
FALCO_MODULE_VERSION = 0.24.0
FALCO_MODULE_SYSDIG_VERSION = 85c88952b018fdbce2464222c3303229f5bfcfad

https://github.com/kubernetes/minikube/tree/master/deploy/iso/minikube-iso/package/falco-module

We still need a variable for the hardcoded tarball location, though. See #1034 (comment)

ExternalProject_Add(
  sysdig
  URL "https://github.com/draios/sysdig/archive/${SYSDIG_VERSION}.tar.gz"
  URL_HASH "${SYSDIG_CHECKSUM}"

[afbjorklund]

See kubernetes/minikube#9068

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Issues labeled "cncf", "roadmap" and "help wanted" will not be automatically closed. Please refer to a maintainer to get such label added if you think this should be kept open.