Falco support

[zmarouf]

What I'd like:
I'd like to run Falco on Bottlerocket.
I already found issues and PR related to this #454, #701, #680

I don't have much experience with low level kernel and filesystem activities but I'm willing to try if I can get some pointers.

I understand that since #701 landed, I can find the Kernel source in /usr/src/kernels/ (e.g /usr/src/kernels/5.4.16/ for current EKS 1.15)

And @sam-aws mentioned that building the probe should be possible in the admin container (#454 (comment))

Which approach would you recommend?

• Using a custom admin container with the build tools installed?

• I'm also wondering if grabbing the Kernel as an archive from /usr/src/kernels/ and the config data from /proc/config.gz would be enough to build the probe outside of the admin container?

Thanks in advance for any pointer/input on the subject.

[sam-aws]

Hi @zmarouf,
Both of those options are fine; the main requirement for building against the kernel sources is to have the same toolchain. The admin container doesn't have that by default since it's in essence an Amazon Linux 2 container which references a different kernel.
Luckily the toolchain used to compile the Bottlerocket kernel is available in the SDK used to compile the image. You can obtain that by downloading it from the cache, eg:
curl https://cache.bottlerocket.aws/bottlerocket/sdk-x86_64:v0.10.0.tar.gz | gunzip | docker load

Issue #819 gives an example of how to consume that, and also a quick workaround for an issue that still has a fix coming.

The SDK image is pretty chunky (around 2GB), so we're looking into trimming it down and reorganising it so we can have a "development SDK" that only includes what we would need to, for example, build a kernel module.

[bcressey]

As a start, we should document a way to build the Falco module on a Bottlerocket host, even if it's not fully automated.

[faarshad]

Falco-0.26.1 is working on bottlerocket-1.0.1 via dkms install at boot time of falco pod

My setup:

• Bottlerocket OS 1.0.1 on EKS
• Deployed Falco chart ver 1.5.0 via Helm Install
• securityContext.privileged: true on the Pod

Logs showing that falco-driver-loader first tries it with gcc-5 and fails but works with gcc-8

* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area.....
'/tmp/falco-dkms-make'.......(bad exit status: 2)
* Running dkms build failed, dumping /var/lib/dkms/falco/2aa88dcf6243982697811df4c1b484bcbe9488a2/build/make.log (with GCC /usr/bin/gcc)
DKMS make.log for falco-2aa88dcf6243982697811df4c1b484bcbe9488a2 for kernel 5.4.50 (x86_64)
Tue Oct 13 17:57:51 UTC 2020
...
...
...
* Trying to dkms install falco module with GCC /usr/bin/gcc-8
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area.....
'/tmp/falco-dkms-make'........................
cleaning build area.....

DKMS: build completed.

falco.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.50/kernel/extra/

depmod...

DKMS: install completed.
* falco module installed in dkms, trying to insmod
* Success: falco module found and loaded in dkms
Tue Oct 13 17:59:29 2020: Falco version 0.26.1 (driver version 2aa88dcf6243982697811df4c1b484bcbe9488a2)
Tue Oct 13 17:59:29 2020: Falco initialized with configuration file /etc/falco/falco.yaml

Question for bottlerocket team:

• are there any plans in future to stop this sort of dkms based support from OS perspective?
• also trying to see the criticality of the dkms status warning

root@ops-falco-5lrnc:/# dkms status
falco, 2aa88dcf6243982697811df4c1b484bcbe9488a2, 5.4.50, x86_64: installed (WARNING! Diff between built and installed module!)

[bcressey]

We don't plan to stop DKMS from working in general. Enabling solutions like that are why we provide the kernel development files in the host OS. We know there are use cases where out-of-tree modules are a popular solution.

In #813 we're looking at a feature to enable kernel lockdown. Lockdown would be set to "none" by default but could be configured through settings at launch. In "integrity" mode it would block out-of-tree modules from being loaded, and DKMS would stop working.

This will allow workloads that don't need these modules to assert that they should not be loaded, trading off some functionality for improved security.

[faarshad]

Thank you for mentioning the kernel LSM feature. not sure if that feature can enable whitelisting/blacklisting the out-of-tree modules by name instead of module signing.

[bcressey]

Once we have lockdown as an option, I'd recommend evaluating Falco's eBPF probe driver instead.

I've confirmed that this works with Bottlerocket v1.0.2 --

helm install falco falcosecurity/falco \
  --set docker.enabled=false \
  --set ebpf.enabled=true

Since the BPF probe doesn't rely on kernel modules, you don't need to align with the toolchain used to build the host kernel, so it's likely to be more robust over time.

[faarshad]

thank you @bcressey - for folks migrating towards bottlerocket, they might need to run falco-ebpf on both AL2 and bottlerocket for easier migration. not sure about ebpf support on AL2 for falco today.

The second point is about performance difference of falco when moving from bottlerocket-kernel-module mode to bottlerocket-ebpf mode. any anecdotal evidence you know of?

[bcressey]

We're using essentially the same kernel as the one that's available in Amazon Linux 2 through the kernel-ng extra, so I'd recommend that for parity.

The only performance comparison I've seen is in the blog post that announced the feature:
https://sysdig.com/blog/sysdig-and-falco-now-powered-by-ebpf/

[gregdek]

Given the conversation here, it seems that we expect this to work; changing this to a docs issue.

[gregdek]

Putting this back into triage to answer the question "how do we document this properly?" We know that Bottlerocket supports the necessary eBPF functionality, but do we document Falco specifically?

[gregdek]

OK, there's nothing here to document from our perspective; we believe that the helm command documented above (#862 (comment)) is sufficient to demonstrate that this issue is resolved.

Thanks @zmarouf and @farshad-hobsons for following along; please feel free to open a new issue or discussion if there are more questions.