Supported Kubernetes Flavours

[simar7]

This issue is keeping track of currently supported and unsupported Kubernetes flavours and setups. This list is a work in progress as we try out new Kubernetes flavours to test Tracee with.

• minikube (v1.23.2+)
• microk8s (v1.22.0+)
• k3s (v1.21.4+k3s1+)
• GKE (both ubuntu and COS)
• EKS
• AKS

Deployment YAMLs are available at: https://github.com/aquasecurity/tracee/tree/main/deploy/kubernetes

Instructions to deploy on Kubernetes: https://aquasecurity.github.io/tracee/dev/install/kubernetes/

[itaysk]

making note here that the interesting part of the test is for the OS the the Kubernetes platform ships with, not for k8s itself, since we want to make sure that ebpf works, and that BTF/kernel headers are available and that our ebpf probe successfully compiles and loads.

[simar7]

Seems to be fine running on EKS (thanks to @krol3):
Version: v1.20.7-eks-135321
Container runtime: docker

# tracee-ebpf  --debug
KConfig: warning: could not check enabled kconfig features
(could not read /boot/config-5.4.141-67.229.amzn2.x86_64: stat /boot/config-5.4.141-67.229.amzn2.x86_64: no such file or directory)
OSInfo: KERNEL_RELEASE: 5.4.141-67.229.amzn2.x86_64
OSInfo: ID: alpine
OSInfo: VERSION_ID: 3.14.2
OSInfo: PRETTY_NAME: "Alpine Linux v3.14"
BTF: bpfenv = false, btfenv = false, vmlinux = true
BPF: using embedded BPF object
unpacked CO:RE bpf object file into memory

[simar7]

On GKE:
Version: v1.20.9-gke.1001
Container runtime: docker

# tracee-ebpf --debug
KConfig: warning: could not check enabled kconfig features
(could not read /boot/config-5.4.0-1049-gke: stat /boot/config-5.4.0-1049-gke: no such file or directory)
OSInfo: PRETTY_NAME: "Alpine Linux v3.14"
OSInfo: KERNEL_RELEASE: 5.4.0-1049-gke
OSInfo: ID: alpine
OSInfo: VERSION_ID: 3.14.2
BTF: bpfenv = false, btfenv = false, vmlinux = false
BPF: no BTF file was found or provided, building BPF object
found bpf object file at: /tmp/tracee/tracee.bpf.5_4_0-1049-gke.v0_6_1.o

[krol3]

On AKS:
Version: v1.20.9-aks
Container Runtime: containerd://1.4.8+azure

KConfig: warning: could not check enabled kconfig features
(could not read /boot/config-5.4.0-1056-azure: stat /boot/config-5.4.0-1056-azure: no such file or directory)
OSInfo: ID: alpine
OSInfo: VERSION_ID: 3.14.2
OSInfo: PRETTY_NAME: "Alpine Linux v3.14"
OSInfo: KERNEL_RELEASE: 5.4.0-1056-azure
BTF: bpfenv = false, btfenv = false, vmlinux = false
BPF: no BTF file was found or provided, building BPF object
found bpf object file at: /tmp/tracee/tracee.bpf.5_4_0-1056-azure.v0_6_1.o

[itaysk]

Happy to see that we're in a good shape in these platforms. does the debug info makes sense to you? could you please explain it/fix the code/fix the test so that it's obvious what happened in each run? Also, I want to make sure that we tested different flavors those cloud distros provide, can you make sure of that?
finally, did you check that e2e with falco worked in each case?

[simar7]

Happy to see that we're in a good shape in these platforms. does the debug info makes sense to you?

More or less. Although I'm not sure if the "pretty name" is correct. It seems to be picking it up from the container image instead of the host.

could you please explain it/fix the code/fix the test so that it's obvious what happened in each run?

What does this mean? I just ran tracee out of the box with a debug flag. There aren't any additional tests or code.

Also, I want to make sure that we tested different flavors those cloud distros provide, can you make sure of that?

Could you list what would be those?

finally, did you check that e2e with falco worked in each case?

We didn't. That should be tracked separately of this regardless.

[itaysk]

I wrote falco but meant falcosidekick, which is part of the core experience of k8s install

[simar7]

I wrote falco but meant falcosidekick, which is part of the core experience of k8s install

Ah yes - it works fine. Here's a snippet:

kubectl logs -f tracee-webhook-66d4dd88c4-9gq52
2021/09/28 23:21:15 [INFO]  : Enabled Outputs : []
2021/09/28 23:21:15 [INFO]  : Falco Sidekick is up and listening on :2801
2021/09/28 23:21:15 [INFO]  : Debug mode : true
2021/09/28 23:23:00 [DEBUG] : Falco's payload : {"output":"Rule \"Anti-Debugging\" detection:\n map[]","priority":"","rule":"Anti-Debugging","time":"2021-09-28T23:23:00Z","output_fields":{"value":0}}

[simar7]

Besides what we've tested so far, I believe we're just left with testing various cloud distributions. So far I've tested amazon linux (EKS) and ubuntu (GKE and AKS).

I think there's some value in standardizing this as an automated test harness. That way we can spin up tracee across different platforms/distros/k8s versions in an automated way. Could be as simple as ssh-ing in, kubectl apply, invoke sigs, assert logs.

We could add this to our long running test cases (think of them as integration/e2e tests across different platforms)

What do you think? @itaysk @danielpacak?

[itaysk]

Also, I want to make sure that we tested different flavors those cloud distros provide, can you make sure of that?

Could you list what would be those?

For example, when you create GKE, you can choose Ubuntu or COS (and also Docker or containerd). IDK about the others but worth checking.

[itaysk]

I think there's some value in standardizing this as an automated test harness. That way we can spin up tracee across different platforms/distros/k8s versions in an automated way. Could be as simple as ssh-ing in, kubectl apply, invoke sigs, assert logs.

Hard to say no to tests :) I think that prior to this we should have a decent e2e test for tracee, then we can discuss testing k8s distros since I think that would be slightly more complicated

[simar7]

GKE:
Version: 1.20.9-gke.1001
OS: Container-Optimized OS (cos_containerd)
container runtime: containerd

OSInfo: KERNEL_RELEASE: 5.4.120+
OSInfo: ID: alpine
OSInfo: VERSION_ID: 3.14.2
OSInfo: PRETTY_NAME: "Alpine Linux v3.14"
BTF: bpfenv = false, btfenv = false, vmlinux = true
BPF: using embedded BPF object
unpacked CO:RE bpf object file into memory
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS

k logs -f tracee-2glmb
Loaded signature(s):  [TRC-1 TRC-2 TRC-3 TRC-4 TRC-5 TRC-6 TRC-7]



*** Detection ***
Time: 2021-10-06T21:53:04Z
Signature ID: TRC-2
Signature: Anti-Debugging
Data: map[]
Command: strace
Hostname: gke-cluster-1-d

[simar7]

GKE:
Version: 1.20.9-gke.1001
OS: Container-Optimized OS (cos)
Container Runtime: docker

OSInfo: ID: alpine
OSInfo: VERSION_ID: 3.14.2
OSInfo: PRETTY_NAME: "Alpine Linux v3.14"
OSInfo: KERNEL_RELEASE: 5.4.120+
BTF: bpfenv = false, btfenv = false, vmlinux = true
BPF: using embedded BPF object
unpacked CO:RE bpf object file into memory

k logs -f tracee-29hv5
Loaded signature(s):  [TRC-1 TRC-2 TRC-3 TRC-4 TRC-5 TRC-6 TRC-7]

*** Detection ***
Time: 2021-10-06T22:14:14Z
Signature ID: TRC-2
Signature: Anti-Debugging
Data: map[]
Command: strace
Hostname: gke-cluster-2-d

[jessequinn]

Hi, I like to report with GKE 1.23:

Container-Optimized OS with containerd (cos_containerd):

INFO: probing tracee-ebpf capabilities...
INFO: starting tracee-ebpf...
INFO: starting tracee-rules...
Loaded 14 signature(s): [TRC-1 TRC-13 TRC-2 TRC-14 TRC-3 TRC-11 TRC-9 TRC-4 TRC-5 TRC-12 TRC-8 TRC-6 TRC-10 TRC-7]
libbpf: kprobe perf_event_open() failed: No such file or directory
libbpf: prog 'trace_vfs_writev': failed to create kprobe 'vfs_writev+0x0' perf event: No such file or directory
2022/06/04 06:42:35 error creating Tracee: error attaching event vfs_writev: failed to attach vfs_writev k(ret)probe to program trace_vfs_writev: no such file or directory

/tracee # /tracee/tracee-ebpf --debug
OSInfo: Security Lockdown is ''
OSInfo: BUILD_ID: 16623.171.1
OSInfo: KERNEL_RELEASE: 5.10.109+
OSInfo: ARCH: x86_64
OSInfo: ID: cos
OSInfo: PRETTY_NAME: "Container-Optimized OS from Google"
OSInfo: VERSION: 93
OSInfo: VERSION_ID: 93
BTF: bpfenv = false, btfenv = false, vmlinux = true
BPF: using embedded BPF object
unpacked CO:RE bpf object file into memory

Ubuntu with containerd (ubuntu_containerd):

INFO: probing tracee-ebpf capabilities...
INFO:
INFO: You should try the FULL tracee container image, it supports
INFO: building, based on your host environment, needed eBPF objects
INFO: so tracee-ebpf may work.
INFO:
Stream closed EOF for monitoring/tracee-9fjnv (tracee)

How exactly can we get tracee working on GKE 1.23+? Docker and Ubuntu image types are no longer allowed.

Is the only solution now to create a new GKE cluster with dataplane v2 enabled?

[itaysk]

@jessequinn thanks for reporting, can you please open this as a new Issue (bug)

[yanivagman]

@NDStrahilevitz is working on a fix for GKE (which fails on vfs_writev symbol) in #1791

[jessequinn]

You still need me to open an issue?