support configuring audit rules from bootstrap container

[bcressey]

Issue number:

Closes #3808

Description of changes:
Run bootstrap containers in the host PID namespace, and with CAP_AUDIT_CONTROL, so that audit rules can be configured.

To facilitate debugging, associate the journal entries to the matching host or bootstrap container by setting the SyslogIdentifier field. This was helpful when troubleshooting issues with my auditctl container.

Testing done:
When adding path-based watches, there's an additional wrinkle in that the path will be based on the current root, which will be the root of the bootstrap container's mount namespace.

To work around this, it's necessary to invoke auditctl like this:

  nsenter -t 1 -m auditctl ...

Or like this:

  chroot /.bottlerocket/rootfs auditctl ...

It's also a good idea to clear out previous rules (auditctl -D) so that the default rules don't filter out events of interest. Syscall auditing is disabled by default, and non-SELinux related messages are discarded.

With that in mind, I used the following script in my bootstrap container:

#!/bin/bash
set -ex

# Write output to stderr so it shows up in the journal.
exec 2>&1

# Delete existing rules.
chroot /.bottlerocket/rootfs auditctl -D

# Don't log user events.
chroot /.bottlerocket/rootfs auditctl -a never,user

# Log syscall exits involving "/etc/passwd".
chroot /.bottlerocket/rootfs auditctl -a always,exit -F arch=b64 -F path=/etc/passwd -F perm=rwa -F key=identity

# Don't log other syscalls.
chroot /.bottlerocket/rootfs auditctl -a never,exit -S all

# From <linux/audit.h>:
# * 1300 - 1399 audit event messages
# * 1400 - 1499 SE Linux use

# Exclude messages besides audit events and SELinux messages.
chroot /.bottlerocket/rootfs auditctl -a always,exclude -F 'msgtype<1300'
chroot /.bottlerocket/rootfs auditctl -a always,exclude -F 'msgtype>1499'

# List updated rules.
chroot /.bottlerocket/rootfs auditctl -l

This produced the following output at launch:

Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: + chroot /.bottlerocket/rootfs auditctl -D
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: time="2024-03-18T23:49:02Z" level=info msg="successfully started container task"
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: + exec
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: No rules
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: + chroot /.bottlerocket/rootfs auditctl -a never,user
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: + chroot /.bottlerocket/rootfs auditctl -a always,exit -F arch=b64 -F path=/etc/passwd -F perm=rwa -F key=identity
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: + chroot /.bottlerocket/rootfs auditctl -a never,exit -S all
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: + chroot /.bottlerocket/rootfs auditctl -a always,exclude -F 'msgtype<1300'
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: + chroot /.bottlerocket/rootfs auditctl -a always,exclude -F 'msgtype>1499'
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: + chroot /.bottlerocket/rootfs auditctl -l
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: -a never,user
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: -a always,exit -F arch=b64 -S all -F path=/etc/passwd -F perm=rwa -F key=identity
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: -a never,exit -S all
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: -a always,exclude -F msgtype<SYSCALL
Mar 18 23:49:02 ip-10-0-20-7.us-west-2.compute.internal bootstrap-containers@setup-audit-rules[1124]: -a always,exclude -F msgtype>1499

Afterwards, the rules were visible on the host and /etc/passwd writes were logged.

# auditctl -l
-a never,user
-a always,exit -F arch=b64 -S all -F path=/etc/passwd -F perm=rwa -F key=identity
-a never,exit -S all
-a always,exclude -F msgtype<SYSCALL
-a always,exclude -F msgtype>1499

# touch /etc/passwd
...

# journalctl | grep audit

Mar 18 23:59:55 ip-10-0-20-7.us-west-2.compute.internal audit[1908]: SYSCALL arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7fff86ff081c a2=941 a3=1b6 items=2 ppid=1868 pid=1908 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="touch" exe="/x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/coreutils" subj=system_u:system_r:super_t:s0-s0:c0.c1023 key="identity"
Mar 18 23:59:55 ip-10-0-20-7.us-west-2.compute.internal audit: CWD cwd="/"
Mar 18 23:59:55 ip-10-0-20-7.us-west-2.compute.internal audit: PATH item=0 name="/etc/" inode=1 dev=00:1d mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
Mar 18 23:59:55 ip-10-0-20-7.us-west-2.compute.internal audit: PATH item=1 name="/etc/passwd" inode=107 dev=00:1d mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
Mar 18 23:59:55 ip-10-0-20-7.us-west-2.compute.internal audit: PROCTITLE proctitle=746F756368002F6574632F706173737764

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[bcressey]

@diranged - any concerns about the SyslogIdentifier change?

It means bootstrap container output will show up in the journal like this:

bootstrap-containers@<name>[PID]: ...

Rather than like this:

host-ctr[PID]: ...

This doesn't address your feature request in #3811 but it may make the logs more useful in the interim.

[diranged]

@diranged - any concerns about the SyslogIdentifier change?

It means bootstrap container output will show up in the journal like this:

bootstrap-containers@<name>[PID]: ...

Rather than like this:

host-ctr[PID]: ...

This doesn't address your feature request in #3811 but it may make the logs more useful in the interim.

Yeah - I think this is an overall improvement, so I support it. I appreciate you checking in, this will not break anything on our end right now.

[yeazelm]

LGTM!

[arnaldo2792]

We should fix the linting suggestions, those are new.

[bcressey]

We should fix the linting suggestions, those are new.

Opened #3840 to track this.