unable to run envoy proxy with selinux enforcing

[dweomer]

Environmental Info:
k3s version v1.18.8+k3s1 (6b59531)
k3s version v1.19.1-rc1+k3s1 (041f18f)

Node(s) CPU architecture, OS, and Version:
Single-node amd64, CentOS 7:

# /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Describe the bug:
Unable to run envoy proxy with SELinux=Enforcing.

Steps To Reproduce

• install k3s:

sudo yum install -y container-selinux selinux-policy-base https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm
curl -fsSL https://get.k3s.io | sudo sh

• create envoy proxy pod:

kubectl run envoy --image=envoyproxy/envoy:v1.14.1

• kubectl get pod envoy

NAME    READY   STATUS             RESTARTS   AGE
envoy   0/1     CrashLoopBackOff   1          10s

Expected behavior:
The pod runs without crashing

Actual behavior:
The pod is in a crash loop.

Additional context / logs:

• sudo ausearch -ts recent -m avc

----
time->Fri Sep 11 23:58:44 2020
type=PROCTITLE msg=audit(1599868724.714:1069): proctitle=656E766F79002D63002F6574632F656E766F792F656E766F792E79616D6C
type=SYSCALL msg=audit(1599868724.714:1069): arch=c000003e syscall=2 success=no exit=-13 a0=7ffc8812c3e0 a1=a00c2 a2=180 a3=121 items=0 ppid=7600 pid=7661 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="envoy" exe="/usr/local/bin/envoy" subj=system_u:system_r:container_t:s0:c715,c888 key=(null)
type=AVC msg=audit(1599868724.714:1069): avc:  denied  { write } for  pid=7661 comm="envoy" name="/" dev="tmpfs" ino=65109 scontext=system_u:system_r:container_t:s0:c715,c888 tcontext=system_u:object_r:container_runtime_tmpfs_t:s0 tclass=dir permissive=0

[dweomer]

In my initial assessment I thought that this was a policy miss in k3s-selinux: https://github.com/rancher/k3s-selinux/blob/470ebf7dc1f53ffcf9020521d6a263bf9f79f4bd/k3s.fc#L16

But, of course, there is no transition yet defined for the shm path as there is for rke2: https://github.com/rancher/rke2-selinux/blob/v0.2.testing.2/rke2.if#L21

Turns out, the transition is in container-selinux: https://github.com/containers/container-selinux/blob/RHEL7.5/container.te#L191

[dweomer]

With attempt to verify this on Fedora 32 on the chance that updated policy solves this (el7 ~ 2.119, f32 ~ 2.144)

[cjellick]

This won't make it into the initial 1.19 release, but we'll address it in a follow up.

[davidnuzik]

@dweomer Are you actually working on a fix now and have a way forward? Or should we put this in the backlog for now?

[dweomer]

@davidnuzik backlog I think but likely high-ish priority. Can maybe get to it later this week.

[davidnuzik]

We plan to take SELinux out of experimental in docs after the fix for this is in.

[ShylajaDevadiga]

Validated using Commit ID c72c186 in CentOS 7.8 as well as RHEL 8.2 on SELinux Enforced systems.

$ k3s -v
k3s version v1.19.3+k3s-c72c1867
$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 (Ootpa)
$ sudo cat  /var/lib/rancher/k3s/etc/containerd/config.toml |grep selinux
  enable_selinux = true
$ kubectl get pod envoy
NAME    READY   STATUS    RESTARTS   AGE
envoy   1/1     Running   0          92s
[ec2-user@ip-172-31-33-149 ~]$ 

[ShylajaDevadiga]

Validated the fix using commit 2c666af on rhel 7.8 as well rhel 8.2
selinux Enforced mode
hostIPC set to true

[ec2-user@ip-172-31-32-116 ~]$ k3s -v
k3s version v1.19.3+k3s-2c666af4 (2c666af4)
[ec2-user@ip-172-31-32-116 ~]$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml  |grep enable
  enable_selinux = true
[ec2-user@ip-172-31-32-116 ~]$ ls -alZ /dev/shm/
total 4
drwxrwxrwt.  2 root root system_u:object_r:tmpfs_t:s0            60 Nov 12 01:38 .
drwxr-xr-x. 18 root root system_u:object_r:device_t:s0         2720 Nov 12 01:27 ..
-rw-------.  1 root root system_u:object_r:container_file_t:s0  104 Nov 12 01:38 envoy_shared_memory_0
[ec2-user@ip-172-31-32-116 ~]$