Group permissions are not set on mounted local-path volumes

[timyates]

Environmental Info:

K3s Version:

k3s version v1.21.3+k3s1 (1d1f220)
go version go1.16.6

Node(s) CPU architecture, OS, and Version:

Ubuntu 20 x86_64

$ uname -a
Linux ip-172-31-30-177 5.4.0-1045-aws #47-Ubuntu SMP Tue Apr 13 07:02:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Same thing happens on Amazon Linux 2

Cluster Configuration:

Single node setup by running

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

on the host.

Describe the bug:

We run containers in non-root mode, and need to support OpenShift so we rely on Group folder permissions for writing data out from our containers. (Under OpenShift, the containers are run as a random user ID, but a group ID of 0)

When we run our containers on K3s v1.21.3+k3s1 using the local-path storage type, the mounted volume gets the permissions:

drwx------    2 root     root        4.0K Aug 13 08:13 test

So our apps cannot write to the volume (as we do not run our containers as root)

Steps To Reproduce:

1. Install v1.21.3+k3s1 (the latest release at time of writing)

   curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
   

2. Add a StatefulSet with a volume claim with:

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     name: test
   spec:
     type: ClusterIP
     ports:
       - port: 1234
         protocol: TCP
         targetPort: 1234
     selector:
       app.kubernetes.io/name: test
   ---
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: test
     labels:
       app.kubernetes.io/name: test
   spec:
     replicas: 1
     selector:
       matchLabels:
         app.kubernetes.io/name: test
     serviceName: test
     template:
       metadata:
         labels:
           app.kubernetes.io/name: test
       spec:
         containers:
           - name: test
             image: busybox
             command: [ "sh", "-c", "sleep 1h" ]
             imagePullPolicy: Always
             ports:
               - containerPort: 1234
             volumeMounts:
             - mountPath: /test
               name: test-volume
         securityContext:
           fsGroup: 0
           runAsGroup: 0
           runAsUser: 999
           runAsNonRoot: true
     volumeClaimTemplates:
       - metadata:
           name: test-volume
         spec:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 20Gi
           storageClassName: local-path
   EOF
   

3. Check the permissions of the /test mount point

   kubectl exec test-0 -- ls -lh /
   

Expected behavior:

I expect the output (note the /test permissions)

total 40K
drwxr-xr-x    2 root     root       12.0K Jun  7 17:34 bin
drwxr-xr-x    5 root     root         360 Aug 13 08:13 dev
drwxr-xr-x    1 root     root        4.0K Aug 13 08:13 etc
drwxr-xr-x    2 nobody   nobody      4.0K Jun  7 17:34 home
dr-xr-xr-x  178 root     root           0 Aug 13 08:13 proc
drwx------    2 root     root        4.0K Jun  7 17:34 root
dr-xr-xr-x   13 root     root           0 Aug 13 08:13 sys
drwxrwxrwx    2 root     root        4.0K Aug 13 08:13 test
drwxrwxrwt    2 root     root        4.0K Jun  7 17:34 tmp
drwxr-xr-x    3 root     root        4.0K Jun  7 17:34 usr
drwxr-xr-x    1 root     root        4.0K Aug 13 08:13 var

Actual behavior:

We actually see this, where only the root user can write to /test:

total 40K
drwxr-xr-x    2 root     root       12.0K Jun  7 17:34 bin
drwxr-xr-x    5 root     root         360 Aug 13 08:13 dev
drwxr-xr-x    1 root     root        4.0K Aug 13 08:13 etc
drwxr-xr-x    2 nobody   nobody      4.0K Jun  7 17:34 home
dr-xr-xr-x  178 root     root           0 Aug 13 08:13 proc
drwx------    2 root     root        4.0K Jun  7 17:34 root
dr-xr-xr-x   13 root     root           0 Aug 13 08:13 sys
drwx------    2 root     root        4.0K Aug 13 08:13 test
drwxrwxrwt    2 root     root        4.0K Jun  7 17:34 tmp
drwxr-xr-x    3 root     root        4.0K Jun  7 17:34 usr
drwxr-xr-x    1 root     root        4.0K Aug 13 08:13 var

Additional context / logs:

If I install the previous version of K3s with

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.21.2+k3s1 sh -s - --write-kubeconfig-mode 644

It works as expected

$ k3s -version
k3s version v1.21.2+k3s1 (5a67e8dc)
go version go1.16.4
$ kubectl exec test-0 -- ls -lh /
total 40K
drwxr-xr-x    2 root     root       12.0K Jun  7 17:34 bin
drwxr-xr-x    5 root     root         360 Aug 13 08:34 dev
drwxr-xr-x    1 root     root        4.0K Aug 13 08:34 etc
drwxr-xr-x    2 nobody   nobody      4.0K Jun  7 17:34 home
dr-xr-xr-x  182 root     root           0 Aug 13 08:34 proc
drwx------    2 root     root        4.0K Jun  7 17:34 root
dr-xr-xr-x   13 root     root           0 Aug 13 08:34 sys
drwxrwxrwx    2 root     root        4.0K Aug 13 08:34 test
drwxrwxrwt    2 root     root        4.0K Jun  7 17:34 tmp
drwxr-xr-x    3 root     root        4.0K Jun  7 17:34 usr
drwxr-xr-x    1 root     root        4.0K Aug 13 08:34 var

Backporting

• Needs backporting to older releases

[timyates]

Ahh, I'm guessing this is the same as #3704

Apologies for the noise if so... I tried to hard with my reproducer too 😉

[dereknola]

Appreciate the digging @timyates. Closing this as it is a duplicate of #3704.
K3s v1.21.4 is now out with a fix.
https://github.com/k3s-io/k3s/releases/tag/v1.21.4%2Bk3s1

[timyates]

Awesome, thanks 😁