SVIDs are not readable by containers running as a different user

[keeganwitt]

When you have a pod with a container running as a different user than the sidecar container, the SVID file can't be read because permission is denied. This is because the file is chmodded as 0600. The only way to avoid this is to either change the running user (not possibly if you have multiple containers that need access to the token and each run as a different user) of the sidecar or set an fsGroup on the pod's securityContext.

In our case, we are adding the spiffe-helper on the workload's behalf (in this case using a mutating admission webhook). We didn't feel it was safe/appropriate to modify the fsGroup for every single pod using the platform.

I'm suggesting we allow the file's chmod to be configured.

[keeganwitt]

If it helps, this reproduces this problem

kind: ConfigMap
apiVersion: v1
metadata:
  name: helper-conf
data:
  helper.conf: |
    agent_address = "/run/spire/agent/socket"
    jwt_svids = [
      {
        jwt_svid_file_name = "/etc/jwts/aws.jwt"
        jwt_audience = "sts.amazonaws.com"
      }
    ]
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: spiffe-job
spec:
  schedule: "*/1 * * * *"
  concurrencyPolicy: Replace
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: main
              image: busybox
              args: ['/bin/sh', '-c', 'cat /etc/jwts/aws.jwt; echo "";']
              volumeMounts:
                - name: jwts
                  mountPath: /etc/jwts
              securityContext:
                runAsUser: 1337
                runAsGroup: 1337
                readOnlyRootFilesystem: true
          initContainers:
            - name: wait-for-svid
              image: ghcr.io/spiffe/spiffe-helper:nightly
              volumeMounts:
                - name: helper-conf
                  mountPath: /etc/spiffe-helper
                - name: spire-agent
                  mountPath: /run/spire/agent/socket
                - name: jwts
                  mountPath: /etc/jwts
              command: ["/spiffe-helper", "-config", "/etc/spiffe-helper/helper.conf", "-daemon-mode=false"]
              securityContext:
                runAsUser: 10000
                runAsGroup: 10000
                readOnlyRootFilesystem: true
            - name: spiffe-helper
              image: ghcr.io/spiffe/spiffe-helper:nightly
              restartPolicy: Always
              volumeMounts:
                - name: helper-conf
                  mountPath: /etc/spiffe-helper
                - name: spire-agent
                  mountPath: /run/spire/agent/socket
                - name: jwts
                  mountPath: /etc/jwts
              command: ["/spiffe-helper", "-config", "/etc/spiffe-helper/helper.conf"]
              securityContext:
                runAsUser: 10000
                runAsGroup: 10000
                readOnlyRootFilesystem: true
          restartPolicy: Never
          volumes:
            - name: helper-conf
              configMap:
                name: helper-conf
                items:
                  - key: helper.conf
                    path: helper.conf
            - name: spire-agent
              hostPath:
                type: Socket
                path: /run/spire/agent/socket
            - name: jwts
              emptyDir:
                medium: Memory
          securityContext:
            runAsUser: 1505
            runAsGroup: 1505
            runAsNonRoot: true

[keeganwitt]

https://github.com/spiffe/spiffe-helper/pull/173/files#diff-81a0c52a44e81d83f6dd8ba18dfbc635501d877dcbcaf058141883987e008542R39 introduced this breakage.