Kubernetes Sidecar Usecase

[kfox1111]

It would be great if spiffe-helper could be used as a sidecar under Kubernetes.

This would require two different modes of operation to function well.

1. a new flag for running in job mode. This would block until it could fetch the cert/key/ca, and then exit 0 on completion.
   This would run as a k8s initContainer and ensure initial cert/key/ca creation before the workload starts.
2. a new flag specifying a pid file. Instead of running a command and then signaling the process it forked, it would just notify the pid that preexists as specified in the file. This would enable it to signal a process in a different container, in the same pod.

A container image would also be needed. Requested here: #107

[kfox1111]

Maybe for a separate issue... To support the new k8s sidecar support in 1.28+, also a health check would be needed.
https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/

[nstott]

1. would definitely be cleaner. definitely a nice to have. we have spiffe-helper running in a sidecar now, and the primary container will generally restart once or twice while it waits for the helper to fetch the svids.

for 2, we'd still have to share the pid namespace though. I don't think I understand that usecase

[kfox1111]

@nstott I ported the mysql bits to work with the bitnami mysql helm chart here:
https://github.com/spiffe/helm-charts-hardened/blob/mysql-using-spire/examples/mysql-using-spire/mysql-values.yaml

The problem is, in order to trigger a reload of the certs in mysql, you need a viable mysql client. So you need spire-helper available in the mysql container. I did mange to do some twisty bits with init containers to piece it together at runtime. But its kind of ugly.

Instead, if you had shared pids enabled and pid signaling, the sidecar could just contain spiffe-helper and send a signal over to a container sidecar that has the mysql client ready to go to adapt the pid signal to a mysql reload certs sql command.

I've prototyped that using the pid pr and a prototype CSI driver for spiffe-helper to make it possible to do this kind of thing:
https://github.com/spiffe/helm-charts-hardened/pull/166/files#diff-95b3020faa9ac7ddba1d84fc7dcf64f0965725c3f3a3d71be001a5d2847ba4c8

That example actually works and signals nginx to properly reload when the certs get updated.

[keeganwitt]

A usecase I'm now looking at is if you have restartPolicy: Always on your pod, and are pre-1.28 Kubernetes, then you need a way to have the sidecar container exit when the main container exits. This is kinda inverting the direction of what's discussed here. Should this need be a separate ticket?

[kfox1111]

I thought a restartPolicy was ignored when its in a terminating state?

But, if thats not the case, maybe a preStop hook would help

[keeganwitt]

I think I mis-stated the scenario. The restartPolicy would be either Never or OnFailure in this case. And let's say the pod is part of a job, so the main container runs and exist normally. But the SPIFFE Helper sidecar is still running. "Terminating" isn't a pod phase. In this case, the pod phase would be "Running" because at least one container (the helper) is still running, the main container status would be "Terminated" and the sidecar container would have a status of "Running".

What we'd want in this scenario is after the main container exits successfully, the sidecar knows it needs to exit too.

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-states