Add fsfreeze example to documentation

[ncdc]

Note that fsfreeze requires a privileged container.

Suggest that 1 option is to run a sidecar pause container that contains fsfreeze and has the appropriate volumeMount(s).

Show using pre/post hooks once #244 is resolved.

[robbyt]

It would be cool if this was written as an operator service, and run as a daemonSet. The service could watch for events to freeze/unfreeze the volumes.

[ncdc]

A daemonset might be challenging (it would need access to all the volumes for all the pods it wants to freeze, and it would be difficult to signal easily), but we are thinking that it would be nice to use an annotation+initializer+controller to make this happen automagically. For example:

1. Set an annotation on your pod, e.g. volume.backup.ark.heptio.com/freeze=true
2. Our initializer controller would see that on the pod and then:
   1. Set up pre and post hooks to freeze & unfreeze
   2. Add a slim container that has fsfreeze that just runs pause

What do you think about something like that?

[robbyt]

That seems simpler from an implementation- but if I'm understanding it all correctly, that might be a bit fragile and prone to race conditions.

Here's why: if ark patches the pod resource directly, what's keeping the replica set from editing the pod while the fs is frozen? If ark patches the RS, what happens if a deployment is updated during the fs freeze? Also, if ark patches the deployment, it causes noise in my audit stream.

I think the even simpler & less magical way is to require users to manually add a sidecar controller to pods that need this feature.

[ncdc]

There's not a race that I can foresee. When a Kubernetes object has an initializer, it means that it's not visible to most of the system. Only components that ask for uninitialized objects (such as initializer controllers) will see them. So when the RS controller submits a pod for creation, our initializer controller would intercept it, mutate it to add the annotations & sidecar container, and then once there were no more pending initializers, the pod would be "created" and visible to everyone.

Does that make more sense?

[robbyt]

Yes it does make sense- I didn't realize that was possible. Is this k8s feature by design? Could this behavior change in the future?

My recommendation from an end-user's perspective is for this to be designed as two features:

1. easily freeze the file system during snapshots
2. minimize end-user effort required to install ark's necessary components

[ncdc]

Initializers is an alpha feature in Kubernetes. The behavior could change in the future given that it's alpha.

This issue is focused on documenting how to fsfreeze using pre/post hooks and a sidecar container. In the future (v0.8.0 or later), we hope to make it easier to do this.

Making it easier to install is part of #52, and will ultimately probably be done with an ark install command.

[skriss]

Fixed by #795