Add liveness and readiness probes for Kubernetes

[arschles]

Describe the solution you'd like

The proxy can implement an endpoint that Kubernetes liveness / readiness probes can use to figure out if the proxy is running and healthy.

I think we should implement the endpoint in the proxy and configure the helm chart to use it.

Additional context

I'm researching and preparing for #772 and found that this would be really helpful for us.

[marpio]

@arschles we already have the /healthz endpoint. Do we need to modify it somehow or do we need something different additionally?

[ghost]

I think the /healthz endpoint covers the liveness probe.

I'm uncertain what we would do different for a readiness probe. Maybe an endpoint that checks outgoing network connectivity or backing storage availability.

[manugupt1]

Would it not be different if you have several proxy server's running and one of them s down, but others are running on the same cluster?

[michalpristas]

for health, I imagined a sweet stats where we can see how many mods are being served/fetched at the moments so in case we have some smart Load balancer it can be configured to balance the work based on these stats.
network check is a good idea, together with a storage ping, VCS ping maybe, you can be in one cluster on different machines and machine can be somehow messed up and not see storage, vcs etc.

[chriscoffee]

I usually do something like DNS resolution check and database check on /readiness for readiness. For our case we could probably probe the storage driver?