helm: update liveness probes

[derekperkins]

Uses mysqladmin ping
cc @enisoc

[derekperkins]

I found these /debug/health endpoints, which seem to work nicely for the most part, but in testing, the vttablet health check started failing when I hadn't run an InitShardMaster yet. I'm not sure if that's what we want or not, but now that I have everything running and set up, it seems to be working nicely. I'm going to try my import again to see if it causes another crash loop.

[enisoc]

As you found, /debug/health can fail for reasons that don't indicate a stuck process, so we shouldn't use it for a livenessProbe. That should make a good readinessProbe for vitess processes though.

For now, it looks like /debug/status may be our best bet for a livenessProbe, since it's the simplest in terms of dependencies. Ideally we should add a handler like /debug/no-op that's hard-coded to immediately return 200 OK, and use that for the livenessProbe.

[derekperkins]

@enisoc I switched /debug/health to be a readiness probe and used /debug/status for the liveness probe, except for vttablet. /debug/health checks the topology, but there is no master on initial deployment, and you can’t initialize one because the pod isn’t ready.

I think this should be ready to merge.

[enisoc]

On second thought, maybe it's more dangerous than helpful to have a livenessProbe on mysqld since it's got a lot more in-memory state than the rest of our processes (you lose a lot on restart). What do you think about having only the readinessProbe here? It seems like a safer default.

[derekperkins]

That's probably a good choice

[enisoc]

We should be able to add /debug/health as a readinessProbe on vttablet if we set podManagementPolicy: Parallel on the StatefulSet. That makes sense anyway because vttablets don't need to be started in any particular order, so there's no need to wait.

[derekperkins]

That's a good choice, but I don't think it solves the problem I encountered. I wasn't able to run InitShardMaster because it never reported as ready. I'll give it another try and report back whether or not it works.

[enisoc]

The Pod being Unready shouldn't prevent InitShardMaster from working, as long as all the Pods are running (which requires Parallel).

[derekperkins]

Ok, I addressed your points and it's working great for me now.

[enisoc]

LGTM