Implement GRPC Healthchecks so only leader-elected GRPC server serves requests

[aliaqel-stripe]

Proposal

We want to run the KEDA Operator in leader elected mode, however, as implemented today, the GRPC server starts only once the leader election lock is taken.

Our envoy mesh relies on healthchecks to determine if a GRPC server is up so without healthchecks, Envoy will panic route between endpoints (if we use the http health checks for service endpoint availability) even if one endpoint is actually health.

What we'd like to do is add GRPC healthchecks such that the server is started before the leader election lock is taken but reports NOT_SERVING, and after the lock is taken, the server returns SERVING.

Use-Case

When running two or more endpoints, we need to have endpoints report GRPC healthcheck status to avoid panic routing on our service mesh.

Is this a feature you are interested in implementing yourself?

Yes

Anything else?

Discussed here in Slack:
https://kubernetes.slack.com/archives/CKZJ36A5D/p1705683700809119