Add error counter for k8s client

[adleong]

We add a http_client_errors_total counter metric to control plane components to measure when requests to the k8s API fail without a response.

[alpeb]

I think clientErrorCounter needs to be registered in the init() function in order to be accounted for.

[adleong]

With some manual testing I can see this counter increase when connection to the k8s api is interrupted.

Surprisingly, the logs are somewhat light in terms of information when this happens. At debug level we see messages like:

INFO[2023-12-18T16:33:33-08:00] GET https://0.0.0.0:32929/apis/discovery.k8s.io/v1/endpointslices?allowWatchBookmarks=true&resourceVersion=753732&timeout=8m14s&timeoutSeconds=494&watch=true  in 0 milliseconds

showing that API responses are returning in 0 milliseconds, but not indicating that this is because of a failed connection.

We have to increase the log level to trace before we see:

INFO[2023-12-18T16:34:54-08:00] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: main/v0.0.0 (darwin/amd64) kubernetes/$Format" 'https://0.0.0.0:32929/api/v1/endpoints?allowWatchBookmarks=true&resourceVersion=753707&timeout=7m5s&timeoutSeconds=425&watch=true'
INFO[2023-12-18T16:34:54-08:00] HTTP Trace: Dial to tcp:0.0.0.0:32929 failed: dial tcp 0.0.0.0:32929: connect: connection refused
INFO[2023-12-18T16:34:54-08:00] GET https://0.0.0.0:32929/api/v1/endpoints?allowWatchBookmarks=true&resourceVersion=753707&timeout=7m5s&timeoutSeconds=425&watch=true  in 0 milliseconds

showing that there was a connection refused.

[alpeb]

LGTM 👍