Don't exit the probe on connection issues

[jsafrane]

What type of PR is this?
/kind bug

What this PR does / why we need it:
Do not exit the liveness probe process when it cannot connect to the CSI driver. The driver could be crashlooping, and we should not crashloop the liveness probe process too.

The process should only fail all probes to /healthz endpoint. Since the HTTP server is not running when connecting to the driver for the first time, "connection refused" must be a good enough failure.

This PR is heavily inspired / copied from #237

Which issue(s) this PR fixes:

Fixes #236

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Liveness probe process does not crash when it cannot access the associated CSI driver. It only fails all kubelet probes, most probably with "connection refused".

[jsafrane]

Tested with a CSI driver that's crashlooping. The liveness probe sidecar is still running and just restarts the driver container. "connection refused" is not great, but it fails the probe just fine:

  Warning  ProbeError  89s (x5 over 2m9s)    kubelet            Liveness probe error: Get "http://10.0.2.31:10300/healthz": dial tcp 10.0.2.31:10300: connect: connection refused
body:
  Warning  Unhealthy  89s (x5 over 2m9s)   kubelet  Liveness probe failed: Get "http://10.0.2.31:10300/healthz": dial tcp 10.0.2.31:10300: connect: connection refused
  Normal   Killing    89s                  kubelet  Container csi-driver failed liveness probe, will be restarted

[ejweber]

I think it makes more sense to simply remove acquireConnection altogether as I did in #237. It seems the main purpose of using it as a wrapper is to add timeout functionality (which connlib now gives us for free). I'm not sure if there's a practical reason someone would set probeTimeout > 30, but if they did, connlib.Connect would timeout before probeTimeout, causing confusion.

That being said, this looks like it fixes the issue identified in #236, so no major complaints if you go this route.

[jsafrane]

I basically copied #237 and updated it with 0 timeout, it looks much better now.

[ejweber]

LGTM. Thanks!

[xing-yang]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ejweber, jsafrane, xing-yang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jsafrane,xing-yang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment