The error log messages for app readiness and liveliness healthcheck failures should be differentiated

[heyjcollins]

Thank you for submitting an issue to the diego-release repository. We appreciate the feedback. To help us address your issue, please fill out the sections in the following template to the best of your ability:

Summary

If the readiness healthcheck fails, the error that's generated in the logs is misleading, making it very difficult to diagnose.
The readiness check timeout is 1s
The liveliness check timeout is 1m0s

Here's an example error generated when the readiness check failed:
Mon Jan 20 2020 17:49:04.680 Timed out after 1m0s: health check never passed. d1e5c9e9-3b2a-481d-8ddf-493df45698c6 2

The error is the same if the liveliness check failes so it's impossible to distinguish between the two without digging deeper into the code.

Expected Result

It would be helpful if readiness and liveliness healthcheck errors were distinct:

readiness:
Mon Jan 20 2020 17:49:04.680 Timed out after 1s: readiness health check never passed. d1e5c9e9-3b2a-481d-8ddf-493df45698c6 2

liveliness:
Mon Jan 20 2020 17:49:04.680 Timed out after 1m0s: liveliness health check never passed. d1e5c9e9-3b2a-481d-8ddf-493df45698c6 2

Actual Result

readiness:
Mon Jan 20 2020 17:49:04.680 Timed out after 1m0s: health check never passed. d1e5c9e9-3b2a-481d-8ddf-493df45698c6 2

liveliness:
Mon Jan 20 2020 17:49:04.680 Timed out after 1m0s: health check never passed. d1e5c9e9-3b2a-481d-8ddf-493df45698c6 2

Context

[Provide other relevant context for the issue, including which version of Diego or other BOSH releases you have deployed, to which infrastructure (such as AWS, GCP, or BOSH-Lite), and other environmental details. If you include a BOSH deployment manifest or section thereof, make sure to include it in a code block and to redact any confidential information such as passwords or private keys.]

Steps to Reproduce

[Provide a minimal sequence of detailed steps to reproduce the issue, including how to observe the effects of the issue.]

Possible Causes or Fixes (optional)

[If you are familiar with the internals of the Diego system or other CF systems, include any conjectures about the causes of the issue or ways to fix it here.]

Additional Text Output or Screenshots (optional)

[If you have additonal text output, such as logs or stack trace output, or screenshots demonstrating the issue, include them in this section. Please use a code block to preserve multi-line text output or output with leading indentation.]

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/170883862

The labels on this github issue will be updated when the story is started.

[mariash]

@heyjcollins Looking closer at the code. It seems like the whole message Timed out after <some_timeout> is misleading. There are might be different reasons why the error happens and it might be not due to timeout. e.g. https://github.com/cloudfoundry/healthcheck/blob/1ea40ee5052f10cd91419f835e309450431bcc3b/healthcheck.go#L102 Suggestion instead of Timed out after say Failed after

[heyjcollins]

Hi @mariash - thanks for pointing that out. If it's possible for other reasons to trigger logging the message, then please do change the message as you suggest above.