Support liveness and readiness state for Kubernetes

[snicoll]

The new health indicator groups feature in Spring Boot 2.2 allows to create arbitrary health indicator groups. It would be nice if Spring Boot provided a default for those when running on Kubernetes.

A programmatic callback (customizer?) with a condition on running with Kubernetes could be provided. If no liveliness and readiness group exists, we could create a liveliness with "ping" only and a readiness with all the rest. That doesn't help customizing the roles and the details but perhaps such thing can be configured via properties.

[bclozel]

After a quick discussion with the team, it seems we need to think about this more.

First, we need to consider making that feature available without actuator on the classpath - that would change our previous plan about leveraging health indicator groups if we decide to do so.

We also need to reconsider the actual checks made by each probe.

readiness probes should tell whether the application is ready to receive traffic. In our case, we should:

• make sure to wait for ApplicationReadyEvent when the application starts
• consider graceful shutdown in that context (see Allow the embedded web server to be shut down gracefully #4657): should we fail that endpoint when shutdown starts?
• if we expose this on a management port, we should check that the actual port responding to application requests is able to do so. There are cases where the management port is ready but the public port is not

liveness probes:

• should be considered carefully and not necessarily exposed by default. It should only be used when the application is stuck with some internal, corrupted state and the best course of action is to kill it
• relying on external checks might could make the problem worse: if an external service/database fails, all application instances depending on it are restarted. We should use this only if the application cannot recover from an error

startup probes:

• should be available soon
• tailored for slow startup times (large database migration, building huge local caches, etc)
• not necessarily a different endpoint from a Spring Boot point of view, but maybe something worth documenting

In all cases, it could be useful to document/point to relevant documentation; each probe needs to be configured with a different spec (failure threshold, period).

[matthyx]

@bclozel I am a Kubernetes reviewer and saw this issue on your side...
Can we work together on defining the best strategy for each probe?

For instance there is more than just setting ReadinessProbe to true when you are ready - it can be used to give your application some breathing space when you process your queue.

The LivenessProbe should fail when your application requires a reset - I have seen cases where that probe relies on a different Spring Context than the application, and continues to work even when the latter crashed after an OOM...

Please let me know how I can help document and implement these. Thanks!

[ttddyy]

Hi,

I just want to put some input for how we implemented our readiness and liveness.
It's pretty much similar to what @bclozel commented above.

Readiness:

We have ReadinessEdnpoint class that simply keep boolean value and also takes HealthIndicator beans for readiness.(Built on spring-boot 2.1, so not using indicator group yet)

This class is an application context event listener that receives our ApplicationReadinessEvent.
Initial value for this readiness bean indicates NOT_READY because we do NOT want traffic until application is ready for serving.
Once application starts-up and bootstrapped necessary things, user fires ApplicationReadinessEvent with value=READY, then readiness starts returning value=READY.
Application needs to decide when to issue readiness event(value=READY) because being ready to serve traffic is upto the application to decide.

Another place that issues readiness event is our graceful shutdown logic.
When graceful shutdown logic is initiated (e.g: by receiving ContextClosedEvent), first thing we do is to fire ApplicationReadinessEvent with value=NOT_READY. This will stop receiving any more requests while shutdown is in progress.

Liveness:

Similar to readiness class, we have simple boolean to indicate LIVE/NOT_LIVE.
The differences from readiness are initial value is set to LIVE and no event is issued to change liveness status right after application is bootstrapped.

There are also consideration for initial delay and frequency(period) for readiness/liveness probes in k8s config. Once initial delay is passed, we check readiness more often than liveness. Also, this frequency(period) may affect duration for graceful shutdown.

Currently, our readiness and liveness are implemented as actuator Endpoint but it is not necessary this way.
As long as there is a boolean value to keep the state and receive application context event, it can be a service bean. Then, if actuator is available, put it into a HealthIndicator(HealthContributor) and be part of each readiness/liveness health groups.

[wilkinsona]

Application needs to decide when to issue readiness event(value=READY) because being ready to serve traffic is upto the application to decide.

I'm intrigued by this, @ttddyy. Thanks for sharing your thoughts. I had hoped that performing work during application context refresh and in application and command-line runners and the subsequent ApplicationReadyEvent would be sufficient for indicating that the application was ready to start handling traffic. What's missing from the current events and startup sequencing that led to you issuing a separate event?

[ttddyy]

Hi @wilkinsona

When application becomes ready to serve traffic is not necessarily tied to ApplicationContext lifecycle.
If application is well behaved to fit in spring lifecycle, application developers would put initialization logic to [Command|Application]Runner; however, it is not something we can enforce.
Also, application received ApplicationReadyEvent doesn't always mean it is ready to serve traffic. It is more like it is ready to perform application logic. The ready to serve flag might depends on external resources. Application may connect to cache cluster upon ApplicationReadyEvent and form the cluster and warm up local cache, then it becomes ready to serve traffic.
So, we think it is application's responsibility to decide when to flip the ready flag.

From spring-boot perspective, I think it is ok to set ready=true at ApplicationReadyEvent by default. But it needs a way to disable the default ready event and let users manually flip the ready flag; so that, application can determine the timing.