Kubernetes API server rolling restarts experience client-side disruption

[nrb]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

An automated test that forces kube API server restarts observed that roughly 30 seconds after the kube api-server returns HTTP 500s from /readyz pre-shutdown, clients are getting ECONNRESET when trying to read responses.

What did you expect to happen:

Clients would not have their connections interrupted.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

The target_health_state.unhealthy.draining_interval_seconds attribute should probably be set to something higher than the default (which is 0) in order to make client interactions more resilient to kube-api-server restarts.

This draining interval is separate from the health check interval seconds we currently set - that one just tells AWS how often to invoke the health check.

See the TargetGroupAttribute docs for specifics.

This was observed during OpenShift testing, but is applicable generally to anyone restarting their KAS.

/area networking
/triage accepted

[nrb]

Reported by @benluddy

[benluddy]

More from the apiserver perspective:

When an apiserver gets SIGTERM, it immediate begins returning HTTP 500 from /readyz, but continues to accept new connections and serve existing connections for a period of time controlled by --shutdown-delay-duration. The purpose of this delay is to give load balancers enough time to mark the instance unhealthy and stop directing new traffic there. After this delay, the apiserver closes established idle connections and allows active connections a period of time to complete any in-flight requests and be closed (https://pkg.go.dev/net/http#Server.Shutdown).

If the load balancer closes established connections the moment an apiserver is considered unhealthy, any clients with in-flight requests observe errors.

This connection termination behavior seems to be unique to NLB (https://docs.aws.amazon.com/elasticloadbalancing/latest/network/edit-target-group-attributes.html#unhealthy-target-connection-termination). I don't think there is a use case for leaving it enabled when the targets are kube-apiservers, since it defeats the mechanism for graceful shutdown.

[nrb]

I think setting a default drain period for the NLBs backing a kube api-server makes sense; as Ben points out, not doing so leads to ungraceful shutdowns and client disruption.

Given that, should we update our default value? And should this option even be exposed for users to change via our CRDs? I think if we allow users to clear it, which is the current default, we're setting them up for failure. If we allow changing it, I think it should be some value above a minimum.

Curious what others think.

cc @richardcase @dlipovetsky @AndiDog