Add RFD for Kubernetes health checks

[rana]

A request for discussion on Kubernetes health checks.

Relates to:

• Kubernetes access health checks #58413

[tigrato]

Kubernetes health checks should go beyond simply verifying that the Kubernetes API server is responsive.
A critical aspect to validate is whether each agent can operate correctly - specifically, that it has the necessary permissions to impersonate users and groups within the cluster.

While not strictly required, it’s also valuable to test Kubernetes GET Pod requests. This ensures that session start/end events are properly populated with fields like kubernetes_pod_labels and kubernetes_pod_image.

Another important area comes from a recent discussion with @programmerq, @webvictim, and a customer. The customer had a script that mistakenly copied Teleport agent state between different clusters. This led to agents across clusters sharing the same host_uid, which caused proxies to make incorrect routing decisions. Instead of forwarding requests to Cluster A, they were sent to Cluster B, since both clusters reported identical host_uid values. Detecting and preventing this scenario - and providing visibility when it occurs - would be a significant improvement.

[rosstimothy]

Kubernetes health checks should go beyond simply verifying that the Kubernetes API server is responsive. A critical aspect to validate is whether each agent can operate correctly - specifically, that it has the necessary permissions to impersonate users and groups within the cluster.

I strongly agree here. These health checks should be able to indicate that the Kubernetes Cluster is usable by Teleport users.

While not strictly required, it’s also valuable to test Kubernetes GET Pod requests. This ensures that session start/end events are properly populated with fields like kubernetes_pod_labels and kubernetes_pod_image.

I don't think that we should be using health checks to validate the presence of data in audit events.

Another important area comes from a recent discussion with @programmerq, @webvictim, and a customer. The customer had a script that mistakenly copied Teleport agent state between different clusters. This led to agents across clusters sharing the same host_uid, which caused proxies to make incorrect routing decisions. Instead of forwarding requests to Cluster A, they were sent to Cluster B, since both clusters reported identical host_uid values. Detecting and preventing this scenario - and providing visibility when it occurs - would be a significant improvement.

I think this should be out of scope for health checks. Duplicate UUIDs are a massive footgun, but I don't know that we should be complicating health checking to catch this scenario.

[TeleLos]

I added c-cv as they are interested in this feature but specifically for database.

[rana]

I added c-cv as they are interested in this feature but specifically for database.

@TeleLos Thank you, health checks for databases happens to already be released in v18 also.