Add health-based connection routing for Kubernetes#60083
Merged
Conversation
rana
force-pushed
the
rana/kube-healthchecks-9
branch
2 times, most recently
from
631d1ee to
36aa44d
Compare
rana
added
kubernetes-access
no-changelog
rana
force-pushed
the
rana/kube-healthchecks-9
branch
from
36aa44d to
4f6b2b0
Compare
github-actions
bot
added
database-access
rosstimothy
reviewed
Oct 9, 2025
GavinFrazar
approved these changes
Oct 9, 2025
rana
force-pushed
the
rana/kube-healthchecks-9
branch
from
4f6b2b0 to
53ce9cb
Compare
rosstimothy
reviewed
Oct 10, 2025
rana
force-pushed
the
rana/kube-healthchecks-9
branch
3 times, most recently
from
1b0ad96 to
d09b2d3
Compare
rosstimothy
reviewed
Oct 14, 2025
rana
force-pushed
the
rana/kube-healthchecks-9
branch
from
d09b2d3 to
7f7fe27
Compare
rosstimothy
approved these changes
Oct 14, 2025
public-teleport-github-review-bot
bot
removed request for
espadolini,
gabrielcorado and
greedy52
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
rana
force-pushed
the
rana/kube-healthchecks-9
branch
from
7f7fe27 to
91feaa5
Compare
rana
added a commit
that referenced
this pull request
Oct 23, 2025
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
37 tasks
rana
added a commit
that referenced
this pull request
Oct 27, 2025
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
rana
added a commit
that referenced
this pull request
Oct 29, 2025
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
rana
added a commit
that referenced
this pull request
Oct 29, 2025
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
rana
added a commit
that referenced
this pull request
Oct 29, 2025
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Oct 29, 2025
* Add protobufs for Kubernetes health checks (#58415) - Add Kubernetes label matchers to `Matcher` for `HealthCheckConfig` - Add message `KubernetesServerStatusV3` - Add `status` field to `KubernetesServerV3` - Add `target_health` field to `Kube` for UI - Regenerate Terraform schema and docs for `HealthCheckConfig` - Add Kubernetes label matchers to Terraform test `TestImportHealthCheckConfig` Relates to #58413 Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> * Refactor `healthcheck` for Kubernetes extensibility with `HealthChecker` interface (#59396) The main intent of refactoring is to provide health check extensibility for Kubernetes while supporting the existing DB health checks. A `HealthChecker` interface is added to support the different health check approaches of DBs and Kubernetes. Existing DB TCP health check logic is moved to a new `TargetDialer` struct. Changes: - Added `HealthChecker` interface with two functions: - `CheckHealth(ctx context.Context) ([]string, error)` - `GetProtocol() types.TargetHealthProtocol` - Added `TargetDialer` struct which encapsulates existing TCP health check logic - Changed `Target` struct to use the `HealthChecker` interface - Changed `worker.checkHealth` to call the new `CheckHealth` function - Removed a `protocol` field from `healthCheckConfig` - Added `TargetHealthProtocolHTTP` for use with Kubernetes health checks - Moved and renamed test `Test_dialEndpoints` to `TestTargetDialer_dialEndpoints` - Added files `net.go` and `net_test.go` for `TargetDialer` Part of #58413 * Add health checks to Kubernetes agent and proxy (#59565) Implements core health check logic for Kubernetes. - Added functions `CheckHealth` and `GetProtocol` to `kubeDetails` - Added a `HealthCheckManager` field to the Kubernetes `TLSServerConfig` - Kube agent and proxy now instantiate a `HealthCheckManager` - Added a `HealthCheckConfigReader` to interfaces `ReadKubernetesAccessPoint` and `ReadProxyAccessPoint` - Added `HealthCheckConfig` read-only permissions for proxy and kube - Added `HealthCheckConfig` watching for proxy and kube - Added functions to `KubeServer` interface and `KubernetesServerV3` struct: - `GetTargetHealth() TargetHealth` - `SetTargetHealth(h TargetHealth)` - `GetTargetHealthStatus() TargetHealthStatus` - `SetTargetHealthStatus(status TargetHealthStatus)` - Health check supports dynamic discovery of Kubernetes clusters. Calls to `startHealthCheck()` and `stopHealthCheck()` were rearranged. - Added functions `startHeartbeatAndHealthCheck()` and `stopHeartbeatAndHealthCheck()` - Moved call to `HealthCheckManager.Start()` outside of the Kubernetes proxy server providing a future option to reuse `HealthCheckManager` in multiple proxy services - Removed `Status` initialization in KubernetesServerV3 `CheckAndSetDefaults()` - Added `kubernetesLabelMatchers` to `healthCheckConfig` struct. Default presets are omitted until the entire kube health check is complete. - Added Kubernetes matcher checking to `ValidateHealthCheckConfig()` - Changed `ValidateHealthCheckConfig()` to allow zero total DB matchers and Kubernetes matchers. - Changed KubernetesServerV3 `GetTargetHealthStatus()` to return `TargetHealthStatusUnknown` instead of an empty string - Changed health check worker `getTargetHealthTimeout` default timeout to `4s` from `10s`. This potentially reduces the response time of the initial heartbeat polling call to `GetServerInfo()`. Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> * Add UI for Kubernetes health checks (#59929) Adds health status indicators for Kubernetes clusters on the Resources page. Unhealthy clusters are highlighted, and clicking them opens a side panel displaying server information. Changes include: - New `KubeServer` protobuf message and `ListKubernetesServers` RPC - Web and Connect API endpoints for fetching Kubernetes server data - Health status filtering in `matchAndFilterKubeClusters` - `TargetHealth` fields added to frontend/backend types - Updated `StatusInfo.tsx` to display `kube_cluster` data Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> * Add health-based connection routing for Kubernetes (#60083) Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> * Allow disabling health check configs (#60447) Health check configuration can be disabled and enabled through the matcher. Disabling the matcher disables health checks for this configuration's resources including databases, Kubernetes clusters, and any future resources. Changes: - Added `disabled` field to health check matcher proto - Updated matcher selection logic - Updated unit tests Part of #58413 * Add documentation for Kubernetes health checks (#60201) A new Kubernetes-specific health check page is added. The existing Kubernetes troubleshooting documentation page is updated with health check specific error resolutions. Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> Co-authored-by: Gavin Frazar <gavin.frazar@goteleport.com> Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> * Enable health checks for Kubernetes with virtual defaults (#60544) Health checks are enabled for all Kubernetes clusters by default. A design of creating one health check config default per resource is implemented. The choice eases adoption of health checks, supports existing clusters that already have database health checks, and avoids migrating the backend database. A new Kubernetes-specific `default-kube` health check config is added. And a database-specific `default` health check config already exists, and is preserved. A virtual default design is implemented by returning health check configs from memory if they don't exist in the backend database. The approach has the benefit of not re-inserting default values to the backend after they're deleted, which a prior approach had. Virtual defaults are added at the local health check service level, and returned from functions `GetHealthCheckConfig` and `ListHealthCheckConfigs`. Virtual defaults may be written, updated, and deleted to and from the backend. While virtual defaults may be deleted, it has the net effect of resetting the config to default settings, and matching all resources of that type (db, kube). Virtual defaults are always returned from health check `get` and `list` functions. Changes: - Added `default-kube` health check config specific to Kubernetes only - Updated local service functions `GetHealthCheckConfig` and `ListHealthCheckConfigs` to return virtual defaults - Added unit tests - Updated health check documentation with `default-kube` and info about virtual defaults Part of #58413 Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> * Add `Resources` function for v18 backport The generic `Resources` function is backported to the generic `Service` type. `Resources` is used in a related commit enabling health checks for Kubernetes (#60544). Part of #58413 * Fix test for terraform health check config (#60640) Filters virtual defaults to allow explicit config references. Part of #58413 --------- Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> Co-authored-by: Gavin Frazar <gavin.frazar@goteleport.com> Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com>
mmcallister
pushed a commit
that referenced
this pull request
Nov 6, 2025
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
mmcallister
pushed a commit
that referenced
this pull request
Nov 19, 2025
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
mmcallister
pushed a commit
that referenced
this pull request
Nov 20, 2025
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution.
The grouping and shuffling is implemented generically in a separate iterator function
OrderByTargetHealthStatusfor reuse and testability.In this PR:
OrderByTargetHealthStatusto thehealthcheckpackageOrderByTargetHealthStatuswhen dialing serversImplementation details:
OrderByTargetHealthStatuswas originally placed in the existing fileapi/types/target_health.go, but encountered a cyclic import for usinglib/utils.ShuffleVisit. The cyclic import is resolved by moving it into thehealthcheckpackage.Relates to: