Enable health checks for Kubernetes with virtual defaults by rana · Pull Request #60544 · gravitational/teleport

rana · 2025-10-24T05:58:35Z

Health checks are enabled for all Kubernetes clusters by default.

A design of creating one health check config default per resource is implemented. The choice eases adoption of health checks, supports existing clusters that already have database health checks, and avoids migrating the backend database. A new Kubernetes-specific default-kube health check config is added. And a database-specific default health check config already exists, and is preserved.

A virtual default design is implemented by returning health check configs from memory if they don't exist in the backend database. The approach has the benefit of not re-inserting default values to the backend after they're deleted, which a prior approach had.

Virtual defaults are added at the local health check service level, and returned from functions GetHealthCheckConfig and ListHealthCheckConfigs. Virtual defaults may be written, updated, and deleted to and from the backend. While virtual defaults may be deleted, it has the net effect of resetting the config to default settings, and matching all resources of that type (db, kube). Virtual defaults are always returned from health check get and list functions.

In this PR:

Added default-kube health check config specific to Kubernetes only
Updated local service functions GetHealthCheckConfig and ListHealthCheckConfigs to return virtual defaults
Added unit tests
Updated health check documentation with default-kube and info about virtual defaults

Part of #58413
Relates to:

Kubernetes access health checks #58413

espadolini

The healthcheck.manager relies on working events for health_check_config, so the approach we used for installers (where we cannot generate events correctly because the virtual resource depends on the state of other resources, so events are broken and virtual resources are reified in the auth grpc layer) doesn't quite work; I've moved things around a bit and added events in #60552, PTAL.

virtual resources are in the service and events work

github-actions · 2025-10-25T06:38:21Z

Amplify deployment status

Branch	Commit	Job ID	Status	Preview	Updated (UTC)
rana/kube-healthchecks-14	`e9d6356`	11	✅SUCCEED	rana-kube-healthchecks-14	2025-10-27 17:04:43

okraport

This looks good to me. Nice job. I would like @espadolini and @rosstimothy stamp before approving to be safe.

espadolini · 2025-10-27T09:47:31Z

+	// we don't need to check if config refers to a potentially virtual resource
+	// because creating on top of a virtual resource is very convenient so it's
+	// a break from the semantics of Create that we want to allow


@hugoShaka will IaC be ok with creating an override for a virtual resource, or will there be checks for existing resources that will result in no write even being attempted? Update will still work correctly regardless.

Health checks are enabled for all Kubernetes clusters by default. A design of creating one health check config default per resource is implemented. The choice eases adoption of health checks, supports existing clusters that already have database health checks, and avoids migrating the backend database. A new Kubernetes-specific `default-kube` health check config is added. And a database-specific `default` health check config already exists, and is preserved. A virtual default design is implemented by returning health check configs from memory if they don't exist in the backend database. The approach has the benefit of not re-inserting default values to the backend after they're deleted, which a prior approach had. Virtual defaults are added at the local health check service level, and returned from functions `GetHealthCheckConfig` and `ListHealthCheckConfigs`. Virtual defaults may be written, updated, and deleted to and from the backend. While virtual defaults may be deleted, it has the net effect of resetting the config to default settings, and matching all resources of that type (db, kube). Virtual defaults are always returned from health check `get` and `list` functions. Changes: - Added `default-kube` health check config specific to Kubernetes only - Updated local service functions `GetHealthCheckConfig` and `ListHealthCheckConfigs` to return virtual defaults - Added unit tests - Updated health check documentation with `default-kube` and info about virtual defaults Part of #58413 Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>

The generic `Resources` function is backported to the generic `Service` type. `Resources` is used in a related commit enabling health checks for Kubernetes (#60544). Part of #58413

Health checks are enabled for all Kubernetes clusters by default. A design of creating one health check config default per resource is implemented. The choice eases adoption of health checks, supports existing clusters that already have database health checks, and avoids migrating the backend database. A new Kubernetes-specific `default-kube` health check config is added. And a database-specific `default` health check config already exists, and is preserved. A virtual default design is implemented by returning health check configs from memory if they don't exist in the backend database. The approach has the benefit of not re-inserting default values to the backend after they're deleted, which a prior approach had. Virtual defaults are added at the local health check service level, and returned from functions `GetHealthCheckConfig` and `ListHealthCheckConfigs`. Virtual defaults may be written, updated, and deleted to and from the backend. While virtual defaults may be deleted, it has the net effect of resetting the config to default settings, and matching all resources of that type (db, kube). Virtual defaults are always returned from health check `get` and `list` functions. Changes: - Added `default-kube` health check config specific to Kubernetes only - Updated local service functions `GetHealthCheckConfig` and `ListHealthCheckConfigs` to return virtual defaults - Added unit tests - Updated health check documentation with `default-kube` and info about virtual defaults Part of #58413 Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>

The generic `Resources` function is backported to the generic `Service` type. `Resources` is used in a related commit enabling health checks for Kubernetes (#60544). Part of #58413

Health checks are enabled for all Kubernetes clusters by default. A design of creating one health check config default per resource is implemented. The choice eases adoption of health checks, supports existing clusters that already have database health checks, and avoids migrating the backend database. A new Kubernetes-specific `default-kube` health check config is added. And a database-specific `default` health check config already exists, and is preserved. A virtual default design is implemented by returning health check configs from memory if they don't exist in the backend database. The approach has the benefit of not re-inserting default values to the backend after they're deleted, which a prior approach had. Virtual defaults are added at the local health check service level, and returned from functions `GetHealthCheckConfig` and `ListHealthCheckConfigs`. Virtual defaults may be written, updated, and deleted to and from the backend. While virtual defaults may be deleted, it has the net effect of resetting the config to default settings, and matching all resources of that type (db, kube). Virtual defaults are always returned from health check `get` and `list` functions. Changes: - Added `default-kube` health check config specific to Kubernetes only - Updated local service functions `GetHealthCheckConfig` and `ListHealthCheckConfigs` to return virtual defaults - Added unit tests - Updated health check documentation with `default-kube` and info about virtual defaults Part of #58413 Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>

The generic `Resources` function is backported to the generic `Service` type. `Resources` is used in a related commit enabling health checks for Kubernetes (#60544). Part of #58413

Health checks are enabled for all Kubernetes clusters by default. A design of creating one health check config default per resource is implemented. The choice eases adoption of health checks, supports existing clusters that already have database health checks, and avoids migrating the backend database. A new Kubernetes-specific `default-kube` health check config is added. And a database-specific `default` health check config already exists, and is preserved. A virtual default design is implemented by returning health check configs from memory if they don't exist in the backend database. The approach has the benefit of not re-inserting default values to the backend after they're deleted, which a prior approach had. Virtual defaults are added at the local health check service level, and returned from functions `GetHealthCheckConfig` and `ListHealthCheckConfigs`. Virtual defaults may be written, updated, and deleted to and from the backend. While virtual defaults may be deleted, it has the net effect of resetting the config to default settings, and matching all resources of that type (db, kube). Virtual defaults are always returned from health check `get` and `list` functions. Changes: - Added `default-kube` health check config specific to Kubernetes only - Updated local service functions `GetHealthCheckConfig` and `ListHealthCheckConfigs` to return virtual defaults - Added unit tests - Updated health check documentation with `default-kube` and info about virtual defaults Part of #58413 Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>

The generic `Resources` function is backported to the generic `Service` type. `Resources` is used in a related commit enabling health checks for Kubernetes (#60544). Part of #58413

* Add protobufs for Kubernetes health checks (#58415) - Add Kubernetes label matchers to `Matcher` for `HealthCheckConfig` - Add message `KubernetesServerStatusV3` - Add `status` field to `KubernetesServerV3` - Add `target_health` field to `Kube` for UI - Regenerate Terraform schema and docs for `HealthCheckConfig` - Add Kubernetes label matchers to Terraform test `TestImportHealthCheckConfig` Relates to #58413 Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> * Refactor `healthcheck` for Kubernetes extensibility with `HealthChecker` interface (#59396) The main intent of refactoring is to provide health check extensibility for Kubernetes while supporting the existing DB health checks. A `HealthChecker` interface is added to support the different health check approaches of DBs and Kubernetes. Existing DB TCP health check logic is moved to a new `TargetDialer` struct. Changes: - Added `HealthChecker` interface with two functions: - `CheckHealth(ctx context.Context) ([]string, error)` - `GetProtocol() types.TargetHealthProtocol` - Added `TargetDialer` struct which encapsulates existing TCP health check logic - Changed `Target` struct to use the `HealthChecker` interface - Changed `worker.checkHealth` to call the new `CheckHealth` function - Removed a `protocol` field from `healthCheckConfig` - Added `TargetHealthProtocolHTTP` for use with Kubernetes health checks - Moved and renamed test `Test_dialEndpoints` to `TestTargetDialer_dialEndpoints` - Added files `net.go` and `net_test.go` for `TargetDialer` Part of #58413 * Add health checks to Kubernetes agent and proxy (#59565) Implements core health check logic for Kubernetes. - Added functions `CheckHealth` and `GetProtocol` to `kubeDetails` - Added a `HealthCheckManager` field to the Kubernetes `TLSServerConfig` - Kube agent and proxy now instantiate a `HealthCheckManager` - Added a `HealthCheckConfigReader` to interfaces `ReadKubernetesAccessPoint` and `ReadProxyAccessPoint` - Added `HealthCheckConfig` read-only permissions for proxy and kube - Added `HealthCheckConfig` watching for proxy and kube - Added functions to `KubeServer` interface and `KubernetesServerV3` struct: - `GetTargetHealth() TargetHealth` - `SetTargetHealth(h TargetHealth)` - `GetTargetHealthStatus() TargetHealthStatus` - `SetTargetHealthStatus(status TargetHealthStatus)` - Health check supports dynamic discovery of Kubernetes clusters. Calls to `startHealthCheck()` and `stopHealthCheck()` were rearranged. - Added functions `startHeartbeatAndHealthCheck()` and `stopHeartbeatAndHealthCheck()` - Moved call to `HealthCheckManager.Start()` outside of the Kubernetes proxy server providing a future option to reuse `HealthCheckManager` in multiple proxy services - Removed `Status` initialization in KubernetesServerV3 `CheckAndSetDefaults()` - Added `kubernetesLabelMatchers` to `healthCheckConfig` struct. Default presets are omitted until the entire kube health check is complete. - Added Kubernetes matcher checking to `ValidateHealthCheckConfig()` - Changed `ValidateHealthCheckConfig()` to allow zero total DB matchers and Kubernetes matchers. - Changed KubernetesServerV3 `GetTargetHealthStatus()` to return `TargetHealthStatusUnknown` instead of an empty string - Changed health check worker `getTargetHealthTimeout` default timeout to `4s` from `10s`. This potentially reduces the response time of the initial heartbeat polling call to `GetServerInfo()`. Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> * Add UI for Kubernetes health checks (#59929) Adds health status indicators for Kubernetes clusters on the Resources page. Unhealthy clusters are highlighted, and clicking them opens a side panel displaying server information. Changes include: - New `KubeServer` protobuf message and `ListKubernetesServers` RPC - Web and Connect API endpoints for fetching Kubernetes server data - Health status filtering in `matchAndFilterKubeClusters` - `TargetHealth` fields added to frontend/backend types - Updated `StatusInfo.tsx` to display `kube_cluster` data Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> * Add health-based connection routing for Kubernetes (#60083) Kubernetes servers are grouped by health, and dialed in order of healthy, unknown, then unhealthy, with random shuffling within each group for load distribution. The grouping and shuffling is implemented generically in a separate iterator function `OrderByTargetHealthStatus()` for reuse and testability. Changes: - Added `healthcheck.OrderByTargetHealthStatus()` function with tests Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> * Allow disabling health check configs (#60447) Health check configuration can be disabled and enabled through the matcher. Disabling the matcher disables health checks for this configuration's resources including databases, Kubernetes clusters, and any future resources. Changes: - Added `disabled` field to health check matcher proto - Updated matcher selection logic - Updated unit tests Part of #58413 * Add documentation for Kubernetes health checks (#60201) A new Kubernetes-specific health check page is added. The existing Kubernetes troubleshooting documentation page is updated with health check specific error resolutions. Part of #58413 Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> Co-authored-by: Gavin Frazar <gavin.frazar@goteleport.com> Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com> * Enable health checks for Kubernetes with virtual defaults (#60544) Health checks are enabled for all Kubernetes clusters by default. A design of creating one health check config default per resource is implemented. The choice eases adoption of health checks, supports existing clusters that already have database health checks, and avoids migrating the backend database. A new Kubernetes-specific `default-kube` health check config is added. And a database-specific `default` health check config already exists, and is preserved. A virtual default design is implemented by returning health check configs from memory if they don't exist in the backend database. The approach has the benefit of not re-inserting default values to the backend after they're deleted, which a prior approach had. Virtual defaults are added at the local health check service level, and returned from functions `GetHealthCheckConfig` and `ListHealthCheckConfigs`. Virtual defaults may be written, updated, and deleted to and from the backend. While virtual defaults may be deleted, it has the net effect of resetting the config to default settings, and matching all resources of that type (db, kube). Virtual defaults are always returned from health check `get` and `list` functions. Changes: - Added `default-kube` health check config specific to Kubernetes only - Updated local service functions `GetHealthCheckConfig` and `ListHealthCheckConfigs` to return virtual defaults - Added unit tests - Updated health check documentation with `default-kube` and info about virtual defaults Part of #58413 Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> * Add `Resources` function for v18 backport The generic `Resources` function is backported to the generic `Service` type. `Resources` is used in a related commit enabling health checks for Kubernetes (#60544). Part of #58413 * Fix test for terraform health check config (#60640) Filters virtual defaults to allow explicit config references. Part of #58413 --------- Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com> Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> Co-authored-by: Gavin Frazar <gavin.frazar@goteleport.com> Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com>

Health checks are enabled for all Kubernetes clusters by default. A design of creating one health check config default per resource is implemented. The choice eases adoption of health checks, supports existing clusters that already have database health checks, and avoids migrating the backend database. A new Kubernetes-specific `default-kube` health check config is added. And a database-specific `default` health check config already exists, and is preserved. A virtual default design is implemented by returning health check configs from memory if they don't exist in the backend database. The approach has the benefit of not re-inserting default values to the backend after they're deleted, which a prior approach had. Virtual defaults are added at the local health check service level, and returned from functions `GetHealthCheckConfig` and `ListHealthCheckConfigs`. Virtual defaults may be written, updated, and deleted to and from the backend. While virtual defaults may be deleted, it has the net effect of resetting the config to default settings, and matching all resources of that type (db, kube). Virtual defaults are always returned from health check `get` and `list` functions. Changes: - Added `default-kube` health check config specific to Kubernetes only - Updated local service functions `GetHealthCheckConfig` and `ListHealthCheckConfigs` to return virtual defaults - Added unit tests - Updated health check documentation with `default-kube` and info about virtual defaults Part of #58413 Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>

rana added kubernetes-access no-changelog Indicates that a PR does not require a changelog entry health-check Resource health check related labels Oct 24, 2025

rana mentioned this pull request Oct 24, 2025

Kubernetes access health checks #58413

Closed

rana force-pushed the rana/kube-healthchecks-14 branch from b122448 to 3525fe9 Compare October 24, 2025 06:28

rana requested review from GavinFrazar, espadolini, okraport and rosstimothy October 24, 2025 06:51

rana marked this pull request as ready for review October 24, 2025 06:51

github-actions bot requested a review from atburke October 24, 2025 06:52

github-actions bot added size/lg tctl tctl - Teleport admin tool labels Oct 24, 2025

espadolini mentioned this pull request Oct 24, 2025

Add a virtual default health check config for kube #60552

Closed

espadolini previously requested changes Oct 24, 2025

View reviewed changes

Comment thread constants.go Outdated

Comment thread lib/auth/healthcheckconfig/healthcheckconfigv1/service.go Outdated

Comment thread tool/tctl/common/resource_health_check_config.go Outdated

rana requested review from okraport and removed request for atburke and okraport October 24, 2025 14:46

rana force-pushed the rana/kube-healthchecks-14 branch from 3525fe9 to 52d6cfd Compare October 24, 2025 16:52

rana mentioned this pull request Oct 24, 2025

[v18] Add health checks to Kubernetes #60492

Merged

37 tasks

rana changed the title ~~Enable health checks for Kubernetes by default~~ Enable health checks for Kubernetes with virtual defaults Oct 24, 2025

rosstimothy reviewed Oct 24, 2025

View reviewed changes

Comment thread lib/auth/init.go

rana force-pushed the rana/kube-healthchecks-14 branch from 52d6cfd to 31d51d4 Compare October 25, 2025 06:37

rana temporarily deployed to docs-amplify October 25, 2025 06:37 — with GitHub Actions Inactive

rana force-pushed the rana/kube-healthchecks-14 branch from 31d51d4 to a582a3a Compare October 25, 2025 07:21

rana temporarily deployed to docs-amplify October 25, 2025 07:21 — with GitHub Actions Inactive

rana force-pushed the rana/kube-healthchecks-14 branch from a582a3a to 1ce1b37 Compare October 25, 2025 23:07

rana temporarily deployed to docs-amplify October 25, 2025 23:07 — with GitHub Actions Inactive

rana force-pushed the rana/kube-healthchecks-14 branch from 263034a to 4294d98 Compare October 26, 2025 21:04

rana temporarily deployed to docs-amplify October 26, 2025 21:04 — with GitHub Actions Inactive

rana requested review from espadolini and rosstimothy October 26, 2025 21:26

okraport reviewed Oct 27, 2025

View reviewed changes

espadolini approved these changes Oct 27, 2025

View reviewed changes

hugoShaka approved these changes Oct 27, 2025

View reviewed changes

public-teleport-github-review-bot bot removed the request for review from GavinFrazar October 27, 2025 14:30

rana force-pushed the rana/kube-healthchecks-14 branch from 4294d98 to f58bcf2 Compare October 27, 2025 16:37

rana temporarily deployed to docs-amplify October 27, 2025 16:37 — with GitHub Actions Inactive

rana force-pushed the rana/kube-healthchecks-14 branch from f58bcf2 to e9d6356 Compare October 27, 2025 16:58

rana temporarily deployed to docs-amplify October 27, 2025 16:58 — with GitHub Actions Inactive

rana added this pull request to the merge queue Oct 27, 2025

Merged via the queue into master with commit 1fcc83a Oct 27, 2025
44 checks passed

rana deleted the rana/kube-healthchecks-14 branch October 27, 2025 17:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable health checks for Kubernetes with virtual defaults#60544

Enable health checks for Kubernetes with virtual defaults#60544
rana merged 1 commit intomasterfrom
rana/kube-healthchecks-14

rana commented Oct 24, 2025 •

edited

Loading

Uh oh!

espadolini left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 25, 2025 •

edited

Loading

Uh oh!

okraport left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

espadolini Oct 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

rana commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

espadolini left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

okraport left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

espadolini Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rana commented Oct 24, 2025 •

edited

Loading

espadolini left a comment •

edited

Loading

github-actions bot commented Oct 25, 2025 •

edited

Loading