Skip to content

Add Prometheus metrics to Kubernetes Access#29363

Merged
tigrato merged 4 commits intomasterfrom
tigrato/kube-add-prometheus-metrics
Jul 27, 2023
Merged

Add Prometheus metrics to Kubernetes Access#29363
tigrato merged 4 commits intomasterfrom
tigrato/kube-add-prometheus-metrics

Conversation

@tigrato
Copy link
Copy Markdown
Contributor

@tigrato tigrato commented Jul 20, 2023

This PR adds several prometheus metrics to the Kubernetes Access package.

The new metrics are:

Name Type Component Description
teleport_kubernetes_client_in_flight_requests gauge Teleport Kubernetes Service In-flight requests waiting for the upstream response.
teleport_kubernetes_client_requests_total counter Teleport Kubernetes Service Total number of requests sent to the upstream teleport proxy, kube_service or Kubernetes Cluster servers.
teleport_kubernetes_client_tls_duration_seconds histogram Teleport Kubernetes Service Latency distribution of TLS handshakes.
teleport_kubernetes_client_got_conn_duration_seconds histogram Teleport Kubernetes Service Latency distribution of time to dial to the upstream server - using reversetunnel or direct dialer.
teleport_kubernetes_client_first_byte_response_duration_seconds histogram Teleport Kubernetes Service Latency distribution of time to receive the first response byte from the upstream server.
teleport_kubernetes_client_request_duration_seconds histogram Teleport Kubernetes Service Latency distribution of the upstream request time.
Name Type Component Description
teleport_kubernetes_server_in_flight_requests gauge Teleport Kubernetes Service In-flight requests currently handled by the server.
teleport_kubernetes_server_api_requests_total counter Teleport Kubernetes Service Total number of requests handled by the server.
teleport_kubernetes_server_request_duration_seconds histogram Teleport Kubernetes Service Latency distribution of the total request time.
teleport_kubernetes_server_response_size_bytes histogram Teleport Kubernetes Service Distribution of the response size.
teleport_kubernetes_server_exec_in_flight_sessions gauge Teleport Kubernetes Service Number of active kubectl exec sessions.
teleport_kubernetes_server_exec_sessions_total counter Teleport Kubernetes Service Total number of kubectl exec sessions.
teleport_kubernetes_server_portforward_in_flight_sessions gauge Teleport Kubernetes Service Number of active kubectl portforward sessions.
teleport_kubernetes_server_portforward_sessions_total counter Teleport Kubernetes Service Number of active kubectl portforward sessions.
teleport_kubernetes_server_join_in_flight_sessions gauge Teleport Kubernetes Service Number of active joining sessions,
teleport_kubernetes_server_join_sessions_total counter Teleport Kubernetes Service Total number of joining sessions.

Closes #29362

This PR adds several prometheus metrics to the Kubernetes Access
package.

The new metrics are:

| Name                                                              | Type      | Component                   | Description                                                                                               |
| ----------------------------------------------------------------- | --------- | --------------------------- | --------------------------------------------------------------------------------------------------------- |
| `teleport_kubernetes_client_in_flight_requests`                   | gauge     | Teleport Kubernetes Service | In-flight requests waiting for the upstream response. |
| `teleport_kubernetes_client_requests_total`                       | counter   | Teleport Kubernetes Service | Total number of requests sent to the upstream teleport proxy, kube_service or Kubernetes Cluster servers. |
| `teleport_kubernetes_client_tls_duration_seconds`                 | histogram | Teleport Kubernetes Service | Latency distribution of TLS handshakes.                                                                   |
| `teleport_kubernetes_client_got_conn_duration_seconds`            | histogram | Teleport Kubernetes Service | Latency distribution of time to dial to the upstream server - using reversetunnel or direct dialer.       |
| `teleport_kubernetes_client_first_byte_response_duration_seconds` | histogram | Teleport Kubernetes Service | Latency distribution of time to receive the first response byte from the upstream server.                 |
| `teleport_kubernetes_client_request_duration_seconds`             | histogram | Teleport Kubernetes Service | Latency distribution of the upstream request time.                                                        |

| Name                                                        | Type      | Component                   | Description                                         |
| ----------------------------------------------------------- | --------- | --------------------------- | --------------------------------------------------- |
| `teleport_kubernetes_server_in_flight_requests`             | gauge     | Teleport Kubernetes Service | In-flight requests currently handled by the server. |
| `teleport_kubernetes_server_api_requests_total`             | counter   | Teleport Kubernetes Service | Total number of requests handled by the server.     |
| `teleport_kubernetes_server_request_duration_seconds`       | histogram | Teleport Kubernetes Service | Latency distribution of the total request time.     |
| `teleport_kubernetes_server_response_size_bytes`            | histogram | Teleport Kubernetes Service | Distribution of the response size.                  |
| `teleport_kubernetes_server_exec_in_flight_sessions`        | gauge     | Teleport Kubernetes Service | Number of active kubectl exec sessions.             |
| `teleport_kubernetes_server_exec_sessions_total`            | counter   | Teleport Kubernetes Service | Total number of kubectl exec sessions.              |
| `teleport_kubernetes_server_portforward_in_flight_sessions` | gauge     | Teleport Kubernetes Service | Number of active kubectl portforward sessions.      |
| `teleport_kubernetes_server_portforward_sessions_total`     | counter   | Teleport Kubernetes Service | Number of active kubectl portforward sessions.      |
| `teleport_kubernetes_server_join_in_flight_sessions`        | gauge     | Teleport Kubernetes Service | Number of active joining sessions,                  |
| `teleport_kubernetes_server_join_sessions_total`            | counter   | Teleport Kubernetes Service | Total number of joining sessions.                   |

Closes #29362

Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>
@tigrato tigrato force-pushed the tigrato/kube-add-prometheus-metrics branch from 733fba0 to 9724c54 Compare July 20, 2023 15:06
Comment thread lib/kube/proxy/prometheus.go Outdated
Comment thread docs/pages/includes/metrics.mdx Outdated
Comment thread docs/pages/includes/metrics.mdx Outdated
Comment thread docs/pages/includes/metrics.mdx Outdated
Comment thread docs/pages/includes/metrics.mdx Outdated
Comment thread docs/pages/includes/metrics.mdx Outdated
Co-authored-by: Alex Fornuto <alex.fornuto@goteleport.com>
@tigrato tigrato requested a review from alexfornuto July 26, 2023 17:07
@tigrato tigrato added this pull request to the merge queue Jul 27, 2023
Merged via the queue into master with commit 9e7154d Jul 27, 2023
@tigrato tigrato deleted the tigrato/kube-add-prometheus-metrics branch July 27, 2023 09:43
tigrato added a commit that referenced this pull request Aug 3, 2023
* Add Prometheus metrics to Kubernetes Access

This PR adds several prometheus metrics to the Kubernetes Access
package.

The new metrics are:

| Name                                                              | Type      | Component                   | Description                                                                                               |
| ----------------------------------------------------------------- | --------- | --------------------------- | --------------------------------------------------------------------------------------------------------- |
| `teleport_kubernetes_client_in_flight_requests`                   | gauge     | Teleport Kubernetes Service | In-flight requests waiting for the upstream response. |
| `teleport_kubernetes_client_requests_total`                       | counter   | Teleport Kubernetes Service | Total number of requests sent to the upstream teleport proxy, kube_service or Kubernetes Cluster servers. |
| `teleport_kubernetes_client_tls_duration_seconds`                 | histogram | Teleport Kubernetes Service | Latency distribution of TLS handshakes.                                                                   |
| `teleport_kubernetes_client_got_conn_duration_seconds`            | histogram | Teleport Kubernetes Service | Latency distribution of time to dial to the upstream server - using reversetunnel or direct dialer.       |
| `teleport_kubernetes_client_first_byte_response_duration_seconds` | histogram | Teleport Kubernetes Service | Latency distribution of time to receive the first response byte from the upstream server.                 |
| `teleport_kubernetes_client_request_duration_seconds`             | histogram | Teleport Kubernetes Service | Latency distribution of the upstream request time.                                                        |

| Name                                                        | Type      | Component                   | Description                                         |
| ----------------------------------------------------------- | --------- | --------------------------- | --------------------------------------------------- |
| `teleport_kubernetes_server_in_flight_requests`             | gauge     | Teleport Kubernetes Service | In-flight requests currently handled by the server. |
| `teleport_kubernetes_server_api_requests_total`             | counter   | Teleport Kubernetes Service | Total number of requests handled by the server.     |
| `teleport_kubernetes_server_request_duration_seconds`       | histogram | Teleport Kubernetes Service | Latency distribution of the total request time.     |
| `teleport_kubernetes_server_response_size_bytes`            | histogram | Teleport Kubernetes Service | Distribution of the response size.                  |
| `teleport_kubernetes_server_exec_in_flight_sessions`        | gauge     | Teleport Kubernetes Service | Number of active kubectl exec sessions.             |
| `teleport_kubernetes_server_exec_sessions_total`            | counter   | Teleport Kubernetes Service | Total number of kubectl exec sessions.              |
| `teleport_kubernetes_server_portforward_in_flight_sessions` | gauge     | Teleport Kubernetes Service | Number of active kubectl portforward sessions.      |
| `teleport_kubernetes_server_portforward_sessions_total`     | counter   | Teleport Kubernetes Service | Number of active kubectl portforward sessions.      |
| `teleport_kubernetes_server_join_in_flight_sessions`        | gauge     | Teleport Kubernetes Service | Number of active joining sessions,                  |
| `teleport_kubernetes_server_join_sessions_total`            | counter   | Teleport Kubernetes Service | Total number of joining sessions.                   |

Closes #29362

Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>

* replace prometheus.MustRegister  with metrics.RegisterPrometheusCollectors

* Apply suggestions from code review

Co-authored-by: Alex Fornuto <alex.fornuto@goteleport.com>

---------

Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>
Co-authored-by: Alex Fornuto <alex.fornuto@goteleport.com>
github-merge-queue Bot pushed a commit that referenced this pull request Aug 3, 2023
* Add Prometheus metrics to Kubernetes Access

This PR adds several prometheus metrics to the Kubernetes Access
package.

The new metrics are:

| Name                                                              | Type      | Component                   | Description                                                                                               |
| ----------------------------------------------------------------- | --------- | --------------------------- | --------------------------------------------------------------------------------------------------------- |
| `teleport_kubernetes_client_in_flight_requests`                   | gauge     | Teleport Kubernetes Service | In-flight requests waiting for the upstream response. |
| `teleport_kubernetes_client_requests_total`                       | counter   | Teleport Kubernetes Service | Total number of requests sent to the upstream teleport proxy, kube_service or Kubernetes Cluster servers. |
| `teleport_kubernetes_client_tls_duration_seconds`                 | histogram | Teleport Kubernetes Service | Latency distribution of TLS handshakes.                                                                   |
| `teleport_kubernetes_client_got_conn_duration_seconds`            | histogram | Teleport Kubernetes Service | Latency distribution of time to dial to the upstream server - using reversetunnel or direct dialer.       |
| `teleport_kubernetes_client_first_byte_response_duration_seconds` | histogram | Teleport Kubernetes Service | Latency distribution of time to receive the first response byte from the upstream server.                 |
| `teleport_kubernetes_client_request_duration_seconds`             | histogram | Teleport Kubernetes Service | Latency distribution of the upstream request time.                                                        |

| Name                                                        | Type      | Component                   | Description                                         |
| ----------------------------------------------------------- | --------- | --------------------------- | --------------------------------------------------- |
| `teleport_kubernetes_server_in_flight_requests`             | gauge     | Teleport Kubernetes Service | In-flight requests currently handled by the server. |
| `teleport_kubernetes_server_api_requests_total`             | counter   | Teleport Kubernetes Service | Total number of requests handled by the server.     |
| `teleport_kubernetes_server_request_duration_seconds`       | histogram | Teleport Kubernetes Service | Latency distribution of the total request time.     |
| `teleport_kubernetes_server_response_size_bytes`            | histogram | Teleport Kubernetes Service | Distribution of the response size.                  |
| `teleport_kubernetes_server_exec_in_flight_sessions`        | gauge     | Teleport Kubernetes Service | Number of active kubectl exec sessions.             |
| `teleport_kubernetes_server_exec_sessions_total`            | counter   | Teleport Kubernetes Service | Total number of kubectl exec sessions.              |
| `teleport_kubernetes_server_portforward_in_flight_sessions` | gauge     | Teleport Kubernetes Service | Number of active kubectl portforward sessions.      |
| `teleport_kubernetes_server_portforward_sessions_total`     | counter   | Teleport Kubernetes Service | Number of active kubectl portforward sessions.      |
| `teleport_kubernetes_server_join_in_flight_sessions`        | gauge     | Teleport Kubernetes Service | Number of active joining sessions,                  |
| `teleport_kubernetes_server_join_sessions_total`            | counter   | Teleport Kubernetes Service | Total number of joining sessions.                   |

Closes #29362



* replace prometheus.MustRegister  with metrics.RegisterPrometheusCollectors

* Apply suggestions from code review



---------

Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>
Co-authored-by: Alex Fornuto <alex.fornuto@goteleport.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add metrics to Kubernetes access flow

4 participants