Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📈 feat(rt): expose tokio runtime metrics #3177

Merged
merged 5 commits into from
Sep 10, 2024

Conversation

cratelyn
Copy link
Collaborator

@cratelyn cratelyn commented Sep 5, 2024

Tokio has an unstable feature supporting runtime metrics.

This allows service operators to export metrics to systems like
Prometheus, so that they can observe how their workloads are performing
on the tokio runtime. This exposes information like the number of worker
threads, queue depth, the number of tasks polled, and so on.

linkerd2-proxy should expose these metrics.

This uses the kubert-prometheus-tokio crate to register a Runtime
metrics worker, and spawn a task to probe these metrics at a fixed,
regular interval.

see: https://github.com/olix0r/kubert/tree/main/kubert-prometheus-tokio

if the tokio_unstable feature is not enabled, this will emit a debug
event and do nothing.

@cratelyn cratelyn added the enhancement New feature or request label Sep 5, 2024
@cratelyn cratelyn self-assigned this Sep 5, 2024
linkerd2-proxy/src/main.rs Outdated Show resolved Hide resolved
@cratelyn
Copy link
Collaborator Author

cratelyn commented Sep 5, 2024

this still needs to wire up changes to release.yml to enable the tokio_unstable flag, but i'm opening this as a draft while i sort that out.

@cratelyn cratelyn force-pushed the kate/expose-tokio-runtime-metrics-wip branch 2 times, most recently from 662c146 to 7c08a0c Compare September 5, 2024 20:27
@cratelyn

This comment was marked as resolved.

@cratelyn cratelyn force-pushed the kate/expose-tokio-runtime-metrics-wip branch 2 times, most recently from 524445c to 415e459 Compare September 6, 2024 12:52
@cratelyn

This comment was marked as resolved.

@cratelyn cratelyn force-pushed the kate/expose-tokio-runtime-metrics-wip branch from 50b89b9 to 34e91ba Compare September 6, 2024 14:07
linkerd/tls/Cargo.toml Outdated Show resolved Hide resolved
@cratelyn cratelyn force-pushed the kate/expose-tokio-runtime-metrics-wip branch from 34e91ba to b65de6a Compare September 6, 2024 14:17
@cratelyn

This comment was marked as resolved.

@cratelyn cratelyn force-pushed the kate/expose-tokio-runtime-metrics-wip branch from 673c77f to 93fde9f Compare September 6, 2024 15:00
@cratelyn cratelyn force-pushed the kate/expose-tokio-runtime-metrics-wip branch 2 times, most recently from 5a9aeee to bc19e6d Compare September 9, 2024 13:58
Tokio has an unstable feature supporting runtime metrics.

This allows service operators to export metrics to systems like
Prometheus, so that they can observe how their workloads are performing
on the tokio runtime. This exposes information like the number of worker
threads, queue depth, the number of tasks polled, and so on.

`linkerd2-proxy` should expose these metrics.

This uses the `kubert-prometheus-tokio` crate to register a `Runtime`
metrics worker, and spawn a task to probe these metrics at a fixed,
regular interval.

see: <https://github.com/olix0r/kubert/tree/main/kubert-prometheus-tokio>

if the `tokio_unstable` feature is not enabled, this will emit a debug
event and do nothing.

Signed-off-by: katelyn martin <[email protected]>
@cratelyn cratelyn force-pushed the kate/expose-tokio-runtime-metrics-wip branch from bc19e6d to b4496a0 Compare September 9, 2024 14:31
@cratelyn cratelyn marked this pull request as ready for review September 9, 2024 14:43
@cratelyn cratelyn requested a review from a team as a code owner September 9, 2024 14:43
@cratelyn cratelyn requested a review from olix0r September 9, 2024 14:43
@cratelyn
Copy link
Collaborator Author

cratelyn commented Sep 9, 2024

@olix0r this branch is now ready for another round of review 🔍 thank you for your assistance in diagnosing runtime panics!

Copy link
Member

@olix0r olix0r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

linkerd2-proxy/src/rt.rs Outdated Show resolved Hide resolved
Co-authored-by: Oliver Gould <[email protected]>
Signed-off-by: katelyn martin <[email protected]>
@cratelyn cratelyn force-pushed the kate/expose-tokio-runtime-metrics-wip branch from 5230cf8 to 0285e8a Compare September 9, 2024 21:04
@olix0r olix0r merged commit 193dcaf into main Sep 10, 2024
17 checks passed
@olix0r olix0r deleted the kate/expose-tokio-runtime-metrics-wip branch September 10, 2024 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants