Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Prometheus metrics for cumulative task transition counts on workers #8697

Open
hendrikmakait opened this issue Jun 13, 2024 · 2 comments
Labels
diagnostics feature Something is missing

Comments

@hendrikmakait
Copy link
Member

Right now, we have the current count of tasks in the various state on the workers exposed as dask_worker_tasks. In some scenarios, we're more interested in the total count and its rate of change, so we should add another metric that tracks cumulative counts.

@hendrikmakait hendrikmakait added diagnostics feature Something is missing labels Jun 13, 2024
@github-actions github-actions bot added needs triage and removed diagnostics feature Something is missing labels Jun 13, 2024
@hendrikmakait hendrikmakait added diagnostics feature Something is missing and removed needs triage labels Jun 13, 2024
@hendrikmakait hendrikmakait changed the title Add Prometheus metrics for cumulative task counts on workers Add Prometheus metrics for cumulative taskn transition counts on workers Jun 13, 2024
@hendrikmakait hendrikmakait changed the title Add Prometheus metrics for cumulative taskn transition counts on workers Add Prometheus metrics for cumulative task transition counts on workers Jun 13, 2024
@fjetter
Copy link
Member

fjetter commented Jun 13, 2024

I'm not sure I agree with this. Why can't we track the rate of change of the existing metric? What additional value would this extra cumulative counter add? I am concerned that this is too much noise. Historically, I found the worker level metrics not to be very valuable

@hendrikmakait
Copy link
Member Author

The existing metric is a point query, so it doesn't provide an accurate picture of the change that happens on the worker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
diagnostics feature Something is missing
Projects
None yet
Development

No branches or pull requests

2 participants