Add gardener_jobs_total error rate to error panel #934
Conversation
There was a problem hiding this comment.
There is a part of me that feels like this might not belong with the other two time series. It's taking the daily rate (not 5 min increase) and the legend shows the datatype (not the status).
Reviewable status:
complete! 1 of 1 approvals obtained
There was a problem hiding this comment.
Thank you. I've updated the time ranges to match, to 10m for all three.
I think the datatype error is more helpful than the other weird status errors from gardener. But, I wanted to surface the weird status errors since they are related to the root causes of some failures. Perhaps a second panel?
Reviewable status:
There was a problem hiding this comment.
Thanks. My only observation is that the legend values are not really homogeneous.
I guess it will make sense to whoever looks at the queries anyway. You can add a second panel or leave them in the same one.
Reviewable status:
There was a problem hiding this comment.
I agree these legend values are strange. In my mind making this more visible encourages fixing the underlying source more than keeping them hidden; I only discovered them recently. And, some interpretation is required by the viewer until they are more helpful. I'll leave the queries in one panel for now, and acknowledge that there is room for improvement here. My main goal for these changes was just to surface more information first and refine second. Prior to #933 this dashboard had no visible gardener errors which made it less helpful for serendipitously noticing issues or investigating known issues.
Reviewable status:
This change adds a new time series to the gardener error panel added in #933. The new time series matches the gardener jobs total error rate used in the
GardenerFailureRateTooHighOrMissingalert.This change is