You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Whenever a service is burning too much of its error budget the alerts fire.
Because we have multi error burn rates for the API server it would be helpful to visualize them in 4 graphs:
For example, in this particular case, my second most critical alert was flaky and here you can clearly see why (second graph).
In my case, since I sadly cannot change my cloud providers api server (other than escalating), the only option is to reduce the objective to a more realistic target (it's pretty much just the latencies).
Not only should the warning alerts stop firing (since less error budget is burned) but also that second alert should stop being as flaky.
In case someone wants to give it a shot implementing these 4 graphs with jsonnet tooling (my screenshot is just written by hand and doesn't adjust to individual objectives), I left this issue. Maybe I'll get to it soon, but until then I'm happy to help contributors instead
The text was updated successfully, but these errors were encountered:
Whenever a service is burning too much of its error budget the alerts fire.
Because we have multi error burn rates for the API server it would be helpful to visualize them in 4 graphs:
For example, in this particular case, my second most critical alert was flaky and here you can clearly see why (second graph).
In my case, since I sadly cannot change my cloud providers api server (other than escalating), the only option is to reduce the objective to a more realistic target (it's pretty much just the latencies).
Not only should the warning alerts stop firing (since less error budget is burned) but also that second alert should stop being as flaky.
The queries for the first panel are:
In case someone wants to give it a shot implementing these 4 graphs with jsonnet tooling (my screenshot is just written by hand and doesn't adjust to individual objectives), I left this issue. Maybe I'll get to it soon, but until then I'm happy to help contributors instead
The text was updated successfully, but these errors were encountered: