Merge pull request #279 from grafana/increase-cortex-request-error-level

Increased CortexRequestErrors alert severity

Author: pracucci

CHANGELOG.md

@@ -15,6 +15,7 @@
   git fetch origin
   git branch -u origin/main main
   ```
+* [CHANGE] `CortexRequestErrors` alert severity raised from `warning` to `critical`. #279
 * [FEATURE] Added "Cortex / Slow queries" dashboard based on Loki logs. #271
 * [ENHANCEMENT] Add `EtcdAllocatingTooMuchMemory` alert for monitoring etcd memory usage. #261
 * [ENHANCEMENT] Sort legend descending in the CPU/memory panels. #271

cortex-mixin/alerts/alerts.libsonnet

@@ -18,7 +18,7 @@
         },
         {
           alert: 'CortexRequestErrors',
-          // Note is alert_aggregation_labels is "job", this will repeat the label.  But
+          // Note if alert_aggregation_labels is "job", this will repeat the label. But
           // prometheus seems to tolerate that.
           expr: |||
             100 * sum by (%s, job, route) (rate(cortex_request_duration_seconds_count{status_code=~"5..",route!~"ready"}[1m]))
@@ -28,7 +28,7 @@
           ||| % [$._config.alert_aggregation_labels, $._config.alert_aggregation_labels],
           'for': '15m',
           labels: {
-            severity: 'warning',
+            severity: 'critical',
           },
           annotations: {
             message: |||