availability and latency slos (#596)

* availability and latency slos * changing latency slo format to match sloth and adding unti tests and make target * new sloth make target and comments for promtool and feedback * Update the sloth-generate target to output PrometheusRules files * update tail in make target and linter (•̀o•́)ง * linter (•̀o•́)ง --------- Co-authored-by: David Martin <[email protected]>
Kuadrant · May 10, 2024 · b3c3813 · b3c3813
1 parent 038f4fb
commit b3c3813
Show file tree

Hide file tree

Showing 8 changed files with 660 additions and 0 deletions.
diff --git a/examples/alerts/README.md b/examples/alerts/README.md
@@ -0,0 +1,41 @@
+## SLO Multi burn rate multi window alerts
+Kuadrant have created two example SLO alerts to help give ideas on the types of SLO alerts that could be used with the operator. We have created one alert for latency and one for availability, both are Multiwindow, Multi-Burn-Rate Alerts. The alerts show a scenario where a 28d rolling window is used and a uptime of 99.95% i.e only 0.05% error budget margin is desired. This in real world time would be downtime of around: 
+
+| Time Frame | Duration   |
+|------------|------------|
+| Daily:     | 43s        |
+| Weekly:    | 5m 2.4s    |
+| Monthly:   | 21m 44s    |
+| Quarterly: | 1h 5m 12s  |
+| Yearly:    | 4h 20m 49s |
+
+These values can be changed to suit different scenarios
+
+### Sloth
+Sloth is a tool to aid in the creation of multi burn rate and multi window SLO alerts and was used to create both the availability and latency alerts. It follows the common standard set out by [Google's SRE book](https://sre.google/workbook/implementing-slos/). Sloth generates alerts based on specific specs given. The specs for our example alerts can be found in the example/sloth folder.
+
+#### Metrics used for the alerts
+
+#### Availability
+For the availability SLO alerts the Istio metric `istio_requests_total` was used as its a counter type metric meaning the values can only increase as well as it gives information on all requests handled by the Istio proxy.
+
+#### Latency
+For the availability SLO alerts the Istio metric `istio_request_duration_milliseconds` was used as its a [Histogram](https://prometheus.io/docs/concepts/metric_types/#histogram).
+
+### Sloth generation
+You can modify the examples Sloth specs we have and regenerate the prometheus rules using the Sloth CLI and the generate command. For more information please the [Sloth website](https://sloth.dev/usage/cli/)
+
+```
+sloth generate -i examples/alerts/sloth/latency.yaml --default-slo-period=28d
+```
+You can also use the make target to generate the rules to.
+
+```
+make sloth-generate
+```
+### Prometheus unit tests
+There are also two matching unit tests to verify and test the alerts that Sloth has generated. These can be run using the make target:
+
+```
+make alerts-tests
+```
diff --git a/examples/alerts/slo-availability.yaml b/examples/alerts/slo-availability.yaml
@@ -0,0 +1,212 @@
+
+---
+# Code generated by Sloth (v0.11.0): https://github.com/slok/sloth.
+# DO NOT EDIT.
+
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  creationTimestamp: null
+  labels:
+    app.kubernetes.io/component: SLO
+    app.kubernetes.io/managed-by: sloth
+  name: availability-slo
+  namespace: monitoring
+spec:
+  groups:
+  - name: sloth-slo-sli-recordings-kuadrant-requests-availability
+    rules:
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[5m])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[5m]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 5m
+      record: slo:sli_error:ratio_rate5m
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[30m])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[30m]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 30m
+      record: slo:sli_error:ratio_rate30m
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[1h])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[1h]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 1h
+      record: slo:sli_error:ratio_rate1h
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[2h])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[2h]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 2h
+      record: slo:sli_error:ratio_rate2h
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[6h])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[6h]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 6h
+      record: slo:sli_error:ratio_rate6h
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[1d])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[1d]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 1d
+      record: slo:sli_error:ratio_rate1d
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[3d])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[3d]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 3d
+      record: slo:sli_error:ratio_rate3d
+    - expr: |
+        sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}[4w])
+        / ignoring (sloth_window)
+        count_over_time(slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}[4w])
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 4w
+      record: slo:sli_error:ratio_rate4w
+  - name: sloth-slo-meta-recordings-kuadrant-requests-availability
+    rules:
+    - expr: vector(0.9995)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:objective:ratio
+    - expr: vector(1-0.9995)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:error_budget:ratio
+    - expr: vector(28)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:time_period:days
+    - expr: |
+        slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+        / on(sloth_id, sloth_slo, sloth_service) group_left
+        slo:error_budget:ratio{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:current_burn_rate:ratio
+    - expr: |
+        slo:sli_error:ratio_rate4w{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+        / on(sloth_id, sloth_slo, sloth_service) group_left
+        slo:error_budget:ratio{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:period_burn_rate:ratio
+    - expr: 1 - slo:period_burn_rate:ratio{sloth_id="kuadrant-requests-availability",
+        sloth_service="kuadrant", sloth_slo="requests-availability"}
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:period_error_budget_remaining:ratio
+    - expr: vector(1)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_mode: cli-gen-k8s
+        sloth_objective: "99.95"
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_spec: sloth.slok.dev/v1
+        sloth_version: v0.11.0
+      record: sloth_slo_info
+  - name: sloth-slo-alerts-kuadrant-requests-availability
+    rules:
+    - alert: KuadrantAvailabilityHighErrorRate
+      annotations:
+        summary: High error rate on HTTPRoute requests responses
+        title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget
+          burn rate is too fast.
+      expr: |
+        (
+            max(slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (13.44 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate1h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (13.44 * 0.0004999999999999716)) without (sloth_window)
+        )
+        or
+        (
+            max(slo:sli_error:ratio_rate30m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (5.6000000000000005 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate6h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (5.6000000000000005 * 0.0004999999999999716)) without (sloth_window)
+        )
+      labels:
+        category: availability
+        severity: critical
+        sloth_severity: page
+    - alert: KuadrantAvailabilityHighErrorRate
+      annotations:
+        summary: High error rate on HTTPRoute requests responses
+        title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error
+          budget burn rate is too fast.
+      expr: |
+        (
+            max(slo:sli_error:ratio_rate2h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (2.8000000000000003 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate1d{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (2.8000000000000003 * 0.0004999999999999716)) without (sloth_window)
+        )
+        or
+        (
+            max(slo:sli_error:ratio_rate6h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (0.9333333333333333 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate3d{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (0.9333333333333333 * 0.0004999999999999716)) without (sloth_window)
+        )
+      labels:
+        category: availability
+        severity: warning
+        sloth_severity: ticket