availability and latency slos

examples/alerts/README.md

@@ -0,0 +1,35 @@
+## SLO Multi burn rate multi window alerts
+Kuadrant have created two example SLO alerts to help give ideas on the types of SLO alerts that could be used with the operator. We have created one alert for latency and one for availability, both are Multiwindow, Multi-Burn-Rate Alerts. The alerts show a scenario where a 28d rolling window is used and a uptime of 99.95% i.e only .1% error margin is desired. This in real world time would be downtime of around. These values can be changed to suit different scenarios
+
+| Time Frame | Duration   |
+|------------|------------|
+| Daily:     | 43s        |
+| Weekly:    | 5m 2.4s    |
+| Monthly:   | 21m 44s    |
+| Quarterly: | 1h 5m 12s  |
+| Yearly:    | 4h 20m 49s |
+
+### Sloth
+Sloth is a tool to aid in the creation of multi burn rate and multi window SLO alerts and was used to create both the availability and latency alerts. It follows the common standard set out by [Google's SRE book](https://sre.google/workbook/implementing-slos/). Sloth generates alerts based on specific specs given. The specs for our example alerts can be found in the example/sloth folder.
+
+#### Metrics used for the alerts
+
+#### Availability
+For the availability SLO alerts the Istio metric `istio_requests_total` was used as its a counter type metric meaning the values can only increase as well as it gives information on all requests handled by the Istio proxy.
+
+#### Latency
+For the availability SLO alerts the Istio metric 'istio_requests_total' was used as its a Distribution type metric meaning values are mapped to different ranges of frequency's as well as it gives information duration of requests.
+
+### Sloth generation
+You can modify the examples Sloth specs we have and regenerate the prometheus rules using the Sloth CLI and the generate command. For more information please the [Sloth website](https://sloth.dev/usage/cli/)
+
+```
+generate -i examples/alerts/sloth/latency.yaml --default-slo-period=28d
+```
+
+### Prometheus unit tests
+There are also two matching unit tests to verify and test the alerts that Sloth has generated. These can be run using the make target:
+
+```
+make alerts-tests
+```

examples/alerts/slo-availability.yaml

@@ -0,0 +1,203 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: availability-slo
+  namespace: monitoring
+spec:
+  groups:
+  - name: sloth-slo-sli-recordings-kuadrant-requests-availability
+    rules:
+    - record: slo:sli_error:ratio_rate5m
+      expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[5m])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[5m]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 5m
+    - record: slo:sli_error:ratio_rate30m
+      expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[30m])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[30m]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 30m
+    - record: slo:sli_error:ratio_rate1h
+      expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[1h])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[1h]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 1h
+    - record: slo:sli_error:ratio_rate2h
+      expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[2h])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[2h]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 2h
+    - record: slo:sli_error:ratio_rate6h
+      expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[6h])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[6h]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 6h
+    - record: slo:sli_error:ratio_rate1d
+      expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[1d])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[1d]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 1d
+    - record: slo:sli_error:ratio_rate3d
+      expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[3d])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[3d]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 3d
+    - record: slo:sli_error:ratio_rate4w
+      expr: |
+        sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}[4w])
+        / ignoring (sloth_window)
+        count_over_time(slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}[4w])
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 4w
+  - name: sloth-slo-meta-recordings-kuadrant-requests-availability
+    rules:
+    - record: slo:objective:ratio
+      expr: vector(0.9995)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+    - record: slo:error_budget:ratio
+      expr: vector(1-0.9995)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+    - record: slo:time_period:days
+      expr: vector(28)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+    - record: slo:current_burn_rate:ratio
+      expr: |
+        slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+        / on(sloth_id, sloth_slo, sloth_service) group_left
+        slo:error_budget:ratio{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+    - record: slo:period_burn_rate:ratio
+      expr: |
+        slo:sli_error:ratio_rate4w{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+        / on(sloth_id, sloth_slo, sloth_service) group_left
+        slo:error_budget:ratio{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+    - record: slo:period_error_budget_remaining:ratio
+      expr: 1 - slo:period_burn_rate:ratio{sloth_id="kuadrant-requests-availability",
+        sloth_service="kuadrant", sloth_slo="requests-availability"}
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+    - record: sloth_slo_info
+      expr: vector(1)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_mode: cli-gen-prom
+        sloth_objective: "99.95"
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_spec: prometheus/v1
+        sloth_version: v0.11.0
+  - name: sloth-slo-alerts-kuadrant-requests-availability
+    rules:
+    - alert: KuadrantAvailabilityHighErrorRate
+      expr: |
+        (
+            max(slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (13.44 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate1h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (13.44 * 0.0004999999999999716)) without (sloth_window)
+        )
+        or
+        (
+            max(slo:sli_error:ratio_rate30m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (5.6000000000000005 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate6h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (5.6000000000000005 * 0.0004999999999999716)) without (sloth_window)
+        )
+      labels:
+        category: availability
+        severity: critical
+        sloth_severity: page
+      annotations:
+        summary: High error rate on HTTPRoute requests responses
+        title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget
+          burn rate is too fast.
+    - alert: KuadrantAvailabilityHighErrorRate
+      expr: |
+        (
+            max(slo:sli_error:ratio_rate2h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (2.8000000000000003 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate1d{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (2.8000000000000003 * 0.0004999999999999716)) without (sloth_window)
+        )
+        or
+        (
+            max(slo:sli_error:ratio_rate6h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (0.9333333333333333 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate3d{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (0.9333333333333333 * 0.0004999999999999716)) without (sloth_window)
+        )
+      labels:
+        category: availability
+        severity: warning
+        sloth_severity: ticket
+      annotations:
+        summary: High error rate on HTTPRoute requests responses
+        title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget
+          burn rate is too fast.