availability and latency slos

examples/alerts/README.md

@@ -0,0 +1,41 @@
+## SLO Multi burn rate multi window alerts
+Kuadrant have created two example SLO alerts to help give ideas on the types of SLO alerts that could be used with the operator. We have created one alert for latency and one for availability, both are Multiwindow, Multi-Burn-Rate Alerts. The alerts show a scenario where a 28d rolling window is used and a uptime of 99.95% i.e only 0.05% error budget margin is desired. This in real world time would be downtime of around: 
+
+| Time Frame | Duration   |
+|------------|------------|
+| Daily:     | 43s        |
+| Weekly:    | 5m 2.4s    |
+| Monthly:   | 21m 44s    |
+| Quarterly: | 1h 5m 12s  |
+| Yearly:    | 4h 20m 49s |
+
+These values can be changed to suit different scenarios
+
+### Sloth
+Sloth is a tool to aid in the creation of multi burn rate and multi window SLO alerts and was used to create both the availability and latency alerts. It follows the common standard set out by [Google's SRE book](https://sre.google/workbook/implementing-slos/). Sloth generates alerts based on specific specs given. The specs for our example alerts can be found in the example/sloth folder.
+
+#### Metrics used for the alerts
+
+#### Availability
+For the availability SLO alerts the Istio metric `istio_requests_total` was used as its a counter type metric meaning the values can only increase as well as it gives information on all requests handled by the Istio proxy.
+
+#### Latency
+For the availability SLO alerts the Istio metric `istio_request_duration_milliseconds` was used as its a [Histogram](https://prometheus.io/docs/concepts/metric_types/#histogram).
+
+### Sloth generation
+You can modify the examples Sloth specs we have and regenerate the prometheus rules using the Sloth CLI and the generate command. For more information please the [Sloth website](https://sloth.dev/usage/cli/)
+
+```
+sloth generate -i examples/alerts/sloth/latency.yaml --default-slo-period=28d
+```
+You can also use the make target to generate the rules to.
+
+```
+make sloth-generate
+```
+### Prometheus unit tests
+There are also two matching unit tests to verify and test the alerts that Sloth has generated. These can be run using the make target:
+
+```
+make alerts-tests
+```

examples/alerts/slo-availability.yaml

@@ -0,0 +1,212 @@
+
+---
+# Code generated by Sloth (v0.11.0): https://github.com/slok/sloth.
+# DO NOT EDIT.
+
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  creationTimestamp: null
+  labels:
+    app.kubernetes.io/component: SLO
+    app.kubernetes.io/managed-by: sloth
+  name: availability-slo
+  namespace: monitoring
+spec:
+  groups:
+  - name: sloth-slo-sli-recordings-kuadrant-requests-availability
+    rules:
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[5m])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[5m]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 5m
+      record: slo:sli_error:ratio_rate5m
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[30m])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[30m]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 30m
+      record: slo:sli_error:ratio_rate30m
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[1h])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[1h]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 1h
+      record: slo:sli_error:ratio_rate1h
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[2h])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[2h]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 2h
+      record: slo:sli_error:ratio_rate2h
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[6h])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[6h]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 6h
+      record: slo:sli_error:ratio_rate6h
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[1d])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[1d]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 1d
+      record: slo:sli_error:ratio_rate1d
+    - expr: |
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy",response_code=~"5.*"}[3d])) by (request_host))
+        /
+        (sum(rate(istio_requests_total{job="ingress-metrics-proxy"}[3d]) )by (request_host))
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 3d
+      record: slo:sli_error:ratio_rate3d
+    - expr: |
+        sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}[4w])
+        / ignoring (sloth_window)
+        count_over_time(slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}[4w])
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_window: 4w
+      record: slo:sli_error:ratio_rate4w
+  - name: sloth-slo-meta-recordings-kuadrant-requests-availability
+    rules:
+    - expr: vector(0.9995)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:objective:ratio
+    - expr: vector(1-0.9995)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:error_budget:ratio
+    - expr: vector(28)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:time_period:days
+    - expr: |
+        slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+        / on(sloth_id, sloth_slo, sloth_service) group_left
+        slo:error_budget:ratio{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:current_burn_rate:ratio
+    - expr: |
+        slo:sli_error:ratio_rate4w{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+        / on(sloth_id, sloth_slo, sloth_service) group_left
+        slo:error_budget:ratio{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"}
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:period_burn_rate:ratio
+    - expr: 1 - slo:period_burn_rate:ratio{sloth_id="kuadrant-requests-availability",
+        sloth_service="kuadrant", sloth_slo="requests-availability"}
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+      record: slo:period_error_budget_remaining:ratio
+    - expr: vector(1)
+      labels:
+        owner: kuadrant-org
+        sloth_id: kuadrant-requests-availability
+        sloth_mode: cli-gen-k8s
+        sloth_objective: "99.95"
+        sloth_service: kuadrant
+        sloth_slo: requests-availability
+        sloth_spec: sloth.slok.dev/v1
+        sloth_version: v0.11.0
+      record: sloth_slo_info
+  - name: sloth-slo-alerts-kuadrant-requests-availability
+    rules:
+    - alert: KuadrantAvailabilityHighErrorRate
+      annotations:
+        summary: High error rate on HTTPRoute requests responses
+        title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget
+          burn rate is too fast.
+      expr: |
+        (
+            max(slo:sli_error:ratio_rate5m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (13.44 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate1h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (13.44 * 0.0004999999999999716)) without (sloth_window)
+        )
+        or
+        (
+            max(slo:sli_error:ratio_rate30m{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (5.6000000000000005 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate6h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (5.6000000000000005 * 0.0004999999999999716)) without (sloth_window)
+        )
+      labels:
+        category: availability
+        severity: critical
+        sloth_severity: page
+    - alert: KuadrantAvailabilityHighErrorRate
+      annotations:
+        summary: High error rate on HTTPRoute requests responses
+        title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error
+          budget burn rate is too fast.
+      expr: |
+        (
+            max(slo:sli_error:ratio_rate2h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (2.8000000000000003 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate1d{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (2.8000000000000003 * 0.0004999999999999716)) without (sloth_window)
+        )
+        or
+        (
+            max(slo:sli_error:ratio_rate6h{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (0.9333333333333333 * 0.0004999999999999716)) without (sloth_window)
+            and
+            max(slo:sli_error:ratio_rate3d{sloth_id="kuadrant-requests-availability", sloth_service="kuadrant", sloth_slo="requests-availability"} > (0.9333333333333333 * 0.0004999999999999716)) without (sloth_window)
+        )
+      labels:
+        category: availability
+        severity: warning
+        sloth_severity: ticket