[loki-distributed] Add sidecar container for loki ruler to fetch AlertingRules from Configmaps/Secrets

charts/loki-distributed/Chart.yaml

@@ -3,7 +3,7 @@ name: loki-distributed
 description: Helm chart for Grafana Loki in microservices mode
 type: application
 appVersion: 2.9.10
-version: 0.80.1
+version: 0.80.2
 home: https://grafana.github.io/helm-charts
 sources:
   - https://github.com/grafana/loki

charts/loki-distributed/README.md

@@ -1,6 +1,6 @@
 # loki-distributed
 
-![Version: 0.80.1](https://img.shields.io/badge/Version-0.80.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.9.10](https://img.shields.io/badge/AppVersion-2.9.10-informational?style=flat-square)
+![Version: 0.80.2](https://img.shields.io/badge/Version-0.80.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.9.10](https://img.shields.io/badge/AppVersion-2.9.10-informational?style=flat-square)
 
 Helm chart for Grafana Loki in microservices mode
 
@@ -24,7 +24,15 @@ helm repo add grafana https://grafana.github.io/helm-charts
 
 Major version upgrades listed here indicate that there is an incompatible breaking change needing manual actions.
 
-### To 0.80.0
+### From 0.79.x to 0.80.0
+The Ruler now supports sidecar container that queries Kubernetes API to find Configmaps and/or Secret that contain Loki AlertingRules.
+In order to enable the sidecar:
+```yaml
+ruler:
+  enabled: true
+  sidecar:
+    enabled: true
+```
 
 Upgrading to chart 0.80.0 will set the memberlist cluster_label config option. During rollout your cluster will temporarilly be split into two memberlist clusters until all components are rolled out. This will interrupt reads and writes. This config option is set to prevent cross talk between Loki and other memberlist clusters.
 
@@ -617,6 +625,28 @@ kubectl delete statefulset RELEASE_NAME-loki-distributed-querier -n LOKI_NAMESPA
 | ruler.replicas | int | `1` | Number of replicas for the ruler |
 | ruler.resources | object | `{}` | Resource requests and limits for the ruler |
 | ruler.serviceLabels | object | `{}` | Labels for ruler service |
+| ruler.sidecar.enableUniqueFilenames | bool | `false` | Ensure that rule files aren't conflicting and being overwritten by prefixing their name with the namespace they are defined in. |
+| ruler.sidecar.enabled | bool | `false` | Whether or not to create a sidecar to ingest rule from specific ConfigMaps and/or Secrets. |
+| ruler.sidecar.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
+| ruler.sidecar.image.repository | string | `"kiwigrid/k8s-sidecar"` | The Docker registry and image for the k8s sidecar |
+| ruler.sidecar.image.sha | string | `""` | Docker image sha. If empty, no sha will be used |
+| ruler.sidecar.image.tag | string | `"1.25.3"` | Docker image tag |
+| ruler.sidecar.livenessProbe | object | `{}` | Liveness probe definition. Probe is disabled on the sidecar by default. |
+| ruler.sidecar.rbac | object | `{"namespaced":false}` | Whether to install RBAC in the namespace only or cluster-wide. Useful if you want to watch ConfigMap globally. |
+| ruler.sidecar.readinessProbe | object | `{}` | Readiness probe definition. Probe is disabled on the sidecar by default. |
+| ruler.sidecar.resources | object | `{}` | Resource requests and limits for the sidecar |
+| ruler.sidecar.rules.folder | string | `"/etc/loki/sc-rules"` | Folder into which the rules will be placed. |
+| ruler.sidecar.rules.label | string | `"loki_rule"` | Label that the configmaps/secrets with rules will be marked with. |
+| ruler.sidecar.rules.labelValue | string | `""` | Label value that the configmaps/secrets with rules will be set to. |
+| ruler.sidecar.rules.logLevel | string | `"INFO"` | Log level of the sidecar container. |
+| ruler.sidecar.rules.resource | string | `"both"` | Search in configmap, secret, or both. |
+| ruler.sidecar.rules.script | string | `nil` | Absolute path to the shell script to execute after a configmap or secret has been reloaded. |
+| ruler.sidecar.rules.searchNamespace | string | `nil` | Comma separated list of namespaces. If specified, the sidecar will search for config-maps/secrets inside these namespaces. Otherwise the namespace in which the sidecar is running will be used. It's also possible to specify 'ALL' to search in all namespaces. |
+| ruler.sidecar.rules.watchClientTimeout | int | `60` | WatchClientTimeout: is a client-side timeout, configuring your local socket. If you have a network outage dropping all packets with no RST/FIN, this is how long your client waits before realizing & dropping the connection. Defaults to 66sec. |
+| ruler.sidecar.rules.watchMethod | string | `"WATCH"` | Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH request, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. |
+| ruler.sidecar.rules.watchServerTimeout | int | `60` | WatchServerTimeout: request to the server, asking it to cleanly close the connection after that. defaults to 60sec; much higher values like 3600 seconds (1h) are feasible for non-Azure K8S. |
+| ruler.sidecar.securityContext | object | `{}` | The SecurityContext for the sidecar. |
+| ruler.sidecar.skipTlsVerify | bool | `false` | Set to true to skip tls verification for kube api calls. |
 | ruler.terminationGracePeriodSeconds | int | `300` | Grace period to allow the ruler to shutdown before it is killed |
 | ruler.tolerations | list | `[]` | Tolerations for ruler pods |
 | runtimeConfig | object | `{}` | Provides a reloadable runtime configuration file for some specific configuration |
@@ -957,3 +987,43 @@ ruler:
                 annotations:
                   summary: High error percentage
 ```
+
+Furthermore, it is possible to enable the sidecar container to load rules from ConfigMaps and Secrets.
+See `values.yaml` for a more detailed example.
+
+```yaml
+ruler:
+  enabled: true
+  sidecar:
+    enabled: true
+```
+
+ConfigMaps/Secrets with Alerting rules must be configured with appropriate labels to be recognized by the sidecar container.
+Exemplary ConfigMap:
+
+```yaml
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: AlertingRule
+  annotations:
+  labels:
+    loki_rule: ""
+data:
+  rules.yaml: |
+    groups:
+      - name: should_fire
+        rules:
+          - alert: HighPercentageError
+            expr: |
+              sum(rate({app="loki"} |= "error" [5m])) by (job)
+                /
+              sum(rate({app="loki"}[5m])) by (job)
+                > 0.05
+            for: 10m
+            labels:
+              severity: warning
+            annotations:
+              summary: High error percentage
+```

charts/loki-distributed/README.md.gotmpl

@@ -22,7 +22,15 @@ helm repo add grafana https://grafana.github.io/helm-charts
 
 Major version upgrades listed here indicate that there is an incompatible breaking change needing manual actions.
 
-### To 0.80.0
+### From 0.79.x to 0.80.0
+The Ruler now supports sidecar container that queries Kubernetes API to find Configmaps and/or Secret that contain Loki AlertingRules.
+In order to enable the sidecar:
+```yaml
+ruler:
+  enabled: true
+  sidecar:
+    enabled: true
+```
 
 Upgrading to chart 0.80.0 will set the memberlist cluster_label config option. During rollout your cluster will temporarilly be split into two memberlist clusters until all components are rolled out. This will interrupt reads and writes. This config option is set to prevent cross talk between Loki and other memberlist clusters.
 
@@ -387,3 +395,43 @@ ruler:
                 annotations:
                   summary: High error percentage
 ```
+
+Furthermore, it is possible to enable the sidecar container to load rules from ConfigMaps and Secrets.
+See `values.yaml` for a more detailed example. 
+
+```yaml
+ruler:
+  enabled: true
+  sidecar:
+    enabled: true
+```
+
+ConfigMaps/Secrets with Alerting rules must be configured with appropriate labels to be recognized by the sidecar container.
+Exemplary ConfigMap:
+
+```yaml
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: AlertingRule
+  annotations:
+  labels:
+    loki_rule: ""
+data:
+  rules.yaml: |
+    groups:
+      - name: should_fire
+        rules:
+          - alert: HighPercentageError
+            expr: |
+              sum(rate({app="loki"} |= "error" [5m])) by (job)
+                /
+              sum(rate({app="loki"}[5m])) by (job)
+                > 0.05
+            for: 10m
+            labels:
+              severity: warning
+            annotations:
+              summary: High error percentage
+```

charts/loki-distributed/templates/role.yaml

@@ -28,4 +28,16 @@ rules:
   resourceNames:
   - {{ include "loki.fullname" . }}
 {{- end }}
+{{- if and .Values.ruler.enabled .Values.ruler.sidecar.enabled .Values.ruler.sidecar.rbac.namespaced }}
+  {{- if or (eq .Values.ruler.sidecar.rules.resource "both") (eq .Values.ruler.sidecar.rules.resource "configmap") }}
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "watch", "list"]
+  {{- end }}
+  {{- if or (eq .Values.ruler.sidecar.rules.resource "both") (eq .Values.ruler.sidecar.rules.resource "secret") }}
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "watch", "list"]
+  {{- end }}
+{{- end}}
 {{- end }}

charts/loki-distributed/templates/ruler/clusterrole.yaml

@@ -0,0 +1,25 @@
+{{- if and .Values.ruler.enabled .Values.ruler.sidecar.enabled }}
+{{- if not .Values.ruler.sidecar.rbac.namespaced }}
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: {{ template "loki.rulerFullname" . }}-clusterrole
+  labels:
+    {{- include "loki.rulerLabels" $ | nindent 4 }}
+    {{- with .Values.loki.annotations }}
+    annotations:
+      {{ toYaml . | indent 4 }}
+    {{- end }}
+rules:
+  {{- if or (eq .Values.ruler.sidecar.rules.resource "both") (eq .Values.ruler.sidecar.rules.resource "configmap") }}
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "watch", "list"]
+  {{- end }}
+  {{- if or (eq .Values.ruler.sidecar.rules.resource "both") (eq .Values.ruler.sidecar.rules.resource "secret") }}
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "watch", "list"]
+  {{- end }}
+{{- end }}
+{{- end }}

charts/loki-distributed/templates/ruler/clusterrolebinding.yaml

@@ -0,0 +1,22 @@
+{{- if and .Values.ruler.enabled .Values.ruler.sidecar.enabled }}
+{{- if not .Values.ruler.sidecar.rbac.namespaced }}
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: {{ template "loki.rulerFullname" . }}-clusterrolebinding
+  labels:
+    {{- include "loki.rulerLabels" $ | nindent 4 }}
+  {{- with .Values.loki.annotations }}
+  annotations:
+    {{ toYaml . | indent 4 }}
+  {{- end }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ template "loki.serviceAccountName" . }}
+    namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: ClusterRole
+  name: {{ template "loki.rulerFullname" . }}-clusterrole
+  apiGroup: rbac.authorization.k8s.io
+{{- end -}}
+{{- end -}}

charts/loki-distributed/templates/ruler/deployment-ruler.yaml

@@ -104,6 +104,10 @@ spec:
               mountPath: /var/loki
             - name: tmp
               mountPath: /tmp/loki
+            {{- if .Values.ruler.sidecar.enabled }}
+            - name: sc-rules
+              mountPath: {{ .Values.ruler.sidecar.rules.folder | quote }}
+            {{- end}}
             {{- range $dir, $_ := .Values.ruler.directories }}
             - name: {{ include "loki.rulerRulesDirName" $dir }}
               mountPath: /etc/loki/rules/{{ $dir }}
@@ -116,6 +120,73 @@ spec:
         {{- if .Values.ruler.extraContainers }}
         {{- toYaml .Values.ruler.extraContainers | nindent 8}}
         {{- end }}
+      {{- if .Values.ruler.sidecar.enabled }}
+        - name: loki-sc-rules
+          {{- if .Values.ruler.sidecar.image.sha }}
+          image: "{{ .Values.ruler.sidecar.image.repository }}:{{ .Values.ruler.sidecar.image.tag }}@sha256:{{ .Values.ruler.sidecar.image.sha }}"
+          {{- else }}
+          image: "{{ .Values.ruler.sidecar.image.repository }}:{{ .Values.ruler.sidecar.image.tag }}"
+          {{- end }}
+          imagePullPolicy: {{ .Values.ruler.sidecar.image.pullPolicy }}
+          env:
+            - name: METHOD
+              value: {{ .Values.ruler.sidecar.rules.watchMethod }}
+            - name: LABEL
+              value: "{{ .Values.ruler.sidecar.rules.label }}"
+            {{- if .Values.ruler.sidecar.rules.labelValue }}
+            - name: LABEL_VALUE
+              value: {{ quote .Values.ruler.sidecar.rules.labelValue }}
+            {{- end }}
+            - name: FOLDER
+              value: "{{ .Values.ruler.sidecar.rules.folder }}"
+            - name: RESOURCE
+              value: {{ quote .Values.ruler.sidecar.rules.resource }}
+            {{- if .Values.ruler.sidecar.enableUniqueFilenames }}
+            - name: UNIQUE_FILENAMES
+              value: "{{ .Values.ruler.sidecar.enableUniqueFilenames }}"
+            {{- end }}
+            {{- if .Values.ruler.sidecar.rules.searchNamespace }}
+            - name: NAMESPACE
+              value: "{{ .Values.ruler.sidecar.rules.searchNamespace | join "," }}"
+            {{- end }}
+            {{- if .Values.ruler.sidecar.skipTlsVerify }}
+            - name: SKIP_TLS_VERIFY
+              value: "{{ .Values.ruler.sidecar.skipTlsVerify }}"
+            {{- end }}
+            {{- if .Values.ruler.sidecar.rules.script }}
+            - name: SCRIPT
+              value: "{{ .Values.ruler.sidecar.rules.script }}"
+            {{- end }}
+            {{- if .Values.ruler.sidecar.rules.watchServerTimeout }}
+            - name: WATCH_SERVER_TIMEOUT
+              value: "{{ .Values.ruler.sidecar.rules.watchServerTimeout }}"
+            {{- end }}
+            {{- if .Values.ruler.sidecar.rules.watchClientTimeout }}
+            - name: WATCH_CLIENT_TIMEOUT
+              value: "{{ .Values.ruler.sidecar.rules.watchClientTimeout }}"
+            {{- end }}
+            {{- if .Values.ruler.sidecar.rules.logLevel }}
+            - name: LOG_LEVEL
+              value: "{{ .Values.ruler.sidecar.rules.logLevel }}"
+            {{- end }}
+          {{- if .Values.ruler.sidecar.livenessProbe }}
+          livenessProbe:
+            {{- toYaml .Values.ruler.sidecar.livenessProbe | nindent 12 }}
+          {{- end }}
+          {{- if .Values.ruler.sidecar.readinessProbe }}
+          readinessProbe:
+            {{- toYaml .Values.ruler.sidecar.readinessProbe | nindent 12 }}
+          {{- end }}
+          resources:
+            {{- toYaml .Values.ruler.sidecar.resources | nindent 12 }}
+          {{- if .Values.ruler.sidecar.securityContext }}
+          securityContext:
+            {{- toYaml .Values.ruler.sidecar.securityContext | nindent 12 }}
+          {{- end }}
+          volumeMounts:
+            - name: sc-rules
+              mountPath: {{ .Values.ruler.sidecar.rules.folder | quote }}
+        {{- end}}
       {{- with .Values.ruler.affinity }}
       affinity:
         {{- tpl . $ | nindent 8 }}
@@ -161,6 +232,15 @@ spec:
           {{- else }}
           emptyDir: {}
           {{- end }}
+        {{- if .Values.ruler.sidecar.enabled }}
+        - name: sc-rules
+        {{- if .Values.ruler.sidecar.rules.sizeLimit }}
+          emptyDir:
+            sizeLimit: {{ .Values.ruler.sidecar.rules.sizeLimit }}
+        {{- else }}
+          emptyDir: {}
+        {{- end -}}
+        {{- end -}}
         {{- with .Values.ruler.extraVolumes }}
         {{- toYaml . | nindent 8 }}
         {{- end }}