[prometheus-operator] feat: add scripts to make upgrading easier (#742)

* chore: move all mesosphere files to patch

This moves all the mesosphere-related templates to the patch folder.
This makes it easier to update the operator with a copy and replace as
none of the mesosphere files will get lost.

Signed-off-by: Sam Tran <[email protected]>

* feat: add patch files

This adds patch files that copy all the mesosphere files into the
templates/ folder. This way they get deployed as part of the chart.

It makes it easier to keep all the patches in one place and know what
changes are needed/differ from upstream.

Signed-off-by: Sam Tran <[email protected]>

* feat: add helpers script for git commands

This adds a helper file so that all the patch files can use the
git_add_and_commit function which adds and commits all the files that
were changed as part of the patch script. This should make it easy to
see the commit history of what changed.

Signed-off-by: Sam Tran <[email protected]>

* chore: update prometheus crd patch script

This commit updates the patch script to use a docker image as the
previous code did not run on OSX.

Also, due to the underlying go-parser lib used in the yq docker image,
updating a yaml file automatically reformats the yaml removing empty
newlines and converting multiline strings into one line[1]. This makes
it hard to see the actual needed change for patch7 which is just to
update the crd volumeClaimTemplate.metadata stanza.

Thus, patch7 is changed to apply two commits, one to simply let yq
format the file with no new changes and a second commit with the needed
change. Similar to what we do with chart revision changes.

Also updated the helper script to accept a commit message.

[1] mikefarah/yq#465 (comment)

Signed-off-by: Sam Tran <[email protected]>

* feat: add update_operator script

This script does a sparse clone of helm/stable/prometheus-operator and
replaces all the files in our fork with latest upstream.

Then it applies all the patches in the /patch folder which should now
make updating our prometheus chart a bit easier.

Signed-off-by: Sam Tran <[email protected]>

* chore: add newlines from PR review

Signed-off-by: Sam Tran <[email protected]>

* chore: add doc for upgrading + patch comments

Add an UPGRADING doc so users know what to do to upgrade.
Add comments to the patch scripts so users know what each one does.
crd patch script needed explaining.

Signed-off-by: Sam Tran <[email protected]>

* chore: add patch for checking CRD exists again

This adds back the check for `monitoring.coreos.com/v1` that was removed
from upstream.

Signed-off-by: Sam Tran <[email protected]>

* chore: fixup shellcheck errors

Co-authored-by: Joe Julian <[email protected]>

staging/prometheus-operator/UPGRADING.md

@@ -0,0 +1,17 @@
+# Upgrading the prometheus-operator fork
+
+We fork the upstream [helm prometheus-operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator) and maintain a set of patches in the [patch/mesospere](./patch/mesosphere) folder. That folder houses all of the templates, hooks, dashboards, etc. we deploy in the prometheus addon in KBA.
+
+## Upgrading
+
+To upgrade the operator, simply run:
+```sh
+./upgrade_operator.sh
+```
+
+The upgrade script:
+- clones the helm/charts repo `master` branch
+- copies all the files under `stable/prometheus-operator`
+- replaces all of the files, of the same name, in this fork with the newer upstream versions; adds the files if they're new.
+- applies all the patches stored in the [/patch](./patch) folder.
+  - note: each of these patch files adds a git commit message when applied. This makes it easier to review as each commit will show what changes were needed.

staging/prometheus-operator/patch/helpers.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+function git_add_and_commit {
+  FILES_PATH=$1
+  git add "${FILES_PATH}"
+  FILENAME=$(basename "$0")
+  git commit -m "chore: apply ${FILENAME}"
+}
+
+function git_add_and_commit_with_msg {
+  FILES_PATH=$1
+  MSG=$2
+  git add "${FILES_PATH}"
+  FILENAME=$(basename "$0")
+  git commit -m "chore: apply ${FILENAME} - ${MSG}"
+}

staging/prometheus-operator/templates/ingress-rbac/grafana.yaml → staging/prometheus-operator/patch/mesosphere/templates/ingress-rbac/grafana.yaml

@@ -60,4 +60,4 @@ rules:
       - head
       - post
       - put
-{{- end }}
+{{- end }}

staging/prometheus-operator/patch/mesosphere/templates/prometheus-operator/crds.yaml

@@ -0,0 +1,8 @@
+{{- if and .Values.prometheusOperator.enabled .Values.prometheusOperator.createCustomResource -}}
+{{- if not (.Capabilities.APIVersions.Has "monitoring.coreos.com/v1") -}}
+{{- range $path, $bytes := .Files.Glob "crds/*.yaml" }}
+{{ $.Files.Get $path }}
+---
+{{- end }}
+{{- end }}
+{{- end }}

staging/prometheus-operator/templates/prometheus/rules/mesosphere-rules/etcd-rules.yaml → staging/prometheus-operator/patch/mesosphere/templates/rules/etcd-rules.yaml

@@ -109,4 +109,4 @@ spec:
       for: 10m
       labels:
         severity: warning
-{{- end }}
+{{- end }}

staging/prometheus-operator/templates/prometheus/rules/mesosphere-rules/velero-rules.yaml → staging/prometheus-operator/patch/mesosphere/templates/rules/velero-rules.yaml

@@ -26,4 +26,4 @@ spec:
       for: 10m
       labels:
         severity: warning
-{{- end }}
+{{- end }}

staging/prometheus-operator/patch/patch.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
 
 for p in patch/patch_*.sh; do
-    $p
+    BASEDIR=${BASEDIR} $p
 done

staging/prometheus-operator/patch/patch_1_mesosphere_dashboards.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+# This patch adds all the custom dashboards that we deploy in Grafana
+
+source $(dirname "$0")/helpers.sh
+
+set -x
+
+TEMPLATES_PATH=${BASEDIR}/templates/grafana/dashboards/mesosphere-dashboards
+
+mkdir -p "${TEMPLATES_PATH}"
+
+cp "${BASEDIR}"/patch/mesosphere/templates/grafana/dashboards/* "${TEMPLATES_PATH}"
+
+git_add_and_commit "${TEMPLATES_PATH}"

staging/prometheus-operator/patch/patch_2_mesosphere_cron_jobs.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+# This patch adds all our custom cronjobs
+
+source $(dirname "$0")/helpers.sh
+
+set -x
+
+GRAFANA_PATH=${BASEDIR}/templates/grafana
+
+for c in "${BASEDIR}"/patch/mesosphere/templates/grafana/cron*; do
+  [[ -e ${c} ]] || break # handle case when no files exist
+  cp "${c}" "${GRAFANA_PATH}"
+done
+
+git_add_and_commit "${GRAFANA_PATH}"

staging/prometheus-operator/patch/patch_3_mesosphere_hooks.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+# This patch adds all of our custom hooks
+
+source $(dirname "$0")/helpers.sh
+
+set -x
+
+TEMPLATES_PATH="${BASEDIR}"/templates/mesosphere-hooks
+
+mkdir -p "${TEMPLATES_PATH}"
+
+cp "${BASEDIR}"/patch/mesosphere/templates/hooks/* "${TEMPLATES_PATH}"
+
+git_add_and_commit "${TEMPLATES_PATH}"

staging/prometheus-operator/patch/patch_4_ingress_rbac.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+# This patch adds all of our ingress rbac rules
+
+source $(dirname "$0")/helpers.sh
+
+set -x
+
+TEMPLATES_PATH="${BASEDIR}"/templates/ingress-rbac
+
+mkdir -p "${TEMPLATES_PATH}"
+
+cp "${BASEDIR}"/patch/mesosphere/templates/ingress-rbac/* "${TEMPLATES_PATH}"
+
+git_add_and_commit "${TEMPLATES_PATH}"

staging/prometheus-operator/patch/patch_5_mesosphere_rules.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+# This patch adds alertmanager rules for certain addons
+
+source $(dirname "$0")/helpers.sh
+
+set -x
+
+TEMPLATES_PATH="${BASEDIR}"/templates/prometheus/rules/mesosphere-rules
+
+mkdir -p "${TEMPLATES_PATH}"
+
+cp "${BASEDIR}"/patch/mesosphere/templates/rules/* "${TEMPLATES_PATH}"
+
+git_add_and_commit "${TEMPLATES_PATH}"

staging/prometheus-operator/patch/patch_6_mesosphere_values.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+# This patch adds mesosphere specific chart values.
+# These are the values we reference in our custom templates.
+
+source $(dirname "$0")/helpers.sh
+
+set -x
+
+cat << EOF >> "${BASEDIR}"/values.yaml
+
+# Create mesosphere specific resources
+mesosphereResources:
+  create: false
+  rules:
+    etcd: true
+    velero: false
+  dashboards:
+    autoscaler: true
+    calico: true
+    elasticsearch: true
+    fluentbit: true
+    grafana: true
+    opsportal: true
+    kibana: true
+    localvolumeprovisioner: true
+    traefik: true
+    velero: true
+  homeDashboard:
+    name: "Kubernetes / Compute Resources / Cluster"
+    cronJob:
+      name: set-grafana-home-dashboard
+      image: dwdraju/alpine-curl-jq
+  hooks:
+    grafana:
+      image: dwdraju/alpine-curl-jq
+      secretKeyRef: ops-portal-credentials
+      # serviceURL is deprecated, do not use
+      serviceURL: http://prometheus-kubeaddons-grafana.kubeaddons:3000
+    prometheus:
+      jobName: prom-get-cluster-id
+      kubectlImage: bitnami/kubectl:1.16.2
+      configmapName: cluster-info-configmap
+  ingressRBAC:
+    enabled: true
+EOF
+
+git_add_and_commit "${BASEDIR}"/values.yaml

staging/prometheus-operator/patch/patch_7_prometheus_crd.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# This patch updates crd values needed to deploy properly in KBA.
+#
+# We use the yq docker image to manipulate the crd file. Due to the image's
+# underlying go-yaml parser, updating a file automatically formats the yaml
+# removing empty newlines and converting multiline strings into one line.
+# See https://github.com/mikefarah/yq/issues/465#issuecomment-643863154
+
+# This makes it hard to see the actual changes needed for the crd.
+# Thus, we break the patches into smaller commits.
+#
+# - let yq format the file with no new added changes
+# - update the volumeClaimTemplate.properties.metadata
+
+source $(dirname "$0")/helpers.sh
+
+set -x
+
+SRCFILE=crds/crd-prometheus.yaml
+TMPFILE=crds/tmp-prom.yaml
+
+# let yq format the file
+docker run --rm -it \
+  -v "${BASEDIR}":/basedir \
+  -w /basedir \
+  -e SRCFILE=${SRCFILE} \
+  -e TMPFILE=${TMPFILE} \
+  mikefarah/yq:3.3.2 \
+  yq read -P "${SRCFILE}" > "${TMPFILE}" && mv "${TMPFILE}" "${SRCFILE}"
+
+git_add_and_commit_with_msg ${SRCFILE} "reformat yaml with yq (no new changes)"
+
+# update volumeClaimTemplate.properties.metadata
+docker run --rm -it \
+  -v "${BASEDIR}":/basedir \
+  -w /basedir \
+  -e SRCFILE=${SRCFILE} \
+  -e TMPFILE=${TMPFILE} \
+  mikefarah/yq:3.3.2 \
+  yq write -i "${SRCFILE}" spec.validation.openAPIV3Schema.properties.spec.properties.storage.properties.volumeClaimTemplate.properties.metadata.properties.name.description "Name is the name used in the PVC claim" && \
+  yq write -i "${SRCFILE}" spec.validation.openAPIV3Schema.properties.spec.properties.storage.properties.volumeClaimTemplate.properties.metadata.properties.name.type "string"
+
+git_add_and_commit_with_msg ${SRCFILE} "update volumeClaimTemplate"

staging/prometheus-operator/patch/patch_8_prometheus_operator_crd.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+
+# This patch adds back the check for `monitoring.coreos.com/v1` that was removed from upstream.
+
+source $(dirname "$0")/helpers.sh
+
+set -x
+
+TEMPLATES_PATH=${BASEDIR}/templates/prometheus-operator/crds.yaml
+
+rm -rf ${BASEDIR}/templates/prometheus-operator/crds.yaml
+cp ${BASEDIR}/patch/mesosphere/templates/prometheus-operator/crds.yaml ${TEMPLATES_PATH}
+
+git_add_and_commit ${TEMPLATES_PATH}

staging/prometheus-operator/upgrade_operator.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+# This script upgrades the prometheus operator by copying all the latest upstream helm files.
+# 
+# It then applies all the needed mesosphere changes from the /patch folder.
+#
+# To upgrade, simply run: 
+#   ./upgrade_operator.sh
+
+set -x
+
+[email protected]:helm/charts.git
+PROMETHEUS_PATH=stable/prometheus-operator
+BASEDIR=$(dirname $(readlink -f "$0"))
+TMPDIR=$(mktemp -d)
+
+cd "${TMPDIR}" || exit
+
+git init
+git remote add origin -f ${UPSTREAM_REPO}
+git config core.sparsecheckout true
+
+echo ${PROMETHEUS_PATH} > .git/info/sparse-checkout
+
+git pull origin master
+
+cd ${PROMETHEUS_PATH} || exit
+
+for f in $(ls -A); do
+  rm -rf "${BASEDIR:?}"/"${f}"
+  cp -R "$f" "${BASEDIR}"
+done
+
+cd "${BASEDIR}" || exit
+
+NEW_VERSION=$(grep version Chart.yaml)
+
+git add .
+git commit -am "chore: copy upstream chart ${NEW_VERSION}"
+
+BASEDIR=${BASEDIR} ./patch/patch.sh
+
+echo "Done upgrading prometheus-operator!"