[prometheus-operator] feat: add scripts to make upgrading easier #742

samvantran · 2020-08-17T14:55:35Z

What type of PR is this?
Feature

What this PR does/ why we need it:
Upgrading prometheus has been problematic because we have a bunch of patches we apply on top of the upstream chart. Each time has been manual and sometimes we've missed something during an upgrade.

This PR adds all the mesosphere-specific files & changes (templates, hooks, etc) to the /patches folder. There are also scripts that apply those patches as a set. This should make it easy to follow which files are added/amended after copying the upstream chart. This is somewhat akin to how DB migrations work in Rails (this was years ago though, things may have changed).

Now we can run ./upgrade_operator.sh and it should take care of all the rest.

Here are the files that are mesophere-specific:

 > tree patch/mesosphere
patch/mesosphere
└── templates
    ├── grafana
    │   ├── cronjob-home-dashboard.yaml
    │   └── dashboards
    │       ├── autoscalerdashboard.yaml
    │       ├── calicodashboard.yaml
    │       ├── elasticsearchdashboard.yaml
    │       ├── fluentbitdashboard.yaml
    │       ├── grafanadashboard.yaml
    │       ├── kibanadashboard.yaml
    │       ├── localvolumeprovisioner.yaml
    │       ├── opsportaldashboard.yaml
    │       ├── traefikdashboard.yaml
    │       └── velerodashboard.yaml
    ├── hooks
    │   └── prometheus-cluster-id-hooks.yaml
    ├── ingress-rbac
    │   ├── alertmanager.yaml
    │   ├── grafana.yaml
    │   └── prometheus.yaml
    └── rules
        ├── etcd-rules.yaml
        └── velero-rules.yaml

6 directories, 17 files

Which issue(s) this PR fixes:
https://jira.d2iq.com/browse/D2IQ-67095

Special notes for your reviewer:
I mostly did this from the diff of copying the latest chart and trying to follow PR #619. It's possible I missed some files so definitely would appreciate anyone who's done a prometheus upgrade to make doubly sure I got everything.

Does this PR introduce a user-facing change?:

NONE

Checklist

The commit message explains the changes and why are needed.
The documentation is updated where needed.

This moves all the mesosphere-related templates to the patch folder. This makes it easier to update the operator with a copy and replace as none of the mesosphere files will get lost. Signed-off-by: Sam Tran <[email protected]>

This adds patch files that copy all the mesosphere files into the templates/ folder. This way they get deployed as part of the chart. It makes it easier to keep all the patches in one place and know what changes are needed/differ from upstream. Signed-off-by: Sam Tran <[email protected]>

This adds a helper file so that all the patch files can use the git_add_and_commit function which adds and commits all the files that were changed as part of the patch script. This should make it easy to see the commit history of what changed. Signed-off-by: Sam Tran <[email protected]>

This commit updates the patch script to use a docker image as the previous code did not run on OSX. Also, due to the underlying go-parser lib used in the yq docker image, updating a yaml file automatically reformats the yaml removing empty newlines and converting multiline strings into one line[1]. This makes it hard to see the actual needed change for patch7 which is just to update the crd volumeClaimTemplate.metadata stanza. Thus, patch7 is changed to apply two commits, one to simply let yq format the file with no new changes and a second commit with the needed change. Similar to what we do with chart revision changes. Also updated the helper script to accept a commit message. [1] mikefarah/yq#465 (comment) Signed-off-by: Sam Tran <[email protected]>

This script does a sparse clone of helm/stable/prometheus-operator and replaces all the files in our fork with latest upstream. Then it applies all the patches in the /patch folder which should now make updating our prometheus chart a bit easier. Signed-off-by: Sam Tran <[email protected]>

joejulian

👍 I wish we did this with all our forks.

alejandroEsc · 2020-08-17T16:50:23Z

wow nice. Could we get a quick readme about how these scripts are run?

gracedo

Amazing 🙌 the only other thing i can think of is this change here that we often forget to apply: https://github.com/mesosphere/charts/pull/486/files cc @alejandroEsc

samvantran · 2020-08-17T22:32:11Z

wow nice. Could we get a quick readme about how these scripts are run?

@alejandroEsc oh yeah for sure. I'll add in comments to the scripts as well so it's easier to understand

Amazing 🙌 the only other thing i can think of is this change here that we often forget to apply:

@gracedo i knew there was something missing! I thought it was this fix but yes, there's more. Okay I'll add a patch for that. Thanks for catching 🙏

Signed-off-by: Sam Tran <[email protected]>

Add an UPGRADING doc so users know what to do to upgrade. Add comments to the patch scripts so users know what each one does. crd patch script needed explaining. Signed-off-by: Sam Tran <[email protected]>

This adds back the check for `monitoring.coreos.com/v1` that was removed from upstream. Signed-off-by: Sam Tran <[email protected]>

… * chore: move all mesosphere files to patch This moves all the mesosphere-related templates to the patch folder. This makes it easier to update the operator with a copy and replace as none of the mesosphere files will get lost. Signed-off-by: Sam Tran <[email protected]> * feat: add patch files This adds patch files that copy all the mesosphere files into the templates/ folder. This way they get deployed as part of the chart. It makes it easier to keep all the patches in one place and know what changes are needed/differ from upstream. Signed-off-by: Sam Tran <[email protected]> * feat: add helpers script for git commands This adds a helper file so that all the patch files can use the git_add_and_commit function which adds and commits all the files that were changed as part of the patch script. This should make it easy to see the commit history of what changed. Signed-off-by: Sam Tran <[email protected]> * chore: update prometheus crd patch script This commit updates the patch script to use a docker image as the previous code did not run on OSX. Also, due to the underlying go-parser lib used in the yq docker image, updating a yaml file automatically reformats the yaml removing empty newlines and converting multiline strings into one line[1]. This makes it hard to see the actual needed change for patch7 which is just to update the crd volumeClaimTemplate.metadata stanza. Thus, patch7 is changed to apply two commits, one to simply let yq format the file with no new changes and a second commit with the needed change. Similar to what we do with chart revision changes. Also updated the helper script to accept a commit message. [1] mikefarah/yq#465 (comment) Signed-off-by: Sam Tran <[email protected]> * feat: add update_operator script This script does a sparse clone of helm/stable/prometheus-operator and replaces all the files in our fork with latest upstream. Then it applies all the patches in the /patch folder which should now make updating our prometheus chart a bit easier. Signed-off-by: Sam Tran <[email protected]> * chore: add newlines from PR review Signed-off-by: Sam Tran <[email protected]> * chore: add doc for upgrading + patch comments Add an UPGRADING doc so users know what to do to upgrade. Add comments to the patch scripts so users know what each one does. crd patch script needed explaining. Signed-off-by: Sam Tran <[email protected]> * chore: add patch for checking CRD exists again This adds back the check for that was removed from upstream. Signed-off-by: Sam Tran <[email protected]> * chore: fixup shellcheck errors Co-authored-by: Joe Julian <[email protected]>

…ier (#742) * chore: move all mesosphere files to patch This moves all the mesosphere-related templates to the patch folder. This makes it easier to update the operator with a copy and replace as none of the mesosphere files will get lost. Signed-off-by: Sam Tran <[email protected]> * feat: add patch files This adds patch files that copy all the mesosphere files into the templates/ folder. This way they get deployed as part of the chart. It makes it easier to keep all the patches in one place and know what changes are needed/differ from upstream. Signed-off-by: Sam Tran <[email protected]> * feat: add helpers script for git commands This adds a helper file so that all the patch files can use the git_add_and_commit function which adds and commits all the files that were changed as part of the patch script. This should make it easy to see the commit history of what changed. Signed-off-by: Sam Tran <[email protected]> * chore: update prometheus crd patch script This commit updates the patch script to use a docker image as the previous code did not run on OSX. Also, due to the underlying go-parser lib used in the yq docker image, updating a yaml file automatically reformats the yaml removing empty newlines and converting multiline strings into one line[1]. This makes it hard to see the actual needed change for patch7 which is just to update the crd volumeClaimTemplate.metadata stanza. Thus, patch7 is changed to apply two commits, one to simply let yq format the file with no new changes and a second commit with the needed change. Similar to what we do with chart revision changes. Also updated the helper script to accept a commit message. [1] mikefarah/yq#465 (comment) Signed-off-by: Sam Tran <[email protected]> * feat: add update_operator script This script does a sparse clone of helm/stable/prometheus-operator and replaces all the files in our fork with latest upstream. Then it applies all the patches in the /patch folder which should now make updating our prometheus chart a bit easier. Signed-off-by: Sam Tran <[email protected]> * chore: add newlines from PR review Signed-off-by: Sam Tran <[email protected]> * chore: add doc for upgrading + patch comments Add an UPGRADING doc so users know what to do to upgrade. Add comments to the patch scripts so users know what each one does. crd patch script needed explaining. Signed-off-by: Sam Tran <[email protected]> * chore: add patch for checking CRD exists again This adds back the check for that was removed from upstream. Signed-off-by: Sam Tran <[email protected]> * chore: fixup shellcheck errors Co-authored-by: Joe Julian <[email protected]>" This reverts commit 54b4815.

* Revert "[prometheus-operator] chore: upgrade to v9.3.1 (#743) * chore: copy upstream chart version: 9.3.1 * chore: apply patch_1_mesosphere_dashboards.sh * chore: apply patch_2_mesosphere_cron_jobs.sh * chore: apply patch_3_mesosphere_hooks.sh * chore: apply patch_4_ingress_rbac.sh * chore: apply patch_5_mesosphere_rules.sh * chore: apply patch_6_mesosphere_values.sh * chore: apply patch_7_prometheus_crd.sh - reformat yaml with yq (no new changes) * chore: apply patch_7_prometheus_crd.sh - update volumeClaimTemplate * chore: apply patch_8_prometheus_operator_crd.sh" This reverts commit b88c046. * Revert "[prometheus-operator] feat: add scripts to make upgrading easier (#742) * chore: move all mesosphere files to patch This moves all the mesosphere-related templates to the patch folder. This makes it easier to update the operator with a copy and replace as none of the mesosphere files will get lost. Signed-off-by: Sam Tran <[email protected]> * feat: add patch files This adds patch files that copy all the mesosphere files into the templates/ folder. This way they get deployed as part of the chart. It makes it easier to keep all the patches in one place and know what changes are needed/differ from upstream. Signed-off-by: Sam Tran <[email protected]> * feat: add helpers script for git commands This adds a helper file so that all the patch files can use the git_add_and_commit function which adds and commits all the files that were changed as part of the patch script. This should make it easy to see the commit history of what changed. Signed-off-by: Sam Tran <[email protected]> * chore: update prometheus crd patch script This commit updates the patch script to use a docker image as the previous code did not run on OSX. Also, due to the underlying go-parser lib used in the yq docker image, updating a yaml file automatically reformats the yaml removing empty newlines and converting multiline strings into one line[1]. This makes it hard to see the actual needed change for patch7 which is just to update the crd volumeClaimTemplate.metadata stanza. Thus, patch7 is changed to apply two commits, one to simply let yq format the file with no new changes and a second commit with the needed change. Similar to what we do with chart revision changes. Also updated the helper script to accept a commit message. [1] mikefarah/yq#465 (comment) Signed-off-by: Sam Tran <[email protected]> * feat: add update_operator script This script does a sparse clone of helm/stable/prometheus-operator and replaces all the files in our fork with latest upstream. Then it applies all the patches in the /patch folder which should now make updating our prometheus chart a bit easier. Signed-off-by: Sam Tran <[email protected]> * chore: add newlines from PR review Signed-off-by: Sam Tran <[email protected]> * chore: add doc for upgrading + patch comments Add an UPGRADING doc so users know what to do to upgrade. Add comments to the patch scripts so users know what each one does. crd patch script needed explaining. Signed-off-by: Sam Tran <[email protected]> * chore: add patch for checking CRD exists again This adds back the check for that was removed from upstream. Signed-off-by: Sam Tran <[email protected]> * chore: fixup shellcheck errors Co-authored-by: Joe Julian <[email protected]>" This reverts commit 54b4815.

…ier (mesosphere#742)" This reverts commit dcdf015.

* Revert "[prometheus-operator] chore: upgrade to v9.3.1 (#743)" This reverts commit 9a2b458. * Revert "[prometheus-operator] feat: add scripts to make upgrading easier (#742)" This reverts commit dcdf015. * chore: Bump chart version

* Revert [prometheus-operator] chore: upgrade to v9.3.1 (#743) This reverts commit b88c046. * Revert [prometheus-operator] feat: add scripts to make upgrading easier (#742) This reverts commit 54b4815.

samvantran added 5 commits August 12, 2020 19:39

chore: move all mesosphere files to patch

be50cc4

This moves all the mesosphere-related templates to the patch folder. This makes it easier to update the operator with a copy and replace as none of the mesosphere files will get lost. Signed-off-by: Sam Tran <[email protected]>

samvantran requested review from branden, joejulian, gracedo and alejandroEsc August 17, 2020 14:55

samvantran requested a review from a team as a code owner August 17, 2020 14:55

samvantran mentioned this pull request Aug 17, 2020

[prometheus-operator] chore: upgrade to v9.3.1 #743

Merged

5 tasks

Merge branch 'master' into svt/prom-operator-upgrade

f92c06c

joejulian approved these changes Aug 17, 2020

View reviewed changes

gracedo requested changes Aug 17, 2020

View reviewed changes

samvantran added 3 commits August 17, 2020 17:21

chore: add newlines from PR review

fa5f30e

Signed-off-by: Sam Tran <[email protected]>

chore: add doc for upgrading + patch comments

9e5e601

Add an UPGRADING doc so users know what to do to upgrade. Add comments to the patch scripts so users know what each one does. crd patch script needed explaining. Signed-off-by: Sam Tran <[email protected]>

chore: add patch for checking CRD exists again

57af579

This adds back the check for `monitoring.coreos.com/v1` that was removed from upstream. Signed-off-by: Sam Tran <[email protected]>

samvantran force-pushed the svt/prom-operator-upgrade branch from e695427 to 57af579 Compare August 18, 2020 00:26

samvantran requested a review from gracedo August 18, 2020 00:36

chore: fixup shellcheck errors

ce81314

shaneutt assigned samvantran Aug 18, 2020

shaneutt added the ready ready label Aug 18, 2020

alejandroEsc approved these changes Aug 18, 2020

View reviewed changes

gracedo approved these changes Aug 18, 2020

View reviewed changes

branden approved these changes Aug 18, 2020

View reviewed changes

samvantran merged commit dcdf015 into master Aug 18, 2020

samvantran deleted the svt/prom-operator-upgrade branch August 18, 2020 23:04

gracedo added a commit to gracedo/charts that referenced this pull request Aug 19, 2020

Revert "[prometheus-operator] feat: add scripts to make upgrading eas…

5b72714

…ier (mesosphere#742)" This reverts commit dcdf015.

samvantran mentioned this pull request Aug 19, 2020

[prometheus-operator] chore: upgrade to v9.3.1 and patch scripts #752

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[prometheus-operator] feat: add scripts to make upgrading easier #742

[prometheus-operator] feat: add scripts to make upgrading easier #742

samvantran commented Aug 17, 2020 •

edited

Loading

joejulian left a comment

alejandroEsc commented Aug 17, 2020

gracedo left a comment

samvantran commented Aug 17, 2020 •

edited

Loading

[prometheus-operator] feat: add scripts to make upgrading easier #742

[prometheus-operator] feat: add scripts to make upgrading easier #742

Conversation

samvantran commented Aug 17, 2020 • edited Loading

joejulian left a comment

Choose a reason for hiding this comment

alejandroEsc commented Aug 17, 2020

gracedo left a comment

Choose a reason for hiding this comment

samvantran commented Aug 17, 2020 • edited Loading

samvantran commented Aug 17, 2020 •

edited

Loading

samvantran commented Aug 17, 2020 •

edited

Loading