diff --git a/CHANGELOG.md b/CHANGELOG.md index be8dd8cbdf..feffeef882 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -444,7 +444,7 @@ removed. Instead, all services that expose metrics will now create `ServiceMonitor` - resources, if their helm chart is applied with `metrics.serviceMonitor.enable` + resources, if their helm chart is applied with `metrics.serviceMonitor.enabled` set to true. This prevents scraping agents from querying services that don't expose metrics diff --git a/Makefile b/Makefile index a60238394f..9ca1cc7478 100644 --- a/Makefile +++ b/Makefile @@ -7,13 +7,13 @@ DOCKER_TAG ?= $(USER) # default helm chart version must be 0.0.42 for local development (because 42 is the answer to the universe and everything) HELM_SEMVER ?= 0.0.42 # The list of helm charts needed on internal kubernetes testing environments -CHARTS_INTEGRATION := wire-server databases-ephemeral redis-cluster fake-aws nginx-ingress-controller nginx-ingress-services wire-server-metrics fluent-bit kibana sftd restund coturn +CHARTS_INTEGRATION := wire-server databases-ephemeral redis-cluster fake-aws nginx-ingress-controller nginx-ingress-services fluent-bit kibana sftd restund coturn # The list of helm charts to publish on S3 # FUTUREWORK: after we "inline local subcharts", # (e.g. move charts/brig to charts/wire-server/brig) # this list could be generated from the folder names under ./charts/ like so: # CHARTS_RELEASE := $(shell find charts/ -maxdepth 1 -type d | xargs -n 1 basename | grep -v charts) -CHARTS_RELEASE := wire-server redis-ephemeral redis-cluster databases-ephemeral fake-aws fake-aws-s3 fake-aws-sqs aws-ingress fluent-bit kibana backoffice calling-test demo-smtp elasticsearch-curator elasticsearch-external elasticsearch-ephemeral minio-external cassandra-external nginx-ingress-controller nginx-ingress-services reaper wire-server-metrics sftd restund coturn inbucket +CHARTS_RELEASE := wire-server redis-ephemeral redis-cluster databases-ephemeral fake-aws fake-aws-s3 fake-aws-sqs aws-ingress fluent-bit kibana backoffice calling-test demo-smtp elasticsearch-curator elasticsearch-external elasticsearch-ephemeral minio-external cassandra-external nginx-ingress-controller nginx-ingress-services reaper sftd restund coturn inbucket BUILDAH_PUSH ?= 0 KIND_CLUSTER_NAME := wire-server BUILDAH_KIND_LOAD ?= 1 diff --git a/changelog.d/0-release-notes/wire-server-metrics-removal b/changelog.d/0-release-notes/wire-server-metrics-removal new file mode 100644 index 0000000000..0f6be3cf66 --- /dev/null +++ b/changelog.d/0-release-notes/wire-server-metrics-removal @@ -0,0 +1,5 @@ +The experimental wire-server-metrics helm chart has been removed. + +These were mostly a wrapper around prometheus operator. It makes more sense to +refer to the upstream docs of Prometheus Operator or Grafana Agent Operator for +installation instead. diff --git a/charts/nginx-ingress-services/README.md b/charts/nginx-ingress-services/README.md index 9aa4dd93fa..50eb736fa8 100644 --- a/charts/nginx-ingress-services/README.md +++ b/charts/nginx-ingress-services/README.md @@ -52,8 +52,3 @@ A: Ensure that your certificate is _valid_ and has _not expired_; trying to serv * the `apiVersion` of all resources based on cert-manager's CRDs, namely `./templates/issuer.yaml` and `./templates/certificate.yaml`, has to be changed to `cert-manager.io/v1alpha3` - - -### Monitoring - -__FUTUREWORK:__ When `wire-server-metrics` is ready, expiration & renewal should be integrated into monitoring. diff --git a/charts/wire-server-metrics/.helmignore b/charts/wire-server-metrics/.helmignore deleted file mode 100644 index f0c1319444..0000000000 --- a/charts/wire-server-metrics/.helmignore +++ /dev/null @@ -1,21 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*~ -# Various IDEs -.project -.idea/ -*.tmproj diff --git a/charts/wire-server-metrics/Chart.yaml b/charts/wire-server-metrics/Chart.yaml deleted file mode 100644 index 9c61aaf42b..0000000000 --- a/charts/wire-server-metrics/Chart.yaml +++ /dev/null @@ -1,5 +0,0 @@ -apiVersion: v1 -appVersion: "1.0" -description: Adds monitoring for the kubernetes cluster and wire-server services -name: wire-server-metrics -version: 0.0.42 diff --git a/charts/wire-server-metrics/README.md b/charts/wire-server-metrics/README.md deleted file mode 100644 index 0fde62f485..0000000000 --- a/charts/wire-server-metrics/README.md +++ /dev/null @@ -1,15 +0,0 @@ -wire-server-metrics -------------------- - - This is mostly a wrapper over https://github.com/helm/charts/tree/master/stable/prometheus-operator -For a full list of overrides, please check the appropriate chart version and its options. - - How to use this chart? ----------------------- - - In its simplest form, install the chart with: -``` -helm upgrade --install --namespace wire/wire-server-metrics [-f ] -``` - -For more detailed information on how to set up monitoring on your cluster, go to the [monitoring page](https://docs.wire.com/how-to/install/monitoring.html) diff --git a/charts/wire-server-metrics/requirements.yaml b/charts/wire-server-metrics/requirements.yaml deleted file mode 100644 index 5b7925e3e1..0000000000 --- a/charts/wire-server-metrics/requirements.yaml +++ /dev/null @@ -1,6 +0,0 @@ ---- -dependencies: - - name: kube-prometheus-stack - version: 19.2.3 - repository: https://prometheus-community.github.io/helm-charts - diff --git a/charts/wire-server-metrics/templates/wire-server-dashboards.yaml b/charts/wire-server-metrics/templates/wire-server-dashboards.yaml deleted file mode 100644 index 6be08a4def..0000000000 --- a/charts/wire-server-metrics/templates/wire-server-dashboards.yaml +++ /dev/null @@ -1,23 +0,0 @@ -{{- $files := .Files.Glob "dashboards/*.json" }} -{{- $relName := .Release.Name }} -{{- $namespace := .Release.Namespace }} -{{- if $files }} -apiVersion: v1 -kind: ConfigMapList -items: -{{- range $path, $fileContents := $files }} -{{- $dashboardName := regexReplaceAll "(^.*/)(.*)\\.json$" $path "${2}" }} -- apiVersion: v1 - kind: ConfigMap - metadata: - name: {{ printf "%s-%s" $relName $dashboardName | trunc 63 | trimSuffix "-" }} - namespace: {{ $namespace }} - labels: - {{- if index $.Values "kube-prometheus-stack" "grafana" "sidecar" "dashboards" "label" }} - {{ index $.Values "kube-prometheus-stack" "grafana" "sidecar" "dashboards" "label" }}: "1" - {{- end }} - app: {{ $.Release.Namespace }}-grafana - data: - {{ $dashboardName }}.json: {{ $.Files.Get $path | toJson }} -{{- end }} -{{- end }} diff --git a/charts/wire-server-metrics/values.yaml b/charts/wire-server-metrics/values.yaml deleted file mode 100644 index 7a9a8b8763..0000000000 --- a/charts/wire-server-metrics/values.yaml +++ /dev/null @@ -1,39 +0,0 @@ -kube-prometheus-stack: - prometheusOperator: - # Don't try to create custom resource types; we prefer to do it manually - # Otherwise we run into race conditions when installing helm charts - createCustomResource: false - - grafana: - adminPassword: "admin" - ingress: - enabled: false - persistence: - storageClassName: "aws-ebs-retained" - enabled: true - accessModes: ["ReadWriteOnce"] - size: 10Gi - sidecar: - dashboards: - label: grafana_dashboard - - prometheusSpec: - storageSpec: - volumeClaimTemplate: - spec: - storageClassName: "aws-ebs-retained" - accessModes: ["ReadWriteOnce"] - resources: - requests: - storage: 10Gi - - alertmanager: - alertmanagerSpec: - storage: - volumeClaimTemplate: - spec: - storageClassName: "aws-ebs-retained" - accessModes: ["ReadWriteOnce"] - resources: - requests: - storage: 10Gi diff --git a/charts/wire-server-metrics/dashboards/message-stats.json b/dashboards/message-stats.json similarity index 100% rename from charts/wire-server-metrics/dashboards/message-stats.json rename to dashboards/message-stats.json diff --git a/charts/wire-server-metrics/dashboards/services.json b/dashboards/services.json similarity index 100% rename from charts/wire-server-metrics/dashboards/services.json rename to dashboards/services.json diff --git a/docs/src/how-to/install/monitoring.rst b/docs/src/how-to/install/monitoring.rst index eb49b1a404..ea900526cc 100644 --- a/docs/src/how-to/install/monitoring.rst +++ b/docs/src/how-to/install/monitoring.rst @@ -3,215 +3,19 @@ Monitoring wire-server using Prometheus and Grafana ======================================================= -Introduction ------------- +All wire-server helm charts offering prometheus metrics expose a +`metrics.serviceMonitor.enabled` option. -The following instructions detail the installation of a monitoring -system consisting of a Prometheus instance and corresponding Alert -Manager in addition to a Grafana instance for viewing dashboards related -to cluster and wire-services health. +If these are set to true, the helm charts will install `ServiceMonitor` +resources, which can be used to mark services for scraping by +[Prometheus Operator](https://prometheus-operator.dev/), +[Grafana Agent Operator](https://grafana.com/docs/grafana-cloud/kubernetes-monitoring/agent-k8s/), +or similar prometheus-compatible tools. -Prerequisites -------------- +Refer to their documentation for installation. -You need to have wire-server installed, see either of - -* :ref:`helm` -* :ref:`helm_prod`. - -How to install Prometheus and Grafana on Kubernetes using Helm ---------------------------------------------------------------- - -.. note:: - - The following makes use of overrides for helm charts. You may wish to read :ref:`understand-helm-overrides` first. - -Create an override file: - -.. code:: bash - - mkdir -p wire-server-metrics - curl -sSL https://raw.githubusercontent.com/wireapp/wire-server-deploy/master/values/wire-server-metrics/demo-values.example.yaml > wire-server-metrics/values.yaml - -And edit this file by editing/uncommenting as needed with respect to the next sections. - -The monitoring system requires disk space if you wish to be resilient to -pod failure. This disk space is given to pods by using a so-called "Storage Class". You have three options: - -* (1) If you deploy on a kubernetes cluster hosted on AWS you may install the ``aws-storage`` helm chart which provides configurations of Storage Classes for AWS's elastic block storage (EBS). For this, install the aws storage classes with ``helm upgrade --install aws-storage wire/aws-storage --wait``. -* (2) If you're not using AWS, but you sill want to have persistent metrics, see :ref:`using-custom-storage-classes`. -* (3) If you don't want persistence at all, see :ref:`using-no-storage-classes`. - -Once you have a storage class configured (or put the override configuration to not use persistence), next we can install the monitoring suite itself. - -There are a few known issues surrounding the ``prometheus-operator`` -helm chart. - -You will likely have to install the Custom Resource Definitions manually -before installing the ``wire-server-metrics`` chart: - -:: - - kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/d34d70de61fe8e23bb21f6948993c510496a0b31/example/prometheus-operator-crd/alertmanager.crd.yaml - kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/d34d70de61fe8e23bb21f6948993c510496a0b31/example/prometheus-operator-crd/prometheus.crd.yaml - kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/d34d70de61fe8e23bb21f6948993c510496a0b31/example/prometheus-operator-crd/prometheusrule.crd.yaml - kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/d34d70de61fe8e23bb21f6948993c510496a0b31/example/prometheus-operator-crd/servicemonitor.crd.yaml - -Now we can install the metrics chart, run the following:: - - helm upgrade --install wire-server-metrics wire/wire-server-metrics --wait -f wire-server-metrics/values.yaml - -See the `Prometheus Operator -README `__ -for more information and troubleshooting help. - -Adding Dashboards ------------------ - -Grafana dashboard configurations are included as JSON inside the -``charts/wire-server-metrics/dashboards`` directory. You may import -these via Grafana's web UI. See `Accessing -grafana <#accessing-grafana>`__. - -Monitoring in a separate namespace ----------------------------------- - -It is advisable to separate your monitoring services from your -application services. To accomplish this you may deploy -``wire-server-metrics`` into a separate namespace from ``wire-server``. -Simply provide a different namespace to the ``helm upgrade --install`` -calls with ``--namespace your-desired-namespace``. - -The wire-server-metrics chart will monitor all wire services across *all* namespaces. - -Accessing grafana +Dashboards ----------------- -Forward a port from your localhost to the grafana service running in -your cluster: - -:: - - kubectl port-forward service/-grafana 3000:80 -n - -Now you can access grafana at ``http://localhost:3000`` - -The username and password are stored in the ``grafana`` secret of your -namespace - -By default this is: - -- username: ``admin`` -- password: ``admin`` - -Accessing prometheus --------------------- - -Forward a port from your localhost to the prometheus service running in -your cluster: - -:: - - kubectl port-forward service/-prometheus 9090:9090 -n - -Now you can access prometheus at ``http://localhost:9090`` - - -Customization ---------------- - -.. _using-no-storage-classes: - -Monitoring without persistent disk -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you wish to deploy monitoring without any persistent disk (not -recommended) you may add the following overrides to your ``values.yaml`` -file. - -.. code:: yaml - - # This configuration switches to use memory instead of disk for metrics services - # NOTE: If the pods are killed you WILL lose all your metrics history - kube-prometheus-stack: - grafana: - persistence: - enabled: false - prometheus: - prometheusSpec: - storageSpec: null - alertmanager: - alertmanagerSpec: - storage: null - -.. _using-custom-storage-classes: - -Using Custom Storage Classes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you're using a provider other than AWS please reference the -`Kubernetes documentation on storage -classes `__ -for configuring a storage class for your kubernetes cluster. - -If you wish to use a different storage class (for instance if you don't -run on AWS) you may add the following overrides to your ``values.yaml`` -file. - -.. code:: yaml - - kube-prometheus-stack: - grafana: - persistence: - storageClassName: "" - prometheus: - prometheusSpec: - storageSpec: - volumeClaimTemplate: - spec: - storageClassName: "" - alertmanager: - alertmanagerSpec: - storage: - volumeClaimTemplate: - spec: - storageClassName: "" - - -Troubleshooting ---------------- - -"validation failed" -^^^^^^^^^^^^^^^^^^^^^ - -If you receive the following error: - -:: - - Error: validation failed: [unable to recognize "": no matches for kind "Alertmanager" in version - "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version - "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version - -Please run the script to install Custom Resource Definitions which is -detailed in the installation instructions above. - -"object is being deleted" -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When upgrading you may see the following error: - -:: - - Error: object is being deleted: customresourcedefinitions.apiextensions.k8s.io "prometheusrules.monitoring.coreos.com" already exists - -Helm sometimes has trouble cleaning up or defining Custom Resource -Definitions. Try manually deleting the resource definitions and trying -your helm install again: - -:: - - kubectl delete customresourcedefinitions \ - alertmanagers.monitoring.coreos.com \ - prometheuses.monitoring.coreos.com \ - servicemonitors.monitoring.coreos.com \ - prometheusrules.monitoring.coreos.com +Grafana dashboard configurations are included as JSON inside the ``dashboards`` +directory. You may import these via Grafana's web UI.