Add Istio Workload Dashboard

[david-martin]

Closes #307

Changes:

• Move the grafana kustomize configuration alongside the kube-prometheus configuration to allow patching the grafana deployment with new dashboard configmaps
• Add the Isito Workload Dashboard json
• Enable scraping of MGC metrics (ServiceMonitor added via kustomize)
• Add dev docs for how to add/update dashboards

TODO:

• Validate the panels & expressions on the Istio Workload Dashboard behave as expected.
• Document (in config/prometheus-for-federation ?) the steps for importing, updating and reloading a dashboard

Follow Up:

• Consider a second (or more) dashboard that are tailored more at the Multi Cluster Gateway administrator persona - e.g. aggregation across gateway instances that isn't shown on the Istio workload dashboard, anything linked from Create an example Grafana dashboard that shows traffic data, leveraging Istio metrics #307 (comment) that would be good to show, any DNS or TLS metrics exposed from mgc that can be visualised (or if none available, suggest some that would be good for that persona), cert-manager metrics (could be an existing dashboard)

Use cases for an Multi Cluster Gateway administrator persona

Please add or suggest more in the comments

• As a Multi Cluster Gateway administrator, I want to see how traffic going to my application across multiple clusters is being balanced (e.g. is weighting behaving as expected)
• As a Multi Cluster Gateway administrator, I want to see where, geographically, traffic is coming from to my app deployed in multiple regions (e.g. is traffic using geo dns as expected)

Notes on importing an existing dashboard

Grafana > Import > Import via grafana.com > input the dashboard ID OR paste json > Load > change any dropdowns until data is shown

Notes on exporting a dashboard as json

• Share symbol at top > Export > Export as json > save to a .json file in config/prometheus-for-federation
• Update ./config/prometheus-for-federation/kustomization.yaml to add a new configMapGenerator for the new dashboard
• Update ./config/prometheus-for-federation/grafana_deployment_patch.yaml to add the new configmap as a volume mount

Notes on reloading a dashboard after changing the json content

# re-build with kustomize, filter out the dashboard configmap & grafana deployment and apply changes
./bin/kustomize build config/prometheus-for-federation/ | docker run --rm -i ryane/kfilt -i kind=ConfigMap,name=<YOUR_DASHBOARD_CONFIG_MAP_NAME> -i kind=Deployment,name=grafana | kubectl apply -f -

# Kill the grafana pod to force a reload of the dashboard configmap from volume
kubectl -n monitoring delete $(kubectl -n monitoring get po -l app.kubernetes.io/component=grafana -o name)

/hold

[david-martin]

@eoinfennessy @Ygnas

Validate the panels & expressions on the Istio Workload Dashboard behave as expected.

I think we're good on this item, marked as done. The only open thing was about the tcp metrics, which is unfortunately expected behaviour at this time.

Document (in config/prometheus-for-federation ?) the steps for importing, updating and reloading a dashboard

I think this PR can merge after this docs piece is done.

Consider a second (or more) dashboard that...

This can be followed up outside this PR/issue, and will likely result in new issues being created based on feedback from the mail sent to kuadrant-dev with subject "Grafana Dashboards: Looking for some feedback and ideas"

I've removed it as a TODO item in the PR, for follow up instead.

[maleck13]

@Ygnas is this ready for a final review and merge?

[maleck13]

changes look good to me, I am having some connectivity issues locally so wasn't able to create request metrics, but I had tried before

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: david-martin, maleck13

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [david-martin,maleck13]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[maleck13]

/lgtm