Skip to content

improve grafana configuration steps #5795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jan 5, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 50 additions & 6 deletions docs/snippets/collecting-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,27 +47,71 @@ aggregating timeseries metrics and alerting. It can also be used to scrape the O
kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/servicemonitor.yaml
```

1. Grafana dashboards can be imported from the [`knative-extensions` repository](https://github.com/knative-extensions/monitoring/tree/main/grafana).
### Access the Prometheus instance locally

By default, the Prometheus instance is only exposed on a private service named `prometheus-operated`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/prometheus-operated/prometheus-kube-prometheus-prometheus

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


To access the console in your web browser:

1. Enter the command:

```bash
kubectl port-forward -n default svc/prometheus-operated 9090
```

1. Access the console in your browser via `http://localhost:9090`.

### Setting up Grafana
Copy link
Contributor

@skonto skonto Dec 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Setting up Grafana dashboards? 🤔
Otherwise it seems like we are installing Grafana, maybe the top level title should be: "Setting up the Prometheus Stack" and mention that it includes Grafana by default.


1. Grafana dashboards can be imported from the [`monitoring` repository](https://github.com/knative-extensions/monitoring/tree/main/grafana).

1. If you are using the Grafana Helm Chart with the Dashboard Sidecar enabled, you can load the dashboards by applying the following configmaps.

```bash
kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/grafana/dashboards.yaml
```

### Access the Prometheus instance locally
!!! caution
You will need to ensure that the helm chart has following values configured, otherwise the dashboards loading will not work.
```yaml
grafana:
sidecar:
dashboards:
enabled: true
searchNamespace: ALL
```
If you have an existing configmaps check the value of `grafana_dashboard` label. In case this is `true`, add the `labelValue: true` attribute to the helm chart above.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/configmaps/configmap

```bash
kubectl -n knative-serving get configmaps/knative-serving-dashboards -o jsonpath="{.metadata.labels.grafana_dashboard}"; echo
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can I adapt namespace and cm name based on the selected section? (Serving/Eventing)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's omit this part about printing that value. Also let's be specific where to add the value. eg after searchNamespace: ALL.

```

By default, the Prometheus instance is only exposed on a private service named `prometheus-operated`.
#### Access the Grafana instance locally
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be one level up, ###


To access the console in your web browser:
By default, the Grafana instance is only exposed on a private service named `prometheus-grafana`.

To access the dashboards in your web browser:

1. Enter the command:

```bash
kubectl port-forward -n default svc/prometheus-operated 9090
kubectl port-forward -n default svc/prometheus-grafana 3000:80
```

1. Access the console in your browser via `http://localhost:9090`.
1. Access the dashboards in your browser via `http://localhost:3000`.

1. Use the default credentials to login:

```text
username: admin
password: prom-operator
```

!!! tip
Copy link
Contributor

@skonto skonto Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although this might be useful if password changes etc (I know I mentioned it on the other PR), let's omit it for now as it seems a bit not that Knative related, we are going a bit too far. I think it belongs at the Prometheus operator side.
In general it would be nice if we could point to a doc page for the Prometheus operator for all these details.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can try to find one that suits us!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did some research, and in my opinion the most useful link is https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md (in particular the Configuration section).

There is also a Further Information section where they linked external resources like Prometheus Operator, Prometheus and Grafana.

What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add a pointer to that instead of the tip, but it does not say much about credentials and not sure if it is going to help a novice. So for now let's not add anything, remove the tip and in the future we can always revise it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done with the latest commit 😄

You can get current credentials as follows or change password adding `adminPassword: <pwd>` attribute to the helm chart above.
```bash
kubectl get secret prometheus-grafana -o jsonpath="{.data.admin-user}" | base64 --decode ; echo
kubectl get secret prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```

## About OpenTelemetry

Expand Down