-
Notifications
You must be signed in to change notification settings - Fork 1.3k
improve grafana configuration steps #5795
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
8e655b1
6827f1a
036680f
c5a3d1b
9f59f97
b8d6af8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -47,27 +47,71 @@ aggregating timeseries metrics and alerting. It can also be used to scrape the O | |
kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/servicemonitor.yaml | ||
``` | ||
|
||
1. Grafana dashboards can be imported from the [`knative-extensions` repository](https://github.com/knative-extensions/monitoring/tree/main/grafana). | ||
### Access the Prometheus instance locally | ||
|
||
By default, the Prometheus instance is only exposed on a private service named `prometheus-operated`. | ||
|
||
To access the console in your web browser: | ||
|
||
1. Enter the command: | ||
|
||
```bash | ||
kubectl port-forward -n default svc/prometheus-operated 9090 | ||
prushh marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
|
||
1. Access the console in your browser via `http://localhost:9090`. | ||
|
||
### Setting up Grafana | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: Setting up Grafana dashboards? 🤔 |
||
|
||
1. Grafana dashboards can be imported from the [`monitoring` repository](https://github.com/knative-extensions/monitoring/tree/main/grafana). | ||
|
||
1. If you are using the Grafana Helm Chart with the Dashboard Sidecar enabled, you can load the dashboards by applying the following configmaps. | ||
|
||
```bash | ||
kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/grafana/dashboards.yaml | ||
``` | ||
|
||
### Access the Prometheus instance locally | ||
!!! caution | ||
You will need to ensure that the helm chart has following values configured, otherwise the dashboards loading will not work. | ||
```yaml | ||
grafana: | ||
sidecar: | ||
dashboards: | ||
enabled: true | ||
searchNamespace: ALL | ||
``` | ||
If you have an existing configmaps check the value of `grafana_dashboard` label. In case this is `true`, add the `labelValue: true` attribute to the helm chart above. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. s/configmaps/configmap |
||
```bash | ||
kubectl -n knative-serving get configmaps/knative-serving-dashboards -o jsonpath="{.metadata.labels.grafana_dashboard}"; echo | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How can I adapt namespace and cm name based on the selected section? (Serving/Eventing) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's omit this part about printing that value. Also let's be specific where to add the value. eg after |
||
``` | ||
|
||
By default, the Prometheus instance is only exposed on a private service named `prometheus-operated`. | ||
#### Access the Grafana instance locally | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should be one level up, ### |
||
|
||
To access the console in your web browser: | ||
By default, the Grafana instance is only exposed on a private service named `prometheus-grafana`. | ||
|
||
To access the dashboards in your web browser: | ||
|
||
1. Enter the command: | ||
|
||
```bash | ||
kubectl port-forward -n default svc/prometheus-operated 9090 | ||
kubectl port-forward -n default svc/prometheus-grafana 3000:80 | ||
``` | ||
|
||
1. Access the console in your browser via `http://localhost:9090`. | ||
1. Access the dashboards in your browser via `http://localhost:3000`. | ||
|
||
1. Use the default credentials to login: | ||
|
||
```text | ||
username: admin | ||
password: prom-operator | ||
``` | ||
|
||
!!! tip | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Although this might be useful if password changes etc (I know I mentioned it on the other PR), let's omit it for now as it seems a bit not that Knative related, we are going a bit too far. I think it belongs at the Prometheus operator side. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I can try to find one that suits us! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did some research, and in my opinion the most useful link is https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md (in particular the Configuration section). There is also a Further Information section where they linked external resources like Prometheus Operator, Prometheus and Grafana. What do you think? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could add a pointer to that instead of the tip, but it does not say much about credentials and not sure if it is going to help a novice. So for now let's not add anything, remove the tip and in the future we can always revise it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done with the latest commit 😄 |
||
You can get current credentials as follows or change password adding `adminPassword: <pwd>` attribute to the helm chart above. | ||
```bash | ||
kubectl get secret prometheus-grafana -o jsonpath="{.data.admin-user}" | base64 --decode ; echo | ||
kubectl get secret prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo | ||
``` | ||
|
||
## About OpenTelemetry | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/prometheus-operated/prometheus-kube-prometheus-prometheus
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done