Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
302 changes: 159 additions & 143 deletions docs/addons/observability.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -30,65 +30,66 @@ The easiest way to get started with production-grade metrics collection is to us

. Create a ServiceMonitor for istiod.

[source,yaml]
----
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: istiod-monitor
namespace: istio-system
spec:
targetLabels:
- app
selector:
matchLabels:
istio: pilot
endpoints:
- port: http-monitoring
interval: 30s
----
[source,yaml]
----
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: istiod-monitor
namespace: istio-system
spec:
targetLabels:
- app
selector:
matchLabels:
istio: pilot
endpoints:
- port: http-monitoring
interval: 30s
----

. Create a PodMonitor to scrape metrics from the istio-proxy containers. Note that *this resource has to be created in all namespaces where you are running sidecars*.

[source,yaml]
----
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: istio-proxies-monitor
namespace: istio-system
spec:
selector:
matchExpressions:
- key: istio-prometheus-ignore
operator: DoesNotExist
podMetricsEndpoints:
- path: /stats/prometheus
interval: 30s
relabelings:
- action: keep
sourceLabels: ["__meta_kubernetes_pod_container_name"]
regex: "istio-proxy"
- action: keep
sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
- action: replace
regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
replacement: "[$2]:$1"
sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
targetLabel: "__address__"
- action: replace
regex: (\d+);((([0-9]+?)(\.|$)){4})
replacement: "$2:$1"
sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
targetLabel: "__address__"
- action: labeldrop
regex: "__meta_kubernetes_pod_label_(.+)"
- sourceLabels: ["__meta_kubernetes_namespace"]
action: replace
targetLabel: namespace
- sourceLabels: ["__meta_kubernetes_pod_name"]
action: replace
targetLabel: pod_name
----
[source,yaml]
----
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: istio-proxies-monitor
namespace: istio-system
spec:
selector:
matchExpressions:
- key: istio-prometheus-ignore
operator: DoesNotExist
podMetricsEndpoints:
- path: /stats/prometheus
interval: 30s
relabelings:
- action: keep
sourceLabels: ["__meta_kubernetes_pod_container_name"]
regex: "istio-proxy"
- action: keep
sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
- action: replace
regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
replacement: "[$2]:$1"
sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
targetLabel: "__address__"
- action: replace
regex: (\d+);((([0-9]+?)(\.|$)){4})
replacement: "$2:$1"
sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
targetLabel: "__address__"
- action: labeldrop
regex: "__meta_kubernetes_pod_label_(.+)"
- sourceLabels: ["__meta_kubernetes_namespace"]
action: replace
targetLabel: namespace
- sourceLabels: ["__meta_kubernetes_pod_name"]
action: replace
targetLabel: pod_name
----

Congratulations! You should now be able to see your control plane and data plane metrics in the OpenShift Console. Just go to Observe -> Metrics and try the query `istio_requests_total`.

Expand All @@ -97,6 +98,7 @@ Congratulations! You should now be able to see your control plane and data plane
This section describes how to setup Istio with OpenShift Distributed Tracing to send distributed traces.

*Prerequisites*

* A Tempo stack is installed and configured
* An instance of an OpenTelemetry collector is already configured in the istio-system namespace
* An Istio instance is created with the `openshift` profile
Expand All @@ -105,37 +107,43 @@ This section describes how to setup Istio with OpenShift Distributed Tracing to
*Steps*

. Configure Istio to enable tracing and include the OpenTelemetry settings:
[source,yaml]
----
meshConfig:
enableTracing: true
extensionProviders:
- name: otel-tracing
opentelemetry:
port: 4317
service: otel-collector.istio-system.svc.cluster.local
----
The *service* field is the OpenTelemetry collector service in the `istio-system` namespace.

[source,yaml]
----
meshConfig:
enableTracing: true
extensionProviders:
- name: otel-tracing
opentelemetry:
port: 4317
service: otel-collector.istio-system.svc.cluster.local
----

The *service* field is the OpenTelemetry collector service in the `istio-system` namespace.

[start=2]
. Create an Istio telemetry resource to active the OpenTelemetry tracer
[source,yaml]
----
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
name: otel-demo
namespace: istio-system
spec:
tracing:
- providers:
- name: otel-tracing
randomSamplingPercentage: 100
----

[source,yaml]
----
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
name: otel-demo
namespace: istio-system
spec:
tracing:
- providers:
- name: otel-tracing
randomSamplingPercentage: 100
----

[start=3]
. Validate the integration: Generate some traffic

We can link:addons.adoc#deploy-gateway-and-bookinfo[Deploy Bookinfo] and generate some traffic.

[start=4]
. Validate the integration: See the traces in the UI

[source,bash,subs="attributes+"]
Expand All @@ -154,93 +162,101 @@ Integration with Kiali really depends on how you collect your metrics and traces
If you followed <<scraping-metrics-using-the-openshift-monitoring-stack>>, you can set up Kiali to retrieve metrics from there.

*Prerequisites*

* User Workload monitoring is link:https://docs.openshift.com/container-platform/latest/observability/monitoring/enabling-monitoring-for-user-defined-projects.html[enabled] and <<scraping-metrics-using-the-openshift-monitoring-stack,configured>>
* Kiali Operator is installed

*Steps*

. Create a ClusterRoleBinding for Kiali, so it can view metrics from user-workload monitoring

[source,yaml]
----
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kiali-monitoring-rbac
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-monitoring-view
subjects:
- kind: ServiceAccount
name: kiali-service-account
namespace: istio-system
----
[source,yaml]
----
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kiali-monitoring-rbac
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-monitoring-view
subjects:
- kind: ServiceAccount
name: kiali-service-account
namespace: istio-system
----

[start=2]
. Find out the revision name of your Istio instance. In our case it is `test`.

[source,console,subs="attributes+"]
----
kubectl get istiorevisions.sailoperator.io
NAME READY STATUS IN USE VERSION AGE
test True Healthy True v{istio_latest_version} 119m
----
[source,console,subs="attributes+"]
----
kubectl get istiorevisions.sailoperator.io
NAME READY STATUS IN USE VERSION AGE
test True Healthy True v{istio_latest_version} 119m
----

[start=3]
. Create a Kiali resource and point it to your Istio instance. Make sure to replace `test` with your revision name in the fields `config_map_name`, `istio_sidecar_injector_config_map_name`, `istiod_deployment_name` and `url_service_version`.

[source,yaml]
----
apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
name: kiali-user-workload-monitoring
namespace: istio-system
spec:
external_services:
istio:
config_map_name: istio-test
istio_sidecar_injector_config_map_name: istio-sidecar-injector-test
istiod_deployment_name: istiod-test
url_service_version: 'http://istiod-test.istio-system:15014/version'
prometheus:
auth:
type: bearer
use_kiali_token: true
thanos_proxy:
enabled: true
url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
----
[source,yaml]
----
apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
name: kiali-user-workload-monitoring
namespace: istio-system
spec:
external_services:
istio:
config_map_name: istio-test
istio_sidecar_injector_config_map_name: istio-sidecar-injector-test
istiod_deployment_name: istiod-test
url_service_version: 'http://istiod-test.istio-system:15014/version'
prometheus:
auth:
type: bearer
use_kiali_token: true
thanos_proxy:
enabled: true
url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
----

[[integrating-kiali-with-openshift-distributed-tracing]]
==== Integrating Kiali with OpenShift Distributed Tracing
This section describes how to setup Kiali with OpenShift Distributed Tracing to read the distributed traces.

*Prerequisites*

* Istio tracing is <<configure-tracing-with-openshift-distributed-tracing,Configured with OpenShift distributed tracing>>

*Steps*

. Setup Kiali to access traces from the Tempo frontend:
[source,yaml]
----
external_services:
grafana:
enabled: true
url: "http://grafana-istio-system.apps-crc.testing/"
tracing:
enabled: true
provider: tempo
use_grpc: false
in_cluster_url: http://tempo-sample-query-frontend.tempo:3200
url: 'https://tempo-sample-query-frontend-tempo.apps-crc.testing'
tempo_config:
org_id: "1"
datasource_uid: "a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00"
----

Where:

[source,yaml]
----
external_services:
grafana:
enabled: true
url: "http://grafana-istio-system.apps-crc.testing/"
tracing:
enabled: true
provider: tempo
use_grpc: false
in_cluster_url: http://tempo-sample-query-frontend.tempo:3200
url: 'https://tempo-sample-query-frontend-tempo.apps-crc.testing'
tempo_config:
org_id: "1"
datasource_uid: "a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00"
----

Where:

* `external_services.grafana` section: Is just needed to see the "View in Tracing" link from the Traces tab
* `external_services.tracing.tempo_config`: Is just needed to see the "View in Tracing" link from the Traces tab and redirect to the proper Tempo datasource

Now, we should be able to see traces from Kiali. For this, you can:

. Select a Workload/Service/App
. Click in the "Traces" tab
. Click in the "Traces" tab
Loading