Bug 1780170: Expose metrics by benjaminapetersen · Pull Request #270 · openshift/console-operator

benjaminapetersen · 2019-08-14T16:23:30Z

We should expose /metrics to cluster monitor for both console & console-operator. Working out the details of this in this PR.

console-operator via library-go already has a /metrics endpoint running, its just a matter of wiring it up
console does not have a /metrics endpoint at this time. This may require a follow on set of PRs, one to the console to enable a prom-client for the console server & collect at least basic heap metrics

Validate that the operator is responding to the /metrics endpoint via:

curl --insecure -H "Authorization: Bearer $(oc whoami --show-token)" https://$(oc get route metrics --template "{{.spec.host}}"/metrics) | grep console_url

Validate that the console_url metric is being reported in the monitoring UI:

open $(oc get route prometheus-k8s -n openshift-monitoring -o jsonpath="{.spec.host}")

Console URL

The console URL metric is tracked in the form of:
console_url{url="https://<url>"} 1

Queried in the prometheus dashboard, will look like:
console_url{endpoint="https",instance="<ip>:8443",job="metrics",namespace="openshift-console-operator",pod="console-operator-<id>",service="metrics",url="https://console-openshift-console.apps.<domain>.devcluster.openshift.com"}

Prometheus UI:

Our UI:

Related PRs to enable telemetry:

console_url approved for telemetry: Add metrics gathering for console_url telemeter#239
console_url in cluster-monitoring-operator: Add console_url metric cluster-monitoring-operator#486
console_url to observatorium: https://github.com/observatorium/configuration/pull/71/files
console_url to sass-telemeter

Note that this process will be simplified into a simpler config for 4.3.

Gist for the RBAC/manifest items to add in the future when we enable console itself:

https://gist.github.com/benjaminapetersen/6929107ba02317051b1e03902e93223d

Gist tracking steps for getting an metric whitelisted for telemetry

https://gist.github.com/benjaminapetersen/3f8eaa31b0abbe60182eb67cf910c428

benjaminapetersen · 2019-09-05T00:50:03Z

/retest

benjaminapetersen · 2019-09-05T17:11:15Z

/retest

benjaminapetersen · 2019-09-10T19:37:38Z

+kind: Role
+metadata:
+  name: prometheus-k8s
+  namespace: openshift-console-operator


Enable metrics to scrape in console-operator and console namespaces.

We dont actually have a /metrics endpoint exposed for console, yet.

benjaminapetersen · 2019-09-18T20:49:12Z

@jhadvig @spadgett ready for review, we are wired up.

A snippet of the other metrics we get automatically:

# HELP Console_adds (Deprecated) Total number of adds handled by workqueue: Console
# TYPE Console_adds counter
Console_adds 17
...
# HELP Console_queue_latency (Deprecated) How long an item stays in workqueueConsole before being requested.
# TYPE Console_queue_latency summary
Console_queue_latency{quantile="0.5"} 9
Console_queue_latency{quantile="0.9"} 83151
Console_queue_latency{quantile="0.99"} 84765
Console_queue_latency_sum 432139
Console_queue_latency_count 17
...
# TYPE ManagementStateController_console_queue_latency summary
ManagementStateController_console_queue_latency{quantile="0.5"} 5
ManagementStateController_console_queue_latency{quantile="0.9"} 5
...
# TYPE ManagementStateController_console_retries counter
ManagementStateController_console_retries 0
...
workqueue_work_duration_seconds_sum{name="ManagementStateController_console"} 0.0017626229999999998
workqueue_work_duration_seconds_count{name="ManagementStateController_console"} 3
...
etc
etc

spadgett

We shouldn't be creating role bindings for things we're not using yet. Let's remove any references to metrics in openshift-console until it's needed.

jhadvig

Couple of questions, otherwise looking good 👍

brancz · 2019-09-19T13:14:01Z

Metrics should be all lower case and word boundaries separated by underscore. In general we adhere to the Kubernetes Instrumentation Guidelines in OpenShift, so we should do the same here.

benjaminapetersen · 2019-09-19T13:40:50Z

@brancz I see that, so I did:

	// metric: console_url{url="https://<url>"} 1
	consoleURLMetric = prometheus.NewGaugeVec(prometheus.GaugeOpts{
		Namespace: "console",
		Name:      "url",
		Help:      "URL of the console exposed on the cluster",
		// one label
	}, []string{"url"})

to give us console_url (using the Namespace). Do you prefer we skip the Namespace & just hard-code console_url as Name?

benjaminapetersen · 2019-09-19T13:41:43Z

@brancz disregard, i'll update to that per reading the doc you shared, thx!

- Add manifest for ServiceMonitor object - Authorize prometheus to scrape metrics from console & console-operator - NOTE: console disabled until we support the /metrics endpoint - RBAC - reenable delegated auth - add rolebinding to kube-system for extension-apiserver-authentication-reader for console-operatoro - add clusterrole system:auth-delegator to console-operator - allows authentication (tokenreview) and authorization (subjectaccessreview) - required as the /metrics endpoint is protected - mount serving-cert in operator deployment - add labels to namespaces to tell openshift monitoring to begin monitoring

spadgett · 2019-09-19T15:18:47Z

/retest

spadgett

/lgtm

openshift-ci-robot · 2019-09-19T17:53:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: benjaminapetersen, spadgett

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [benjaminapetersen,spadgett]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

benjaminapetersen · 2019-12-04T21:16:43Z

/cherry-pick release-4.1

openshift-cherrypick-robot · 2019-12-04T21:16:53Z

@benjaminapetersen: #270 failed to apply on top of branch "release-4.1":

.git/rebase-apply/patch:80: new blank line at EOF.
+
.git/rebase-apply/patch:174: new blank line at EOF.
+
warning: 2 lines add whitespace errors.
error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
A	manifests/04-rbac-rolebinding-cluster.yaml
M	manifests/04-rbac-rolebinding.yaml
M	manifests/07-operator.yaml
M	pkg/console/operator/sync_v400.go
Falling back to patching base and 3-way merge...
Auto-merging vendor/k8s.io/apiextensions-apiserver/artifacts/example/auth-delegator.yaml
CONFLICT (content): Merge conflict in vendor/k8s.io/apiextensions-apiserver/artifacts/example/auth-delegator.yaml
Auto-merging pkg/console/operator/sync_v400.go
CONFLICT (content): Merge conflict in pkg/console/operator/sync_v400.go
Auto-merging manifests/07-operator.yaml
Auto-merging manifests/04-rbac-rolebinding.yaml
CONFLICT (content): Merge conflict in manifests/04-rbac-rolebinding.yaml
Patch failed at 0001 Expose /metrics endpoint for monitoring

Details

In response to this:

/cherry-pick release-4.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

benjaminapetersen · 2019-12-04T21:17:18Z

kinda figured.

openshift-ci-robot · 2019-12-05T14:31:15Z

@benjaminapetersen: All pull requests linked via external trackers have merged. Bugzilla bug 1780170 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1780170: Expose metrics

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Adds a new ServiceMonitor to allow the Helm metrics being scraped from console /metrics endpoint by prometheus-k8s. Closes: https://issues.redhat.com/browse/HELM-235 Reference: openshift#270 Signed-off-by: Allen Bai <abai@redhat.com>

benjaminapetersen changed the title ~~Expose metrics~~ [WIP] Expose metrics Aug 14, 2019

openshift-ci-robot requested review from jhadvig and spadgett August 14, 2019 16:23

brancz reviewed Aug 30, 2019

View reviewed changes

Comment thread manifests/0000_90_console-operator_02_servicemonitor.yaml Outdated

benjaminapetersen force-pushed the monitoring/expose-metrics branch from a92d35f to 596a07f Compare September 10, 2019 19:22