add pause metric in prometheus for scaledobject #4446

eladmotola · 2023-04-12T09:41:44Z

Provide a description of what has been changed

Checklist

When introducing a new scaler, I agree with the scaling governance policy
I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #

Relates to #4430
Relates to kedacore/keda-docs#1111

pkg/scaling/scale_handler.go

pkg/prommetrics/prommetrics.go

tomkerkhove · 2023-04-14T13:12:35Z

Can you open a PR for our docs on kedacore/keda-docs please?

eladmotola · 2023-04-17T20:04:48Z

@JorTurFer Hey, can you please review again?
I will fix the ci(dco) on the weekend

pkg/prommetrics/prommetrics.go

tomkerkhove

LGTM

pkg/prommetrics/prommetrics.go

JorTurFer · 2023-04-24T19:03:18Z

Your latest commit isn't signed, could you fix it? https://github.com/kedacore/keda/blob/main/CONTRIBUTING.md#i-didnt-sign-my-commit-now-what

JorTurFer · 2023-04-24T19:07:33Z

And there is an error in the code https://github.com/kedacore/keda/actions/runs/4790038051/jobs/8518637396?pr=4446#step:11:20

JorTurFer · 2023-04-24T19:30:39Z

We still need the DCO fixed 🙏
https://github.com/kedacore/keda/blob/main/CONTRIBUTING.md?#i-didnt-sign-my-commit-now-what

Signed-off-by: Elad Motola <[email protected]>

eladmotola · 2023-04-24T20:24:37Z

@JorTurFer I fixed everything 😁

JorTurFer · 2023-04-24T20:41:10Z

/run-e2e prometheus
Update: You can check the progress here

tomkerkhove · 2023-04-25T06:07:46Z

/run-e2e prometheus
Update: You can check the progress here

tests/sequential/prometheus_metrics/prometheus_metrics_test.go

Signed-off-by: Elad Motola <[email protected]> Signed-off-by: eladmotola <[email protected]>

JorTurFer · 2023-04-25T14:14:08Z

/run-e2e prometheus
Update: You can check the progress here

zroubalik

I can see some inconsistencies in the code:

	scaledObjectPausedTotal = prometheus.NewGaugeVec(
		prometheus.GaugeOpts{
			Namespace: DefaultPromMetricsNamespace,
			Subsystem: "scaled_object",
			Name:      "paused",
			Help:      "Total number of paused scaled_objects",
		},
		[]string{"namespace", "scaledObject"},

vs

	if _, ok := family["keda_scaleobject_pause"]; ok {
		t.Errorf("metric should not be available")
	}

etc.

So what is the intention behind this PR? It reports total number of ScaledObjects that are in a paused state?

I think that it would be great if we add a metric, that acutally shows whether a specific ScaledObject is paused or not? At least that's what was the idea behind #4430

I think, that similar metric to keda_scaler_activity, but in this case on ScaledObject level and not Scaler level. ie. keda_scaled_object_paused should tell us whether a specific SO is paused or not.

eladmotola · 2023-04-25T15:17:05Z

here

oh wow i missed that...
and yes this is the purpose of this pr. do you think i implement it the wrong way?

Signed-off-by: Elad Motola <[email protected]> Signed-off-by: eladmotola <[email protected]>

JorTurFer · 2023-04-25T19:27:43Z

I think that it would be great if we add a metric, that acutally shows whether a specific ScaledObject is paused or not? At least that's what was the idea behind #4430

Why do you think that this PR doesn't do it? I mean, the metric is updated on each reconciliation based on if it's paused or not

zroubalik · 2023-04-26T12:08:52Z

I am not saying it doesn't do that, it was just super confusing given the metric name mismatch and also this:

Help: "Total number of paused scaled_objects",

I just briefly went through the code, I will do a proper review later :)

eladmotola · 2023-04-28T19:23:29Z

Hey, can you trigger the e2e tests?

JorTurFer · 2023-05-01T21:57:47Z

/run-e2e prometheus
Update: You can check the progress here

eladmotola · 2023-05-05T20:17:01Z

The branch was merged from main

JorTurFer · 2023-05-07T18:47:21Z

/run-e2e prometheus
Update: You can check the progress here

zroubalik

This PR should be rebased once this: #4470 is merged.

zroubalik · 2023-05-17T15:05:36Z

pkg/prommetrics/prommetrics.go

@@ -204,3 +215,9 @@ func DecrementCRDTotal(crdType, namespace string) {

 	crdTotalsGaugeVec.WithLabelValues(crdType, namespace).Dec()
 }
+
+func RecordScalerPaused(namespace string, scaledObject string, value float64) {


could you please rewrite this function to accept bool instead of float64? the same way RecordScalerActive() is written.

I don't mind doing it. I just can't figure out why the tests fails

zroubalik · 2023-05-17T15:08:26Z

pkg/prommetrics/prommetrics.go

+			Namespace: DefaultPromMetricsNamespace,
+			Subsystem: "scaled_object",
+			Name:      "paused",
+			Help:      "Total number of paused scaled_objects",


The help message is wrong.

eladmotola · 2023-07-01T13:19:46Z

Any chance that one of you guys can see why what I did is wrong?

JorTurFer · 2023-07-05T21:19:37Z

/run-e2e prometheus
Update: You can check the progress here

JorTurFer · 2023-07-05T21:21:17Z

It looks like the expected metric is 0:

JorTurFer · 2023-07-05T21:24:03Z

tests/sequential/prometheus_metrics/prometheus_metrics_test.go

+	KubectlApplyWithTemplate(t, data, "deploymentForPausedTemplate", deploymentForPausedTemplate)
+	KubectlApplyWithTemplate(t, data, "scaledObjectPausedTemplate", scaledObjectPausedTemplate)
+
+	family = fetchAndParsePrometheusMetrics(t, fmt.Sprintf("curl --insecure %s", kedaOperatorPrometheusURL))


I think that the metric is checked too fast and KEDA doesn't have enough time to process the ScaledObject. I guess that you can introduce a sleep here (5 seconds should be enough) or maybe deploying this resources in the beginning of the test file instead of inside the test case

JorTurFer · 2023-09-24T18:21:10Z

Any update? We aim to cut a release next week

eladmotola · 2023-09-26T07:20:18Z

Any update? We aim to cut a release next week

Hey,
I'm on vacation at the moment. Feel free to take it from here

geoffrey1330 · 2023-10-03T13:21:52Z

Hi @zroubalik I want to finish up this PR.
could you please assign the issue to me

zroubalik · 2023-10-03T15:47:51Z

Any update? We aim to cut a release next week

Hey, I'm on vacation at the moment. Feel free to take it from here

@eladmotola we will continue in this PR - @geoffrey1330 volunteered to do so.

zroubalik · 2023-11-13T18:57:04Z

Done in #5045

eladmotola requested a review from a team as a code owner April 12, 2023 09:41

eladmotola force-pushed the feat/pause-metric branch from 1213fad to ee44129 Compare April 12, 2023 10:07

JorTurFer reviewed Apr 12, 2023

View reviewed changes

pkg/scaling/scale_handler.go Outdated Show resolved Hide resolved

tomkerkhove reviewed Apr 14, 2023

View reviewed changes

pkg/prommetrics/prommetrics.go Outdated Show resolved Hide resolved

eladmotola mentioned this pull request Apr 22, 2023

add keda_scaled_object_paused to the docs kedacore/keda-docs#1111

Closed

1 task

eladmotola force-pushed the feat/pause-metric branch from dc1278d to 759a45a Compare April 22, 2023 08:42

tomkerkhove reviewed Apr 22, 2023

View reviewed changes

pkg/prommetrics/prommetrics.go Outdated Show resolved Hide resolved

tomkerkhove approved these changes Apr 23, 2023

View reviewed changes

JorTurFer reviewed Apr 24, 2023

View reviewed changes

pkg/prommetrics/prommetrics.go Outdated Show resolved Hide resolved

one commit

a3ae5e2

Signed-off-by: Elad Motola <[email protected]>

eladmotola force-pushed the feat/pause-metric branch from 0b3de51 to a3ae5e2 Compare April 24, 2023 19:53

JorTurFer requested a review from zroubalik April 24, 2023 20:40

JorTurFer approved these changes Apr 24, 2023

View reviewed changes

tomkerkhove approved these changes Apr 25, 2023

View reviewed changes

JorTurFer reviewed Apr 25, 2023

View reviewed changes

tests/sequential/prometheus_metrics/prometheus_metrics_test.go Outdated Show resolved Hide resolved

fix scaledObjectPausedTemplate template

4941b65

Signed-off-by: Elad Motola <[email protected]> Signed-off-by: eladmotola <[email protected]>

zroubalik reviewed Apr 25, 2023

View reviewed changes

fix metric name in tests

bd1c9d7

Signed-off-by: Elad Motola <[email protected]> Signed-off-by: eladmotola <[email protected]>

Merge branch 'main' into feat/pause-metric

bdae457

zroubalik reviewed May 17, 2023

View reviewed changes

JorTurFer reviewed Jul 5, 2023

View reviewed changes

zroubalik assigned geoffrey1330 Oct 3, 2023

This was referenced Oct 4, 2023

add pause metric in prometheus for scaledobject #5044

Closed

add pause metric in prometheus for scaledobject #5045

Merged

zroubalik closed this Nov 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add pause metric in prometheus for scaledobject #4446

add pause metric in prometheus for scaledobject #4446

eladmotola commented Apr 12, 2023 •

edited by tomkerkhove

Loading

tomkerkhove commented Apr 14, 2023

eladmotola commented Apr 17, 2023

tomkerkhove left a comment

JorTurFer commented Apr 24, 2023

JorTurFer commented Apr 24, 2023

JorTurFer commented Apr 24, 2023

eladmotola commented Apr 24, 2023

JorTurFer commented Apr 24, 2023 •

edited by github-actions bot

Loading

tomkerkhove commented Apr 25, 2023 •

edited by github-actions bot

Loading

JorTurFer commented Apr 25, 2023 •

edited by github-actions bot

Loading

zroubalik left a comment •

edited

Loading

eladmotola commented Apr 25, 2023

JorTurFer commented Apr 25, 2023 •

edited

Loading

zroubalik commented Apr 26, 2023 •

edited

Loading

eladmotola commented Apr 28, 2023

JorTurFer commented May 1, 2023 •

edited by github-actions bot

Loading

eladmotola commented May 5, 2023

JorTurFer commented May 7, 2023 •

edited by github-actions bot

Loading

zroubalik left a comment

zroubalik May 17, 2023

eladmotola May 18, 2023

zroubalik May 17, 2023

eladmotola commented Jul 1, 2023

JorTurFer commented Jul 5, 2023 •

edited by github-actions bot

Loading

JorTurFer commented Jul 5, 2023

JorTurFer Jul 5, 2023

JorTurFer commented Sep 24, 2023

eladmotola commented Sep 26, 2023

geoffrey1330 commented Oct 3, 2023 •

edited

Loading

zroubalik commented Oct 3, 2023

zroubalik commented Nov 13, 2023

add pause metric in prometheus for scaledobject #4446

add pause metric in prometheus for scaledobject #4446

Conversation

eladmotola commented Apr 12, 2023 • edited by tomkerkhove Loading

Checklist

tomkerkhove commented Apr 14, 2023

eladmotola commented Apr 17, 2023

tomkerkhove left a comment

Choose a reason for hiding this comment

JorTurFer commented Apr 24, 2023

JorTurFer commented Apr 24, 2023

JorTurFer commented Apr 24, 2023

eladmotola commented Apr 24, 2023

JorTurFer commented Apr 24, 2023 • edited by github-actions bot Loading

tomkerkhove commented Apr 25, 2023 • edited by github-actions bot Loading

JorTurFer commented Apr 25, 2023 • edited by github-actions bot Loading

zroubalik left a comment • edited Loading

Choose a reason for hiding this comment

eladmotola commented Apr 25, 2023

JorTurFer commented Apr 25, 2023 • edited Loading

zroubalik commented Apr 26, 2023 • edited Loading

eladmotola commented Apr 28, 2023

JorTurFer commented May 1, 2023 • edited by github-actions bot Loading

eladmotola commented May 5, 2023

JorTurFer commented May 7, 2023 • edited by github-actions bot Loading

zroubalik left a comment

Choose a reason for hiding this comment

zroubalik May 17, 2023

Choose a reason for hiding this comment

eladmotola May 18, 2023

Choose a reason for hiding this comment

zroubalik May 17, 2023

Choose a reason for hiding this comment

eladmotola commented Jul 1, 2023

JorTurFer commented Jul 5, 2023 • edited by github-actions bot Loading

JorTurFer commented Jul 5, 2023

JorTurFer Jul 5, 2023

Choose a reason for hiding this comment

JorTurFer commented Sep 24, 2023

eladmotola commented Sep 26, 2023

geoffrey1330 commented Oct 3, 2023 • edited Loading

zroubalik commented Oct 3, 2023

zroubalik commented Nov 13, 2023

eladmotola commented Apr 12, 2023 •

edited by tomkerkhove

Loading

JorTurFer commented Apr 24, 2023 •

edited by github-actions bot

Loading

tomkerkhove commented Apr 25, 2023 •

edited by github-actions bot

Loading

JorTurFer commented Apr 25, 2023 •

edited by github-actions bot

Loading

zroubalik left a comment •

edited

Loading

JorTurFer commented Apr 25, 2023 •

edited

Loading

zroubalik commented Apr 26, 2023 •

edited

Loading

JorTurFer commented May 1, 2023 •

edited by github-actions bot

Loading

JorTurFer commented May 7, 2023 •

edited by github-actions bot

Loading

JorTurFer commented Jul 5, 2023 •

edited by github-actions bot

Loading

geoffrey1330 commented Oct 3, 2023 •

edited

Loading