Prometheus' metrics in the operator are only exposed when the metrics server requests the metrics

[JorTurFer]

Report

I have seen that metric keda_scaler_metrics_value is updated when the value is queried by the ms (and the same for keda_scaler_error_totals and keda_scaled_object_error_totals), I think that it makes sense to update the metric after GetMetricsAndActivity (inside GetScaledObjectState) because it has been read

KEDA Version

2.9.1

[nunoadrego]

Is this related to the fact that apart from Metrics exposed by the Operator SDK, I only see keda_resource_totals and keda_trigger_totals metrics?

I was expecting to see keda_scaler_metrics_value, keda_scaler_errors, keda_scaler_error_totals and keda_scaled_object_error_totals, even if they were zero.

[zroubalik]

Is this related to the fact that apart from Metrics exposed by the Operator SDK, I only see keda_resource_totals and keda_trigger_totals metrics?

I was expecting to see keda_scaler_metrics_value, keda_scaler_errors, keda_scaler_error_totals and keda_scaled_object_error_totals, even if they were zero.

So you don't see these values once you create a ScaledObject?

[nunoadrego]

I was able to see them after following this https://keda.sh/docs/2.9/operate/metrics-server/. Since most of the SOs only use CPU and Memory scalers, I read in another issue this is intended.

I can now see:

• keda_resource_totals
• keda_scaled_errors
• keda_scaler_errors (don't appear in the docs or have a different name)
• keda_scaler_metrics_value
• keda_trigger_totals

Missing (according to docs):

• keda_scaler_error_totals
• keda_scaled_object_error_totals

The latest official Grafana dashboard depends on some of these metrics, in particular, keda_scaled_object_errors. What's the best approach to deal with this behavior?

[zroubalik]

So the supported metrics are (for 2.9.1.)

keda_scaler_metrics_value
keda_scaler_errors
keda_scaler_error_totals
keda_scaled_object_errors     # due to a bug this is now reported as keda_scaled_errors, has been fixed already
keda_resource_totals
keda_trigger_totals

[zroubalik]

We will most likely release 2.9.2 soon, this should also fix the exported metrics.

[zroubalik]

We will most likely release 2.9.2 soon, this should also fix the exported metrics.

2.9.2 including fix this has been just released!

[nunoadrego]

Will try it and provide feedback @zroubalik. Thanks!

[vmm-afonso]

Is this related to the fact that apart from Metrics exposed by the Operator SDK, I only see keda_resource_totals and keda_trigger_totals metrics?

I was expecting to see keda_scaler_metrics_value, keda_scaler_errors, keda_scaler_error_totals and keda_scaled_object_error_totals, even if they were zero.

Not sure if this was supposed to be fixed since I'm rolling with the latest release v.2.9.3 and I am having the same problem.

This is what I'm getting

I was trying to get the grafana dashboard to work but as @nunoadrego pointed out, the dashboard depends on some of these metrics. I'm also unable to set some most of the variables for the dashboard apart from the data-source.

Any tips?