Record metrics for scaler latency

[kevinteng525]

Proposal

For production monitoring, we want to actively monitoring different kinds of metrics, KEDA already provided a lot, and one more metric I would like to have is the latency of retrieve metrics from scalar. That will be helpful for us to diagnose and locate the root cause.

Suggested to add a metric calling keda_metrics_adapter_scaler_metrics_latency
And record the time of
metrics, err := cache.GetMetricsForScaler(ctx, scalerIndex, info.Metric, metricSelector)

Use-Case

No response

Anything else?

No response

[JorTurFer]

Hi @kevinteng525
This sounds really useful IMO, but not in the adapter (at least not only). Since v2.9, all the metric requesting load has been moved from metrics server (adapter) to the operator, now the adapter just makes a gRPC request to the operator for getting the value. I think this metric could be more useful getting it from the operator itself, to have the querying time to the upstream.
WDYT, are you willing to contribute it?

[kevinteng525]

Hi @JorTurFer ,

Sure, I will contribute on it.

Thanks,
Kevin

[kevinteng525]

#4040 submitted, PTAL

[zroubalik]

A great suggestion, I like it.

[tomkerkhove]

+1