KEDA metric name issue

[krishna-kariya]

Hi,
I am testing HTTP-triggered Azure Functions on Azure Arc with Keda version 2.5 and 2.6.1. We have created our external HTTP scaler on Keda for autoscaling.

I am getting the error below in the metrics API server
E0207 02:55:09.412436 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"No matching metrics found for s0-upstream_rq_total"}: No matching metrics found for s0-upstream_rq_total.

The hpa shows a similar error:
invalid metrics (4 invalid out of 4), first error is: failed to get s0-upstream_rq_active external metric: unable to get external metric k8se-apps/s0-upstream_rq_active/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: myapp,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-upstream_rq_active

I see that from Keda v2.5 and later, the external metric names are prefixed with "sX-". Is this a new addition?
We have created our external HTTP scaler. Do we need to update our external HTTP scaler with these new metric names?

Any help would be appreciated. Thanks.

[tomkerkhove]

@zroubalik @JorTurFer This feels similar to #2632, right?

I see that from Keda v2.5 and later, the external metric names are prefixed with "sX-". Is this a new addition?
We have created our external HTTP scaler. Do we need to update our external HTTP scaler with these new metric names?

This was introduced a few versions back to ensure metric names are unique when using multiple triggers. In theory this should not break you but, afaik, you can always override the name with your own (but make sure it's unique)

[JatinSanghvi]

I am still unclear on how adding s0- before the metric name helps with uniqueness, what does the 0 or 1 after the s signify, will KEDA not prefix s0- if user themselves have added s0- to the metric name, and what different use cases are impacted with the breaking change introduced in KEDA 2.5.0? Any clarification or links to documentation are much appreciated.

[JorTurFer]

hi @JatinSanghvi
You can find how the name should be generated here. Basically, KEDA add the prefix sX where X is the index of the scaler inside the ScaledObject.

The reason behind this decision is that metric names must be unique per each scaled object (no problem if they are duplicated with other ScaledObjets). When the HPA does the metric request, it sends the ScaledObject labelSelector as a query param, that's why inside the ScaledObject the metric name must be unique.
BTW, if the user adds the s0 to metric name in scalers which allow yet the metric name setting, the final metric will be like sX-{scalerName}-s0-myMetricName. This is because KEDA always add the prefix sX- before all metrics.

[tomkerkhove]

In theory, though, nobody should have to care about this as KEDA core would do translation from metric name to trigger and then call your external scaler - Isn't that correct @JorTurFer? This is similar issue as #2632 I presume?

[JorTurFer]

I think that this is different from #2632. In this case the external scaler is generating the metric wrong because the prefix. In the other case the metric is correctly generated and the SQS scaler supports it because all is internal.

[JorTurFer]

I'm reviewing and probably we have made a mistake in external scaler. I can see that on GetMetrics we are passing the received metric name to the external scaler (with sX- prefix) and but probably we should pass the metadata metricName (the name given by user). the prefix usage is something internal from KEDA that we are making external in this call.

Other option could be in this specific scaler, remove the prefix (which is known inside the scaler) and add it again after the external query. Thoughts @zroubalik ?

[tomkerkhove]

Created #2640

[JatinSanghvi]

Just to update, I did not find any problem scaling deployments with KEDA 2.6.1 and Cosmos DB external scaler.

[tomkerkhove]

Thanks for letting us know. That is with the workaround I presume or not because you don't rely on the metric name? 🤔

[JatinSanghvi]

Without the workaround. I still think the regression only affects the external scalers that return values for metrics with names are self-generated inside the external scaler application. They may have a function that generates metric names from scaler-metadata. The handlers for both GetMetricSpec and GetMetrics requests call this common function.

External scalers like Cosmos DB, which return single metric value per query may opt to not regenerate metric name, but return the same metric name as extracted from the incoming GetMetrics request. This category of scalers should not be impacted. Let me know if I have made incorrect assumption.

[tomkerkhove]

That's what I figured as well - Just wanted to verify :) Thanks.

[v-shenoy]

Just putting some of my findings here, trying to understand the behavior.

Let's say I have an external scaler, and KEDA calls GetMetricSpecForScaling and I return two specs with names metricSpec1 and metricSpec2

KEDA passes these on to the HPA as s0-metricSpec1 and s0-metricSpec2 (assuming I only have one scaler in the Scaled Object). Now, the HPA fetches the metrics via a call to the provider if am right, as per this.

Executing kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/" to get the available metrics gives me

{
"kind":"APIResourceList", "apiVersion":"v1", "groupVersion":"external.metrics.k8s.io/v1beta1",
"resources":
    [
        {"name":"s0-metricSpec1","singularName":"","namespaced":true,"kind":"ExternalMetricValueList","verbs":["get"]},
        {"name":"s0-metricSpec2","singularName":"","namespaced":true,"kind":"ExternalMetricValueList","verbs":["get"]}
    ]
}

At some point the provider.GetExternalMetric method is called with info.Metric = s0-metricSpec1 the provider will then make a call to the scaler's GetMetrics method with this as the metric name. Say the method returns an array with [{MetricName: metric1, metricValue: <value1>}, {MetricName: metric2, metricValue: <value2>}]

I can see both these values when executing kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/<scaled-object-namespace>/s0-metricSpec1" as

{
"kind":"ExternalMetricValueList","apiVersion":"external.metrics.k8s.io/v1beta1","metadata":{},
"items":
     [
         {"metricName":"metric1","metricLabels":null,"timestamp":"2022-04-07T08:16:21Z","value":"<value1>"}, 
         {"metricName":"metric2","metricLabels":null,"timestamp":"2022-04-07T08:16:21Z","value":"<value2>"}
    ]
}

My question here is, what are the individual metric names used for? Aren't these both just values for the metric s0-metricSpec1? And since they are both returned for s0-metricSpec1, how does the HPA pick the current value between <value1> and <value2>?

@tomkerkhove @JorTurFer @zroubalik

[v-shenoy]

As for why our external scaler in Arc doesn't work is we internally use a map to store the list of metrics corresponding to metric names. So, when KEDA tries to call GetMetrics with s0-dummyMetricName this name isn't in our map (but the key dummyMetricName is in our map), so we return an empty metric response. This results in the provider (and thus the HPA) not receiving any metrics and KEDA raises an error .

So the only case this really causes an error is when the external scaler has a condition on the metric name which might cause it to not return any values.

If the external scaler is agnostic of the metric name passed to it in GetMetrics and returns a response (with whatever individual metric names, these do not matter), these will show up under s0-dummyMetricName in the metrics server.