Upgrading from KEDA 2.5.0 to 2.7.0 with external scaler #3032

miaooss · 2022-05-10T20:37:58Z

Report

We are using au external scaler
Keda seems to be connecting to the service correctly.

But some expected metrics or metadata are missing

Here is our KEDA configuration:

  scalingStrategy:
    strategy: "accurate"                        # Optional. Default: default. Which Scaling Strategy to use.
  triggers:
    - type: external
      metadata:
        scalerAddress: servicename.<env>:3009
        service: servicename

This is the C# code we are using


    public async override Task<GetMetricSpecResponse> GetMetricSpec(ScaledObjectRef request, ServerCallContext context)
    {
        var resp = new GetMetricSpecResponse();

        if (request.ScalerMetadata.TryGetValue("service", out string serviceValue) && serviceValue == "servicename")
        {
            resp.MetricSpecs.Add(new MetricSpec
            {
                MetricName = "targetAverageValue",
                TargetSize = 1
            });
        }

        return await Task.FromResult(resp);
    }
    
    
    public override async Task<GetMetricsResponse> GetMetrics(GetMetricsRequest request, ServerCallContext context)
    {
        var resp = new GetMetricsResponse();

        if (request.ScalerMetadata.TryGetValue("service", out string serviceValue) && serviceValue == "servicename")
        {
            if (request.MetricName == "queueLength")
            {
                long pendingJobsCount = await GetPendingJobsCount();
                resp.MetricValues.Add(new MetricValue
                {
                    MetricName = "queueLength",
                    MetricValue_ = pendingJobsCount
                });
            }
        }

        return resp;
    }

Expected Behavior

Expect KEDA to scale up the service

Actual Behavior

We do return no actual job.

Steps to Reproduce the Problem

We update KEDA version to use 2.7.0
We can tell that the old version is working fine 2.5.0

Logs from KEDA operator

1.6522127784315143e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "servicename", "scaledJob.Namespace": "dev", "Number of running Jobs": 0}

KEDA Version

2.7.0

Kubernetes Version

1.23

Platform

Microsoft Azure

Scaler Details

External

Anything else?

No response

The text was updated successfully, but these errors were encountered:

JorTurFer · 2022-05-12T17:16:39Z

hey @miaooss
sorry for the slow response :(
We had problems related with the name generation in v2.6.0 but they have been solved in v2.7.0.
Is there any error in the external metric server?
In the log that you shared seems all normal, could you share more lines please?

miaooss · 2022-05-13T15:57:24Z

Hi,

Thank you for the assistance.

Here is the logs for the keda-operator-metrics-apiserver

I0505 18:50:52.872453       1 main.go:147] keda_metrics_adapter "msg"="KEDA Version: 2.7.0"
I0505 18:50:52.872505       1 main.go:148] keda_metrics_adapter "msg"="KEDA Commit: 5f8af5cd90805d8b9c5964e1619a8b2bdcec6cfd"
I0505 18:50:52.872512       1 main.go:149] keda_metrics_adapter "msg"="Go Version: go1.17.3"
I0505 18:50:52.872518       1 main.go:150] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64"
I0505 18:50:55.024359       1 request.go:665] Waited for 1.047381127s due to client-side throttling, not priority and fairness, request: GET:https://10.0.0.1:443/apis/security.istio.io/v1beta1?timeout=32s
I0505 18:50:56.379197       1 logr.go:261] keda_metrics_adapter/controller-runtime/metrics "msg"="Metrics server is starting to listen"  "addr"=":8080"
I0505 18:50:56.379673       1 provider.go:65] keda_metrics_adapter/provider "msg"="starting"
I0505 18:50:56.379697       1 main.go:214] keda_metrics_adapter "msg"="starting adapter..."
2022/05/05 18:50:56 Starting metrics server at :9022
I0505 18:50:56.380078       1 internal.go:362] keda_metrics_adapter "msg"="Starting server" "addr"={"IP":"::","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics"
I0505 18:50:56.380194       1 controller.go:178] keda_metrics_adapter/controller/scaledobject "msg"="Starting EventSource" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject" "source"="kind source: *v1alpha1.ScaledObject"
I0505 18:50:56.380211       1 controller.go:178] keda_metrics_adapter/controller/scaledobject "msg"="Starting EventSource" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject" "source"="kind source: *v1alpha1.ScaledObject"
I0505 18:50:56.380221       1 controller.go:186] keda_metrics_adapter/controller/scaledobject "msg"="Starting Controller" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject"
I0505 18:50:56.481278       1 controller.go:220] keda_metrics_adapter/controller/scaledobject "msg"="Starting workers" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject" "worker count"=1
I0505 18:50:56.885629       1 serving.go:342] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0505 18:50:57.606025       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0505 18:50:57.606114       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0505 18:50:57.606148       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0505 18:50:57.606156       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0505 18:50:57.606164       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0505 18:50:57.606166       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0505 18:50:57.606394       1 secure_serving.go:266] Serving securely on [::]:6443
I0505 18:50:57.606475       1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0505 18:50:57.606769       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0505 18:50:57.706800       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0505 18:50:57.706845       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0505 18:50:57.706978       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file

here is more log for the keda-operator

1.6524569446550982e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569446551404e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569446554346e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652456944655462e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652456958601352e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569586013892e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569595817766e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652456959581809e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569596552165e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569596552806e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569596554308e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569596554503e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569736168468e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569736168823e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569745867527e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569745868163e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652456974658725e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569746587565e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569746588607e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569746588833e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569886073647e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569886074188e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652456989584188e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569895844314e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569896585844e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569896586394e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652456989658696e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569896587121e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457003604612e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570036046746e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570045903165e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570045903482e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570046629522e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570046629887e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570046630619e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457004663086e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457018608771e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457018608892e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570195849993e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570195858319e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570196627512e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570196627936e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570196629586e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570196629784e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570335978518e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570335978868e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457034582769e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570345828042e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570346581707e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570346582224e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570346662562e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570346662822e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457048595017e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570485950663e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457049587505e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570495875382e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570496693807e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570496694207e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570496695542e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570496700332e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.65245706359464e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570635946712e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457064585202e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457064585233e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457064656624e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570646566572e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457064659847e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570646598754e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570786188848e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570786189187e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570795922773e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.65245707959231e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457079659782e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570796598115e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570796600237e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570796600382e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570936210456e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570936210818e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570945882676e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570945882998e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570946588528e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457094658886e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570946590633e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570946590824e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457108618759e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457108618796e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571095873528e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571095873878e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457109631762e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571096317909e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571096811817e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571096812131e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.65245712359978e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571235998347e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457124635557e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571246355932e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571246629527e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571246629891e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571246710186e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571246710455e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571386051843e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571386052206e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571395921571e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457139592191e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457139661193e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571396612282e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457139661309e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571396613333e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571536284606e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571536285079e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571545890434e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571545890923e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457154661795e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571546618304e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571546619413e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457154661958e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571686101148e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571686101503e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457169594603e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457169594639e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571696704307e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457169670483e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457169670598e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571696706245e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571836137216e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571836137826e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571845879164e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571845879683e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571846665828e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457184666625e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457184666774e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571846667974e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571986211176e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571986211545e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571995916958e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571995917401e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571996617348e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571996617699e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571996751537e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571996751778e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457213617805e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572136178405e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572145926697e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572145927079e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457214664278e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572146643088e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572146645582e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572146645832e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457228600856e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572286008997e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572295974908e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572295975227e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457229668759e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572296687949e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572296689763e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572296689982e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572436069596e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572436070082e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457244596044e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572445960753e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572446663008e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457244666337e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572446665025e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457244666519e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572586117945e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.65245725861183e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572595968733e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572595969095e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457259672644e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572596726744e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572596727903e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457259672809e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.65245727361469e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572736147256e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572745920768e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572745921156e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457274664208e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457274664243e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572746644068e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572746644235e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572886217391e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572886217892e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457289598154e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572895981982e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457289655974e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457289656167e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572896771553e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572896771848e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573036131463e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573036131837e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457304597814e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573045978465e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573046702592e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573046702964e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573046704085e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573046704376e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573186286345e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457318628688e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457319596347e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573195963778e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573196797385e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573196797764e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573196800365e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573196800718e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457333612797e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573336128378e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457334596943e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457334596976e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573346715412e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573346715832e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457334671983e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573346720085e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457348621807e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573486218617e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573495978022e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573495978377e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573496727686e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573496728072e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573496730149e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457349673032e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573636113615e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457363611409e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573645989597e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573645989912e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573646701136e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.65245736467015e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457364670288e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573646703088e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}

miaooss · 2022-05-13T15:59:17Z

Actually all the logs seems normal

In our code, we have a condition if Keda does not request the correct metric we send nothing.

It seems like Keda does not send the expected metric.

Thank you,
Jeremy.

JorTurFer · 2022-05-13T16:29:09Z

I have an idea about why this can be happening, I'm checking it, I'm checking it locally

JorTurFer · 2022-05-14T00:00:02Z

I have found the problem and unfortunately there is not any workaround.
Basically, it's due to an internal changes introduced in v2.6 to generate unique metric names and how we use the metric names in ScaledJobs. I have opened a PR with a possible fix but next week is KubeCon so we'll be quite offline and we have to discuss the implementation I have done.
Sorry for the inconvenience 😞

JorTurFer · 2022-05-26T09:11:12Z

hey @miaooss ,
I have just merged a PR I expect solves your problem, if you can test it and give feedback about if it's solved or not, we can go deeper if it's not solved (just for not waiting till next release)

miaooss · 2022-05-26T15:31:27Z

@JorTurFer
Thank you, I will give it a try
Can you let me know what version the fix is applied to?

JorTurFer · 2022-05-26T17:14:29Z

Can you let me know what version the fix is applied to?

it's in main

miaooss · 2022-07-12T15:23:38Z

It seems like the issue is still present on the 2.7.1.

We are using a ScaledJob with an external trigger

Return metrics for GetMetricSpec

resp.MetricSpecs.Add(new MetricSpec { MetricName = "testmetrics", TargetSize = 1 });

Return metrics for GetMetrics

resp.MetricValues.Add(new MetricValue { MetricName = "testmetrics", MetricValue_ = pendingJobsCount });

Logs of the service
We can see that the GetMetrics endpoint is called

info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 1.2699ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 7.2223ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.3122ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 0.6693ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 6.3065ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.2627ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.2627ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 45.4623ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 6.6638ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.2603ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 1.2903ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 6.1950ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.3217ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 1.0164ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 8.1591ms

JorTurFer · 2022-07-12T15:57:33Z

Hey,
Yes I know it, v2.7.1 was released on May 10 and the fix was merged on May 26. That's why I said to give a try using main tag because next version will be in August. If you still can give a try using main tag to ensure that v2.8 fixes it, would be nice. If not, don't worry, next version will be released in a month

miaooss · 2022-07-12T17:02:03Z

Thank you for the clarification.

JorTurFer · 2022-07-12T17:05:58Z

No worries, happy to help :)

miaooss added the bug Something isn't working label May 10, 2022

tomkerkhove added this to Roadmap - KEDA Core May 10, 2022

tomkerkhove moved this to Proposed in Roadmap - KEDA Core May 10, 2022

JorTurFer mentioned this issue May 13, 2022

fix: use metricName from GetMetricsSpec in ScaledJobs instead of queueLength #3046

Merged

3 tasks

JorTurFer moved this from Proposed to Ready To Ship in Roadmap - KEDA Core May 26, 2022

JorTurFer moved this from Ready To Ship to To Do in Roadmap - KEDA Core May 26, 2022

JorTurFer moved this from To Do to In Review in Roadmap - KEDA Core May 26, 2022

JorTurFer closed this as completed May 26, 2022

Repository owner moved this from In Review to Ready To Ship in Roadmap - KEDA Core May 26, 2022

tomkerkhove moved this from Ready To Ship to Done in Roadmap - KEDA Core Aug 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading from KEDA 2.5.0 to 2.7.0 with external scaler #3032

Upgrading from KEDA 2.5.0 to 2.7.0 with external scaler #3032

miaooss commented May 10, 2022 •

edited by tomkerkhove

Loading

JorTurFer commented May 12, 2022

miaooss commented May 13, 2022 •

edited

Loading

miaooss commented May 13, 2022

JorTurFer commented May 13, 2022

JorTurFer commented May 14, 2022

JorTurFer commented May 26, 2022

miaooss commented May 26, 2022

JorTurFer commented May 26, 2022

miaooss commented Jul 12, 2022 •

edited

Loading

JorTurFer commented Jul 12, 2022

miaooss commented Jul 12, 2022

JorTurFer commented Jul 12, 2022

Upgrading from KEDA 2.5.0 to 2.7.0 with external scaler #3032

Upgrading from KEDA 2.5.0 to 2.7.0 with external scaler #3032

Comments

miaooss commented May 10, 2022 • edited by tomkerkhove Loading

Report

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Logs from KEDA operator

KEDA Version

Kubernetes Version

Platform

Scaler Details

Anything else?

JorTurFer commented May 12, 2022

miaooss commented May 13, 2022 • edited Loading

miaooss commented May 13, 2022

JorTurFer commented May 13, 2022

JorTurFer commented May 14, 2022

JorTurFer commented May 26, 2022

miaooss commented May 26, 2022

JorTurFer commented May 26, 2022

miaooss commented Jul 12, 2022 • edited Loading

JorTurFer commented Jul 12, 2022

miaooss commented Jul 12, 2022

JorTurFer commented Jul 12, 2022

miaooss commented May 10, 2022 •

edited by tomkerkhove

Loading

miaooss commented May 13, 2022 •

edited

Loading

miaooss commented Jul 12, 2022 •

edited

Loading