ScaledJob: introduce MultipleScalersCalculation

[TsuyoshiUshio]

This PR fix issue of multi scalers on Scaled Job. And introduce a new option.
The current login assume only one scaler is active and available.
The most of the cases, it is ok for that. That is why we don't get any issues about it.

However @zroubalik pointed out this issue, I come up with the fix and do some refactoring to make it clear.

I'm not writing documentation yet, before that, I'd like to confirm this logic if fine for you guys.
Once It looks, ok, I'll go ahead full testing and write the documentation and make this PR as not draft.

Spec

Introduce a new option MultipleScalersCalculation on ScaledJob.

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: {scaled-job-name}
spec:
  jobTargetRef:
         :
  scalingStrategy:
    strategy: "custom"                        # Optional. Default: default. Which Scaling Strategy to use. 
    customScalingQueueLengthDeduction: 1      # Optional. A parameter to optimize custom ScalingStrategy.
    customScalingRunningJobPercentage: "0.5"  # Optional. A parameter to optimize custom ScalingStrategy.
    pendingPodConditions:                     # Optional. A parameter to calculate pending job count per the specified pod conditions
      - "Ready"
      - "PodScheduled"
      - "AnyOtherCustomPodCondition"
    MultipleScalersCalculation: "max" # Optional. Default: max. If multiple scaler is exists, how to calculate the queueLength and maxValue (that is the expected number of pods that will be created)
  triggers:

Available Options

The option is how keda calculate queueLength and maxValue that is the number of pod will be created.

max : default, pick a scaler that has the max number of queueLength.
min: pick a scaler that has the min number of queueLength.
avg: sum up all the active scalers metrics and devided by the number of the active scalers.
sum: sum up all the active scalers metrics

Checklist

• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• Tests have been added
• A PR is opened to update the documentation on (repo) (if applicable)
• Changelog has been updated

Fixes #

Document #
kedacore/keda-docs#505

[TsuyoshiUshio]

I tested, and I add test case and improve logging for clarifying on the multiple scalers scenario.
One question is, the change of controller-gen.kubebuilder.io/version: v0.5.0 from v0.3.0 Should I keep it to v0.3.0 ? It is auto generated, so that I keep it for now.

[TsuyoshiUshio]

Hi @zroubalik
I update this PR as you requested. Could you have a look?
I also update the documentation according to the name suggestion.
kedacore/keda-docs#505

[TsuyoshiUshio]

Hi @zroubalik
I updated the PR. Could you have a look?

[zroubalik]

Looking good! I have a few very minor nits. Thanks @TsuyoshiUshio !

[TsuyoshiUshio]

Hi @zroubalik I accept all your suggestion, however, CI / Validate PR looks start to fail. I looked the code change again, however, I just changed ScaledJob related part. The CI fails with the ScaledObjectController. Do you know this issue?

ScaledObjectController
/__w/keda/keda/controllers/keda/scaledobject_controller_test.go:50
  functional tests
  /__w/keda/keda/controllers/keda/scaledobject_controller_test.go:204
    deploys ScaledObject and creates HPA, when IdleReplicaCount, MinReplicaCount and MaxReplicaCount is defined [It]
    /__w/keda/keda/controllers/keda/scaledobject_controller_test.go:269

    Timed out after 20.001s.
    Expected
        <v1.ConditionStatus>: Unknown
    to equal
        <v1.ConditionStatus>: True

[zroubalik]

Hi @zroubalik I accept all your suggestion, however, CI / Validate PR looks start to fail. I looked the code change again, however, I just changed ScaledJob related part. The CI fails with the ScaledObjectController. Do you know this issue?

ScaledObjectController
/__w/keda/keda/controllers/keda/scaledobject_controller_test.go:50
  functional tests
  /__w/keda/keda/controllers/keda/scaledobject_controller_test.go:204
    deploys ScaledObject and creates HPA, when IdleReplicaCount, MinReplicaCount and MaxReplicaCount is defined [It]
    /__w/keda/keda/controllers/keda/scaledobject_controller_test.go:269

    Timed out after 20.001s.
    Expected
        <v1.ConditionStatus>: Unknown
    to equal
        <v1.ConditionStatus>: True

That is just a flaky test :( I rerun the job and it fine now.

[zroubalik]

Looking good, great job @TsuyoshiUshio !

[taylorchu]

is there a reason why we use queuelength instead of maxscale in max/min/avg calculation?

For example consider 2 scalers:

1. queuelength=10, target=4 will produce maxscale=3
2. queuelength=5, target=1 will produce maxscale=5

Right now max in this implementation, will use maxscale=3. To me it makes more sense to use maxscale=5. Similarly min seems more intuitive to pick the one that will produce maxscale=3.

[taylorchu]

@TsuyoshiUshio @zroubalik thanks!