Support Pod conditions for calculating PendingJobCount in accurate scaling startegy

[yaronya]

Proposal

Currently KEDA looks for the Pod phase (when scaling strategy is set to accurate) to determine whether a pod is pending or not in order to avoid excess triggers when pod is waiting to start or create. This feature was introduced in #1391.

I believe this could be enhanced by not only looking at the Pod's phase, but also its conditions. The conditions should be configurable like:

scalingStategy:
  strategy: "accurate"
  customPendingPodConditions: 
    - "Ready"
    - "PodScheduled"
    - "SomeOtherCustomConditionDefinedByReadinessGate"

Another refactor can be to not look at the Pod's Running phase at all since the native PodScheduled: true condition means the Pod phase is Running so PodScheduled can always be set in customPendingPodConditions (but I'd leave this to whoever implements it)

Use-Case

SQS scaler with a polling interval of 5s (needs to react ASAP to new messages).
Sometimes the pod is already marked with a Running phase but haven't deleted the message from the queue just yet so KEDA triggers another redundant job in the meantime 😢

Using Readiness Probes (or custom readinessGate) would allow me to mark the pod as "not pending" only after assuring the message is no longer in the queue, helping me to avoid those redundant KEDA triggers 💯

Anything else?

Thank you for this great solution. Go KEDA! 🥇

[zroubalik]

@yaronya this is a great proposal, it sounds reasonable for me. Are you willing to contribute this one?

[yaronya]

Sure thing.
I'm a little busy lately so I'll try to get to it in the forthcoming week or two.