fix(nats-jetstream): correctly count messages that should be redelivered (waiting for ack) towards keda value

[toniopelo]

Until now, the keda nats jetstream scaler did only use the num_pending value returned by the nats monitoring endpoint for a specifc consumer. The problem was that messages that should be re-delivered (retried) are not returned as part of the num_pending counter but they are in a separate counter called num_ack_pending.
This PR use the sum of these two counter to determine the value that should be used by keda instead of only the num_pending value.

This fixes two problems:

• Keda would never scale up a deployment/job based on a consumer if this consumer only has messages that are waiting for a retry.
• Keda would scale down deployment/job too fast because when a consumer pulls a message from nats, it decrements immediatly the num_pending counter and increment the num_ack_pending counter. Keda would then think that there is no work to be done and scale down the deployment/job that are still processing the messages (after a cooldown time if setup).

Checklist

• When introducing a new scaler, I agree with the scaling governance policy
• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• Tests have been added
• A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
• A PR is opened to update the documentation on (repo) (if applicable)
• Changelog has been updated and is aligned with our changelog requirements

Fixes #3787

Relates to #

[toniopelo]

@JorTurFer That was a pretty straightforward fix after all :).
I didn't add tests because this specific line was not tested before (so it seems) and I don't have the golang knowledge to setup tests, I hope this is not a no-go. Else, we would have to find somebody to add tests on this PR or at least provide me with some guidance on how to do it :). But as it's a single line change it should leave the code coverage untouched.

[JorTurFer]

/run-e2e nats*
Update: You can check the progress here

[toniopelo]

@JorTurFer e2e passed, I checked the last point in the list to make the PR checks happy and updated the branch.
Do you think this can be merged an released ?

[zroubalik]

@toniopelo could you please rebase this PR? I think we can merge it then. Thanks!

[toniopelo]

@zroubalik Nice! Just rebased it :)

[JorTurFer]

/run-e2e nats*
Update: You can check the progress here

[toniopelo]

When can I expect this to be released @JorTurFer ?
The scaler is unusable in production right now :/

[JorTurFer]

Hi @toniopelo ,
You can see the expected release dates in the roadmap.md file. Due to KubeCon we delayed next release to December, the only suggestion I can give if you need this fix immediately, is to use main tag directly. main tag is generated on every commit, so it has these changes, but it could be not stable, so if you use it, I'd suggest to pull from main and push to another registry in order to freeze the version and reduce the chance of having errors.

[toniopelo]

Hi @JorTurFer, thanks for the information, that's crystal clear!
I'll see what I do, thanks for everything :)

[JorTurFer]

You're welcome, happy to help