Retry with backoff on cluster connection failures

[walles]

Before this change, if there were connection failures to the cluster, we did all our retries without any backoff.

With this change in place:

• We first do the previous no-backoff tactic for one third of our maxAttempts (see the shouldBackOff() method)
• Then we start backing off as determined by the getBackoffSleepMillis() method

Additionally, this change adds unit tests for the retries / backoff logic.

This change is based on the changes in #2355 (approved, not yet merged, currently waiting for more reviewers).

[walles]

✅ 👀 Ready for review!

[sazzad16]

This PR breaks backward compatibility. Breaking backward compatibility means it won't be released until next major release. As of this moment, next major release for Jedis is 4.0.0 which, you can imagine, is a long away.

Try to find a backward compatible solution. Don't make the code too ugly for that purpose though :)

[walles]

Thank you for your short turnaround time in reviewing, I really appreciate that @sazzad16!

[walles]

Try to find a backward compatible solution. Don't make the code too ugly for that purpose though :)

Another constructor is needed either way (I think).

But if #2364 would get merged before this PR, that constructor could be made private and wouldn't have to clutter the public API.

[walles]

public makes these reachable from JedisClusterCommand.java for its default timeout.

[yangbodong22011]

I disagree. Firstly, it doesn't suit there. Secondly, when we'd try to improve this (targeting Jedis 4.0.0), this would mess the config interface and/or could be bottlenecked by it.

@sazzad16 Okay, If we have an improvement plan, then I agree to continue, but I still think the default value of maxTotalRetriesDuration should be: maxAttempts * soTimeout, not equal to soTimeout.

It's just that those commands were implemented & merged after this PR is crafted and simple git merge doesn't add those. We'll always have time to add those.

This is the responsibility of this PR, and maxTotalRetriesDuration should be added to the new command before merged.

[sazzad16]

@yangbodong22011

the default value of maxTotalRetriesDuration should be: maxAttempts * soTimeout

agreed

maxTotalRetriesDuration should be added to the new command before merged

We can do this after the PR is approved.

[sazzad16]

@gkorland @yangbodong22011 Please check #2490. Hopefully that PR addresses your concerns.

[walles]

🥳

[nitinware]

@walles we are seeing similar for jedis version:5.1.0, I see this PR has fixed tis issue, which jedis version we need to use to address the error.

Error:
org.springframework.dao.InvalidDataAccessApiUsageException: No more cluster attempts left.
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67)
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:42)
	at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:40)

[ggivo]

Hi @nitinware

Took a brief look at the history of commits and it shows that this PR is already part of 5.1.0.

"No more cluster attempts left." is a pretty generic error thrown when there is a persisting error even after retries are exhausted. Looking at Jedis code the actual cause is stored as a suppressed exception inside JedisClusterOperationException here.

I see you are using spring framework and it wraps the original Jedis exception, probably somewhere down the stack there should be JedisClusterOperationException with the actual error causing the failure inside suppressed.

Hope it helps