[JDBC] Add retries with delay

[singhpk234]

About the change

This change attempts to add retries in the JDBC persistence layer, these retries with jitter are tunable in the following ways :
a. max_retries : total number of retries we expect the persistence to do before giving up.
b. max_duaration_in_ms : time in ms since the first attempt this retries should be done. For ex on configured 500 ms the total time spent in retrying should not exceed 500ms (optimistically)
c. initial_delay_in_ms : intial delay before the first attempt

---

As a result of which we see the following improvements, with the setup of getting-started guide

Make 100% of the call pass

========================================================================================================================
---- Global Information -------------------------------------------------------------|---Total---|-----OK----|----KO----
> request count                                                                      |    30,222 |    30,222 |         0
> min response time (ms)                                                             |         1 |         1 |         -
> max response time (ms)                                                             |       786 |       786 |         -
> mean response time (ms)                                                            |        17 |        17 |         -
> response time std deviation (ms)                                                   |        24 |        24 |         -
> response time 25th percentile (ms)                                                 |         5 |         5 |         -
> response time 50th percentile (ms)                                                 |        11 |        11 |         -
> response time 75th percentile (ms)                                                 |        23 |        23 |         -
> response time 99th percentile (ms)                                                 |        96 |        96 |         -
> mean throughput (rps)                                                              |     83.72 |     83.72 |         -
---- Response Time Distribution ----------------------------------------------------------------------------------------
> OK: t < 800 ms                                                                                         30,222   (100%)
> OK: 800 ms <= t < 1200 ms                                                                                   0     (0%)
> OK: t >= 1200 ms                                                                                            0     (0%)
> KO                                                                                                          0     (0%)
========================================================================================================================

Increased the TPS to 125

========================================================================================================================
2025-05-05 21:01:49 GMT                                                                             360s elapsed
---- Requests -----------------------------------------------------------------------|---Total---|-----OK----|----KO----
> Global                                                                             |    37,305 |    37,305 |         0
> Authenticate                                                                       |         6 |         6 |         0
> Read / Check Namespace Exists                                                      |     2,059 |     2,059 |         0
> Read / Fetch all Views under parent namespace                                      |     2,095 |     2,095 |         0
> Write / Update Namespace Properties                                                |     6,199 |     6,199 |         0
> Read / Fetch all Tables under parent namespace                                     |     2,098 |     2,098 |         0
> Read / Check View Exists                                                           |     2,036 |     2,036 |         0
> Read / Fetch Namespace                                                             |     2,082 |     2,082 |         0
> Read / Check Table Exists                                                          |     2,111 |     2,111 |         0
> Read / Fetch Table                                                                 |     2,085 |     2,085 |         0
> Write / Update View metadata                                                       |     6,182 |     6,182 |         0
> Write / Update table metadata                                                      |     6,166 |     6,166 |         0
> Read / Fetch all Namespaces under specific parent                                  |     2,097 |     2,097 |         0
> Read / Fetch View                                                                  |     2,089 |     2,089 |         0

---- Read and write entities using the Iceberg REST API ----------------------------------------------------------------
[################################################################################################################]  100%
          waiting:         0 / active:         0  / done:    37,299
---- Stop refreshing the authentication token --------------------------------------------------------------------------
[################################################################################################################]  100%
          waiting:         0 / active:         0  / done:         1
---- Authenticate every minute using the Iceberg REST API --------------------------------------------------------------
[################################################################################################################]  100%
          waiting:         0 / active:         0  / done:         1
---- Wait for the authentication token to be available -----------------------------------------------------------------
[################################################################################################################]  100%
          waiting:         0 / active:         0  / done:         1
========================================================================================================================

Parsing log file(s)...
Parsing log file(s) done in 0s.
Generating reports...

========================================================================================================================
---- Global Information -------------------------------------------------------------|---Total---|-----OK----|----KO----
> request count                                                                      |    37,305 |    37,305 |         0
> min response time (ms)                                                             |         1 |         1 |         -
> max response time (ms)                                                             |     1,166 |     1,166 |         -
> mean response time (ms)                                                            |        24 |        24 |         -
> response time std deviation (ms)                                                   |        44 |        44 |         -
> response time 25th percentile (ms)                                                 |         7 |         7 |         -
> response time 50th percentile (ms)                                                 |        14 |        14 |         -
> response time 75th percentile (ms)                                                 |        27 |        27 |         -
> response time 99th percentile (ms)                                                 |       197 |       197 |         -
> mean throughput (rps)                                                              |    103.34 |    103.34 |         -
---- Response Time Distribution ----------------------------------------------------------------------------------------
> OK: t < 800 ms                                                                                         37,296 (99.98%)
> OK: 800 ms <= t < 1200 ms                                                                                   9  (0.02%)
> OK: t >= 1200 ms                                                                                            0     (0%)
> KO                                                                                                          0     (0%)
========================================================================================================================

[singhpk234]

cc @dimas-b @flyrain

[pingtimeout]

@singhpk234 I think the code should be implemented using SmallRye @Retry annotation instead of a custom logic. This is a very common feature and I don't think we should reimplement it. You could benefit from the randomized delay strategy or even an exponential backoff (which could be better).

That being said, it would require some logic around the driver execute(...) method. Something like:

• Create a new exception class called RetryableSQLException
• if the query results in a SQLException, and the error code is considered retryable, wrap it in RetryableSQLException
• otherwise, re-throw the original exception
• Use the @Retry annotation with the retryOn attribute set to RetryableSQLException

Benefits from doing so include less code to maintain, but also metrics on the amount of retries.

Note that I am only -0.9 here. If you feel strongly against using @Retry, feel free to ignore.

[singhpk234]

Thank you @pingtimeout, Really appreciate your inputs.

are you suggesting using this module from quarkus https://quarkus.io/guides/smallrye-fault-tolerance
if thats the case the jdbc module was supposed to be independent of any quarkus code and Honestly i didn't knew about this functionality in first place either (thank you for pointing this out), but if i re-evaluate only because of this IMHO we should not use it.

This is a very common feature and I don't think we should reimplement it

I agree with this in principal, i did evaluate options like FailSafe lib also but we don't want very sophisticated retry logic, its simple one we need with simple knobs if we want like Adaptive Retries or like callbacks i would considered using lib, but our use-case it seemed like an over kill, as bringing new lib will bring lib / notice, cve etc, hence refrained from it only for this use case.

That being said, if you strongly feel about this i am happy to re evaluate and find the best path forward !

[pingtimeout]

@singhpk234 no problem, that was just a suggestion, not a requirement :-). That being said, could you add a javadoc or a TODO around the withRetries method and mention the library? That way, in the future, if somebody needs to improve the retry logic, they know about the library and can consider refactoring the code.

[snazy]

About the change

[1] Add retries (make this configurable) [2] add random delays

Make 100% of the call pass

...

Could you add more details what the description represents and a summary of the actual changes. Two Gatling summaries w/o context do not tell anything.

[snazy]

Retry-loops, especially fair retries, are very tricky. What I'm missing in this PR are concurrency-tests and specific tests vetting the retry loop itself.

[snazy]

The tests look appropriate now from a brief look.

[snazy]

There is

  id("polaris-quarkus")

for this?

[dimas-b]

LGTM 👍 Please allow one more day for other people to comment before merging.

[singhpk234]

I re-ran polaris benchmarks (with the setting @pingtimeout used https://docs.google.com/document/d/1RLYaAtNUkgNW3Ef7-BWfF_8RkSK7B7oR/edit#bookmark=id.von5ayuoga6), happy to share that we now have 100% success with this.

[dimas-b]

nit: Don't defaults work for "getting started"?

[singhpk234]

They do ! these are just for demo purpose !