Add retry loop to block handler.

[winder]

Summary

If the Postgres database handles are exhausted due to many pending queries, block import will fail. With the existing code that would cause Indexer to terminate.

This PR adds a retry loop to ensure the process eventually recover after this sort of transient failure.

Test Plan

New unit test.

[tolikzinovyev]

How about creating 2 dedicated connections for block import? (AddBlock() needs 2).

[winder]

How about creating 2 dedicated connections for block import? (AddBlock() needs 2).

I think this is still useful as a basic safety net, but your idea is even better.

Another possible issue is that Aurora replication (presumably) needs its own connection handle(s). One suspicion is this is why our mobile wallet deployments fall behind from time to time. To resolve that one we might need to make the pgx pool size configurable. Seems like ConnectConfig has plenty of options for doing that.

[tolikzinovyev]

I'm just not sure ignoring an error is a good solution.

Not following about aurora replication.

[winder]

I'm just not sure ignoring an error is a good solution.

It doesn't ignore the error. An error is logged, it's up to the service runner to decide whether or not to ignore errors.

Not following about aurora replication.

I can follow up on this separately.

[codecov-commenter]

Codecov Report

Merging #823 (3e4bbe0) into develop (59e181f) will decrease coverage by 1.67%.
The diff coverage is 70.00%.

@@             Coverage Diff             @@
##           develop     #823      +/-   ##
===========================================
- Coverage    59.17%   57.49%   -1.68%     
===========================================
  Files           32       35       +3     
  Lines         4108     4327     +219     
===========================================
+ Hits          2431     2488      +57     
- Misses        1379     1539     +160     
- Partials       298      300       +2     

Impacted Files	Coverage Δ
cmd/algorand-indexer/daemon.go	25.43% <70.00%> (ø)
cmd/algorand-indexer/import.go	12.50% <0.00%> (ø)
cmd/algorand-indexer/main.go	29.21% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59e181f...3e4bbe0. Read the comment docs.

[winder]

@tolikzinovyev I created some followup tickets to reduce the chance that this will happen:
#826
#827

[shiqizng]

looks fine.

[tolikzinovyev]

Looks good!