services/horizon/internal: Improve horizon history reaper

[tamirms]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

Close #5339, #5338

This PR implements the following improvements to the horizon history reaper:

• Add metrics for how many rows we delete during a reaping invocation and the duration of the reaping invocation
• Ensure reaping is not run on multiple horizon nodes at the same time
• Ensure reaping is only run on ingesting nodes (otherwise reaping will not work on horizon architectures where requests are served using read only replica dbs)
• Remove statement timeout on reaping db connection to allow long running reaping DB statements to finish
• Improve reaping so it can skip gaps
• Reap history in ascending order so we can minimize gaps in case a reap invocation only partially succeeds

Known limitations

[N/A]

[sreuland]

left minor comments for consideration, your call, nice work!

[tamirms]

@sreuland @urvisavla could you take another look at the PR? I believe I have addressed all code review feedback. thanks!

[sreuland]

@sreuland @urvisavla could you take another look at the PR? I believe I have addressed all code review feedback. thanks!

yes, lgtm, approved.