refactor: remove memory storage and indexes

[glevco]

Depends on #1218

Motivation

The exclusively in-memory storage and indexes haven't been very useful for a while, and generate maintenance overhead. Its implementation has to be fully compatible with its RocksDB counterpart, which is not true for some cases. For this and for simplicity, this PR completely removes support for in-memory storage/indexes, and updates all tests to run using RocksDB instead. This also allows us to have better guarantees that we're testing the same behavior that we observe in production.

A new CLI option --temp-data is provided to be used instead of --memory-storage. It requires no arguments and creates a temporary directory that is automatically cleaned up when the full node exits.

Acceptance Criteria

• Changes in CLI args:
  • Deprecate --memory-storage and --memory-indexes.
  • Remove --rocksdb-storage and --x-rocksdb-indexes (because they're now the only possibility).
• Remove all classes that were exclusively being used by memory storage or indexes.
• Update Builder and CLIBuilder accordingly.
• Refactor settings management on indexes so they all use injected settings.
• Remove TimestampIndex.get_hashes_and_next_idx() as it's unused (this was a Sync-v1 only method).
• All tests are changed to use RocksDB instead of memory.
  • This surfaced a few bugs/incompatible implementations that were also fixed in this PR. See the comments in the respective test files.

Checklist

• If you are requesting a merge into master, confirm this code is production-ready and can be included in future releases as soon as it gets merged

[codecov]

Codecov Report

Attention: Patch coverage is 88.46154% with 15 lines in your changes missing coverage. Please review.

Project coverage is 84.14%. Comparing base (6f3124a) to head (80c5ae3).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
hathor/builder/builder.py	63.15%	7 Missing and 7 partials ⚠️
hathor/indexes/rocksdb_mempool_tips_index.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1219      +/-   ##
==========================================
- Coverage   84.36%   84.14%   -0.23%     
==========================================
  Files         320      311       -9     
  Lines       24471    23868     -603     
  Branches     3748     3651      -97     
==========================================
- Hits        20646    20083     -563     
+ Misses       3098     3071      -27     
+ Partials      727      714      -13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jansegre]

The proposal looks good to me. I'm only a bit worried about deploys that still use --memory-indexes, but I guess they can wait to update when it works well with rocksdb indexes, which we might have to prioritize for the next releases.

[github-actions]

Bencher Report

Branch	chore/remove-memory
Testbed	ubuntu-22.04

⚠️ WARNING: No Threshold found!

Without a Threshold, no Alerts will ever be generated.

• Latency (nanoseconds (ns))

Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds flag.

Click to view all benchmark results

Benchmark	Latency	minutes (m)
sync-v2 (up to 20000 blocks)	📈 view plot ⚠️ NO THRESHOLD	1.64 m

🐰 View full continuous benchmarking report in Bencher