feat(metadata): create basic static metadata structures [part 1/12]

[glevco]

Motivation

This PR is the first one in a new series of PRs that will lead up to the implementation of the Multiprocess Verification project.

In order to implement that project, we need all verification dependencies to be pre-fetched before the verification is sent to another process, for it to be a pure function without side effects (access to the storage). Another complication factor is that we need Sync-V2 to work asynchronously, meaning it can keep retrieving new vertices before their dependencies have been completely verified, through the consensus, and saved.

Considering that verification dependencies include some metadata attributes, those two requirements impose that we must be able to calculate/retrieve those attributes before verification, and that we must do that without access to a storage (as not-yet-verified vertices will only exist in Sync-V2 memory). It is not trivial to implement this with the current metadata structure, as it heavily depends on access to a storage.

We've discussed multiple times that there are different types of metadata attributes mixed together in the current TransactionMetadata class, and that they should be separated. Not coincidentally, all metadata attributes that are used in verification are static attributes, meaning that they can be calculated with only the vertex itself and its dependencies, and once set, their values never change.

Therefore, this PR introduces the concept of static metadata dataclasses, laying the necessary structure to move those static attributes from existing metadata to static metadata. This will allow us to easily calculate all verification dependencies with optional access to a storage, laying a fundamental change that is necessary for Multiprocess Verification. Also not coincidentally, this solves many other problems that we've faced in the past.

Acceptance Criteria

• Introduce Static Metadata dataclasses for vertices, blocks, and transactions.
• Change BaseTransaction to GenericVertex, accepting a generic static metadata type.
  • Introduce the static_metadata property and initialization methods.
  • Introduce Vertex and BaseTransaction type aliases for easiness of use and backwards compatibility.
  • Update Block and Transaction accordingly.
• Introduce _save_static_metadata() and _get_static_metadata() methods in TransactionStorage and its subclasses.
• Miscellaneous smaller changes:
  • Add TransacionStorage to the VerificationService (it's not used in this PR, but will be used in the next one).
  • Change TransactionStorage.get_transaction_by_height() to get_block_by_height().
  • Improve and update tests.

Checklist

• If you are requesting a merge into master, confirm this code is production-ready and can be included in future releases as soon as it gets merged

[codecov]

Codecov Report

Attention: Patch coverage is 84.76190% with 16 lines in your changes missing coverage. Please review.

Project coverage is 84.79%. Comparing base (5972abc) to head (ae9fe2d).
Report is 1 commits behind head on master.

Files	Patch %	Lines
hathor/transaction/static_metadata.py	72.00%	7 Missing ⚠️
hathor/transaction/storage/rocksdb_storage.py	75.00%	3 Missing ⚠️
hathor/transaction/base_transaction.py	90.47%	2 Missing ⚠️
hathor/transaction/storage/cache_storage.py	75.00%	2 Missing ⚠️
hathor/transaction/storage/memory_storage.py	81.81%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1013      +/-   ##
==========================================
- Coverage   84.85%   84.79%   -0.06%     
==========================================
  Files         314      315       +1     
  Lines       23971    24059      +88     
  Branches     3627     3644      +17     
==========================================
+ Hits        20341    20402      +61     
- Misses       2916     2938      +22     
- Partials      714      719       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Bencher

Report	Wed, August 14, 2024 at 16:44:50 UTC
Project	hathor-core
Branch	feat/metadata/create-static-metadata-1
Testbed	ubuntu-22.04

Click to view all benchmark results

Benchmark	Latency	Latency Results nanoseconds (ns) | (Δ%)	Latency Lower Boundary nanoseconds (ns) | (%)	Latency Upper Boundary nanoseconds (ns) | (%)
sync-v2 (up to 20000 blocks)	✅ (view plot)	103,932,344,720.04 (-10.82%)	93,228,797,009.22 (89.70%)	139,843,195,513.82 (74.32%)

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help