feat(sync-v2): Implement new protocol for synchronizing transactions

[msbrogli]

Motivation

This PR addresses two significant issues in the current implementation:

1. In Sync-v2, a global dependencies index was utilized across all connections.
2. Sync-v2 agents insisted on the fulfillment of all dependencies before progressing with the synchronization.

These combined challenges led to potential deadlocks. Specifically, if one peer introduced a dependency that was unavailable from others, it could cause all agents to endlessly attempt to satisfy this elusive dependency from their peers—a futile endeavor.

To counter this, we've established a new transaction synchronization protocol from blocks. With this enhanced approach, every agent manages its own unique dependency list, allowing for independent synchronization with each peer. Furthermore, it allows the synchronization loop to function even with outstanding dependencies.

This PR changes the behavior of the GET-TRANSACTIONS-BFS message. The new parameters are start_from, first_block_hash, and end_block_hash. The response is a sequence of TRANSACTION messages, each containing one transaction.

New protocol

The new protocol syncs blocks first and keeps track of partial_blocks which are the blocks that cannot be fully validated because of a missing dependency. Then, the partial_blocks list is passed to the transaction streamer. The streamer will then send all transactions confirmed by the first block followed by all transactions confirmed by the second block and so on. To determine the precise order of transactions within each block, the streamer implements a Breadth-First Search (BFS) algorithm, tracing back from the block in question.

To illustrate the ordering, consider partial_blocks enumerated as [b0, b1, b2, ..., bn], where tx_i_j represents the i-th transaction within the j-th block (b_j). The transaction streamer would then relay transactions in the ensuing sequence: [tx_0_0, tx_1_0, tx_2_0, ... tx_N0_0, tx_0_1, tx_1_1, tx_2_1, ..., tx_N1_1, ..., tx_0_n, tx_1_n, tx_2_n, ..., tx_Nn_n]. Here, N0, N1, ..., Nn correspond to the total number of transactions validated by blocks b0, b1, ..., bn respectively.

Acceptance Criteria

1. Remove DepsIndex, MemoryDepsIndex, RocksDBDepsIndex because this index is not used anymore.
2. Remove HathorManager._sync_v2_resume_validations() and HathorManager.sync_v2_step_validations() because they are not used anymore.
3. Add STOP_TRANSACTIONS_STREAMING message to allow clients to request servers to stop streaming.
4. Remove sync_v2.NodeBlockSync.run_sync_transactions() because it is not used anymore.
5. Stop using partially validated vertex. All vertices added to the DAG are immediately fully validated.
6. Modify TransactionsStreamingServer to send transactions according to the new protocol.
7. Change fixed seed in tests/consensus/test_soft_voided.py because this test was failing.

Checklist

• If you are requesting a merge into master, confirm this code is production-ready and can be included in future releases as soon as it gets merged

[codecov]

Codecov Report

Attention: 34 lines in your changes are missing coverage. Please review.

Comparison is base (f528a06) 85.16% compared to head (acb577d) 85.45%.

❗ Current head acb577d differs from pull request most recent head d0a1ff2. Consider uploading reports for the commit d0a1ff2 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #850      +/-   ##
==========================================
+ Coverage   85.16%   85.45%   +0.28%     
==========================================
  Files         285      282       -3     
  Lines       22652    22242     -410     
  Branches     3443     3361      -82     
==========================================
- Hits        19292    19007     -285     
+ Misses       2677     2570     -107     
+ Partials      683      665      -18     

Files	Coverage Δ
hathor/indexes/manager.py	91.90% <ø> (+0.16%)	⬆️
hathor/manager.py	80.51% <100.00%> (-1.70%)	⬇️
hathor/p2p/manager.py	72.58% <ø> (-0.07%)	⬇️
hathor/p2p/messages.py	100.00% <100.00%> (ø)
hathor/p2p/sync_v2/exception.py	100.00% <100.00%> (ø)
hathor/p2p/sync_v2/mempool.py	93.42% <ø> (ø)
hathor/p2p/sync_v2/streamers.py	90.86% <96.87%> (+0.80%)	⬆️
hathor/p2p/sync_v2/blockchain_streaming_client.py	81.01% <68.75%> (+4.38%)	⬆️
hathor/p2p/sync_v2/transaction_streaming_client.py	90.09% <91.30%> (+14.68%)	⬆️
hathor/p2p/sync_v2/agent.py	79.96% <78.12%> (+2.33%)	⬆️

... and 9 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.