feat(indexes): remove sync-v1 indexes by jansegre · Pull Request #1200 · HathorNetwork/hathor-core

jansegre · 2025-01-08T16:38:05Z

Motivation

After removing sync-v1 (#1197) and removing block ties (#673) there is no need to have the sync-v1 indexes (tx-tips, block-tips and all-tips). There should be a slight runtime benefit (no measurement has been made), but more noticeably, there will be a substantial improvement in startup time.

Acceptance Criteria

Remove tx_tips, block_tips and all_tips indexes
Refactor relevant methods
Refactor tests

Checklist

If you are requesting a merge into master, confirm this code is production-ready and can be included in future releases as soon as it gets merged

github-actions · 2025-01-08T16:45:05Z

Bencher Report

Branch	feat/remove-sync-v1-indexes
Testbed	ubuntu-22.04

Click to view all benchmark results

Benchmark	Latency	Benchmark Result minutes (m) (Result Δ%)	Lower Boundary minutes (m) (Limit %)	Upper Boundary minutes (m) (Limit %)
sync-v2 (up to 20000 blocks)	📈 view plot 🚷 view threshold	1.68 m (-3.01%) Baseline: 1.73 m	1.56 m (92.79%)	2.08 m (80.82%)

🐰 View full continuous benchmarking report in Bencher

codecov · 2025-01-24T16:35:14Z

Codecov Report

❌ Patch coverage is 97.36842% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.23%. Comparing base (5e3d848) to head (0f8cb8d).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
hathor/transaction/resources/mempool.py	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1200      +/-   ##
==========================================
+ Coverage   86.19%   86.23%   +0.04%     
==========================================
  Files         440      437       -3     
  Lines       34056    33725     -331     
  Branches     5327     5265      -62     
==========================================
- Hits        29353    29082     -271     
+ Misses       3677     3628      -49     
+ Partials     1026     1015      -11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

glevco · 2025-03-20T01:30:27Z

This PR greatly reduces memory usage, see https://github.com/HathorNetwork/on-call-incidents/issues/196.

hathor/indexes/memory_timestamp_index.py

hathor/indexes/mempool_tips_index.py

hathor/p2p/resources/status.py

hathor/reward_lock/reward_lock.py

hathor/manager.py

hathor_tests/poa/test_poa_simulation.py

hathor/transaction/storage/transaction_storage.py

The base branch was changed.

glevco · 2025-12-11T18:59:07Z

hathor/indexes/manager.py

        # We need to check r1 as well to make sure we don't count twice the transactions/blocks that are
        # just changing from voided to executed or vice-versa
-        if r1 and r3:
+        if r1:


Before, this was r1 and r3 which was equivalent to r2 and r4 which is equivalent to self.sorted_all.add_tx(tx) and self.sorted_blocks.add_tx(tx) (in the case of blocks, and analogous for txs).

Then new code is just equivalent to the sorted_all parts and is missing the sorted_blocks parts, which would mean r1 and r2 now. Is this correct?

glevco · 2025-12-11T19:09:59Z

hathor/simulator/fake_connection.py

+        tips1 = {tx.hash for tx in state1.protocol.node.tx_storage.iter_mempool()}
+        tips2 = {tx.hash for tx in state2.protocol.node.tx_storage.iter_mempool()}


iter_mempool_tips instead?

jansegre requested a review from msbrogli as a code owner January 8, 2025 16:38

jansegre self-assigned this Jan 8, 2025

jansegre changed the base branch from master to feat/consensus-change January 24, 2025 16:15

jansegre force-pushed the feat/remove-sync-v1-indexes branch from b6bdf57 to 0e6013a Compare January 24, 2025 16:19

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 0e6013a to 752b56a Compare January 24, 2025 16:38

jansegre force-pushed the feat/consensus-change branch from 0a38b06 to ea7d72c Compare January 24, 2025 18:45

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 752b56a to cbfd6f1 Compare January 24, 2025 18:46

jansegre requested a review from glevco January 27, 2025 18:55

This was referenced Jan 29, 2025

feat(consensus): eliminate block ties #673

Merged

feat(indexes): make sync-v1 indexes optional #540

Closed

jansegre force-pushed the feat/remove-sync-v1-indexes branch from cbfd6f1 to 105ee8d Compare February 26, 2025 15:16

jansegre force-pushed the feat/consensus-change branch 2 times, most recently from eb49f63 to f32a3f9 Compare April 1, 2025 10:05

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 105ee8d to 11d5dd1 Compare April 1, 2025 10:05

jansegre force-pushed the feat/consensus-change branch from f32a3f9 to e11dfd3 Compare July 1, 2025 16:21

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 11d5dd1 to 796f9a2 Compare July 1, 2025 16:33

jansegre force-pushed the feat/consensus-change branch from e11dfd3 to 4b3736f Compare August 7, 2025 17:39

jansegre force-pushed the feat/consensus-change branch from 4b3736f to 5e88bb8 Compare August 21, 2025 22:31

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 796f9a2 to 9692db5 Compare August 22, 2025 17:29

jansegre force-pushed the feat/consensus-change branch from 5e88bb8 to b65c060 Compare September 11, 2025 15:34

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 9692db5 to 86d519d Compare September 11, 2025 15:41

jansegre force-pushed the feat/consensus-change branch from b65c060 to 299608f Compare September 24, 2025 18:07

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 86d519d to 221618e Compare September 24, 2025 19:16

msbrogli force-pushed the feat/consensus-change branch from 299608f to dd0b9fe Compare September 30, 2025 16:54

msbrogli force-pushed the feat/consensus-change branch 3 times, most recently from 9341e49 to dda0697 Compare December 9, 2025 12:24

msbrogli requested changes Dec 11, 2025

View reviewed changes

github-project-automation bot moved this from In Progress (Done) to In Review (WIP) in Hathor Network Dec 11, 2025

msbrogli force-pushed the feat/consensus-change branch from 248b99e to 08a48da Compare December 11, 2025 04:04

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 4e7c26c to 941485d Compare December 11, 2025 12:05

msbrogli previously approved these changes Dec 11, 2025

View reviewed changes

msbrogli force-pushed the feat/consensus-change branch 2 times, most recently from b06697d to 2942f66 Compare December 11, 2025 14:57

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 941485d to 52e64ef Compare December 11, 2025 16:42

msbrogli force-pushed the feat/consensus-change branch from 2942f66 to 5e3d848 Compare December 11, 2025 17:12

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 52e64ef to 342762e Compare December 11, 2025 18:21

jansegre changed the base branch from feat/consensus-change to master December 11, 2025 18:28

jansegre force-pushed the feat/remove-sync-v1-indexes branch from 342762e to a6245b8 Compare December 11, 2025 18:33

msbrogli previously approved these changes Dec 11, 2025

View reviewed changes

glevco reviewed Dec 11, 2025

View reviewed changes

feat(indexes): remove sync-v1 indexes

0f8cb8d

jansegre dismissed msbrogli’s stale review via 0f8cb8d December 11, 2025 19:29

jansegre force-pushed the feat/remove-sync-v1-indexes branch from a6245b8 to 0f8cb8d Compare December 11, 2025 19:29

glevco approved these changes Dec 11, 2025

View reviewed changes

jansegre mentioned this pull request Dec 11, 2025

Better coverage of multiple scenarios for initialziation tests. #1513

Open

msbrogli approved these changes Dec 11, 2025

View reviewed changes

jansegre moved this from In Review (WIP) to In Review (Done) in Hathor Network Dec 11, 2025

jansegre merged commit a579e7f into master Dec 11, 2025
5 of 7 checks passed

jansegre deleted the feat/remove-sync-v1-indexes branch December 11, 2025 19:59

github-project-automation bot moved this from In Review (Done) to Waiting to be deployed in Hathor Network Dec 11, 2025

jansegre mentioned this pull request Dec 11, 2025

Release-candidate v0.68.2-rc.1 #1514

Merged

2 tasks

jansegre moved this from Waiting to be deployed to Done in Hathor Network Dec 12, 2025

jansegre mentioned this pull request Dec 12, 2025

Release v0.68.2 #1515

Merged

2 tasks

glevco mentioned this pull request Jan 9, 2026

refactor: improve loading logs #1549

Merged

1 task

jansegre mentioned this pull request Jan 14, 2026

Release-candidate v0.69.0-rc.1 #1553

Merged

2 tasks

		tips1 = {tx.hash for tx in state1.protocol.node.tx_storage.iter_mempool()}
		tips2 = {tx.hash for tx in state2.protocol.node.tx_storage.iter_mempool()}

Conversation

jansegre commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Acceptance Criteria

Checklist

Uh oh!

github-actions bot commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

codecov bot commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

glevco commented Mar 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

glevco Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

jansegre Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

jansegre Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

glevco Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

jansegre Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

jansegre Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jansegre commented Jan 8, 2025 •

edited

Loading

github-actions bot commented Jan 8, 2025 •

edited

Loading

codecov bot commented Jan 24, 2025 •

edited

Loading