Skip to content

refactor: improve loading logs#1549

Merged
glevco merged 1 commit intomasterfrom
refactor/improve-loading
Jan 20, 2026
Merged

refactor: improve loading logs#1549
glevco merged 1 commit intomasterfrom
refactor/improve-loading

Conversation

@glevco
Copy link
Contributor

@glevco glevco commented Jan 9, 2026

Motivation

PR #1200 removed sync-v1 indexes, and with that the indexes that require initialization changed. Before, we had indexes interested in blocks, but now we only load the mempool index which is interested in transactions only. This means the Scope iterator skipped a lot of blocks before yielding to the progress log iterator, causing the log to appear stuck for several minutes.

This PR fixes this and also makes some other small improvements on loading logs.

Before:

2026-01-09 18:21:25 [info     ] [hathor.indexes.manager] there are indexes that need initialization indexes_to_init=[<hathor.indexes.memory_mempool_tips_index.MemoryMempoolTipsIndex object at 0x7f0ddd9f6120>]
2026-01-09 18:21:25 [info     ] [hathor.indexes.base_index] loaded info-index              block_count=6196394 first_timestamp=1578075305 latest_timestamp=1767982595 tx_count=1410269

< ~25 minutes with no logs >

2026-01-09 18:47:17 [warning  ] [hathor.indexes.manager] iterator was slow to yield     took_sec=~1551s
2026-01-09 18:47:17 [info     ] [hathor.indexes.manager] loading...  0%                 dt=~1551s height=0 latest_ts=2022-09-07 05:31:04 progress=0.0 remaining_time=? total=0 tx_new=0 tx_rate=0.0
2026-01-09 18:47:47 [info     ] [hathor.indexes.manager] loading...  0%                 dt=~30s height=0 latest_ts=2026-01-07 12:10:48 progress=0.005423797531190747 remaining_time=~289993s total=41257 tx_new=41257 tx_rate=1375.2048410493485

After:

2026-01-09 20:42:27 [info     ] [hathor.indexes.manager] there are indexes that need initialization indexes_to_init=['MemoryMempoolTipsIndex']
2026-01-09 20:42:27 [info     ] [hathor.indexes.base_index] loaded info-index              block_count=6155424 first_timestamp=1578075305 latest_timestamp=1766752000 tx_count=1408999
2026-01-09 20:42:27 [info     ] [hathor.indexes.manager] initializing indexes...
2026-01-09 20:42:27 [info     ] [hathor.indexes.manager] loading...  0%                 dt=<1ms remaining_time=? total=0 tx_new=0 tx_rate=0.0
2026-01-09 20:42:37 [info     ] [hathor.indexes.manager] loading...  2%                 dt=~10s remaining_time=~483s total=153400 tx_new=153400 tx_rate=15339.935631068443

Acceptance Criteria

  • Refactor Scope.get_iterator so it yields all txs, not only those that match the scope. It was only used in one place.
  • In loading logs:
    • Add names to list of indexes that require initialization.
    • Change progress dt from 30s to 10s.
    • Remove block height and tx timestamp (it only made sense when the load was topological, which it's not anymore).
    • Add initial log on 0%.
    • Remove progress float (this information is already logged as a percentage).

Checklist

  • If you are requesting a merge into master, confirm this code is production-ready and can be included in future releases as soon as it gets merged

@glevco glevco self-assigned this Jan 9, 2026
@glevco glevco requested a review from jansegre as a code owner January 9, 2026 20:41
@glevco glevco requested a review from msbrogli as a code owner January 9, 2026 20:41
@glevco glevco force-pushed the refactor/improve-loading branch from 3e2983d to 8afc0b1 Compare January 9, 2026 20:45
@github-actions
Copy link

github-actions bot commented Jan 9, 2026

🐰 Bencher Report

Branchrefactor/improve-loading
Testbedubuntu-22.04

🚨 1 Alert

BenchmarkMeasure
Units
ViewBenchmark Result
(Result Δ%)
Lower Boundary
(Limit %)
sync-v2 (up to 20000 blocks)Latency
minutes (m)
📈 plot
🚷 threshold
🚨 alert (🔔)
1.52 m
(-11.64%)Baseline: 1.72 m
1.55 m
(101.85%)

Click to view all benchmark results
BenchmarkLatencyBenchmark Result
minutes (m)
(Result Δ%)
Lower Boundary
minutes (m)
(Limit %)
Upper Boundary
minutes (m)
(Limit %)
sync-v2 (up to 20000 blocks)📈 view plot
🚷 view threshold
🚨 view alert (🔔)
1.52 m
(-11.64%)Baseline: 1.72 m
1.55 m
(101.85%)

2.06 m
(73.64%)
🐰 View full continuous benchmarking report in Bencher

@glevco glevco moved this from Todo to In Progress (Done) in Hathor Network Jan 9, 2026
@codecov
Copy link

codecov bot commented Jan 9, 2026

Codecov Report

❌ Patch coverage is 92.30769% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.30%. Comparing base (b164a87) to head (1901010).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
hathor/indexes/scope.py 66.66% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1549      +/-   ##
==========================================
+ Coverage   86.25%   86.30%   +0.04%     
==========================================
  Files         438      438              
  Lines       33776    33780       +4     
  Branches     5270     5269       -1     
==========================================
+ Hits        29134    29153      +19     
+ Misses       3627     3612      -15     
  Partials     1015     1015              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-project-automation github-project-automation bot moved this from In Progress (Done) to In Review (WIP) in Hathor Network Jan 12, 2026
@glevco glevco force-pushed the refactor/improve-loading branch from 9f9133d to ba0e523 Compare January 12, 2026 15:02
@glevco glevco force-pushed the refactor/improve-loading branch from ba0e523 to dbf1c5a Compare January 12, 2026 16:19
@glevco glevco force-pushed the refactor/improve-loading branch from dbf1c5a to 1681d91 Compare January 14, 2026 14:30
@glevco glevco moved this from In Review (WIP) to In Review (Done) in Hathor Network Jan 14, 2026
@jansegre jansegre mentioned this pull request Jan 14, 2026
2 tasks
@glevco glevco force-pushed the refactor/improve-loading branch from 1681d91 to 1901010 Compare January 20, 2026 16:14
@glevco glevco merged commit 932ce5f into master Jan 20, 2026
10 of 12 checks passed
@glevco glevco deleted the refactor/improve-loading branch January 20, 2026 16:43
@github-project-automation github-project-automation bot moved this from In Review (Done) to Waiting to be deployed in Hathor Network Jan 20, 2026
@jansegre jansegre moved this from Waiting to be deployed to Done in Hathor Network Jan 21, 2026
@jansegre jansegre mentioned this pull request Jan 21, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants