Add voting statistics during maintenance time

[syalon]

User Story

Many times we need to query the actual voting power of users. Currently, we only rely on some centralized timed tasks for voting power statistics, which is inconvenient and easy to make mistakes. This PR is calculated during the maintenance period. And provide some query API.

Performance data

After testing, the current voting statistics can be completed within 1s and can be received.

More

Do we need to provide some apis to query the data of all followers.

Impacts

Describe which portion(s) of BitShares Core may be impacted by your request. Please tick at least one box.

• API (the application programming interface)
• UI (the user interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

[abitmore]

Thanks.

I think the new code would more or less impact performance due to

• sorting due to new indices, and
• updating more data.

It would be a surprise if it does not. A full replay would show the difference more clearly.

BTW please bump DB_VERSION since the database scheme is updated.

[syalon]

added a new commit

[abitmore]

So the only thing missing is a performance report. See #1401 (comment) for an example. Thanks.

[syalon]

develop#a758bb0 + this pr replay log

2020-08-27T03:10:13        th_a:?unnamed?              reindex ] Replaying blocks, starting at 1...     db_management.cpp:70
2020-08-27T03:10:13        th_a:?unnamed?              reindex ]    [by size: 0.00185%   1568884 of 84781831723]   [by num: 0.01981%   10000 of 50479432]     db_management.cpp:141
2020-08-27T03:10:13        th_a:?unnamed?              reindex ]    [by size: 0.00371%   3146036 of 84781831723]   [by num: 0.03962%   20000 of 50479432]     db_management.cpp:141

...

2020-08-27T05:43:39        th_a:?unnamed?              reindex ]    [by size: 99.86162%   84664509470 of 84781831723]   [by num: 99.44644%   50200000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:40        th_a:?unnamed?              reindex ]    [by size: 99.86715%   84669203105 of 84781831723]   [by num: 99.46625%   50210000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:40        th_a:?unnamed?              reindex ]    [by size: 99.87346%   84674546459 of 84781831723]   [by num: 99.48606%   50220000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:41        th_a:?unnamed?              reindex ]    [by size: 99.87737%   84677863511 of 84781831723]   [by num: 99.50587%   50230000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:41        th_a:?unnamed?              reindex ]    [by size: 99.88066%   84680653285 of 84781831723]   [by num: 99.52568%   50240000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:42        th_a:?unnamed?              reindex ]    [by size: 99.88389%   84683395361 of 84781831723]   [by num: 99.54549%   50250000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:42        th_a:?unnamed?              reindex ]    [by size: 99.88722%   84686215567 of 84781831723]   [by num: 99.56530%   50260000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:43        th_a:?unnamed?              reindex ]    [by size: 99.89192%   84690199477 of 84781831723]   [by num: 99.58511%   50270000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:43        th_a:?unnamed?              reindex ]    [by size: 99.89817%   84695498328 of 84781831723]   [by num: 99.60492%   50280000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:44        th_a:?unnamed?              reindex ]    [by size: 99.90269%   84699330527 of 84781831723]   [by num: 99.62473%   50290000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:44        th_a:?unnamed?              reindex ]    [by size: 99.90754%   84703444041 of 84781831723]   [by num: 99.64454%   50300000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:45        th_a:?unnamed?              reindex ]    [by size: 99.91335%   84708371898 of 84781831723]   [by num: 99.66435%   50310000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:46        th_a:?unnamed?              reindex ]    [by size: 99.91886%   84713036778 of 84781831723]   [by num: 99.68416%   50320000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:46        th_a:?unnamed?              reindex ]    [by size: 99.92300%   84716547411 of 84781831723]   [by num: 99.70397%   50330000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:47        th_a:?unnamed?              reindex ]    [by size: 99.92639%   84719420735 of 84781831723]   [by num: 99.72378%   50340000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:47        th_a:?unnamed?              reindex ]    [by size: 99.92997%   84722461062 of 84781831723]   [by num: 99.74359%   50350000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:48        th_a:?unnamed?              reindex ]    [by size: 99.93416%   84726011214 of 84781831723]   [by num: 99.76340%   50360000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:49        th_a:?unnamed?              reindex ]    [by size: 99.94252%   84733102294 of 84781831723]   [by num: 99.78321%   50370000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:49        th_a:?unnamed?              reindex ]    [by size: 99.94757%   84737384489 of 84781831723]   [by num: 99.80302%   50380000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:50        th_a:?unnamed?              reindex ]    [by size: 99.95157%   84740774032 of 84781831723]   [by num: 99.82283%   50390000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:51        th_a:?unnamed?              reindex ]    [by size: 99.95715%   84745499073 of 84781831723]   [by num: 99.84264%   50400000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:51        th_a:?unnamed?              reindex ]    [by size: 99.96206%   84749664671 of 84781831723]   [by num: 99.86245%   50410000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:52        th_a:?unnamed?              reindex ]    [by size: 99.96606%   84753060055 of 84781831723]   [by num: 99.88226%   50420000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:52        th_a:?unnamed?              reindex ]    [by size: 99.96920%   84755722305 of 84781831723]   [by num: 99.90207%   50430000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:53        th_a:?unnamed?              reindex ]    [by size: 99.97239%   84758421664 of 84781831723]   [by num: 99.92189%   50440000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:53        th_a:?unnamed?              reindex ]    [by size: 99.97610%   84761568589 of 84781831723]   [by num: 99.94170%   50450000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:54        th_a:?unnamed?              reindex ]    [by size: 99.98111%   84765817956 of 84781831723]   [by num: 99.96151%   50460000 of 50479432]      db_management.cpp:141
2020-08-27T05:43:55        th_a:?unnamed?              reindex ] Writing database to disk at block 50469432     db_management.cpp:145
2020-08-27T05:43:57        th_a:?unnamed?              reindex ] Done     db_management.cpp:147
2020-08-27T05:43:57        th_a:?unnamed?              reindex ]    [by size: 100.00000%   84782071724 of 84782071724]   [by num: 99.98132%   50470000 of 50479432]     db_management.cpp:141
2020-08-27T05:43:59        th_a:?unnamed?              reindex ] Done reindexing, elapsed time: 9225.84765700000025390 sec      db_management.cpp:162

4.0.0 replay log

2020-08-27T00:18:30        th_a:?unnamed?              reindex ] Replaying blocks, starting at 1...     db_management.cpp:70
2020-08-27T00:18:30        th_a:?unnamed?              reindex ]    [by size: 0.00185%   1568884 of 84775689161]   [by num: 0.01981%   10000 of 50479432]     db_management.cpp:138
2020-08-27T00:18:30        th_a:?unnamed?              reindex ]    [by size: 0.00371%   3146036 of 84775689161]   [by num: 0.03962%   20000 of 50479432]     db_management.cpp:138

...

2020-08-27T00:18:30
2020-08-27T02:47:51        th_a:?unnamed?              reindex ]    [by size: 99.86885%   84664509470 of 84775689161]   [by num: 14.36293%   50200000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:52        th_a:?unnamed?              reindex ]    [by size: 99.87439%   84669203105 of 84775689161]   [by num: 14.38274%   50210000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:53        th_a:?unnamed?              reindex ]    [by size: 99.88069%   84674546459 of 84775689161]   [by num: 14.40255%   50220000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:53        th_a:?unnamed?              reindex ]    [by size: 99.88461%   84677863511 of 84775689161]   [by num: 14.42236%   50230000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:53        th_a:?unnamed?              reindex ]    [by size: 99.88790%   84680653285 of 84775689161]   [by num: 14.44217%   50240000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:54        th_a:?unnamed?              reindex ]    [by size: 99.89113%   84683395361 of 84775689161]   [by num: 14.46198%   50250000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:54        th_a:?unnamed?              reindex ]    [by size: 99.89446%   84686215567 of 84775689161]   [by num: 14.48179%   50260000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:55        th_a:?unnamed?              reindex ]    [by size: 99.89916%   84690199477 of 84775689161]   [by num: 14.50160%   50270000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:55        th_a:?unnamed?              reindex ]    [by size: 99.90541%   84695498328 of 84775689161]   [by num: 14.52141%   50280000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:56        th_a:?unnamed?              reindex ]    [by size: 99.90993%   84699330527 of 84775689161]   [by num: 14.54122%   50290000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:56        th_a:?unnamed?              reindex ]    [by size: 99.91478%   84703444041 of 84775689161]   [by num: 14.56103%   50300000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:57        th_a:?unnamed?              reindex ]    [by size: 99.92059%   84708371898 of 84775689161]   [by num: 14.58084%   50310000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:58        th_a:?unnamed?              reindex ]    [by size: 99.92610%   84713036778 of 84775689161]   [by num: 14.60065%   50320000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:58        th_a:?unnamed?              reindex ]    [by size: 99.93024%   84716547411 of 84775689161]   [by num: 14.62046%   50330000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:59        th_a:?unnamed?              reindex ]    [by size: 99.93363%   84719420735 of 84775689161]   [by num: 14.64027%   50340000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:59        th_a:?unnamed?              reindex ]    [by size: 99.93721%   84722461062 of 84775689161]   [by num: 14.66008%   50350000 of 50479432]      db_management.cpp:138
2020-08-27T02:47:59        th_a:?unnamed?              reindex ]    [by size: 99.94140%   84726011214 of 84775689161]   [by num: 14.67989%   50360000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:00        th_a:?unnamed?              reindex ]    [by size: 99.94977%   84733102294 of 84775689161]   [by num: 14.69970%   50370000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:01        th_a:?unnamed?              reindex ]    [by size: 99.95482%   84737384489 of 84775689161]   [by num: 14.71951%   50380000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:01        th_a:?unnamed?              reindex ]    [by size: 99.95881%   84740774032 of 84775689161]   [by num: 14.73932%   50390000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:02        th_a:?unnamed?              reindex ]    [by size: 99.96439%   84745499073 of 84775689161]   [by num: 14.75913%   50400000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:02        th_a:?unnamed?              reindex ]    [by size: 99.96930%   84749664671 of 84775689161]   [by num: 14.77894%   50410000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:03        th_a:?unnamed?              reindex ]    [by size: 99.97331%   84753060055 of 84775689161]   [by num: 14.79875%   50420000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:03        th_a:?unnamed?              reindex ]    [by size: 99.97645%   84755722305 of 84775689161]   [by num: 14.81856%   50430000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:04        th_a:?unnamed?              reindex ]    [by size: 99.97963%   84758421664 of 84775689161]   [by num: 14.83837%   50440000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:04        th_a:?unnamed?              reindex ]    [by size: 99.98334%   84761568589 of 84775689161]   [by num: 14.85818%   50450000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:05        th_a:?unnamed?              reindex ]    [by size: 99.98836%   84765817956 of 84775689161]   [by num: 14.87799%   50460000 of 50479432]      db_management.cpp:138
2020-08-27T02:48:06        th_a:?unnamed?              reindex ] Writing database to disk at block 50469432     db_management.cpp:142
2020-08-27T02:48:08        th_a:?unnamed?              reindex ] Done     db_management.cpp:144
2020-08-27T02:48:08        th_a:?unnamed?              reindex ]    [by size: 100.00028%   84775929162 of 84775689161]   [by num: 14.89780%   50470000 of 50479432]     db_management.cpp:138
2020-08-27T02:48:10        th_a:?unnamed?              reindex ] Done reindexing, elapsed time: 8980.39084600000023784 sec      db_management.cpp:159

[syalon]

Some performance test data, the previous comment has a detailed replay log.

Version	4.0.0	develop#a758bb0 + this pr
replay	8980s	9226s

[abitmore]

So it increases replay time by around 2.7%, or 4 minutes out of 2.5 hours with your setup. Seems acceptable.