database api changes in witnesses, committee members and workers calls. #352

oxarbitrage · 2017-08-09T23:59:14Z

In the database api there are some functions around witness, worker and committee_member objects. objects are very similar but the calls are inconsistent.

This is one of many of the problems the api_database haves described at: #95

The witness object is currently the one with more number of calls and they are:

get_witnesses
get_witness_by_account
lookup_witness_accounts
get_witness_count

The pull makes the 4 same calls for the other objects so in addition to the 4 witness calls we now have:

get_committee_members
get_committee_member_by_account
lookup_committee_member_accounts
get_committee_count

get_workers
get_workers_by_account
lookup_worker_accounts
get_worker_count

Some were already there, others were added.

Testing was done while syncing a chain at different states, your numbers will vary from the results presented here.

the 3 counters:

alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "get_worker_count", []], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": 19
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "get_witness_count", []], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": 58
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "get_committee_count", []], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": 25
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $

get witnesses:

alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "get_witnesses", [["1.6.2", "1.6.1"]]], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": [
    {
      "id": "1.6.2",
      "witness_account": "1.2.90743",
      "last_aslot": 1531,
      "signing_key": "BTS8Mxnmn6AxGCvJ4miXt4H3T65KN2GvYeuQXrbaHyKJFeH8Xbzq1",
      "vote_id": "1:1",
      "total_votes": 0,
      "url": "",
      "total_missed": 693,
      "last_confirmed_block_num": 84
    },
    {
      "id": "1.6.1",
      "witness_account": "1.2.90742",
      "last_aslot": 172345,
      "signing_key": "BTS6eTWdfBXvgmfY1VLSRc6iiFk4qMbmsLcLCEEbbLYJBgAYw3C7L",
      "pay_vb": "1.13.8",
      "vote_id": "1:0",
      "total_votes": 0,
      "url": "",
      "total_missed": 982,
      "last_confirmed_block_num": 167184
    }
  ]
}

get committee_members:

alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "get_committee_members", [["1.5.2", "1.5.1"]]], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": [
    {
      "id": "1.5.2",
      "committee_member_account": "1.2.90744",
      "vote_id": "0:13",
      "total_votes": "20599522639597",
      "url": ""
    },
    {
      "id": "1.5.1",
      "committee_member_account": "1.2.90743",
      "vote_id": "0:12",
      "total_votes": "921211352795",
      "url": ""
    }
  ]
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $

get workers:

alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "get_workers", [["1.14.2", "1.14.1"]]], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": [
    {
      "id": "1.14.2",
      "worker_account": "1.2.22517",
      "work_begin_date": "2015-10-23T00:00:00",
      "work_end_date": "2015-11-23T00:00:00",
      "daily_pay": 1000000000,
      "worker": [
        1,
        {
          "balance": "1.13.233"
        }
      ],
      "vote_for": "2:69",
      "vote_against": "2:70",
      "total_votes_for": 0,
      "total_votes_against": 2830810772,
      "name": "mindphlux-bitasset-poolfund",
      "url": ""
    },
    {
      "id": "1.14.1",
      "worker_account": "1.2.22517",
      "work_begin_date": "2015-10-23T00:00:00",
      "work_end_date": "2015-11-23T00:00:00",
      "daily_pay": 10000,
      "worker": [
        1,
        {
          "balance": "1.13.232"
        }
      ],
      "vote_for": "2:67",
      "vote_against": "2:68",
      "total_votes_for": "11743077642696",
      "total_votes_against": 0,
      "name": "mindphlux-bitasset-poolfund",
      "url": ""
    }
  ]
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $

get witness by account:

alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "get_witness_by_account", ["1.2.167"]], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": {
    "id": "1.6.16",
    "witness_account": "1.2.167",
    "last_aslot": 3635877,
    "signing_key": "BTS6bcMW7N4dMCYXc8LM9evJ1J2jWCDUFYiW2L1RWHqVxHH3pq2U7",
    "pay_vb": "1.13.32",
    "vote_id": "1:26",
    "total_votes": "36781493604751",
    "url": "",
    "total_missed": 1507,
    "last_confirmed_block_num": 3608703
  }
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $

committee by account:

alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "get_committee_member_by_account", ["1.2.282"]], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": {
    "id": "1.5.27",
    "committee_member_account": "1.2.282",
    "vote_id": "0:147",
    "total_votes": "21626722904708",
    "url": "http://bitshares.eu"
  }
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $

worker by account:

alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "get_workers_by_account", ["1.2.22517"]], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": [
    {
      "id": "1.14.1",
      "worker_account": "1.2.22517",
      "work_begin_date": "2015-10-23T00:00:00",
      "work_end_date": "2015-11-23T00:00:00",
      "daily_pay": 10000,
      "worker": [
        1,
        {
          "balance": "1.13.232"
        }
      ],
      "vote_for": "2:67",
      "vote_against": "2:68",
      "total_votes_for": 0,
      "total_votes_against": 0,
      "name": "mindphlux-bitasset-poolfund",
      "url": ""
    },
    {
      "id": "1.14.2",
      "worker_account": "1.2.22517",
      "work_begin_date": "2015-10-23T00:00:00",
      "work_end_date": "2015-11-23T00:00:00",
      "daily_pay": 1000000000,
      "worker": [
        1,
        {
          "balance": "1.13.233"
        }
      ],
      "vote_for": "2:69",
      "vote_against": "2:70",
      "total_votes_for": 0,
      "total_votes_against": 0,
      "name": "mindphlux-bitasset-poolfund",
      "url": ""
    },
    {
      "id": "1.14.3",
      "worker_account": "1.2.22517",
      "work_begin_date": "2015-10-23T00:00:00",
      "work_end_date": "2015-11-23T00:00:00",
      "daily_pay": 1000000000,
      "worker": [
        1,
        {
          "balance": "1.13.234"
        }
      ],
      "vote_for": "2:71",
      "vote_against": "2:72",
      "total_votes_for": 0,
      "total_votes_against": 0,
      "name": "mindphlux-bitasset-poolfund",
      "url": "https://bitsharestalk.org/index.php/topic,19317.0.html"
    },
    {
      "id": "1.14.4",
      "worker_account": "1.2.22517",
      "work_begin_date": "2015-10-21T11:00:00",
      "work_end_date": "2015-11-21T11:00:00",
      "daily_pay": 1000000000,
      "worker": [
        1,
        {
          "balance": "1.13.235"
        }
      ],
      "vote_for": "2:73",
      "vote_against": "2:74",
      "total_votes_for": "1698106100813",
      "total_votes_against": 0,
      "name": "bitasset-fund-pool",
      "url": "https://bitsharestalk.org/index.php/topic,19317.0.html"
    }
  ]
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $

/////////// the 3 lookups:

alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "lookup_worker_accounts", ["aaa", 100]], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": [
    [
      "init0",
      "1.14.0"
    ],
    [
      "maqifrnswa",
      "1.14.5"
    ],
    [
      "mindphlux",
      "1.14.1"
    ]
  ]
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "lookup_committee_member_accounts", ["aaa", 100]], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": [
    [
      "bhuz",
      "1.5.16"
    ],
    [
      "bitcrab",
      "1.5.15"
    ],
    [
      "bitcube",
      "1.5.14"
    ],
    [
      "clayop",
      "1.5.17"
    ],
    [
      "dele-puppy",
      "1.5.12"
    ],
    [
      "fav",
      "1.5.13"
    ],
    [
      "init0",
      "1.5.0"
    ],
    [
      "init1",
      "1.5.1"
    ],
    [
      "init10",
      "1.5.10"
    ],
    [
      "init2",
      "1.5.2"
    ],
    [
      "init3",
      "1.5.3"
    ],
    [
      "init4",
      "1.5.4"
    ],
    [
      "init5",
      "1.5.5"
    ],
    [
      "init6",
      "1.5.6"
    ],
    [
      "init7",
      "1.5.7"
    ],
    [
      "init8",
      "1.5.8"
    ],
    [
      "init9",
      "1.5.9"
    ],
    [
      "logxing",
      "1.5.18"
    ],
    [
      "mindphlux",
      "1.5.11"
    ]
  ]
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $ curl --silent --data '{"jsonrpc": "2.0", "params": ["database", "lookup_worker_accounts", ["aaa", 100]], "method": "call", "id": 10}' http://localhost:8090/rpc | jq
{
  "id": 10,
  "jsonrpc": "2.0",
  "result": [
    [
      "init0",
      "1.14.0"
    ],
    [
      "maqifrnswa",
      "1.14.5"
    ],
    [
      "mindphlux",
      "1.14.1"
    ]
  ]
}
alfredo@alfredo-Inspiron-5559 ~/CLionProjects/database_api_changes $

more discussion about database_api improves can be found at: #95

abitmore · 2017-08-10T00:07:33Z

An account can own multiple workers but only one witness and one committee member, it's different. Witnesses and committee members are currently permanent, but workers may expire, so the new API's for workers (count and etc) don't make much sense, instead, it's better to have time range queries for workers.

By the way, current implementation of lookup APIs are inefficient, however it's not a big deal right now because we don't have many witnesses or committee members.

oxarbitrage · 2017-08-10T00:15:49Z

it is considered, check the worker by account sample, it was returning 2 workers for the requested account. if it is expired/active/not started/etc can be decided client side with the dates. the get_workers_by_account is different from get_witness_by_account and get_committee_member_by_account, this returns only 1 result.

oxarbitrage · 2017-08-11T12:14:24Z

we discussed this on telegram and it is true that we can just have 1 call per section and it will be probably enough with our current network basically because we do not have a lot of records in this places. so a get_witnesses, get_committee_members and get_workers returning all the data in the indexes will be enough.

problem is that we need to keep the calls that are already there, for instance witness haves the 4 calls, committee had 3 and only the count was added. workers just had 1 so 3 calls were added.

i need at least to add the committee_count and the workers_count to the explorer, that is the main reason of the pull, the other 2 calls added to worker was just to have the same calls in the 3 sections. also, the get_workers_by_account to get the worker proposals an account have submitted is useful. I agree that the lookups in the 3 sections don't make too much sense right now but we can't remove them from committee or witness as some clients may be using them.

let me know @abitmore how do you think we may proceed here.

abitmore · 2017-08-11T14:30:25Z

libraries/app/database_api.cpp

+    vector<optional<worker_object>> result;
+    const auto& workers_idx = _db.get_index_type<worker_index>().indices().get<by_account>();
+
+    for( const auto& w : workers_idx )


Please avoid iterating through the whole index. Use lower_bound and check for end while iterating.

abitmore · 2017-08-11T14:41:02Z

libraries/app/database_api.cpp

+    for (const worker_object& worker : workers_by_id)
+        if (auto account_iter = _db.find(worker.worker_account))
+            if (account_iter->name >= lower_bound_name) // we can ignore anything below lower_bound_name
+                workers_by_account_name.insert(std::make_pair(account_iter->name, worker.id));


As mentioned before, this implementation is inefficient. I'd rather have a simple API to list all worker objects right now, or only the ones are active. Even range query with an account_id lower bound is better. If we really want to query with an account_name lower bound, best redundantly add the account_name field into worker_object.

abitmore · 2017-08-16T22:04:20Z

libraries/app/database_api.cpp

-    auto itr = idx.find(account);
-    vector<worker_object> result;
-
-    if( itr != idx.end() && itr->worker_account == account )


There was a bug here, if should be while. Fixed in this PR.

oxarbitrage · 2017-08-17T00:46:34Z

added get_all_workers to get all the worker objects. removed lookup_worker_accounts. kept the count calls.

oxarbitrage · 2017-08-17T23:15:33Z

removed optional.

database api changes in witnesses, committee members and workers calls.

8e22263

abitmore requested changes Aug 11, 2017

View reviewed changes

abitmore reviewed Aug 16, 2017

View reviewed changes

requested changes

ecffbc5

removing optional

51fee3f

abitmore approved these changes Aug 19, 2017

View reviewed changes

oxarbitrage merged commit 8b56fc5 into bitshares:develop Aug 20, 2017

startailcoon mentioned this pull request Mar 20, 2018

[1] workers with id > 100 not shown! bitshares/bitshares-ui#1349

Closed

abitmore mentioned this pull request Mar 21, 2018

Improve worker query API #759

Open

oxarbitrage deleted the database_api_changes branch November 15, 2018 23:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

database api changes in witnesses, committee members and workers calls. #352

database api changes in witnesses, committee members and workers calls. #352

oxarbitrage commented Aug 9, 2017 •

edited

Loading

abitmore commented Aug 10, 2017 •

edited

Loading

oxarbitrage commented Aug 10, 2017

oxarbitrage commented Aug 11, 2017

abitmore Aug 11, 2017

abitmore Aug 11, 2017

abitmore Aug 16, 2017

oxarbitrage commented Aug 17, 2017

oxarbitrage commented Aug 17, 2017

database api changes in witnesses, committee members and workers calls. #352

database api changes in witnesses, committee members and workers calls. #352

Conversation

oxarbitrage commented Aug 9, 2017 • edited Loading

abitmore commented Aug 10, 2017 • edited Loading

oxarbitrage commented Aug 10, 2017

oxarbitrage commented Aug 11, 2017

abitmore Aug 11, 2017

Choose a reason for hiding this comment

abitmore Aug 11, 2017

Choose a reason for hiding this comment

abitmore Aug 16, 2017

Choose a reason for hiding this comment

oxarbitrage commented Aug 17, 2017

oxarbitrage commented Aug 17, 2017

oxarbitrage commented Aug 9, 2017 •

edited

Loading

abitmore commented Aug 10, 2017 •

edited

Loading