INDY-1205: Use RocksDB as a key-value storage #561

sergey-shilov · 2018-03-06T13:30:22Z

No description provided.

Signed-off-by: Sergey Shilov <[email protected]>

ghost · 2018-03-06T13:30:24Z

Could one of the committers please verify this patch?

dsurnin · 2018-03-06T13:34:51Z

plenum/config.py

@@ -59,7 +60,7 @@
 clientBootStrategy = ClientBootStrategy.PoolTxn

 hashStore = {
-    "type": HS_FILE
+    "type": HS_LEVELDB


could you please use rocksdb here

No, thanks)

Seriously speaking, I've made temporary roll back to leveldb as rocksdb building procedure is not ready yet.

ashcherbakov · 2018-03-06T13:47:40Z

plenum/persistence/rocksdb_hash_store.py

+logger = getlogger()
+
+
+class RocksDbHashStore(HashStore):


The implementation looks exactly the same as LevelDbHashStore. Maybe use just one implementation with Dependency Injection of a proper key-value implementation?

Yes, but I propose to merge hash stores implementations after rocksdb build procedure is ready.

ashcherbakov · 2018-03-06T13:49:27Z

plenum/server/node.py

@@ -940,7 +932,7 @@ def onStopping(self):

    def closeAllKVStores(self):
        # Clear leveldb lock files
-        logger.debug("{} closing level dbs".format(self), extra={"cli": False})
+        logger.debug("{} closing level/rocks dbs".format(self), extra={"cli": False})


A better comment: closing key-value storages

ashcherbakov · 2018-03-06T13:50:27Z

plenum/test/storage/test_rocksdb_hash_store.py

+
+
+@pytest.yield_fixture(scope="module")
+def rocksdbHashStore(tdir):


Are there the same tests as for LevelDbHashStore? maybe just use the same test with parametrized fixtures?

ashcherbakov · 2018-03-06T13:53:25Z

storage/kv_store_rocksdb.py


    def close(self):
-        raise NotImplementedError
+        del self._db


Why do we call del here?

Signed-off-by: Sergey Shilov <[email protected]>

…ksdb. Signed-off-by: Sergey Shilov <[email protected]>

…torage. Signed-off-by: Sergey Shilov <[email protected]>

Signed-off-by: Sergey Shilov <[email protected]>

sergey-shilov · 2018-03-07T13:01:46Z

Test this please.

Signed-off-by: Sergey Shilov <[email protected]>

sergey-shilov · 2018-03-12T07:53:29Z

Test this please.

Signed-off-by: Sergey Shilov <[email protected]>

…deb from sovrin. Signed-off-by: Sergey Shilov <[email protected]>

Signed-off-by: Sergey Shilov <[email protected]>

…bIntKeys. Signed-off-by: Sergey Shilov <[email protected]>

Signed-off-by: Sergey Shilov <[email protected]>

ashcherbakov · 2018-03-20T15:48:44Z

storage/kv_store_rocksdb_int_keys.py

+        opts.comparator = IntegerComparator()
+        self._db = rocksdb.DB(self._db_path, opts)
+
+    def get_equal_or_prev(self, key):


Why don't we implement it in KeyValueStorageRocksdb?

Because it is correct just in case of integer keys with corresponding comparator. Otherwise the order is lexicographical where '2' is greater than '100', so get_equal_or_prev() does not make sense.

* INDY-1205: Add rocksdb as a key-value backend. Signed-off-by: Sergey Shilov <[email protected]> * Add tests for rocksdb. Signed-off-by: Sergey Shilov <[email protected]> * Add rocksdb support. Signed-off-by: Sergey Shilov <[email protected]> * Use rocksdb as a backend storage. Signed-off-by: Sergey Shilov <[email protected]> * Fix initialisation of rocksdb kvstore. Signed-off-by: Sergey Shilov <[email protected]> * Fix test_kv_rocksdb. Signed-off-by: Sergey Shilov <[email protected]> * Fix comparator of KeyValueStorageRocksdbIntKeys class. Signed-off-by: Sergey Shilov <[email protected]> * Fix test_state_rocksdb. Signed-off-by: Sergey Shilov <[email protected]> * Add unified config-based creation of hash store. Signed-off-by: Sergey Shilov <[email protected]> * Change default hash storage from file to rocksdb. Signed-off-by: Sergey Shilov <[email protected]> * Integrate rocksdb into state tests. Signed-off-by: Sergey Shilov <[email protected]> * Merge kv storages tests into single module. Signed-off-by: Sergey Shilov <[email protected]> * Temporary rollback to leveldb. Signed-off-by: Sergey Shilov <[email protected]> * Re-factor tests. Signed-off-by: Sergey Shilov <[email protected]> * Implement the first version of installation of rocksdb and python-rocksdb. Signed-off-by: Sergey Shilov <[email protected]> * Merge leveldb and rocksdb hash storages implementations into single storage. Signed-off-by: Sergey Shilov <[email protected]> * Use RocksDB as a key-value storage. Signed-off-by: Sergey Shilov <[email protected]> * Tempoprary use leveldb as a default storage for the ledger. Signed-off-by: Sergey Shilov <[email protected]> * Adopt getAllTxn() for working with rocksdb iterator. Signed-off-by: Sergey Shilov <[email protected]> * Fix db_path property for leveldb and rocksdb, fix test. Signed-off-by: Sergey Shilov <[email protected]> * Add build procedure for python-rocksdb and setuptool, use librocksdb deb from sovrin. Signed-off-by: Sergey Shilov <[email protected]> * Add missed libs to docker file. Signed-off-by: Sergey Shilov <[email protected]> * Change rocksdb package. Signed-off-by: Sergey Shilov <[email protected]> * Change rocksdb package for 3d parties build. Signed-off-by: Sergey Shilov <[email protected]> * Implement get_equal_or_prev() functionality for KeyValueStorageRocksdbIntKeys. Signed-off-by: Sergey Shilov <[email protected]> * Add a helper for init of k/v storage with int keys. Signed-off-by: Sergey Shilov <[email protected]> * Add rocksdb tests for the equal-or-prev functionality. Signed-off-by: Sergey Shilov <[email protected]> * Fallback to leveldb as we do not want to migrate to rocksdb right now. Signed-off-by: Sergey Shilov <[email protected]>

Sergey Shilov added 13 commits March 5, 2018 18:41

INDY-1205: Add rocksdb as a key-value backend.

2fc293c

Signed-off-by: Sergey Shilov <[email protected]>

Add tests for rocksdb.

904751b

Signed-off-by: Sergey Shilov <[email protected]>

Add rocksdb support.

6a694a8

Signed-off-by: Sergey Shilov <[email protected]>

Use rocksdb as a backend storage.

747de27

Signed-off-by: Sergey Shilov <[email protected]>

Fix initialisation of rocksdb kvstore.

fd127f6

Signed-off-by: Sergey Shilov <[email protected]>

Fix test_kv_rocksdb.

7f9b666

Signed-off-by: Sergey Shilov <[email protected]>

Fix comparator of KeyValueStorageRocksdbIntKeys class.

186f773

Signed-off-by: Sergey Shilov <[email protected]>

Fix test_state_rocksdb.

c9e7a19

Signed-off-by: Sergey Shilov <[email protected]>

Add unified config-based creation of hash store.

b2359fa

Signed-off-by: Sergey Shilov <[email protected]>

Change default hash storage from file to rocksdb.

4271498

Signed-off-by: Sergey Shilov <[email protected]>

Integrate rocksdb into state tests.

8b6399f

Signed-off-by: Sergey Shilov <[email protected]>

Merge kv storages tests into single module.

6ed63c3

Signed-off-by: Sergey Shilov <[email protected]>

Temporary rollback to leveldb.

ebed50f

Signed-off-by: Sergey Shilov <[email protected]>

dsurnin suggested changes Mar 6, 2018

View reviewed changes

ashcherbakov reviewed Mar 6, 2018

View reviewed changes

Re-factor tests.

3e88988

Signed-off-by: Sergey Shilov <[email protected]>

sergey-shilov force-pushed the feature/INDY-1205 branch 3 times, most recently from 7bfc05e to f45ab67 Compare March 7, 2018 10:01

Implement the first version of installation of rocksdb and python-roc…

a551c09

…ksdb. Signed-off-by: Sergey Shilov <[email protected]>

sergey-shilov force-pushed the feature/INDY-1205 branch from f45ab67 to a551c09 Compare March 7, 2018 11:05

Sergey Shilov added 3 commits March 7, 2018 15:05

Merge leveldb and rocksdb hash storages implementations into single s…

268398b

…torage. Signed-off-by: Sergey Shilov <[email protected]>

Use RocksDB as a key-value storage.

4945aba

Signed-off-by: Sergey Shilov <[email protected]>

Merge remote-tracking branch 'base/master' into feature/INDY-1205

b7319ef

ashcherbakov previously approved these changes Mar 7, 2018

View reviewed changes

Tempoprary use leveldb as a default storage for the ledger.

b2190a8

Signed-off-by: Sergey Shilov <[email protected]>

sergey-shilov dismissed ashcherbakov’s stale review via b2190a8 March 7, 2018 13:57

Adopt getAllTxn() for working with rocksdb iterator.

677d669

Signed-off-by: Sergey Shilov <[email protected]>

Sergey Shilov added 11 commits March 12, 2018 12:57

Fix db_path property for leveldb and rocksdb, fix test.

814bf04

Signed-off-by: Sergey Shilov <[email protected]>

Merge remote-tracking branch 'base/master' into feature/INDY-1205

b45fe58

Add build procedure for python-rocksdb and setuptool, use librocksdb …

237ce3b

…deb from sovrin. Signed-off-by: Sergey Shilov <[email protected]>

Add missed libs to docker file.

1b84c91

Signed-off-by: Sergey Shilov <[email protected]>

Merge remote-tracking branch 'base/master' into feature/INDY-1205

6f63c64

Change rocksdb package.

2cd538e

Signed-off-by: Sergey Shilov <[email protected]>

Change rocksdb package for 3d parties build.

d9f34a1

Signed-off-by: Sergey Shilov <[email protected]>

Implement get_equal_or_prev() functionality for KeyValueStorageRocksd…

39ae3d8

…bIntKeys. Signed-off-by: Sergey Shilov <[email protected]>

Add a helper for init of k/v storage with int keys.

b976a51

Signed-off-by: Sergey Shilov <[email protected]>

Add rocksdb tests for the equal-or-prev functionality.

3308804

Signed-off-by: Sergey Shilov <[email protected]>

Fallback to leveldb as we do not want to migrate to rocksdb right now.

4a2469a

Signed-off-by: Sergey Shilov <[email protected]>

ashcherbakov reviewed Mar 20, 2018

View reviewed changes

ashcherbakov approved these changes Mar 20, 2018

View reviewed changes

ashcherbakov merged commit b656455 into hyperledger:master Mar 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

INDY-1205: Use RocksDB as a key-value storage #561

INDY-1205: Use RocksDB as a key-value storage #561

sergey-shilov commented Mar 6, 2018

ghost commented Mar 6, 2018

dsurnin Mar 6, 2018

sergey-shilov Mar 6, 2018

sergey-shilov Mar 6, 2018

ashcherbakov Mar 6, 2018

sergey-shilov Mar 7, 2018

sergey-shilov Mar 7, 2018

ashcherbakov Mar 6, 2018

sergey-shilov Mar 7, 2018

ashcherbakov Mar 6, 2018

sergey-shilov Mar 7, 2018

ashcherbakov Mar 6, 2018

sergey-shilov commented Mar 7, 2018

sergey-shilov commented Mar 12, 2018

ashcherbakov Mar 20, 2018

sergey-shilov Mar 20, 2018



		@pytest.yield_fixture(scope="module")
		def rocksdbHashStore(tdir):

		logger = getlogger()


		class RocksDbHashStore(HashStore):

INDY-1205: Use RocksDB as a key-value storage #561

INDY-1205: Use RocksDB as a key-value storage #561

Conversation

sergey-shilov commented Mar 6, 2018

ghost commented Mar 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergey-shilov commented Mar 7, 2018

sergey-shilov commented Mar 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment