Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

state-db key value storage #4002

Closed
cheme wants to merge 85 commits into from
Closed

state-db key value storage #4002

cheme wants to merge 85 commits into from

Conversation

@cheme
Copy link
Contributor

@cheme cheme commented Nov 2, 2019

This PR extract the state-db logic from #3774 to try to make it more reviewable.
For general goal of key value storage, please refer to #3774.
This PR cover state-db additions.

As an extract from #3774, some TODO are present on state machine and client interfaces and tests are lacking.

  • dedicated journal (KvJournalRecord) are added (this could be merge in existing journal but focus here on non breaking change) to store block change, and pruning.
  • fork are handled with a sequential 'branch id' used as a per value index for memory historic of data (see historical-data tree).
    The branch id are not persisted since this is state db.
  • pruning which is more a client problematic is handled in this pr in the state machine test over a test backend.
  • apply commit operation got new variants, because some key value content do not make sense.

Note that some pruning enhancement described in #3774 could be added.

cheme added 30 commits September 23, 2019 18:55
@cheme
init from other pr, have some code up to client (not client db),
af6a8fd 
complete some test method by providing child access to values.
@cheme
Dumb state_db implementation, mostly boiler plate, but cannot work as
4db0a7f 
is.
Need to use historied values, maybe remove pinned, and put some context
as parameter for get_offstate
@cheme
TODO2 variant for in memory
19df6e9
@cheme
traits in place, still need to parameterized with blocknb &| branchix
3989fea
@cheme
passing block hash as state in a convoluted way
6a6606d
@cheme
passing block hash as state in a convoluted way
934c800
@cheme
Recover previous branch index code without persistence.
5d3d47e
@cheme
let's remove branch range set of changed and removed in next commit
aeab064
@cheme
removed delta in branch (not needed if no persistence)
57e862e
@cheme
Before using range instead of hash (we move range to upper client on
4edbf27 
pin).
@cheme
use also block number for query (when not in statedb we need number).
91b87d7
@cheme
Fixing finalize state algo.
966a09f
@cheme
remove keys from commit set, this likely indicates broken code, will
a37a32f 
probably need to store state, apply gc on state and depending on result
restore state or actually gc historied values.
@cheme
put a client test, for rest of test reset_storage and genesis build
4714b70 
should get updated.
@cheme
Get needed component from branch history-data
5e90b20
@cheme
use history from utils
f6956b1
@cheme
Test issue is that query on remove hash do not work
9ba8057
@cheme
test commit set
31c1ee3
@cheme
fix unpin gc
6209f4d
@cheme
missing offstate prunnig handling
70f69b6
@cheme
previous pruning did not make sense, switch to only storing touched key
6803c86 
for gc. Need to change commit set to contain an history of values and
adjust test db.
@cheme
Figure out pruning (see prune_one/prune_two tests).
2497575
@cheme
end fixing pruning test, remainings root lib tests to adjust
e86fdd2
@cheme
fix gc call block index.
e48cf43
@cheme
state_db tests passing.
0e3c65e
@cheme
fix underflow
13ea088
@cheme
before switching to absolute indexing (option<u64> to u64).
92c699c
@cheme
complete first test in client db, new todo.
bf79879
@cheme
Merge branch 'master' into chain-aux-merge
3913ba7
@cheme
serialized with default
4dc2c8d
cheme and others added 26 commits October 18, 2019 09:25
@cheme
Merge branch 'master' into chain-aux2
5ce414c
@cheme
Merge branch 'master' into chain-aux2
31400ae
@cheme
Merge branch 'master' into chain-aux2
2fcc675
@cheme
rename pairs to in_memory when it is not pairs but inmemory storage for
704178b 
a state.
@cheme
clean client crates code, add a FullStorageContent struct, not
5b0f4c2 
integrated into storage cache (no handling of kv there yet).
@cheme
cleaning in state-db
6dc9ef0
@cheme
end clean state_db
9ee40a7
@cheme
clean state-machine
b3e7d00
@cheme
renaming as in state-machine, and use only range
b642c69
@cheme
Move tree implementation state to test.
dd686c1
@cheme
Fix get on tree history (and iterator).
7713d8b
@cheme
Rename historied to historical
f0909fa
@cheme
A few renaming and documentation precision.
19f183c
@cheme
Switch test from Vec<u8> to Ser and removing Serialized buf mut variant.
4a834dd
@cheme
Merge branch 'master' into chain-aux2
b4f7805
@cheme
use copy_within
4268daf
@cheme
Remove truncate specific implementation.
397a77f
@cheme @gui1117
Apply documentation suggestions from code review
6caadf2 
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
@cheme
Remove into_owned function.
7410029 
Break big line.
@cheme
Merge branch 'master' into chain-aux2
5e93dec
@cheme
strip down client an state machine
06790bf
@cheme
switch linear in state test to remove linear code from this pr.
914e1c1
@cheme
Merge branch 'master' into chain-aux2
979db3f
@cheme
Merge branch 'chain-aux2' into kv-statedb
90b3837
@cheme
Merge branch 'master' into chain-aux2
4ba47b5
@cheme
Merge branch 'chain-aux2' into kv-statedb
37ad244
@cheme cheme added the A0-please_review Pull request needs code review. label Nov 2, 2019
@cheme
Copy link
Contributor Author

cheme commented Nov 11, 2019

Closing this PR because Child trie associated issue will be work around in a way that does not require this.
It make this a non prioritary thing.
Generally it adds complexity, and would require its own crate separation.
Some other part could maybe get merge.
Rethinking the design could also be interesting.

@cheme cheme closed this Nov 11, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

A0-please_review Pull request needs code review.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cheme