Merklize accounts internal state

[t-nelson]

Problem

We need a way to generate proofs of account state changes

Summary of Changes

Replace hash of concatenated accounts state hashes with a Merkle root

[t-nelson]

@mvines I think this is the only place where SPV and verifying accounts loaded from snapshots overlap. Should be good to go on cutting someone loose on the latter

[codecov]

Codecov Report

Merging #4999 into master will decrease coverage by 4.3%.
The diff coverage is 100%.

@@           Coverage Diff            @@
##           master   #4999     +/-   ##
========================================
- Coverage    76.9%   72.5%   -4.4%     
========================================
  Files         200     197      -3     
  Lines       37494   39023   +1529     
========================================
- Hits        28834   28312    -522     
- Misses       8660   10711   +2051

[codecov]

Codecov Report

Merging #4999 into master will decrease coverage by 16.8%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master   #4999      +/-   ##
=========================================
- Coverage    78.2%   61.3%   -16.9%     
=========================================
  Files         196     197       +1     
  Lines       35789   45635    +9846     
=========================================
+ Hits        27993   27995       +2     
- Misses       7796   17640    +9844

[sakridge]

Speed difference?

[t-nelson]

@sakridge runtime/benches/accounts.rs is my friend for that info?

[sakridge]

@t-nelson I don't think there exists a micro-bench for this at the moment.

[t-nelson]

@sakridge Yeah saw that. Adding one now.

[sakridge]

@t-nelson You might check this one out:
#5075
run with cd banking_bench; RUST_LOG=solana_banking_bench=info cargo run --bin solana-banking-bench --release

There should be a print of "average: xx us tx_average: xx us" average contains the time with the accounts hash_internal_state.

[t-nelson]

@sakridge I used the "test" crate, as was already in place for some benches under runtime. t-nelson@22fc268#diff-7348e822a67e92dad03464ab694a595eR61. No problem to write up something more thorough if it's insufficient

It's takes about twice as long

Hash of concatenated account hashes (old):
test test_accounts_hash_internal_state ... bench: 285,918 ns/iter (+/- 28,519)

Merklized account hashes (new):
test test_accounts_hash_internal_state ... bench: 584,399 ns/iter (+/- 255,462)

[sakridge]

@t-nelson Yea I ran it on the benchmark I referenced, it is about 15% reduction in performance just from this. That's pretty huge. Can we make this any faster?

[t-nelson]

Assuming we don't need the actual account hash later (they appear to be discarded now), I think I might be able to tweak it to pass the serialization of the account state directly to the Merkle Tree ctor. This would eliminate N hashing operations on something that is doing ~3N now

[t-nelson]

@sakridge is the above bench something we want in tree? I can PR it separately if so

I'll look into reducing hashing operations later today

[t-nelson]

@sakridge ~20% improvement with the proposed reduction of hashes

Without proposal:
test test_accounts_hash_internal_state ... bench: 578,833 ns/iter (+/- 57,388)

With proposal:
test test_accounts_hash_internal_state ... bench: 471,129 ns/iter (+/- 78,149)

[sakridge]

@t-nelson yea we should add the bench to the tree. separate PR is great.

[garious]

@t-nelson, what's going on with this PR?

[t-nelson]

@garious I need to reduce negative effects on perf

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[aeyakovenko]

@t-nelson @sakridge close this one?

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[t-nelson]

Superseded by #5573