Book: Update SPV section to reflect new account state query mechanism by t-nelson · Pull Request #5399 · solana-labs/solana

t-nelson · 2019-08-01T22:48:47Z

As per Discord convo with @aeyakovenko and @rob-solana

aeyakovenko · 2019-08-01T23:18:13Z

-such as a credit of 0 Lamports.
+An account's state (balance or other data) can be verified by submitting a
+transaction with a ___TBD___ Instruction to the cluster. This instruction contains
+the slot boundary of interest and the expected state hash at that boundary. The


We can’t enforce the slot boundary for this query. I think it’s basically is the query true when it’s executed by the cluster.

Hrm. That's a pretty narrow window to hit on a high throughput account, basically the next slot and 🤞, right?

to be able to do "slot of interest": would it be terribly expensive to hang onto the account hash vector at the end of every slot for 256 slots? worst case is 256*2.5*sizeof(Hash)*TPS, similar to StatusCache

@t-nelson why would the slot boundary provide any safety? The state can change immediately on the next slot. It’s up to the contract to have some meaningful terminal state, like “game over”

@rob-solana by "hash vector" you mean the "inclusion proof" or "merkle path"? They'd be log(N) and mergable per-slot, so that's certainly something to explore

@aeyakovenko there's no safety at the boundary. I was getting at the race between receiving the "Don't squash this slot" instruction and the slot being squashed, though after review Carl's snapshot work, I realize we have until the bank in question falls out of the locktower. This will require us scanning all bank forks and taking action on an unfinalized transaction, right?

@t-nelson we can take action on it after that bank is rooted, and provide the result way later, maybe 1024 slots after or something long enough, or at an epoch boundary.

aeyakovenko · 2019-08-01T23:19:27Z

Cool! This is missing the checkpoint proof generation.

t-nelson · 2019-08-02T00:19:32Z

Oof. I got to thinking about whether there was potential for a cheap resource exhaustion attack here and forgot to write it up 😆

aeyakovenko · 2019-08-05T20:42:33Z

@t-nelson so I think this would work on Ethereum.
vote for BankHash N, links to BankHash N -1

Since the BankHash's are connected, the SPV check on ethereum can skip PoH.

@pgarg66 I think this would replace the "Ancestor Verification" section in:
https://solana-labs.github.io/book/vote-signing-to-implement.html#ancestor-verification

garious · 2019-08-12T21:42:19Z

@t-nelson, what's this PR blocked by?

t-nelson · 2019-08-13T21:01:45Z

@aeyakovenko Sorry, this fell off my radar with the colo setup stuff. Last two commits address the issues I think. LMK what you think

t-nelson · 2019-08-21T15:45:08Z

@rob-solana, since @aeyakovenko is travelling, would you mind giving this a once over to see if it jives with your understanding of the proposal?

rob-solana · 2019-08-21T16:30:03Z

@rob-solana, since @aeyakovenko is travelling, would you mind giving this a once over to see if it jives with your understanding of the proposal?

I think @sakridge has better information about the current plan.

sakridge · 2019-08-21T22:01:48Z

So we are considering making the accounts hash value as:

expand_fn(hash(account0)) ^ expand_fn(hash(account1)) ^ ...

where expand_fn(hash) -> [u8; 440] is generating 440 bytes of random data from a crypto RNG with the account hash as the seed value. This allows one to add/remove accounts from the state value without re-computing the entire hash value.

sakridge · 2019-08-21T22:51:14Z

440 bytes comes from this paper, avoiding xor collision with 0 (or thus any other given bit pattern):
https://link.springer.com/content/pdf/10.1007%2F3-540-45708-9_19.pdf

O(k · 2^(n/(1+lg(k)))
k=2^40 accounts
n=440
2^(40) * 2^(448 * 8 / 41) ~= O(2^(128))

t-nelson · 2019-08-21T23:12:23Z

@sakridge OK thanks! Funny, I was actually typing up some clarifications that your second reply addressed 🙂

I get that it's cheaper to calculate and add/remove entries from this construction, but I don't see how we can build an inclusion proof? Or is that a tangential requirement?

sakridge · 2019-08-22T01:46:36Z

@t-nelson well, I was mainly after the performance side of it compared to what we are doing now or potentially computing a merkle tree of the accounts. @aeyakovenko had some idea that it would reduce the need to compute a merkle tree with old account states but I can't say I fully grok how that works yet.

t-nelson · 2019-08-22T01:56:59Z

@sakridge Gotcha! I think you and I are on the same page. We'll have to await input from the big guy 🙃

rob-solana · 2019-08-22T03:55:47Z

So we are considering making the accounts hash value as:

expand_fn(hash(account0)) ^ expand_fn(hash(account1)) ^ ...

where expand_fn(hash) -> [u8; 440] is generating 440 bytes of random data from a crypto RNG with the account hash as the seed value. This allows one to add/remove accounts from the state value without re-computing the entire hash value.

oh man, this is gonna get messy... have a PR yet?

sakridge · 2019-08-22T16:14:32Z

@rob-solana #5573 this is the PR. Yea, it's not super clean, but not the worst. Not sure how it combines with what changes you are planning.

aeyakovenko · 2019-08-26T16:40:34Z

-such as a credit of 0 Lamports.
+A complication of this method is avoiding bank accounts squashing discarding
+information required to verify blocks referenced by an inclusion proof. A solution
+is to instead snapshot/checkpoint these banks by submitting a ___TBDb_Instruction___,


This is no longer necessary. The bank hash is derived from the snapshot image

But we may want the bank hash to be the hash of the block transaction merkle root, hash of the snapshot image and the previous bank hash

What about the accounts XOR? I think I missed some convo on that somewhere (didn't seem to be more in discord/#consensus, anyway). Do we not need inclusion proofs on the account state anymore?

Oh, and if you're on vacation, feel free to tell me to shove off 🙂

we don’t have an account inclusion proof. So you still need a tx. Shove off :)

carllin · 2019-08-26T21:34:49Z

-with the Previous Bank-Merkle, and the Block-Merkle.
+An account's state (balance or other data) can be verified by submitting a
+transaction with a ___TBDa_Instruction___ to the cluster. The client can then use
+a [Transaction Inclusion Proof](#transaction-inclusion-proof) to verify whether


Is the purpose of the account state verification to prove that an account has some expected state, at some given point in the past (some block)?

ah ok, I see, the Accounts-Hash only includes accounts that were modified during a slot, which is why a transaction inclusion proof of the transaction that caused the modification is necessary. Thanks!

carllin · 2019-08-26T21:40:30Z

-or
-* Transaction -> Entry-Merkle -> Block-Merkle -> Bank-Merkle
+
+* Transaction -> Entry-Merkle -> Block-Merkle -> Bank-Hash


If we are generating the SPV of a transaction in a block A, for the vector of validator vote entries mentioned below, do we include all future votes for each validator up until MAX_LOCKOUT? If not, what is the locktower depth that we are interested in for confirmation?

The confirmation depth should be configurable (with a safe min and default) and would be dependent on the economic value of the transaction. $10 TX, low confirmation, $10M TX, high confirmation.

Note that this is a little WIP ATM as typically an SPV client would be validating block headers on its own and not have to reveal this information to the full node providing the TX proof. On Solana, this would require them to be capable of keeping up with and validating PoH, which is an atypically high standard for such a client.

stale · 2019-09-25T23:45:38Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

garious · 2019-09-26T00:00:44Z

I haven't been following this one. Is there a subset of stuff you guys agree on that we can get merged?

t-nelson · 2019-09-26T15:40:00Z

This needs updated for #5885. I'll try to sync it up today.

garious · 2019-10-15T15:07:49Z

@t-nelson, can you wrap this up?

t-nelson · 2019-10-15T15:18:24Z

Can do

stale · 2019-11-22T22:58:09Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

stale · 2019-11-29T23:10:31Z

This stale pull request has been automatically closed. Thank you for your contributions.

stale · 2019-12-17T21:45:53Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

stale · 2019-12-24T21:54:46Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

stale · 2019-12-31T21:58:11Z

This stale pull request has been automatically closed. Thank you for your contributions.

stale · 2020-01-07T22:50:33Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

garious · 2020-01-10T00:12:04Z

Can you format to 80 char lines?

t-nelson · 2020-01-10T02:50:14Z

Absolutely

garious · 2020-01-10T17:22:51Z

@t-nelson, waiting on action from anyone here?

t-nelson requested a review from aeyakovenko August 1, 2019 22:48

aeyakovenko reviewed Aug 1, 2019

View reviewed changes

t-nelson force-pushed the book-spv_acct_state_verify branch from 7edb070 to afd7dee Compare August 13, 2019 17:19

t-nelson requested a review from aeyakovenko August 19, 2019 16:23

aeyakovenko reviewed Aug 26, 2019

View reviewed changes

carllin reviewed Aug 26, 2019

View reviewed changes

stale Bot added the stale [bot only] Added to stale content; results in auto-close after a week. label Sep 25, 2019

stale Bot removed the stale [bot only] Added to stale content; results in auto-close after a week. label Sep 26, 2019

t-nelson force-pushed the book-spv_acct_state_verify branch from afd7dee to f5b4350 Compare October 23, 2019 22:36

aeyakovenko mentioned this pull request Nov 29, 2019

Fix BankHash calculation, which doesn't follow the SPV design. #7053

Open

stale Bot closed this Nov 29, 2019

garious reopened this Nov 29, 2019

stale Bot removed the stale [bot only] Added to stale content; results in auto-close after a week. label Nov 29, 2019

stale Bot added the stale [bot only] Added to stale content; results in auto-close after a week. label Dec 17, 2019

t-nelson removed the stale [bot only] Added to stale content; results in auto-close after a week. label Dec 17, 2019

stale Bot added the stale [bot only] Added to stale content; results in auto-close after a week. label Dec 24, 2019

stale Bot closed this Dec 31, 2019

t-nelson removed the stale [bot only] Added to stale content; results in auto-close after a week. label Dec 31, 2019

t-nelson reopened this Dec 31, 2019

stale Bot added the stale [bot only] Added to stale content; results in auto-close after a week. label Jan 7, 2020

t-nelson force-pushed the book-spv_acct_state_verify branch from f5b4350 to 828dda3 Compare January 9, 2020 22:51

stale Bot removed the stale [bot only] Added to stale content; results in auto-close after a week. label Jan 9, 2020

t-nelson added 3 commits January 9, 2020 15:55

Book: Update SPV section to reflect new account state query mechanism

68014f9

Book: SPV - Rename Bank-Merkle diagram

0d5b50d

Relax specificity of inclusion proof resolution

5a7523b

t-nelson force-pushed the book-spv_acct_state_verify branch from 828dda3 to 5a7523b Compare January 9, 2020 22:55

Cosmetic: re-wrap at 80

c473868

garious added the problem description doesn't describe a problem label Jan 10, 2020

t-nelson merged commit 2356b25 into solana-labs:master Jan 10, 2020

t-nelson deleted the book-spv_acct_state_verify branch January 10, 2020 17:56

ryoqun mentioned this pull request Jan 14, 2020

Clean up simple-payment-and-state-verification #7697

Closed

Conversation

t-nelson commented Aug 1, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rob-solana Aug 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aeyakovenko commented Aug 1, 2019

Uh oh!

t-nelson commented Aug 2, 2019

Uh oh!

aeyakovenko commented Aug 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

garious commented Aug 12, 2019

Uh oh!

t-nelson commented Aug 13, 2019

Uh oh!

t-nelson commented Aug 21, 2019

Uh oh!

rob-solana commented Aug 21, 2019

Uh oh!

sakridge commented Aug 21, 2019

Uh oh!

sakridge commented Aug 21, 2019

Uh oh!

t-nelson commented Aug 21, 2019

Uh oh!

sakridge commented Aug 22, 2019

Uh oh!

t-nelson commented Aug 22, 2019

Uh oh!

rob-solana commented Aug 22, 2019

Uh oh!

sakridge commented Aug 22, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carllin Aug 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stale Bot commented Sep 25, 2019

Uh oh!

garious commented Sep 26, 2019

Uh oh!

t-nelson commented Sep 26, 2019

Uh oh!

garious commented Oct 15, 2019

Uh oh!

t-nelson commented Oct 15, 2019

Uh oh!

stale Bot commented Nov 22, 2019

Uh oh!

stale Bot commented Nov 29, 2019

Uh oh!

rob-solana Aug 2, 2019 •

edited

Loading

aeyakovenko commented Aug 5, 2019 •

edited

Loading

carllin Aug 26, 2019 •

edited

Loading