Add Binary64 option for account data

[sakridge]

Problem

bs58 encoding of large data is slow

Summary of Changes

Add base64 option and limit bs58 requests to account data less that 128 bytes.

Fixes #10140

[mvines]

We'll need an update to docs/src/apps/jsonrpc-api.md here eventually. Also let's convert solana account/solana vote-account/solana stake-account over to use base64 (there may be a couple other commands in the cli that will now die with the 128 byte limit too)

[mvines]

Allowing the user to optionally supply an "offset"/"length" seems pretty nice too, for when they know they only need a slice of a large account

[CriesofCarrots]

Approach looks fine to me. I haven't examined to see why the cli/deploy test is unhappy yet.
I like the idea of an offset/length specification, and could help with that in a follow-up

[sakridge]

Approach looks fine to me. I haven't examined to see why the cli/deploy test is unhappy yet.
I like the idea of an offset/length specification, and could help with that in a follow-up

rpc_client was trying to decode as Binary instead of Binary64. After I fixed that it seems to be ok, although my fix is a bit hacky.
773f002

[CriesofCarrots]

rpc_client was trying to decode as Binary instead of Binary64. After I fixed that it seems to be ok, although my fix is a bit hacky.

Ah, serde_json must go with the first enum variant that matches type 😕
I'm fine with your fix; the less-hacky ways I've thought to do this are a lot less concise.

Looks like you just need to update this line to Binary64 to fix the CI failure:

solana/cli/src/offline/blockhash_query.rs

Line 353 in a527021

let rpc_nonce_account = UiAccount::encode(nonce_account, UiAccountEncoding::Binary, None);

[mvines]

I like the idea of an offset/length specification, and could help with that in a follow-up

@CriesofCarrots - sure, can you queue this up for next week. We now do have a real use case where it would be more efficient for the client (and the node) if they were able to only request partial account data

[sakridge]

test bench_encode_base58 ... bench:  20,546,592 ns/iter (+/- 65,220)
test bench_encode_base64 ... bench:   6,253,972 ns/iter (+/- 107,235)

base64 does seem quite a bit faster.

[codecov]

Codecov Report

Merging #11474 into master will decrease coverage by 0.0%.
The diff coverage is 78.5%.

@@            Coverage Diff            @@
##           master   #11474     +/-   ##
=========================================
- Coverage    81.8%    81.7%   -0.1%     
=========================================
  Files         323      323             
  Lines       74965    74982     +17     
=========================================
- Hits        61325    61307     -18     
- Misses      13640    13675     +35     

[mvines]

Definitely faster. I did a ./run.sh with/without this PR and timed solana accounts TokenSVp5gheXUvJ6jGWGeCsgPKgnE3YgdGKRVCMY9o -- 8 seconds for base58, 3.5 seconds for base64.

I wonder if we could do even better with just hex encoding (as an alternative for when increase in encoding size doesn't matter, like small partial account reads) , would that be quick to bench @sakridge?

[sakridge]

Definitely faster. I did a ./run.sh with/without this PR and timed solana accounts TokenSVp5gheXUvJ6jGWGeCsgPKgnE3YgdGKRVCMY9o -- 8 seconds for base58, 3.5 seconds for base64.

I wonder if we could do even better with just hex encoding (as an alternative for when increase in encoding size doesn't matter, like small partial account reads) , would that be quick to bench @sakridge?

Hex seems worse:

test bench_encode_base58 ... bench:  73,587,849 ns/iter (+/- 531,971)
test bench_encode_base64 ... bench:       3,518 ns/iter (+/- 31)
test bench_encode_hex    ... bench:      26,636 ns/iter (+/- 84)

That's 8k of random data which with base58 is all I can stand to wait for. It really falls apart with random data.

[mvines]

Hex is worse 🤯.

[sakridge]

Hex is worse 🤯.

yep @mvines any blockers for merging this?

[mvines]

lgtm, thanks!