eth: add debug_storageRangeAt

[fjl]

The new RPC method allows downloading storage in chunks.

debug_storageRangeAt takes parameters blockHash, txIndex, account, startKey, limit. The startKey applies to the hashed key (instead of the preimage as originally suggested in #3407) . The returned storage values contain the preimage if available. The returned object also contains the nextKey field, which will be non-null if there are more keys after the range that was returned.

Example:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "debug_storageRangeAt",
  "params": [
    "0xced82545131cdb6253f183a62dc3eaf81259be5cc744767798bcfb95664dc32c", // blockHash
    4,                                                                    // txIndex
    "0xB4A5BB81C6962Fa87Af6886ABE59b3cc56ec44b8",                         // account
    "0x",                                                                 // startKey
    2                                                                     // limit
  ]
}
// Response:
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "storage": {
      "0x1ab0c6948a275349ae45a06aad66a8bd65ac18074615d53676c09b67809099e0": {
        "key": "0x405787fa12a823e0f2b7631cc41b3ba8828b3321ca811111fa75cd3aa3bb5ace",
        "value": "0x00000000000000000000009413739a6ef2d14b5433465dbeffafbc2f2d610551"
      },
      "0x1d3954ea05fac9c4f44f290d32ed11be13922e2c8162b3594bb3bb0405546e8e": {
        "key": "0x2a359015afbfbe3b36db751c88f03aed63209927e8cf9ca7f0b69158e29e28f5",
        "value": "0x0000000000000000000000000000000000000000000000000000000000000004"
      }
    },
    "nextKey": "0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563"
  }
}

// Continuing the download using nextKey as startKey:
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "debug_storageRangeAt",
  "params": [
    "0xced82545131cdb6253f183a62dc3eaf81259be5cc744767798bcfb95664dc32c",
    4,
    "0xB4A5BB81C6962Fa87Af6886ABE59b3cc56ec44b8",
    "0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563",
    100
  ]
}
// The response has nextKey == null because there aren't any more values:
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "storage": {
      "0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563": {
        "key": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "value": "0x0000000000000000000000945e237bf1540eea146f5e830593d1d54c279616a9"
      },
      ... // 16 more entries elided
    },
    "nextKey": null
  }
}

We agreed on a preliminary spec in #3407. This supersedes that PR.
The earlier commits leading up to the new API add support for seeking in the trie iterator.

[Arachnid]

Could you define typedefs for hex and compact encoding?

[fjl]

No, because both types of key can be stored in shortNode.Key.

[Arachnid]

We could always use casts there. Typedefs would be helpful to avoid confusion elsewhere - though the function naming accomplishes some of that at least.

[fjl]

I don't want to change it in this PR. We can do it in another PR.

[Arachnid]

preorder. :)

[Arachnid]

What was the motivation behind making this internal?

[fjl]

The motivation was to have a single constructor for each iterator. Please see the commit message.

[Arachnid]

Why isn't this part of the loop initializer any longer?

[fjl]

Accident

[Arachnid]

Can you add tests for seeking to the empty string, and for seeking to a key past the end?

[fjl]

No need to test empty string, all other tests use NodeIterator(nil) ;). I've added one that seeks past the end.

[Arachnid]

I assume these are the hashes of the keys in the array below? Could you add a note to that effect for anyone coming across this later?

[karalabe]

Minor issues mostly. Please check and consider whether to fix or not.

One potentially bigger issue I saw was in the state object deep copy. Please check hat part out in detail and explain why it works if it works.

[karalabe]

❤️ I hate these labels so much :P

[karalabe]

Why return the previous "item" if peeking at the next one fails?

[fjl]

It has to return something. The idea was that the iterator wouldn't advance if there was an error fetching the next item.

[karalabe]

onE byte

[karalabe]

Ah sweet, I've never seen this part of the code in 2 years :))

[karalabe]

Could you add a test for encoding/decoding empty keys? The previous code had explicit checks whereas the new code does this implicitly by relying on the calculation returning an empty slice. There's even a slight API change, where the old code returned nil in hexToKeybytes whereas the new returns []byte{}. I'd be happy to see an explicit test that verifies this weird case too.

[karalabe]

Could you add a test for encoding/decoding empty keys?

[karalabe]

Isn't stateObject.trie a trie.SecureTrie? Because that one is a much fancier construct that won't "deep copy" so easily https://github.com/ethereum/go-ethereum/blob/master/trie/secure_trie.go#L44

[fjl]

Shallow copy support for SecureTrie was added in 710435b51, by yours truly.

[karalabe]

Don't trust that guy.... he looks shady af ;)

[karalabe]

Doesn't look like an iterator to me :P

[karalabe]

Can this ever happen? If the tx is already in our chain, I think it's safe to say this won't ever fail. (Mentioning because you deleted a similar check at tx.AsMessage).

[fjl]

The around AsMessage is for deriving the sender. I don't want to remove this one because many more things can go wrong (including DB errors, etc.).

[karalabe]

LGTM