Substrate RPC : Response too big, Exceeded max limit of 15728640

[muddlebee]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

error response ⬇️

{
    "jsonrpc": "2.0",
    "error": {
        "code": -32702,
        "message": "Response is too big",
        "data": "Exceeded max limit of 15728640"
    },
    "id": 1
}

curl request ⬇️

curl --location --request POST 'https://westend-rpc.polkadot.io' \
--header 'Content-Type: application/json' \
--data-raw '{
    "id": 1,
    "jsonrpc": "2.0",
    "method": "state_getKeys",
    "params": [
        "0x26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9"
    ]
}'

Steps to reproduce

Hit the above curl request

[jacogr]

The state_getKeys should rather not be used, the state_getKeysPaged version is preferred. See

substrate/client/rpc-api/src/state/mod.rs

Lines 40 to 43 in 37cca71

	/// Returns the keys with prefix, leave empty to get all the keys.
	#[method(name = "state_getKeys", blocking)]
	#[deprecated(since = "2.0.0", note = "Please use `getKeysPaged` with proper paging support")]
	fn storage_keys(&self, prefix: StorageKey, hash: Option<Hash>) -> RpcResult<Vec<StorageKey>>;

Like the unsafe state_getPairs this version adds a lot of load on the Node since it has to iterate the trie to retrieve all matching keys, in this case it is even blowing through the assigned response limits.

IMHO these load-bearing, free-for-all (and thus deprecated) methods should to be hidden behind the unsafe flag because of the overhead. In this case the key provided is System.Account, which you really want to page if you have to iterate through - however nothing is stopping you from not paging and placing undue/immediate/blocking load on the RPC Node.

[dcolley]

polkadot/utlls/staking-miner v0.9.27 has the same problem on Polkadot with state_getStorage:
(v0.9.29 on Kusama is fine)

2022-09-22T09:45:10.946190Z DEBUG jsonrpsee_client_transport::ws: send: [{"jsonrpc":"2.0","id":10,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e30bb1d35a8b4d31acec2fccbcf0172fc4","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]},{"jsonrpc":"2.0","id":11,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e313792e785168f725b60e2969c7fc2552","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]},{"jsonrpc":"2.0","id":12,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e314ce3a66063250f71e0a11a517ea4062","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]},{"jsonrpc":"2.0","id":13,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e32a3e5f1d461bf763a76013e062b46c0e","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]},{"jsonrpc":"2.0","id":14,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e34e7b9012096b41c4eb3aaf947f6ea429","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]},{"jsonrpc":"2.0","id":15,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e36d8bbda3bbbad46cc57d43825dd040c5","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]},{"jsonrpc":"2.0","id":16,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e3720b70fd47fbed875a3a2dad4378ec4d","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]},{"jsonrpc":"2.0","id":17,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e396d38fd45bc038faa9586fa93aa03ef7","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]},{"jsonrpc":"2.0","id":18,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e3b3d1c643c0d45e2bb9269ac86c1dcda0b4def25cfda6ef3a00000000","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]},
{"jsonrpc":"2.0","id":19,"method":"state_getStorage","params":
["0xede8e4fdc3c8b556f0ce2f77fc2575e3d9764401941df7f707a47ba7db64a6ea","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]}]
2022-09-22T09:45:11.030616Z DEBUG jsonrpsee_core::client::async_client: [backend]: recv unparseable message: Ok(Array([Object({"id": Number(10), "jsonrpc": String("2.0"), "result": 
String("0x04f5518899384042000000000000000000d7c8606453bcbd5800000000000000006f7c25610c8e98c315909
2776b021b0000000000")}), Object({"id": Number(11), "jsonrpc": String("2.0"), "result": String("0x22020000")}), Object({"id": Number(12), "jsonrpc": String("2.0"), "result": String("0x01000000")}), Object({"id": Number(13), "jsonrpc": 
String("2.0"), "result": String("0x925f01006d10")}), Object({"id": Number(14), "jsonrpc": String("2.0"), "result": 
String("0x0000")}), Object({"id": Number(15), "jsonrpc": String("2.0"), "result": 

String("0x257d35561c1b1e000000000000000000847d60d281237a2d00000000000000009ea660a6735ba073d310da480c012f00")}), Object({"id": Number(16), "jsonrpc": String("2.0"), "result": String("0x29010000")}), Object({"error": 

Object({"code": Number(-32702), "data": String("Exceeded max limit of 15728640"), "message": String("Response is too big")}), "id": Number(17), "jsonrpc": String("2.0")}), Object({"id": Number(18), "jsonrpc": String("2.0"), "result": 

String("0xc4181e01fbb27d6e4792397efe9becf4a3e4c535be6112f60b5ece445a76552ec0dceb5d5f00000000000000000
000000e4e0100d0fd0fdca906d8b106ec710fe05104f0f909f8b90df4c9010d0115030101150e09012110050195092501cd08
1d01f10f29018d042d01e9054501990d3d01dd014d0105019101d10d5d0109027d01bd06690191056d01e908710175077501
910c7901e505950151029901c5039d0171038d0149038901550285015502a501a903a101cd0681017506b5014903c10104c
501c10cb101f101c901cd08b901c10be501bd04d5016109d101350ed9016901e901c90ccd012907f1014d0dfd015d04f9018d0
5f501110219025d0a09020d021d0205030102c9011102c1021502c103050295042902c908310215010d0221073502c10345
028d054902c90c5902910b3902fd0c5d023d0b81026d0f850271038902850a6902910c710275077502e9087902bd068d0
221079502d5076502c49d020501a1026d0fa9023d0bb102d106a5029509ad029509c1026109c902190cbd02310bb902150
ed902c10bdd025d04cd025509e1028504b502d903d502d502fd026901f90285070903d104e902d1020503d903bd03a90
6ed02c10b0103350e110345031503e905190389061d03150e0d03b9082103e90525032905750305018d034d0d4d03750
6390335046503bd044903910b6103a10c6d031d077103350e8103310b29039903310305063d03250f4103190c69036901
5903dd012d033d0b7d03c10c4503490d9503550a8503c10b9103b9088903e102d503150951039c5d03ed0ac903d50779
03b10835037506a5030504b1037d05cd038d099d038906a9030d02b5034d0db9039504d1032905c5032905ad03910bc
103290505049504dd03bd0ff1030902e1036109e903bd0f9903d903ed03350e1104bd0f2d04c10215040d08fd037506410
429050d040d083504003904d903f503d507f903990319041509210415031d04c5032904d10d310415013d0421100904c9
0c250455094d04190c510409025904050675046d0f6d04fd0c65040d0a8104490d7104a10c8d04c46904dd0355045d0a
8904090449045d0a5d04c10ca904c90c8504b5047904bd0f61042110e9043d06b504f501e1040506b9...

[jacogr]

That is from the node itself, not the API layers interacting with RPC - Object({"code": Number(-32702), "data": String("Exceeded max limit of 15728640")

Generally the RPC responses are limited as to not DDOS the nodes. staking state can indeed be "quite" large, as seen here, so unless the limit is adjusted upwards, there will be a problem if a single state entry exceeds the limits.

[dcolley]

this call will get the response:

curl --location --request POST 'https://rpc.polkadot.io' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","id":18,"method":"state_getStorage","params":["0xede8e4fdc3c8b556f0ce2f77fc2575e3b3d1c643c0d45e2bb9269ac86c1dcda0b4def25cfda6ef3a00000000","0x5e55c8c7b5f523cd710e29516f2f0c559c24b7c0ae819926f451e9428f01c4d3"]}'

Above works on command line, but jsonrpsee throws "Response is too big" error

[ggwpez]

polkadot/utlls/staking-miner v0.9.27 has the same problem on Polkadot with state_getStorage:

So you mean there is a but in the staking miner? Then please open an issue for it.
I think the staking miner needs additional flags on its connected node, something like --ws-max-out-buffer-capacity 1000 --rpc-max-payload 1000.
Not sure where that is documented.

[dcolley]

@ggwpez thanks for that, I'll try that on the polkadot validator and see if it helps.

[kianenigma]

As noted this is fixable with --rpc-max-payload.

[dcolley]

polkadot -h reveals

so better to use

  --rpc-max-request-size <size in MB> \
  --rpc-max-response-size <size in MB> \