trie: reduce the memory allocation in trie hashing#31902
trie: reduce the memory allocation in trie hashing#31902rjl493456442 merged 6 commits intoethereum:masterfrom
Conversation
|
After running a bit more, it turns out the IOwait is not relevant with the change. The PR is slightly slower though. 
|
| // hashData hashes the provided data | ||
| func (h *hasher) hashData(data []byte) hashNode { | ||
| n := make(hashNode, 32) | ||
| func (h *hasher) hashData(data []byte) []byte { |
There was a problem hiding this comment.
Could we have made the return type here and hashNode a [32]byte? It probably wouldn't save much on the total size of allocations, but it will reduce the number of individual allocations.
There was a problem hiding this comment.
Sounds reasonable to me. But I don't want to do it in this pr, otherwise it will end up as a giant PR I think.
| func (h *hasher) hashFullNodeChildren(n *fullNode) *fullNode { | ||
| var children [17]node | ||
| func (h *hasher) encodeFullNode(n *fullNode) []byte { | ||
| fn := fnEncoderPool.Get().(*fullnodeEncoder) |
There was a problem hiding this comment.
this introduces a synchronization point for all parallel hashers, which is probably why the performance degraded slightly. I think we can do better.
There was a problem hiding this comment.
Really? I think sync.Pool should be efficient for concurrent usage.
There was a problem hiding this comment.
Well, it might be efficient in what it's doing but it is certainly not free.
rjl493456442
force-pushed
the
trie-hasher-2
branch
from
6b5ccf9 to
17b0a10
Compare
|
My 2c is, optimizing to reduce GC churn at the cost of increased runtime is not really an optimization. We just moved the cost of GC churn to somewhere else (the sync pool in this case). |
rjl493456442
force-pushed
the
trie-hasher-2
branch
from
79a934b to
b7aa034
Compare
|
This pull request optimizes trie hashing by reducing memory allocation overhead. Specifically: - define a fullNodeEncoder pool to reuse encoders and avoid memory allocations. - simplify the encoding logic for shortNode and fullNode by getting rid of the Go interfaces.
This pull request optimizes trie hashing by reducing memory allocation overhead. Specifically: - define a fullNodeEncoder pool to reuse encoders and avoid memory allocations. - simplify the encoding logic for shortNode and fullNode by getting rid of the Go interfaces.
3 participants
This pull request optimizes trie hashing by reducing memory allocation overhead. Specifically:
Benchmark results
Memory profile