core trie rlp: patches from go-ethereum #1068
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change adds a code generator tool for creating EncodeRLP method implementations. The generated methods will behave identically to the reflect-based encoder, but run faster because there is no reflection overhead. Package rlp now provides the EncoderBuffer type for incremental encoding. This is used by generated code, but the new methods can also be useful for hand-written encoders. There is also experimental support for generating DecodeRLP, and some new methods have been added to the existing Stream type to support this. Creating decoders with rlpgen is not recommended at this time because the generated methods create very poor error reporting. More detail about package rlp changes: * rlp: externalize struct field processing / validation This adds a new package, rlp/internal/rlpstruct, in preparation for the RLP encoder generator. I think the struct field rules are subtle enough to warrant extracting this into their own package, even though it means that a bunch of adapter code is needed for converting to/from rlpstruct.Type. * rlp: add more decoder methods (for rlpgen) This adds new methods on rlp.Stream: - Uint64, Uint32, Uint16, Uint8, BigInt - ReadBytes for decoding into []byte - MoreDataInList - useful for optional list elements * rlp: expose encoder buffer (for rlpgen) This exposes the internal encoder buffer type for use in EncodeRLP implementations. The new EncoderBuffer type is a sort-of 'opaque handle' for a pointer to encBuffer. It is implemented this way to ensure the global encBuffer pool is handled correctly.
This change makes use of the new code generator rlp/rlpgen to improve the
performance of RLP encoding for Header and StateAccount. It also speeds up
encoding of ReceiptForStorage using the new rlp.EncoderBuffer API.
The change is much less transparent than I wanted it to be, because Header and
StateAccount now have an EncodeRLP method defined with pointer receiver. It
used to be possible to encode non-pointer values of these types, but the new
method prevents that and attempting to encode unadressable values (even if
part of another value) will return an error. The error can be surprising and may
pop up in places that previously didn't expect any errors.
To make things work, I also needed to update all code paths (mostly in unit tests)
that lead to encoding of non-pointer values, and pass a pointer instead.
Benchmark results:
name old time/op new time/op delta
EncodeRLP/legacy-header-8 328ns ± 0% 237ns ± 1% -27.63% (p=0.000 n=8+8)
EncodeRLP/london-header-8 353ns ± 0% 247ns ± 1% -30.06% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 237ns ± 0% 123ns ± 0% -47.86% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 297ns ± 0% 301ns ± 1% +1.39% (p=0.000 n=8+8)
name old speed new speed delta
EncodeRLP/legacy-header-8 1.66GB/s ± 0% 2.29GB/s ± 1% +38.19% (p=0.000 n=8+8)
EncodeRLP/london-header-8 1.55GB/s ± 0% 2.22GB/s ± 1% +42.99% (p=0.000 n=8+8)
EncodeRLP/receipt-for-storage-8 38.0MB/s ± 0% 64.8MB/s ± 0% +70.48% (p=0.000 n=8+7)
EncodeRLP/receipt-full-8 910MB/s ± 0% 897MB/s ± 1% -1.37% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
EncodeRLP/legacy-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/london-header-8 0.00B 0.00B ~ (all equal)
EncodeRLP/receipt-for-storage-8 64.0B ± 0% 0.0B -100.00% (p=0.000 n=8+8)
EncodeRLP/receipt-full-8 320B ± 0% 320B ± 0% ~ (all equal)
This change speeds up trie hashing and all other activities that require RLP encoding of trie nodes by approximately 20%. The speedup is achieved by avoiding reflection overhead during node encoding. The interface type trie.node now contains a method 'encode' that works with rlp.EncoderBuffer. Management of EncoderBuffers is left to calling code. trie.hasher, which is pooled to avoid allocations, now maintains an EncoderBuffer. This means memory resources related to trie node encoding are tied to the hasher pool. Co-authored-by: Felix Lange <[email protected]>
unclezoro
approved these changes
Aug 26, 2022
brilliant-lx
approved these changes
Aug 31, 2022
tolunghk
added a commit
to tolunghk/bsc
that referenced
this pull request
Oct 7, 2022
commit f3fd0f8 Author: zjubfd <[email protected]> Date: Fri Oct 7 11:54:15 2022 +0800 release: prepare for release v1.1.15 (bnb-chain#1112) commit 5b35b39 Author: zjubfd <[email protected]> Date: Fri Oct 7 11:46:38 2022 +0800 nanofork: add black list account (bnb-chain#1109) commit 3354924 Merge: 3d3f0a8 781e567 Author: zjubfd <[email protected]> Date: Wed Sep 21 18:15:33 2022 +0800 Merge pull request bnb-chain#1090 from bnb-chain/develop prepare for release v1.1.14 commit 781e567 Author: zjubfd <[email protected]> Date: Wed Sep 21 15:54:03 2022 +0800 release: update version and changelogs commit 8375939 Author: yutianwu <[email protected]> Date: Wed Sep 21 15:44:34 2022 +0800 overwrite HTTPHost in config.toml via cli (bnb-chain#999) commit 3c7bccb Author: Leon <[email protected]> Date: Mon Sep 19 11:49:31 2022 +0800 core: fix potential goroutine leak (bnb-chain#1079) * core: fix potential goroutine leak * core: fix 0 index, and add ut for routineleaking commit b051dc6 Author: zhaochonghe <[email protected]> Date: Thu May 19 15:25:22 2022 +0800 core: fix the order of address in queue (#24907) reverse the order of address in queue commit 4fb1e9d Author: Håvard Anda Estensen <[email protected]> Date: Tue May 17 13:56:52 2022 +0200 all: use strings.EqualFold for string comparison (#24890) commit 57da9c0 Author: s7v7nislands <[email protected]> Date: Mon May 16 17:39:07 2022 +0800 core/asm: use strings.Builder and fix godoc issues (#24861) commit ef24fa3 Author: Sina Mahmoodi <[email protected]> Date: Thu May 12 21:10:08 2022 +0200 graphql: fix long literal passed in a variable (#24864) commit 461835a Author: s7v7nislands <[email protected]> Date: Wed May 11 13:03:35 2022 +0800 core/vm: clean up some dead functions (#24851) commit 4feeaf3 Author: aaronbuchwald <[email protected]> Date: Tue May 10 10:26:48 2022 -0400 core/vm: update benchmark to use Errorf instead of Sprintf (#24845) commit 5d980a0 Author: Sina Mahmoodi <[email protected]> Date: Tue May 10 10:46:26 2022 +0200 internal: drop TestSignCliqueBlock (#24837) commit e8b9338 Author: Marius van der Wijden <[email protected]> Date: Fri May 6 17:20:41 2022 +0200 core/state/snapshot: fix race condition (#24685) Fixes three race conditions found through fuzzing by David Theodore commit 5a06ca3 Author: Rachel Franks <[email protected]> Date: Fri May 6 09:45:57 2022 -0500 abi: fix checks when all fields are indexed (#24792) This PR fixes abi checks in the edge case where all arguments are indexed commit 8d19185 Author: Martin Holst Swende <[email protected]> Date: Fri May 6 15:23:43 2022 +0200 eth/fetcher: avoid hang in tests (partial fix for #23331) (#23351) * eth/fetcher: fix test to avoid hanging. Partial fix for #23331 * eth/filters: avoid dangling goroutines * eth/fetcher: revert closing of proceed commit f781ea8 Author: ImanSharaf <[email protected]> Date: Thu May 5 10:44:36 2022 -0700 p2p/simulations: escape mockerType value from request (#24822) Co-authored-by: Felix Lange <[email protected]> commit 0981812 Author: EXEC <[email protected]> Date: Thu May 5 23:58:43 2022 +0800 eth/filters: remove explicit continue label in filterLogs (#24795) The loop label can be removed because this 'continue' statement is not in a nested loop. commit bd22292 Author: Evgeny Kolyakov <[email protected]> Date: Wed May 4 00:37:37 2022 +0300 build: fix formatted logs (#24807) Changed `log.Fatal` to `log.Fatalf()` as it has a parameter... commit cea8319 Author: Martin Holst Swende <[email protected]> Date: Tue May 3 12:12:40 2022 +0200 cmd/utils: double limit on free-disk monitor (#24781) This PR doubles the limit on which to trigger automatic shutdown, and also changes the timer to run once every 30s instead of 60s. commit 58f1e8b Author: EXEC <[email protected]> Date: Tue May 3 17:17:24 2022 +0800 docker: speed up docker image build (#24796) This PR improves the docker build speed for repeated builds where go.mod and go.sum do no change, by placing the downloaded dependencies in a lower layer commit f67e303 Author: hero5512 <[email protected]> Date: Fri Apr 29 16:07:52 2022 +0800 eth/filters: remove unused struct fields (#24782) commit a181eee Author: Joshua Gutow <[email protected]> Date: Wed Apr 27 02:51:24 2022 -0700 ethclient/gethclient: return storage proofs in GetProof (#24697) Storage proofs were being unmarshalled from the RPC form to the go struct, but were not being included in the final returned struct. commit 34420d0 Author: tia-99 <[email protected]> Date: Wed Apr 27 16:56:50 2022 +0800 core/types: fix unhandled errors in TestTransactionCoding (#24692) commit ae31b5f Author: henopied <[email protected]> Date: Mon Apr 25 04:05:47 2022 -0500 p2p: fix type of DiscSubprotocolError (#24747) It was 'int' accidentally, should be DiscReason instead. commit 6aae1bc Author: Nikita Kozhemyakin <[email protected]> Date: Mon Apr 25 10:35:04 2022 +0300 mobile: fix receipt encoding to json (#24701) commit 30c695c Author: jwasinger <[email protected]> Date: Sun Apr 24 21:16:49 2022 -1000 cmd/evm: ensure input length is even (#24721) * cmd/evm: ensure input length is even * cmd/evm: minor nit + lintfix Co-authored-by: Martin Holst Swende <[email protected]> commit 3766168 Author: Emmanuel T Odeke <[email protected]> Date: Fri Apr 22 00:30:15 2022 +0100 eth/tracers/logger: remove unnecessary comparisons in accessList.equal (#24663) This change removes extraneous/unnecessary checks for equality when comparing 2 accessList values A and B. Given that we validate that their lengths of A and B are equal, if so and if every element in A is in B, reflexively every element in B is already in A. If that weren't the case and an element g existed in A but not in B, that would mean that there is an extra element and hence a mathematical contradiction. Fixes #24658 commit 3081421 Author: Darioush Jalali <[email protected]> Date: Wed Apr 20 09:12:06 2022 -0500 trie: remove unused makeHashNode (#24702) commit a723243 Author: rjl493456442 <[email protected]> Date: Tue Apr 12 16:34:07 2022 +0800 core: fix benchmark panic (#24657) This PR fixes a few panics in the chain marker benchmarks. The root cause for panic is in chain marker the genesis header/block is not accessible, while it's expected to be obtained in tests. So this PR avoids touching genesis header at all to avoid panic. commit f37f2a9 Author: Felix Lange <[email protected]> Date: Mon Apr 11 20:19:13 2022 +0200 core/types: make "miner" optional in Header JSON (#24666) "miner" is not set for pending block responses in some cases. Fixes #24632 commit a41619f Author: rjl493456442 <[email protected]> Date: Fri Apr 8 21:48:52 2022 +0800 les: fix panic in ultralight client sync (#24641) commit f310921 Author: Péter Szilágyi <[email protected]> Date: Thu Apr 7 09:36:44 2022 +0300 core/state/snapshot: remove noop map item assignment commit 211ab45 Author: Péter Szilágyi <[email protected]> Date: Wed Apr 6 10:18:57 2022 +0300 eth/fetcher: if peers never respond, drop them commit c54fd7f Author: Nic Jansma <[email protected]> Date: Wed Mar 30 04:28:27 2022 -0400 eth: change snapshot extension registration failure to warning instead of error (#24475) * core: Change Snapshot extension registration failed to Debug * Update eth/handler.go Co-authored-by: Martin Holst Swende <[email protected]> commit 3b4c683 Author: Martin Holst Swende <[email protected]> Date: Tue Mar 29 22:36:55 2022 +0200 eth/tracers/logger: use omitempty to reduce log bloat (#24547) Makes the evm json output less verbose: omitting output of `memory` and `returndata` in case they are empty. commit 3d5896a Author: ucwong <[email protected]> Date: Wed Mar 30 04:33:19 2022 +0800 go.mod : upnp 1.0.3 stable version (#24573) commit 3d3f0a8 Author: zjubfd <[email protected]> Date: Wed Sep 14 14:49:07 2022 +0800 ci: add curl in docker file commit b8020b9 Author: zjubfd <[email protected]> Date: Tue Sep 6 16:42:21 2022 +0800 fix: remove fixed curl version in docker file commit 6f6bbcf Merge: 723863e aa835e0 Author: zjubfd <[email protected]> Date: Fri Sep 2 11:31:36 2022 +0800 Merge pull request bnb-chain#1056 from bnb-chain/develop prepare for release v1.1.13 commit aa835e0 Author: Roshan <[email protected]> Date: Wed Aug 31 17:04:48 2022 +0800 upgrade: update version and CHANGELOG.md (bnb-chain#1073) commit fd6fa7c Author: Roshan <[email protected]> Date: Wed Aug 31 14:42:23 2022 +0800 fix: update cross chain logic of native stake (bnb-chain#1066) commit 755df16 Merge: 64549a7 24da9ec Author: zjubfd <[email protected]> Date: Wed Aug 31 13:30:59 2022 +0800 Merge pull request bnb-chain#1068 from qinglin89/dev-rlp commit 64549a7 Author: Leon <[email protected]> Date: Wed Aug 31 13:30:25 2022 +0800 statedb: get rid of activestate, stopPrefetcher manually (bnb-chain#1006) commit b485651 Author: bittrainee <[email protected]> Date: Wed Aug 31 12:38:09 2022 +0800 fix: fix potential goroutine leak (bnb-chain#1067) commit f457598 Author: dylanhuang <[email protected]> Date: Wed Aug 31 12:37:20 2022 +0800 txpool: reheap the priced list if london fork not enabled (bnb-chain#1070) commit 24da9ec Author: qinglin89 <[email protected]> Date: Fri Aug 26 14:38:01 2022 +0800 fix: gomod add packages commit da57d11 Author: Qian Bin <[email protected]> Date: Wed Mar 9 21:45:17 2022 +0800 rlp trie: faster trie node encoding (#24126) This change speeds up trie hashing and all other activities that require RLP encoding of trie nodes by approximately 20%. The speedup is achieved by avoiding reflection overhead during node encoding. The interface type trie.node now contains a method 'encode' that works with rlp.EncoderBuffer. Management of EncoderBuffers is left to calling code. trie.hasher, which is pooled to avoid allocations, now maintains an EncoderBuffer. This means memory resources related to trie node encoding are tied to the hasher pool. Co-authored-by: Felix Lange <[email protected]> commit 25337f5 Author: Felix Lange <[email protected]> Date: Fri Feb 18 08:10:26 2022 +0100 core/types: faster RLP encoding of Header, StateAcccount, Re... (#24420) This change makes use of the new code generator rlp/rlpgen to improve the performance of RLP encoding for Header and StateAccount. It also speeds up encoding of ReceiptForStorage using the new rlp.EncoderBuffer API. The change is much less transparent than I wanted it to be, because Header and StateAccount now have an EncodeRLP method defined with pointer receiver. It used to be possible to encode non-pointer values of these types, but the new method prevents that and attempting to encode unadressable values (even if part of another value) will return an error. The error can be surprising and may pop up in places that previously didn't expect any errors. To make things work, I also needed to update all code paths (mostly in unit tests) that lead to encoding of non-pointer values, and pass a pointer instead. Benchmark results: name old time/op new time/op delta EncodeRLP/legacy-header-8 328ns ± 0% 237ns ± 1% -27.63% (p=0.000 n=8+8) EncodeRLP/london-header-8 353ns ± 0% 247ns ± 1% -30.06% (p=0.000 n=8+8) EncodeRLP/receipt-for-storage-8 237ns ± 0% 123ns ± 0% -47.86% (p=0.000 n=8+7) EncodeRLP/receipt-full-8 297ns ± 0% 301ns ± 1% +1.39% (p=0.000 n=8+8) name old speed new speed delta EncodeRLP/legacy-header-8 1.66GB/s ± 0% 2.29GB/s ± 1% +38.19% (p=0.000 n=8+8) EncodeRLP/london-header-8 1.55GB/s ± 0% 2.22GB/s ± 1% +42.99% (p=0.000 n=8+8) EncodeRLP/receipt-for-storage-8 38.0MB/s ± 0% 64.8MB/s ± 0% +70.48% (p=0.000 n=8+7) EncodeRLP/receipt-full-8 910MB/s ± 0% 897MB/s ± 1% -1.37% (p=0.000 n=8+8) name old alloc/op new alloc/op delta EncodeRLP/legacy-header-8 0.00B 0.00B ~ (all equal) EncodeRLP/london-header-8 0.00B 0.00B ~ (all equal) EncodeRLP/receipt-for-storage-8 64.0B ± 0% 0.0B -100.00% (p=0.000 n=8+8) EncodeRLP/receipt-full-8 320B ± 0% 320B ± 0% ~ (all equal) commit 1d6c01a Author: Felix Lange <[email protected]> Date: Wed Feb 16 18:14:12 2022 +0100 rlp/rlpgen: RLP encoder code generator (#24251) This change adds a code generator tool for creating EncodeRLP method implementations. The generated methods will behave identically to the reflect-based encoder, but run faster because there is no reflection overhead. Package rlp now provides the EncoderBuffer type for incremental encoding. This is used by generated code, but the new methods can also be useful for hand-written encoders. There is also experimental support for generating DecodeRLP, and some new methods have been added to the existing Stream type to support this. Creating decoders with rlpgen is not recommended at this time because the generated methods create very poor error reporting. More detail about package rlp changes: * rlp: externalize struct field processing / validation This adds a new package, rlp/internal/rlpstruct, in preparation for the RLP encoder generator. I think the struct field rules are subtle enough to warrant extracting this into their own package, even though it means that a bunch of adapter code is needed for converting to/from rlpstruct.Type. * rlp: add more decoder methods (for rlpgen) This adds new methods on rlp.Stream: - Uint64, Uint32, Uint16, Uint8, BigInt - ReadBytes for decoding into []byte - MoreDataInList - useful for optional list elements * rlp: expose encoder buffer (for rlpgen) This exposes the internal encoder buffer type for use in EncodeRLP implementations. The new EncoderBuffer type is a sort-of 'opaque handle' for a pointer to encBuffer. It is implemented this way to ensure the global encBuffer pool is handled correctly. commit f2b1f3f Merge: 2e22fe3 94885ae Author: zjubfd <[email protected]> Date: Tue Aug 23 10:43:14 2022 +0800 Merge pull request bnb-chain#1061 from qinglin89/dev-merge fix: upstream patches commit 94885ae Author: Zhang Zhuo <[email protected]> Date: Tue Mar 29 16:21:11 2022 +0800 core/vm: fix sstore gas comment type (#24583) commit 9c72720 Author: rjl493456442 <[email protected]> Date: Wed Mar 23 01:58:05 2022 +0800 eth/downloader: ignore zero size header batch for importing (#24569) * eth/downloader: ignore zero size header batch for importing * core, light: reject empty header batch for importing commit c1d9690 Author: rjl493456442 <[email protected]> Date: Wed Mar 16 18:13:10 2022 +0800 les, tests: fix vflux fuzzer by removing unnecessary panic (#24537) commit 3572eb7 Author: Guifel <[email protected]> Date: Tue Mar 15 10:27:30 2022 +0100 eth/tracers: use hex encoding for debug_traceBlock argument (#24517) commit f8e2104 Author: ligi <[email protected]> Date: Tue Mar 15 10:24:45 2022 +0100 build: fix linter install on windows (#24523) commit d130c0e Author: rjl493456442 <[email protected]> Date: Tue Mar 15 17:23:37 2022 +0800 trie: fix two issues in trie iterator (#24539) * trie: fix memory leak in trie iterator In the trie iterator, live nodes are tracked in a stack while iterating. Popped node states should be explictly set to nil in order to get garbage-collected. * trie: fix empty trie iterator commit 799f041 Author: s7v7nislands <[email protected]> Date: Fri Mar 11 23:24:13 2022 +0800 rpc: fix defer in test (#24490) Co-authored-by: Felix Lange <[email protected]> commit 0e65495 Author: Martin Holst Swende <[email protected]> Date: Thu Mar 10 12:46:48 2022 +0100 eth/protocols/snap: avoid estimating infinite percentage commit 9843a36 Author: Zachinquarantine <[email protected]> Date: Tue Mar 8 18:42:14 2022 -0500 mobile: remove deprecated Stop function (#24369) commit 413a42d Author: Ceyhun Onur <[email protected]> Date: Wed Mar 9 02:39:34 2022 +0300 core/rawdb: add HasCode, HashTrieNode and use them where possible (#24454) commit 2e22fe3 Author: dylanhuang <[email protected]> Date: Mon Aug 22 11:25:31 2022 +0800 ci: fix the pattern of commit lint (bnb-chain#1063) commit c3e3b11 Author: dylanhuang <[email protected]> Date: Mon Aug 22 11:24:50 2022 +0800 consensus: fix the GasLimitBoundDivisor (bnb-chain#1060) commit 02b31ab Author: dylanhuang <[email protected]> Date: Mon Aug 22 11:24:18 2022 +0800 test: fix TestOfflineBlockPrune (bnb-chain#1062) commit e5f4fde Author: Felföldi Zsolt <[email protected]> Date: Tue Mar 1 10:36:42 2022 +0100 les/vflux/client: fix goroutine leak in testIter (#24449) commit 62d66e4 Author: Guruprasad Kamath <[email protected]> Date: Tue Mar 1 10:36:11 2022 +0100 appveyor.yml: fetch sub-modules recursively (#24451) commit 7bb1463 Author: Sina Mahmoodi <[email protected]> Date: Tue Feb 22 10:45:11 2022 +0100 graphql: fix nonce for pending accounts (#24443) commit c1dd16f Author: b1ackd0t <[email protected]> Date: Tue Feb 15 12:44:22 2022 +0300 internal/ethapi: fix incorrect type on empty slice (#24372) * Fixes #24368 Signed-off-by: 0x6f736f646f <[email protected]> * Update internal/ethapi/api.go Co-authored-by: Martin Holst Swende <[email protected]> commit a1db38a Author: zhiqiangxu <[email protected]> Date: Mon Feb 14 19:05:48 2022 +0800 p2p: reduce the scope of variable dialPubkey (#24385) dialPubkey isn't used anywhere else after dialDest.Load, so it should be safe to restrict its scope to the if clause. commit 788b771 Author: Martin Holst Swende <[email protected]> Date: Mon Feb 14 09:22:57 2022 +0100 core/state: fix read-meters + simplify code (#24304) commit e64fde1 Author: setunapo <[email protected]> Date: Thu Aug 18 15:28:13 2022 +0800 ci: allow merge pull request commit 11d2bae Author: Roshan <[email protected]> Date: Thu Aug 18 14:36:45 2022 +0800 bep153: Native Staking on BSC (bnb-chain#1051) commit ddbe6c4 Author: setunapo <[email protected]> Date: Wed Aug 17 15:19:50 2022 +0800 ci: no type-case check for scope. commit dbb047b Merge: 05d8399 895c077 Author: joeycli <[email protected]> Date: Wed Aug 17 14:29:11 2022 +0800 Merge pull request bnb-chain#1053 from bnb-chain/fix_opendatabase_fail state: fix offline tool start failed when start with pruneancient commit 05d8399 Author: dylanhuang <[email protected]> Date: Tue Aug 16 19:01:24 2022 +0800 ci: add commit message lint (bnb-chain#1044) commit 895c077 Author: joeycli <[email protected]> Date: Tue Aug 16 14:02:45 2022 +0800 fix bug bnb-chain#1034, offline tool start failed when start with pruneancient commit 76e3c90 Author: Leon <[email protected]> Date: Fri Jul 29 14:21:37 2022 +0800 [R4R] Pipecommit enable trie prefetcher (bnb-chain#992) commit 77c8372 Author: setunapo <[email protected]> Date: Tue Jul 26 22:23:55 2022 +0800 Trie prefetch on state pretch (bnb-chain#996) * feature: do trie prefetch on state prefetch Currently, state prefetch just pre execute the transactions and discard the results. It is helpful to increase the snapshot cache hit rate. It would be more helpful, if it can do trie prefetch at the same time, since the it will preload the trie node and build the trie tree in advance. This patch is to implement it, by reusing the main trie prefetch and doing finalize after transaction is executed. * some code improvements for trie prefetch ** increase pendingSize before dispatch tasks ** use throwaway StateDB for TriePrefetchInAdvance and remove the prefetcherLock ** remove the necessary drain operation in trie prefetch mainloop, trie prefetcher won't be used after close. commit df3e1be Author: setunapo <[email protected]> Date: Fri Jul 15 19:17:08 2022 +0800 [Feature]: Improve trie prefetch (bnb-chain#952) * trie prefetcher for From/To address in advance We found that trie prefetch could be not fast enough, especially trie prefetch of the outer big state trie tree. Instead of do trie prefetch until a transaction is finalized, we could do trie prefetch in advance. Try to prefetch the trie node of the From/To accounts, since their root hash are most likely to be changed. * Parallel TriePrefetch for large trie update. Currently, we create a subfetch for each account address to do trie prefetch. If the address has very large state change, trie prefetch could be not fast enough, e.g. a contract modified lots of KV pair or a large number of account's root hash is changed in a block. With this commit, there will be children subfetcher created to do trie prefetch in parallell if the parent subfetch's workload exceed the threshold. * some improvemnts of parallel trie prefetch implementation 1.childrenLock is removed, since it is not necessary APIs of triePrefetcher is not thread safe, they should be used sequentially. A prefetch will be interrupted by trie() or clos(), so we only need mark it as interrupted and check before call scheduleParallel to avoid the concurrent access to paraChildren 2.rename subfetcher.children to subfetcher.paraChildren 3.use subfetcher.pendingSize to replace totalSize & processedIndex 4.randomly select the start child to avoid always feed the first one 5.increase threshold and capacity to avoid create too many child routine * fix review comments ** nil check refine ** create a separate routine for From/To prefetch, avoid blocking the cirtical path * remove the interrupt member * not create a signer for each transaction * some changes to triePrefetcher ** remove the abortLoop, move the subfetcher abort operation into mainLoop since we want to make subfetcher's create & schedule & abort within a loop to avoid concurrent access locks. ** no wait subfetcher's term signal in abort() it could speed up the close by closing subfetcher concurrently. we send stop signnal to all subfetchers in burst and wait their term signal later. * some coding improve for subfetcher.scheduleParallel * fix a UT crash of s.prefetcher == nil * update parallel trie prefetcher configuration tested with different combination of parallelTriePrefetchThreshold & parallelTriePrefetchCapacity, found the most efficient configure could be: parallelTriePrefetchThreshold = 10 parallelTriePrefetchCapacity = 20 * fix review comments: code refine commit 51bfeca Author: kyrie-yl <[email protected]> Date: Thu Jul 14 17:09:44 2022 +0800 broadcast block before commit block and add metrics (bnb-chain#975) Signed-off-by: cryyl <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Apply some patches from [email protected]
Rationale
N/A
Example
N/A
Changes
Notable changes: