Skip to content

Pull gossip queues for better throughput - #5350

Merged
wemeetagain merged 6 commits into
unstablefrom
tuyen/new_gossip_queues_merge_to_unstable
Apr 7, 2023
Merged

Pull gossip queues for better throughput#5350
wemeetagain merged 6 commits into
unstablefrom
tuyen/new_gossip_queues_merge_to_unstable

Conversation

@twoeths

@twoeths twoeths commented Apr 7, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Motivation

Get back #5195 with new improvement that make it comparable performance to stable

Description

This is just an official merge of tuyen/new_gossip_queues branch to unstable. It includes the below PRs:

Closes #5247

Note: merge this with Merge Commit strategy

Test Result on feat1

  • Gossip Block Process Time

Screenshot 2023-04-07 at 15 49 59

vs. stable
Screenshot 2023-04-07 at 15 50 27

  • Notify New Payload duration

Screenshot 2023-04-07 at 15 55 45

vs stable
Screenshot 2023-04-07 at 15 56 09

twoeths and others added 6 commits April 7, 2023 15:03
@twoeths
Revert "Revert "Pull gossip queues for better throughput (#5195)" (#5305
415d120 
)"

This reverts commit b861ab8.
@twoeths
Process gossip block immediately (#5263)
34b8b92 
* Process gossip block immediately

* Process beacon_block_and_blobs_sidecar gossip object immediately

* New WorkOpts type

* Check canAceptWork once per executeWork()
@twoeths
Handle gossip messages asynchronously (#5316)
a02353c 
* Handle gossip messages async

* Add comments
@twoeths @wemeetagain
Cache gossip validation by attestation data (#5333)
c7390ca 
* Add computeSigningRoot benchmark test

* Add utilities to extract AttestationData hash from serialized data

* Implement SeenAttestationDatas

* Do not cache state in AttestationDataCacheEntry

* Add memory test

* Fix getGossipDataHashFns()

* Cache signingRoot and attDataRootHex

* Apply SeenAttestationDatas cache to validateGossipAggregateAndProof()

* Cache AttestationDatas of 3 most recent slots

* Unit test SeenAttestationDatas

* chain.attDataCacheSlotDistance cli option

* Type gossip validation functions result

* Use cached attDataRootHex in attestation opPools

* Pass gossip serialized data to gossip validation function

* slight refactor

* Fix MockBeaconChain

* Fix lint in MockBeaconChain

* Fix validateGossipAttestation unit tests

---------

Co-authored-by: Cayman <caymannava@gmail.com>
@twoeths
Handle unknown block root attestations (#5327)
8f5a769 
* Add sszBytes utils

* Unknown Block Root queues

* Use Uint8Array.subarray()

* Merge tuyen/new_gossip_queues

* Avoid forEach pattern

* Process unknown attestations: yield to event loop

* Fix metrics

* Name variable and function as *attDataBase64 everywhere

* Handle invalid serialized data

* Check ssz data length of gossip messages
@twoeths
Fix lint
d940f3b
@github-actions

github-actions Bot commented Apr 7, 2023

Copy link
Copy Markdown
Contributor

Performance Report

鉁旓笍 no performance regression detected

Full benchmark results
Benchmark suite Current: 15f5818 Previous: 83b29d7 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 584.07 us/op 532.79 us/op 1.10
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 62.326 us/op 45.572 us/op 1.37
BLS verify - blst-native 1.2968 ms/op 1.2161 ms/op 1.07
BLS verifyMultipleSignatures 3 - blst-native 2.6392 ms/op 2.4880 ms/op 1.06
BLS verifyMultipleSignatures 8 - blst-native 5.6155 ms/op 5.3015 ms/op 1.06
BLS verifyMultipleSignatures 32 - blst-native 20.485 ms/op 19.284 ms/op 1.06
BLS aggregatePubkeys 32 - blst-native 27.649 us/op 26.094 us/op 1.06
BLS aggregatePubkeys 128 - blst-native 105.76 us/op 100.66 us/op 1.05
getAttestationsForBlock 59.745 ms/op 55.172 ms/op 1.08
isKnown best case - 1 super set check 276.00 ns/op 259.00 ns/op 1.07
isKnown normal case - 2 super set checks 267.00 ns/op 250.00 ns/op 1.07
isKnown worse case - 16 super set checks 270.00 ns/op 247.00 ns/op 1.09
CheckpointStateCache - add get delete 5.7000 us/op 5.0540 us/op 1.13
validate gossip signedAggregateAndProof - struct 2.8565 ms/op 2.7805 ms/op 1.03
validate gossip attestation - struct 1.3575 ms/op 1.3336 ms/op 1.02
pickEth1Vote - no votes 1.3909 ms/op 1.3820 ms/op 1.01
pickEth1Vote - max votes 10.988 ms/op 11.137 ms/op 0.99
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.6186 ms/op 9.1947 ms/op 1.05
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 16.404 ms/op 15.669 ms/op 1.05
pickEth1Vote - Eth1Data fastSerialize value x2048 717.14 us/op 722.30 us/op 0.99
pickEth1Vote - Eth1Data fastSerialize tree x2048 8.9775 ms/op 5.4928 ms/op 1.63
bytes32 toHexString 583.00 ns/op 570.00 ns/op 1.02
bytes32 Buffer.toString(hex) 415.00 ns/op 448.00 ns/op 0.93
bytes32 Buffer.toString(hex) from Uint8Array 614.00 ns/op 628.00 ns/op 0.98
bytes32 Buffer.toString(hex) + 0x 407.00 ns/op 420.00 ns/op 0.97
Object access 1 prop 0.17600 ns/op 0.17400 ns/op 1.01
Map access 1 prop 0.17000 ns/op 0.16200 ns/op 1.05
Object get x1000 6.8360 ns/op 6.7840 ns/op 1.01
Map get x1000 0.58900 ns/op 0.65300 ns/op 0.90
Object set x1000 58.753 ns/op 58.210 ns/op 1.01
Map set x1000 46.344 ns/op 45.764 ns/op 1.01
Return object 10000 times 0.24550 ns/op 0.24440 ns/op 1.00
Throw Error 10000 times 4.2840 us/op 4.3166 us/op 0.99
fastMsgIdFn sha256 / 200 bytes 3.4590 us/op 3.5640 us/op 0.97
fastMsgIdFn h32 xxhash / 200 bytes 312.00 ns/op 306.00 ns/op 1.02
fastMsgIdFn h64 xxhash / 200 bytes 447.00 ns/op 443.00 ns/op 1.01
fastMsgIdFn sha256 / 1000 bytes 11.984 us/op 11.745 us/op 1.02
fastMsgIdFn h32 xxhash / 1000 bytes 440.00 ns/op 415.00 ns/op 1.06
fastMsgIdFn h64 xxhash / 1000 bytes 532.00 ns/op 518.00 ns/op 1.03
fastMsgIdFn sha256 / 10000 bytes 105.77 us/op 110.28 us/op 0.96
fastMsgIdFn h32 xxhash / 10000 bytes 2.0070 us/op 2.0320 us/op 0.99
fastMsgIdFn h64 xxhash / 10000 bytes 1.4470 us/op 1.5250 us/op 0.95
enrSubnets - fastDeserialize 64 bits 1.4130 us/op 1.5290 us/op 0.92
enrSubnets - ssz BitVector 64 bits 537.00 ns/op 557.00 ns/op 0.96
enrSubnets - fastDeserialize 4 bits 200.00 ns/op 202.00 ns/op 0.99
enrSubnets - ssz BitVector 4 bits 631.00 ns/op 578.00 ns/op 1.09
prioritizePeers score -10:0 att 32-0.1 sync 2-0 122.55 us/op 111.23 us/op 1.10
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 153.72 us/op 156.20 us/op 0.98
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 189.71 us/op 199.76 us/op 0.95
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 354.27 us/op 360.73 us/op 0.98
prioritizePeers score 0:0 att 64-1 sync 4-1 426.07 us/op 449.33 us/op 0.95
array of 16000 items push then shift 1.7305 us/op 1.7162 us/op 1.01
LinkedList of 16000 items push then shift 9.5900 ns/op 9.4980 ns/op 1.01
array of 16000 items push then pop 118.12 ns/op 120.24 ns/op 0.98
LinkedList of 16000 items push then pop 9.1560 ns/op 8.9230 ns/op 1.03
array of 24000 items push then shift 2.4760 us/op 2.3803 us/op 1.04
LinkedList of 24000 items push then shift 9.6490 ns/op 9.1720 ns/op 1.05
array of 24000 items push then pop 84.033 ns/op 82.713 ns/op 1.02
LinkedList of 24000 items push then pop 9.0720 ns/op 8.9810 ns/op 1.01
intersect bitArray bitLen 8 14.185 ns/op 13.980 ns/op 1.01
intersect array and set length 8 108.03 ns/op 85.866 ns/op 1.26
intersect bitArray bitLen 128 46.865 ns/op 44.977 ns/op 1.04
intersect array and set length 128 1.3258 us/op 1.2986 us/op 1.02
Buffer.concat 32 items 2.9400 us/op 3.4330 us/op 0.86
Uint8Array.set 32 items 2.2760 us/op 3.3200 us/op 0.69
pass gossip attestations to forkchoice per slot 2.9965 ms/op 3.2605 ms/op 0.92
computeDeltas 3.0017 ms/op 3.4222 ms/op 0.88
computeProposerBoostScoreFromBalances 1.8179 ms/op 1.9207 ms/op 0.95
altair processAttestation - 250000 vs - 7PWei normalcase 2.7634 ms/op 3.3193 ms/op 0.83
altair processAttestation - 250000 vs - 7PWei worstcase 4.1292 ms/op 4.2066 ms/op 0.98
altair processAttestation - setStatus - 1/6 committees join 141.47 us/op 146.70 us/op 0.96
altair processAttestation - setStatus - 1/3 committees join 288.28 us/op 289.00 us/op 1.00
altair processAttestation - setStatus - 1/2 committees join 371.34 us/op 379.14 us/op 0.98
altair processAttestation - setStatus - 2/3 committees join 469.96 us/op 476.01 us/op 0.99
altair processAttestation - setStatus - 4/5 committees join 656.51 us/op 666.29 us/op 0.99
altair processAttestation - setStatus - 100% committees join 788.56 us/op 778.51 us/op 1.01
altair processBlock - 250000 vs - 7PWei normalcase 17.334 ms/op 18.252 ms/op 0.95
altair processBlock - 250000 vs - 7PWei normalcase hashState 28.117 ms/op 27.620 ms/op 1.02
altair processBlock - 250000 vs - 7PWei worstcase 48.696 ms/op 50.597 ms/op 0.96
altair processBlock - 250000 vs - 7PWei worstcase hashState 68.557 ms/op 67.655 ms/op 1.01
phase0 processBlock - 250000 vs - 7PWei normalcase 2.3259 ms/op 2.3326 ms/op 1.00
phase0 processBlock - 250000 vs - 7PWei worstcase 29.860 ms/op 29.658 ms/op 1.01
altair processEth1Data - 250000 vs - 7PWei normalcase 501.29 us/op 516.36 us/op 0.97
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 10.218 us/op 9.5070 us/op 1.07
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 30.646 us/op 30.056 us/op 1.02
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 11.824 us/op 11.577 us/op 1.02
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 11.526 us/op 9.7920 us/op 1.18
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 116.63 us/op 108.29 us/op 1.08
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 783.91 us/op 671.64 us/op 1.17
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 925.55 us/op 910.71 us/op 1.02
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 913.95 us/op 918.29 us/op 1.00
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.4326 ms/op 2.4098 ms/op 1.01
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.5137 ms/op 1.5037 ms/op 1.01
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 4.3206 ms/op 3.9678 ms/op 1.09
Tree 40 250000 create 343.66 ms/op 345.63 ms/op 0.99
Tree 40 250000 get(125000) 196.33 ns/op 186.09 ns/op 1.05
Tree 40 250000 set(125000) 1.0760 us/op 935.87 ns/op 1.15
Tree 40 250000 toArray() 22.833 ms/op 17.971 ms/op 1.27
Tree 40 250000 iterate all - toArray() + loop 22.535 ms/op 20.496 ms/op 1.10
Tree 40 250000 iterate all - get(i) 78.318 ms/op 72.277 ms/op 1.08
MutableVector 250000 create 11.134 ms/op 11.434 ms/op 0.97
MutableVector 250000 get(125000) 6.5190 ns/op 6.3670 ns/op 1.02
MutableVector 250000 set(125000) 264.25 ns/op 261.81 ns/op 1.01
MutableVector 250000 toArray() 3.4522 ms/op 3.1852 ms/op 1.08
MutableVector 250000 iterate all - toArray() + loop 3.7077 ms/op 3.2803 ms/op 1.13
MutableVector 250000 iterate all - get(i) 1.5540 ms/op 1.5396 ms/op 1.01
Array 250000 create 3.0631 ms/op 3.1382 ms/op 0.98
Array 250000 clone - spread 1.0994 ms/op 1.2129 ms/op 0.91
Array 250000 get(125000) 0.57300 ns/op 0.58800 ns/op 0.97
Array 250000 set(125000) 0.68100 ns/op 0.66100 ns/op 1.03
Array 250000 iterate all - loop 87.152 us/op 86.474 us/op 1.01
effectiveBalanceIncrements clone Uint8Array 300000 34.256 us/op 34.986 us/op 0.98
effectiveBalanceIncrements clone MutableVector 300000 328.00 ns/op 350.00 ns/op 0.94
effectiveBalanceIncrements rw all Uint8Array 300000 170.62 us/op 168.03 us/op 1.02
effectiveBalanceIncrements rw all MutableVector 300000 84.793 ms/op 85.124 ms/op 1.00
phase0 afterProcessEpoch - 250000 vs - 7PWei 116.96 ms/op 118.04 ms/op 0.99
phase0 beforeProcessEpoch - 250000 vs - 7PWei 43.695 ms/op 49.380 ms/op 0.88
altair processEpoch - mainnet_e81889 338.11 ms/op 344.47 ms/op 0.98
mainnet_e81889 - altair beforeProcessEpoch 54.800 ms/op 78.458 ms/op 0.70
mainnet_e81889 - altair processJustificationAndFinalization 19.268 us/op 26.763 us/op 0.72
mainnet_e81889 - altair processInactivityUpdates 5.8449 ms/op 6.2258 ms/op 0.94
mainnet_e81889 - altair processRewardsAndPenalties 49.657 ms/op 67.685 ms/op 0.73
mainnet_e81889 - altair processRegistryUpdates 3.0930 us/op 6.0180 us/op 0.51
mainnet_e81889 - altair processSlashings 567.00 ns/op 1.1170 us/op 0.51
mainnet_e81889 - altair processEth1DataReset 713.00 ns/op 1.1300 us/op 0.63
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2745 ms/op 1.9814 ms/op 0.64
mainnet_e81889 - altair processSlashingsReset 5.9110 us/op 7.3560 us/op 0.80
mainnet_e81889 - altair processRandaoMixesReset 6.9720 us/op 14.167 us/op 0.49
mainnet_e81889 - altair processHistoricalRootsUpdate 1.0430 us/op 1.1490 us/op 0.91
mainnet_e81889 - altair processParticipationFlagUpdates 3.9660 us/op 5.9790 us/op 0.66
mainnet_e81889 - altair processSyncCommitteeUpdates 1.0300 us/op 936.00 ns/op 1.10
mainnet_e81889 - altair afterProcessEpoch 131.52 ms/op 136.88 ms/op 0.96
phase0 processEpoch - mainnet_e58758 374.28 ms/op 377.33 ms/op 0.99
mainnet_e58758 - phase0 beforeProcessEpoch 149.00 ms/op 133.11 ms/op 1.12
mainnet_e58758 - phase0 processJustificationAndFinalization 23.231 us/op 19.926 us/op 1.17
mainnet_e58758 - phase0 processRewardsAndPenalties 67.936 ms/op 47.443 ms/op 1.43
mainnet_e58758 - phase0 processRegistryUpdates 13.317 us/op 9.4580 us/op 1.41
mainnet_e58758 - phase0 processSlashings 1.1710 us/op 520.00 ns/op 2.25
mainnet_e58758 - phase0 processEth1DataReset 838.00 ns/op 860.00 ns/op 0.97
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.0656 ms/op 1.0393 ms/op 1.03
mainnet_e58758 - phase0 processSlashingsReset 7.2840 us/op 4.4490 us/op 1.64
mainnet_e58758 - phase0 processRandaoMixesReset 6.0510 us/op 4.6010 us/op 1.32
mainnet_e58758 - phase0 processHistoricalRootsUpdate 820.00 ns/op 698.00 ns/op 1.17
mainnet_e58758 - phase0 processParticipationRecordUpdates 5.6210 us/op 5.4510 us/op 1.03
mainnet_e58758 - phase0 afterProcessEpoch 101.57 ms/op 96.071 ms/op 1.06
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2873 ms/op 1.2664 ms/op 1.02
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.6687 ms/op 1.8209 ms/op 0.92
altair processInactivityUpdates - 250000 normalcase 27.011 ms/op 22.606 ms/op 1.19
altair processInactivityUpdates - 250000 worstcase 26.290 ms/op 25.582 ms/op 1.03
phase0 processRegistryUpdates - 250000 normalcase 8.6690 us/op 10.651 us/op 0.81
phase0 processRegistryUpdates - 250000 badcase_full_deposits 414.61 us/op 277.40 us/op 1.49
phase0 processRegistryUpdates - 250000 worstcase 0.5 135.84 ms/op 128.62 ms/op 1.06
altair processRewardsAndPenalties - 250000 normalcase 67.962 ms/op 50.698 ms/op 1.34
altair processRewardsAndPenalties - 250000 worstcase 66.036 ms/op 52.594 ms/op 1.26
phase0 getAttestationDeltas - 250000 normalcase 8.0736 ms/op 7.9978 ms/op 1.01
phase0 getAttestationDeltas - 250000 worstcase 6.9737 ms/op 7.0334 ms/op 0.99
phase0 processSlashings - 250000 worstcase 3.4546 ms/op 3.7110 ms/op 0.93
altair processSyncCommitteeUpdates - 250000 180.24 ms/op 191.48 ms/op 0.94
BeaconState.hashTreeRoot - No change 281.00 ns/op 291.00 ns/op 0.97
BeaconState.hashTreeRoot - 1 full validator 52.722 us/op 52.016 us/op 1.01
BeaconState.hashTreeRoot - 32 full validator 557.14 us/op 546.66 us/op 1.02
BeaconState.hashTreeRoot - 512 full validator 5.6843 ms/op 5.3215 ms/op 1.07
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 68.591 us/op 68.169 us/op 1.01
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 888.54 us/op 927.90 us/op 0.96
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.921 ms/op 12.501 ms/op 0.95
BeaconState.hashTreeRoot - 1 balances 49.350 us/op 51.725 us/op 0.95
BeaconState.hashTreeRoot - 32 balances 447.27 us/op 481.94 us/op 0.93
BeaconState.hashTreeRoot - 512 balances 4.5939 ms/op 4.9974 ms/op 0.92
BeaconState.hashTreeRoot - 250000 balances 79.744 ms/op 77.452 ms/op 1.03
aggregationBits - 2048 els - zipIndexesInBitList 18.598 us/op 21.310 us/op 0.87
regular array get 100000 times 33.906 us/op 34.621 us/op 0.98
wrappedArray get 100000 times 34.215 us/op 34.656 us/op 0.99
arrayWithProxy get 100000 times 16.361 ms/op 16.459 ms/op 0.99
ssz.Root.equals 591.00 ns/op 598.00 ns/op 0.99
byteArrayEquals 594.00 ns/op 622.00 ns/op 0.95
shuffle list - 16384 els 7.0598 ms/op 7.3206 ms/op 0.96
shuffle list - 250000 els 103.51 ms/op 104.33 ms/op 0.99
processSlot - 1 slots 8.8090 us/op 9.6100 us/op 0.92
processSlot - 32 slots 1.4190 ms/op 1.4823 ms/op 0.96
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 36.861 ms/op 34.683 ms/op 1.06
getCommitteeAssignments - req 1 vs - 250000 vc 3.0352 ms/op 3.0179 ms/op 1.01
getCommitteeAssignments - req 100 vs - 250000 vc 4.2965 ms/op 4.3091 ms/op 1.00
getCommitteeAssignments - req 1000 vs - 250000 vc 4.6161 ms/op 4.7578 ms/op 0.97
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.8200 ns/op 5.5300 ns/op 1.05
state getBlockRootAtSlot - 250000 vs - 7PWei 870.22 ns/op 772.54 ns/op 1.13
computeProposers - vc 250000 11.180 ms/op 11.747 ms/op 0.95
computeEpochShuffling - vc 250000 105.50 ms/op 109.04 ms/op 0.97
getNextSyncCommittee - vc 250000 187.67 ms/op 187.71 ms/op 1.00
computeSigningRoot for AttestationData 15.240 us/op
hash AttestationData serialized data then Buffer.toString(base64) 2.6374 us/op
toHexString serialized data 1.2901 us/op
Buffer.toString(base64) 356.56 ns/op

by benchmarkbot/action

@wemeetagain
wemeetagain marked this pull request as ready for review April 7, 2023 16:41
@wemeetagain
wemeetagain requested a review from a team as a code owner April 7, 2023 16:42
@wemeetagain
wemeetagain merged commit 2d2d0c0 into unstable Apr 7, 2023
@wemeetagain
wemeetagain deleted the tuyen/new_gossip_queues_merge_to_unstable branch April 7, 2023 16:42
This was referenced Apr 9, 2023
Closed
Closed
Merged
@dapplion dapplion mentioned this pull request May 24, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

v1.6.0-rc.0: Increased block process time

2 participants

@twoeths @wemeetagain