Faster fastMsgIdFn using xxhash by twoeths · Pull Request #4649 · ChainSafe/lodestar

twoeths · 2022-10-07T10:07:49Z

Motivation

We want faster fastMsgIdFn(), it turns out it has a big impact on the I/O lagged issue

Description

Same to Use xxhash with salt for fastMsgIdFn #4630 (use xx-hash for better performance compared to as-sha256) but use h64 instead of h32 to reduce hash collision, hence no memory issue

Closes #4603
could close #4600 due to its significantly faster sumitPoolAttestations request time

TODO

monitor the metrics on feat1 group

github-actions · 2022-10-07T10:20:43Z

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: `04f82c0`	Previous: `ca27458`	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	2.5423 ms/op	1.7624 ms/op	1.44
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	75.813 us/op	64.241 us/op	1.18
BLS verify - blst-native	1.8556 ms/op	2.1652 ms/op	0.86
BLS verifyMultipleSignatures 3 - blst-native	3.8043 ms/op	4.4787 ms/op	0.85
BLS verifyMultipleSignatures 8 - blst-native	8.1872 ms/op	9.6628 ms/op	0.85
BLS verifyMultipleSignatures 32 - blst-native	29.698 ms/op	35.144 ms/op	0.85
BLS aggregatePubkeys 32 - blst-native	39.281 us/op	46.721 us/op	0.84
BLS aggregatePubkeys 128 - blst-native	153.14 us/op	182.19 us/op	0.84
getAttestationsForBlock	92.794 ms/op	75.866 ms/op	1.22
isKnown best case - 1 super set check	425.00 ns/op	491.00 ns/op	0.87
isKnown normal case - 2 super set checks	411.00 ns/op	479.00 ns/op	0.86
isKnown worse case - 16 super set checks	410.00 ns/op	480.00 ns/op	0.85
CheckpointStateCache - add get delete	9.0140 us/op	8.7250 us/op	1.03
validate gossip signedAggregateAndProof - struct	4.2638 ms/op	5.0209 ms/op	0.85
validate gossip attestation - struct	2.0367 ms/op	2.3771 ms/op	0.86
pickEth1Vote - no votes	2.2551 ms/op	2.1786 ms/op	1.04
pickEth1Vote - max votes	23.662 ms/op	18.038 ms/op	1.31
pickEth1Vote - Eth1Data hashTreeRoot value x2048	11.808 ms/op	12.287 ms/op	0.96
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	22.558 ms/op	19.746 ms/op	1.14
pickEth1Vote - Eth1Data fastSerialize value x2048	1.6088 ms/op	1.4400 ms/op	1.12
pickEth1Vote - Eth1Data fastSerialize tree x2048	15.834 ms/op	12.626 ms/op	1.25
bytes32 toHexString	1.1770 us/op	980.00 ns/op	1.20
bytes32 Buffer.toString(hex)	758.00 ns/op	739.00 ns/op	1.03
bytes32 Buffer.toString(hex) from Uint8Array	919.00 ns/op	998.00 ns/op	0.92
bytes32 Buffer.toString(hex) + 0x	734.00 ns/op	752.00 ns/op	0.98
Object access 1 prop	0.40000 ns/op	0.37000 ns/op	1.08
Map access 1 prop	0.29700 ns/op	0.30200 ns/op	0.98
Object get x1000	18.473 ns/op	11.894 ns/op	1.55
Map get x1000	1.0250 ns/op	0.95800 ns/op	1.07
Object set x1000	128.01 ns/op	71.139 ns/op	1.80
Map set x1000	74.795 ns/op	47.155 ns/op	1.59
Return object 10000 times	0.36660 ns/op	0.43350 ns/op	0.85
Throw Error 10000 times	6.1772 us/op	6.0625 us/op	1.02
fastMsgIdFn sha256 / 200 bytes	4.3070 us/op
fastMsgIdFn h32 xxhash / 200 bytes	559.00 ns/op
fastMsgIdFn h64 xxhash / 200 bytes	687.00 ns/op
fastMsgIdFn sha256 / 1000 bytes	13.350 us/op
fastMsgIdFn h32 xxhash / 1000 bytes	696.00 ns/op
fastMsgIdFn h64 xxhash / 1000 bytes	875.00 ns/op
fastMsgIdFn sha256 / 10000 bytes	113.03 us/op
fastMsgIdFn h32 xxhash / 10000 bytes	2.2630 us/op
fastMsgIdFn h64 xxhash / 10000 bytes	1.8040 us/op
enrSubnets - fastDeserialize 64 bits	2.8840 us/op	2.5040 us/op	1.15
enrSubnets - ssz BitVector 64 bits	763.00 ns/op	814.00 ns/op	0.94
enrSubnets - fastDeserialize 4 bits	429.00 ns/op	376.00 ns/op	1.14
enrSubnets - ssz BitVector 4 bits	819.00 ns/op	830.00 ns/op	0.99
prioritizePeers score -10:0 att 32-0.1 sync 2-0	102.10 us/op	81.436 us/op	1.25
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	140.28 us/op	123.28 us/op	1.14
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	254.28 us/op	206.14 us/op	1.23
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	393.60 us/op	334.37 us/op	1.18
prioritizePeers score 0:0 att 64-1 sync 4-1	467.83 us/op	404.53 us/op	1.16
RateTracker 1000000 limit, 1 obj count per request	215.40 ns/op	182.42 ns/op	1.18
RateTracker 1000000 limit, 2 obj count per request	164.98 ns/op	131.98 ns/op	1.25
RateTracker 1000000 limit, 4 obj count per request	139.50 ns/op	107.09 ns/op	1.30
RateTracker 1000000 limit, 8 obj count per request	124.33 ns/op	94.564 ns/op	1.31
RateTracker with prune	4.8590 us/op	3.8560 us/op	1.26
array of 16000 items push then shift	3.1579 us/op	51.590 us/op	0.06
LinkedList of 16000 items push then shift	19.067 ns/op	12.274 ns/op	1.55
array of 16000 items push then pop	262.06 ns/op	202.26 ns/op	1.30
LinkedList of 16000 items push then pop	17.676 ns/op	12.216 ns/op	1.45
array of 24000 items push then shift	4.6062 us/op	77.345 us/op	0.06
LinkedList of 24000 items push then shift	23.969 ns/op	12.314 ns/op	1.95
array of 24000 items push then pop	220.27 ns/op	192.82 ns/op	1.14
LinkedList of 24000 items push then pop	20.045 ns/op	12.189 ns/op	1.64
intersect bitArray bitLen 8	11.620 ns/op	10.958 ns/op	1.06
intersect array and set length 8	178.42 ns/op	134.67 ns/op	1.32
intersect bitArray bitLen 128	72.222 ns/op	57.218 ns/op	1.26
intersect array and set length 128	2.4137 us/op	1.8024 us/op	1.34
Buffer.concat 32 items	1.9540 ns/op	1.8030 ns/op	1.08
pass gossip attestations to forkchoice per slot	4.1182 ms/op	5.1841 ms/op	0.79
computeDeltas	5.9953 ms/op	4.8346 ms/op	1.24
computeProposerBoostScoreFromBalances	921.27 us/op	807.90 us/op	1.14
altair processAttestation - 250000 vs - 7PWei normalcase	4.2640 ms/op	3.3314 ms/op	1.28
altair processAttestation - 250000 vs - 7PWei worstcase	6.1838 ms/op	5.1085 ms/op	1.21
altair processAttestation - setStatus - 1/6 committees join	218.51 us/op	179.51 us/op	1.22
altair processAttestation - setStatus - 1/3 committees join	429.13 us/op	352.05 us/op	1.22
altair processAttestation - setStatus - 1/2 committees join	579.01 us/op	502.88 us/op	1.15
altair processAttestation - setStatus - 2/3 committees join	750.09 us/op	663.82 us/op	1.13
altair processAttestation - setStatus - 4/5 committees join	1.0292 ms/op	922.43 us/op	1.12
altair processAttestation - setStatus - 100% committees join	1.2274 ms/op	1.1121 ms/op	1.10
altair processBlock - 250000 vs - 7PWei normalcase	29.725 ms/op	25.206 ms/op	1.18
altair processBlock - 250000 vs - 7PWei normalcase hashState	41.495 ms/op	38.511 ms/op	1.08
altair processBlock - 250000 vs - 7PWei worstcase	84.768 ms/op	77.254 ms/op	1.10
altair processBlock - 250000 vs - 7PWei worstcase hashState	102.30 ms/op	102.99 ms/op	0.99
phase0 processBlock - 250000 vs - 7PWei normalcase	3.9904 ms/op	3.0701 ms/op	1.30
phase0 processBlock - 250000 vs - 7PWei worstcase	48.149 ms/op	50.249 ms/op	0.96
altair processEth1Data - 250000 vs - 7PWei normalcase	936.39 us/op	662.50 us/op	1.41
Tree 40 250000 create	916.17 ms/op	675.49 ms/op	1.36
Tree 40 250000 get(125000)	294.46 ns/op	262.31 ns/op	1.12
Tree 40 250000 set(125000)	2.7275 us/op	2.2658 us/op	1.20
Tree 40 250000 toArray()	33.745 ms/op	27.395 ms/op	1.23
Tree 40 250000 iterate all - toArray() + loop	34.137 ms/op	27.629 ms/op	1.24
Tree 40 250000 iterate all - get(i)	112.82 ms/op	109.44 ms/op	1.03
MutableVector 250000 create	16.579 ms/op	14.482 ms/op	1.14
MutableVector 250000 get(125000)	13.414 ns/op	11.140 ns/op	1.20
MutableVector 250000 set(125000)	706.95 ns/op	541.26 ns/op	1.31
MutableVector 250000 toArray()	7.7176 ms/op	5.9968 ms/op	1.29
MutableVector 250000 iterate all - toArray() + loop	7.8503 ms/op	6.0070 ms/op	1.31
MutableVector 250000 iterate all - get(i)	3.2823 ms/op	2.8564 ms/op	1.15
Array 250000 create	7.2739 ms/op	5.3576 ms/op	1.36
Array 250000 clone - spread	3.7890 ms/op	2.3122 ms/op	1.64
Array 250000 get(125000)	1.5330 ns/op	1.1330 ns/op	1.35
Array 250000 set(125000)	1.5350 ns/op	1.1300 ns/op	1.36
Array 250000 iterate all - loop	167.84 us/op	150.93 us/op	1.11
effectiveBalanceIncrements clone Uint8Array 300000	94.391 us/op	35.161 us/op	2.68
effectiveBalanceIncrements clone MutableVector 300000	1.1200 us/op	697.00 ns/op	1.61
effectiveBalanceIncrements rw all Uint8Array 300000	252.50 us/op	247.20 us/op	1.02
effectiveBalanceIncrements rw all MutableVector 300000	240.12 ms/op	128.79 ms/op	1.86
phase0 afterProcessEpoch - 250000 vs - 7PWei	191.13 ms/op	201.07 ms/op	0.95
phase0 beforeProcessEpoch - 250000 vs - 7PWei	87.629 ms/op	58.005 ms/op	1.51
altair processEpoch - mainnet_e81889	542.32 ms/op	559.89 ms/op	0.97
mainnet_e81889 - altair beforeProcessEpoch	170.17 ms/op	112.09 ms/op	1.52
mainnet_e81889 - altair processJustificationAndFinalization	31.313 us/op	17.082 us/op	1.83
mainnet_e81889 - altair processInactivityUpdates	12.061 ms/op	8.8775 ms/op	1.36
mainnet_e81889 - altair processRewardsAndPenalties	99.178 ms/op	77.482 ms/op	1.28
mainnet_e81889 - altair processRegistryUpdates	5.9500 us/op	2.6220 us/op	2.27
mainnet_e81889 - altair processSlashings	1.1280 us/op	617.00 ns/op	1.83
mainnet_e81889 - altair processEth1DataReset	1.3700 us/op	587.00 ns/op	2.33
mainnet_e81889 - altair processEffectiveBalanceUpdates	2.2746 ms/op	2.0588 ms/op	1.10
mainnet_e81889 - altair processSlashingsReset	9.0630 us/op	4.4700 us/op	2.03
mainnet_e81889 - altair processRandaoMixesReset	9.7590 us/op	4.7180 us/op	2.07
mainnet_e81889 - altair processHistoricalRootsUpdate	1.4570 us/op	693.00 ns/op	2.10
mainnet_e81889 - altair processParticipationFlagUpdates	6.8380 us/op	2.3080 us/op	2.96
mainnet_e81889 - altair processSyncCommitteeUpdates	1.2240 us/op	509.00 ns/op	2.40
mainnet_e81889 - altair afterProcessEpoch	200.50 ms/op	199.75 ms/op	1.00
phase0 processEpoch - mainnet_e58758	562.33 ms/op	480.85 ms/op	1.17
mainnet_e58758 - phase0 beforeProcessEpoch	253.72 ms/op	175.28 ms/op	1.45
mainnet_e58758 - phase0 processJustificationAndFinalization	27.751 us/op	16.114 us/op	1.72
mainnet_e58758 - phase0 processRewardsAndPenalties	82.103 ms/op	71.658 ms/op	1.15
mainnet_e58758 - phase0 processRegistryUpdates	15.875 us/op	8.3080 us/op	1.91
mainnet_e58758 - phase0 processSlashings	1.3550 us/op	545.00 ns/op	2.49
mainnet_e58758 - phase0 processEth1DataReset	1.2550 us/op	618.00 ns/op	2.03
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	2.0162 ms/op	1.6985 ms/op	1.19
mainnet_e58758 - phase0 processSlashingsReset	7.7450 us/op	3.4600 us/op	2.24
mainnet_e58758 - phase0 processRandaoMixesReset	9.6630 us/op	6.5510 us/op	1.48
mainnet_e58758 - phase0 processHistoricalRootsUpdate	1.5230 us/op	620.00 ns/op	2.46
mainnet_e58758 - phase0 processParticipationRecordUpdates	9.2490 us/op	3.3930 us/op	2.73
mainnet_e58758 - phase0 afterProcessEpoch	165.05 ms/op	164.23 ms/op	1.01
phase0 processEffectiveBalanceUpdates - 250000 normalcase	2.6894 ms/op	2.0384 ms/op	1.32
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	3.4639 ms/op	2.3031 ms/op	1.50
altair processInactivityUpdates - 250000 normalcase	43.818 ms/op	34.260 ms/op	1.28
altair processInactivityUpdates - 250000 worstcase	41.903 ms/op	41.118 ms/op	1.02
phase0 processRegistryUpdates - 250000 normalcase	12.883 us/op	7.2980 us/op	1.77
phase0 processRegistryUpdates - 250000 badcase_full_deposits	449.57 us/op	385.89 us/op	1.17
phase0 processRegistryUpdates - 250000 worstcase 0.5	252.13 ms/op	175.94 ms/op	1.43
altair processRewardsAndPenalties - 250000 normalcase	137.97 ms/op	74.677 ms/op	1.85
altair processRewardsAndPenalties - 250000 worstcase	90.865 ms/op	72.667 ms/op	1.25
phase0 getAttestationDeltas - 250000 normalcase	14.376 ms/op	12.011 ms/op	1.20
phase0 getAttestationDeltas - 250000 worstcase	14.707 ms/op	12.231 ms/op	1.20
phase0 processSlashings - 250000 worstcase	5.4441 ms/op	4.9218 ms/op	1.11
altair processSyncCommitteeUpdates - 250000	288.44 ms/op	289.22 ms/op	1.00
BeaconState.hashTreeRoot - No change	503.00 ns/op	560.00 ns/op	0.90
BeaconState.hashTreeRoot - 1 full validator	60.925 us/op	66.039 us/op	0.92
BeaconState.hashTreeRoot - 32 full validator	854.54 us/op	761.60 us/op	1.12
BeaconState.hashTreeRoot - 512 full validator	6.0243 ms/op	6.8300 ms/op	0.88
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	77.362 us/op	88.991 us/op	0.87
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	1.1918 ms/op	1.3021 ms/op	0.92
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	16.385 ms/op	17.995 ms/op	0.91
BeaconState.hashTreeRoot - 1 balances	64.138 us/op	67.072 us/op	0.96
BeaconState.hashTreeRoot - 32 balances	585.65 us/op	640.34 us/op	0.91
BeaconState.hashTreeRoot - 512 balances	6.0611 ms/op	6.2114 ms/op	0.98
BeaconState.hashTreeRoot - 250000 balances	100.04 ms/op	101.42 ms/op	0.99
aggregationBits - 2048 els - zipIndexesInBitList	43.710 us/op	28.342 us/op	1.54
regular array get 100000 times	67.551 us/op	60.530 us/op	1.12
wrappedArray get 100000 times	67.443 us/op	60.565 us/op	1.11
arrayWithProxy get 100000 times	29.323 ms/op	28.752 ms/op	1.02
ssz.Root.equals	571.00 ns/op	490.00 ns/op	1.17
byteArrayEquals	543.00 ns/op	483.00 ns/op	1.12
shuffle list - 16384 els	12.781 ms/op	11.354 ms/op	1.13
shuffle list - 250000 els	166.14 ms/op	168.52 ms/op	0.99
processSlot - 1 slots	11.771 us/op	13.849 us/op	0.85
processSlot - 32 slots	1.8049 ms/op	2.0069 ms/op	0.90
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	602.53 us/op	388.20 us/op	1.55
getCommitteeAssignments - req 1 vs - 250000 vc	5.3462 ms/op	5.3828 ms/op	0.99
getCommitteeAssignments - req 100 vs - 250000 vc	7.3242 ms/op	7.8650 ms/op	0.93
getCommitteeAssignments - req 1000 vs - 250000 vc	7.7942 ms/op	8.4669 ms/op	0.92
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	10.060 ns/op	9.0300 ns/op	1.11
state getBlockRootAtSlot - 250000 vs - 7PWei	1.2289 us/op	982.38 ns/op	1.25
computeProposers - vc 250000	18.016 ms/op	16.783 ms/op	1.07
computeEpochShuffling - vc 250000	171.69 ms/op	170.52 ms/op	1.01
getNextSyncCommittee - vc 250000	283.08 ms/op	280.33 ms/op	1.01

by benchmarkbot/action

twoeths · 2022-10-07T14:43:20Z

it's been 8h since I tested this branch on feat1 (vs feat2, both configured after_block_delay_slot_fraction as 3 in order to submit attestations right at 1/3 of slot, that's a very busy time to reproduce the missed attestation issue on goerli)

feat1 (this branch)

feat2 (unstable)

feat1 (this branch) - submitPoolsAttestation request time

feat2 (unstable) - submitPoolsAttestation request time

all other metrics are the same

twoeths · 2022-10-07T15:14:20Z

also posting the benchmark in this PR

network / gossip / fastMsgIdFn
    ✔ fastMsgIdFn sha256 / 200 bytes                                      232180.2 ops/s    4.307000 us/op        -      84581 runs  0.404 s
    ✔ fastMsgIdFn h32 xxhash / 200 bytes                                   178[89](https://github.com/ChainSafe/lodestar/actions/runs/3205610042/jobs/5238344436#step:9:90)09 ops/s    559.0000 ns/op        -    1215000 runs   1.11 s
    ✔ fastMsgIdFn h64 xxhash / 200 bytes                                   1455604 ops/s    687.0000 ns/op        -     [90](https://github.com/ChainSafe/lodestar/actions/runs/3205610042/jobs/5238344436#step:9:91)8438 runs   1.01 s
    ✔ fastMsgIdFn sha256 / 1000 bytes                                     74906.37 ops/s    13.35000 us/op        -      57568 runs  0.808 s
    ✔ fastMsgIdFn h32 xxhash / 1000 bytes                                  1436782 ops/s    696.0000 ns/op        -     666281 runs  0.707 s
    ✔ fastMsgIdFn h64 xxhash / 1000 bytes                                  1142857 ops/s    875.0000 ns/op        -     731729 runs  0.[91](https://github.com/ChainSafe/lodestar/actions/runs/3205610042/jobs/5238344436#step:9:92)4 s
    ✔ fastMsgIdFn sha256 / 10000 bytes                                    8846.896 ops/s    113.0340 us/op        -       1779 runs  0.316 s
    ✔ fastMsgIdFn h32 xxhash / 10000 bytes                                441891.3 ops/s    2.263000 us/op        -     109369 runs  0.303 s
    ✔ fastMsgIdFn h64 xxhash / 10000 bytes                                554323.7 ops/s    1.804000 us/op        -     275290 runs  0.610 s

wemeetagain

LGTM, great work

philknows · 2022-10-08T15:36:00Z

<3 thanks @tuyennhv !

twoeths · 2022-10-09T05:26:20Z

This PR only reduce fastMsgIdFn time, it does not help I/O lag issue in #4600
For some reasons, metrics in feat1 is better than feat2 even I deploy unstable to both of them

dapplion and others added 5 commits October 6, 2022 15:51

Use xxhash with salt for fastMsgIdFn

53fdd95

fastMsgIdFn to return a number

ec287f0

Use putUnsafe() for fastMsgIdCache of gossipsub

c383cfe

Use h64() version of xx-hash

d80e4d9

Remove unused unit tests

221ef64

twoeths marked this pull request as ready for review October 7, 2022 14:43

twoeths requested a review from a team as a code owner October 7, 2022 14:43

Fix lint in beacon-node

dc0b1d8

wemeetagain approved these changes Oct 7, 2022

View reviewed changes

wemeetagain merged commit 081afbb into unstable Oct 7, 2022

wemeetagain deleted the tuyen/dapplion/gossip-fast-msg-id branch October 7, 2022 20:45

Navigation Menu

Uh oh!

Uh oh!

Faster fastMsgIdFn using xxhash - #4649

Faster fastMsgIdFn using xxhash#4649
wemeetagain merged 6 commits into
unstablefrom
tuyen/dapplion/gossip-fast-msg-id

twoeths commented Oct 7, 2022

Uh oh!

github-actions Bot commented Oct 7, 2022 •

edited

Loading

Uh oh!

twoeths commented Oct 7, 2022

Uh oh!

twoeths commented Oct 7, 2022

Uh oh!

wemeetagain left a comment

Uh oh!

philknows commented Oct 8, 2022

Uh oh!

twoeths commented Oct 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

twoeths commented Oct 7, 2022

Uh oh!

github-actions Bot commented Oct 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Report

Uh oh!

twoeths commented Oct 7, 2022

Uh oh!

twoeths commented Oct 7, 2022

Uh oh!

wemeetagain left a comment

Choose a reason for hiding this comment

Uh oh!

philknows commented Oct 8, 2022

Uh oh!

twoeths commented Oct 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Oct 7, 2022 •

edited

Loading