Skip to content

[DNM] Reprocess gossip attestations - #5286

Closed
twoeths wants to merge 1 commit into
unstablefrom
tuyen/reprocess_attestations
Closed

[DNM] Reprocess gossip attestations#5286
twoeths wants to merge 1 commit into
unstablefrom
tuyen/reprocess_attestations

Conversation

@twoeths

@twoeths twoeths commented Mar 21, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Motivation

  • There are so many unknown block root attestations passing through executeWork and they are reprocessed in uncontrollable way because when block come, they are processed immediately
  • On a mainnet node subscribing to all subnets, there are too many of them and the pool reached limit

Screenshot 2023-03-21 at 13 11 10

Description

  • All attestations pass through gossip queues
  • If an attestation has unknown block root, send it to a new queue in block processor (implemented as a Map of unkown block root)
  • When block comes, get from this new queue and send to the gossip queue again
  • On 2nd validation, do not need to deserialize the gossip message again

part of #5247

Note: the new gossip queue will be reverted in unstable when we merge v1.6.0 so making this PR as Draft

@github-actions

github-actions Bot commented Mar 21, 2023

Copy link
Copy Markdown
Contributor

Performance Report

鉁旓笍 no performance regression detected

Full benchmark results
Benchmark suite Current: a76e415 Previous: df1ef97 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 821.20 us/op 511.75 us/op 1.60
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 44.236 us/op 46.057 us/op 0.96
BLS verify - blst-native 1.1975 ms/op 1.2111 ms/op 0.99
BLS verifyMultipleSignatures 3 - blst-native 2.4368 ms/op 2.4663 ms/op 0.99
BLS verifyMultipleSignatures 8 - blst-native 5.2396 ms/op 5.3115 ms/op 0.99
BLS verifyMultipleSignatures 32 - blst-native 19.083 ms/op 19.409 ms/op 0.98
BLS aggregatePubkeys 32 - blst-native 25.292 us/op 25.587 us/op 0.99
BLS aggregatePubkeys 128 - blst-native 99.205 us/op 101.20 us/op 0.98
getAttestationsForBlock 51.843 ms/op 52.708 ms/op 0.98
isKnown best case - 1 super set check 261.00 ns/op 263.00 ns/op 0.99
isKnown normal case - 2 super set checks 248.00 ns/op 258.00 ns/op 0.96
isKnown worse case - 16 super set checks 259.00 ns/op 253.00 ns/op 1.02
CheckpointStateCache - add get delete 4.9220 us/op 4.8180 us/op 1.02
validate gossip signedAggregateAndProof - struct 2.7584 ms/op 2.7351 ms/op 1.01
validate gossip attestation - struct 1.3177 ms/op 1.3063 ms/op 1.01
pickEth1Vote - no votes 1.2188 ms/op 1.3189 ms/op 0.92
pickEth1Vote - max votes 11.292 ms/op 8.3392 ms/op 1.35
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.9746 ms/op 8.4065 ms/op 1.07
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.586 ms/op 13.111 ms/op 1.11
pickEth1Vote - Eth1Data fastSerialize value x2048 624.18 us/op 705.55 us/op 0.88
pickEth1Vote - Eth1Data fastSerialize tree x2048 7.2515 ms/op 4.9644 ms/op 1.46
bytes32 toHexString 477.00 ns/op 518.00 ns/op 0.92
bytes32 Buffer.toString(hex) 326.00 ns/op 357.00 ns/op 0.91
bytes32 Buffer.toString(hex) from Uint8Array 521.00 ns/op 544.00 ns/op 0.96
bytes32 Buffer.toString(hex) + 0x 330.00 ns/op 377.00 ns/op 0.88
Object access 1 prop 0.15500 ns/op 0.16900 ns/op 0.92
Map access 1 prop 0.14900 ns/op 0.16200 ns/op 0.92
Object get x1000 6.7300 ns/op 7.2050 ns/op 0.93
Map get x1000 0.59100 ns/op 0.63000 ns/op 0.94
Object set x1000 49.388 ns/op 54.747 ns/op 0.90
Map set x1000 41.381 ns/op 46.374 ns/op 0.89
Return object 10000 times 0.22760 ns/op 0.23770 ns/op 0.96
Throw Error 10000 times 4.0011 us/op 4.1452 us/op 0.97
fastMsgIdFn sha256 / 200 bytes 3.2920 us/op 3.4720 us/op 0.95
fastMsgIdFn h32 xxhash / 200 bytes 268.00 ns/op 285.00 ns/op 0.94
fastMsgIdFn h64 xxhash / 200 bytes 372.00 ns/op 392.00 ns/op 0.95
fastMsgIdFn sha256 / 1000 bytes 11.230 us/op 11.474 us/op 0.98
fastMsgIdFn h32 xxhash / 1000 bytes 395.00 ns/op 399.00 ns/op 0.99
fastMsgIdFn h64 xxhash / 1000 bytes 449.00 ns/op 453.00 ns/op 0.99
fastMsgIdFn sha256 / 10000 bytes 100.17 us/op 103.30 us/op 0.97
fastMsgIdFn h32 xxhash / 10000 bytes 1.8150 us/op 1.8980 us/op 0.96
fastMsgIdFn h64 xxhash / 10000 bytes 1.2890 us/op 1.3040 us/op 0.99
enrSubnets - fastDeserialize 64 bits 1.2190 us/op 1.2850 us/op 0.95
enrSubnets - ssz BitVector 64 bits 466.00 ns/op 500.00 ns/op 0.93
enrSubnets - fastDeserialize 4 bits 163.00 ns/op 179.00 ns/op 0.91
enrSubnets - ssz BitVector 4 bits 465.00 ns/op 488.00 ns/op 0.95
prioritizePeers score -10:0 att 32-0.1 sync 2-0 100.36 us/op 101.28 us/op 0.99
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 128.30 us/op 131.88 us/op 0.97
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 160.71 us/op 166.54 us/op 0.96
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 304.29 us/op 304.73 us/op 1.00
prioritizePeers score 0:0 att 64-1 sync 4-1 362.25 us/op 363.57 us/op 1.00
array of 16000 items push then shift 1.6189 us/op 1.6329 us/op 0.99
LinkedList of 16000 items push then shift 8.7240 ns/op 8.8340 ns/op 0.99
array of 16000 items push then pop 78.063 ns/op 81.262 ns/op 0.96
LinkedList of 16000 items push then pop 8.4930 ns/op 8.6500 ns/op 0.98
array of 24000 items push then shift 2.3384 us/op 2.3471 us/op 1.00
LinkedList of 24000 items push then shift 8.7430 ns/op 8.7760 ns/op 1.00
array of 24000 items push then pop 76.929 ns/op 74.835 ns/op 1.03
LinkedList of 24000 items push then pop 8.4340 ns/op 8.5470 ns/op 0.99
intersect bitArray bitLen 8 13.299 ns/op 13.144 ns/op 1.01
intersect array and set length 8 76.894 ns/op 80.266 ns/op 0.96
intersect bitArray bitLen 128 44.068 ns/op 43.834 ns/op 1.01
intersect array and set length 128 1.0488 us/op 1.0889 us/op 0.96
Buffer.concat 32 items 2.9080 us/op 2.6830 us/op 1.08
Uint8Array.set 32 items 2.7900 us/op 3.0450 us/op 0.92
pass gossip attestations to forkchoice per slot 2.6910 ms/op 3.2323 ms/op 0.83
computeDeltas 2.8323 ms/op 2.9235 ms/op 0.97
computeProposerBoostScoreFromBalances 1.7783 ms/op 1.7683 ms/op 1.01
altair processAttestation - 250000 vs - 7PWei normalcase 2.0337 ms/op 2.1019 ms/op 0.97
altair processAttestation - 250000 vs - 7PWei worstcase 3.1786 ms/op 3.2527 ms/op 0.98
altair processAttestation - setStatus - 1/6 committees join 136.91 us/op 138.10 us/op 0.99
altair processAttestation - setStatus - 1/3 committees join 274.17 us/op 271.82 us/op 1.01
altair processAttestation - setStatus - 1/2 committees join 371.01 us/op 366.47 us/op 1.01
altair processAttestation - setStatus - 2/3 committees join 449.13 us/op 457.54 us/op 0.98
altair processAttestation - setStatus - 4/5 committees join 653.34 us/op 639.24 us/op 1.02
altair processAttestation - setStatus - 100% committees join 738.01 us/op 749.72 us/op 0.98
altair processBlock - 250000 vs - 7PWei normalcase 19.133 ms/op 16.295 ms/op 1.17
altair processBlock - 250000 vs - 7PWei normalcase hashState 26.272 ms/op 29.056 ms/op 0.90
altair processBlock - 250000 vs - 7PWei worstcase 54.873 ms/op 45.768 ms/op 1.20
altair processBlock - 250000 vs - 7PWei worstcase hashState 67.456 ms/op 63.494 ms/op 1.06
phase0 processBlock - 250000 vs - 7PWei normalcase 2.0005 ms/op 2.0193 ms/op 0.99
phase0 processBlock - 250000 vs - 7PWei worstcase 27.630 ms/op 29.210 ms/op 0.95
altair processEth1Data - 250000 vs - 7PWei normalcase 472.32 us/op 461.83 us/op 1.02
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 6.5920 us/op 7.1510 us/op 0.92
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 21.262 us/op 21.301 us/op 1.00
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 8.4060 us/op 8.8640 us/op 0.95
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 6.3640 us/op 7.4370 us/op 0.86
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 73.976 us/op 84.720 us/op 0.87
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 600.18 us/op 633.04 us/op 0.95
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 871.29 us/op 902.37 us/op 0.97
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 901.16 us/op 857.86 us/op 1.05
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.3164 ms/op 2.2780 ms/op 1.02
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.5038 ms/op 1.4975 ms/op 1.00
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.8732 ms/op 3.7820 ms/op 1.02
Tree 40 250000 create 312.08 ms/op 300.01 ms/op 1.04
Tree 40 250000 get(125000) 180.77 ns/op 182.08 ns/op 0.99
Tree 40 250000 set(125000) 886.96 ns/op 896.50 ns/op 0.99
Tree 40 250000 toArray() 16.990 ms/op 16.680 ms/op 1.02
Tree 40 250000 iterate all - toArray() + loop 19.357 ms/op 17.385 ms/op 1.11
Tree 40 250000 iterate all - get(i) 67.379 ms/op 66.774 ms/op 1.01
MutableVector 250000 create 9.8759 ms/op 9.6544 ms/op 1.02
MutableVector 250000 get(125000) 6.4030 ns/op 6.1900 ns/op 1.03
MutableVector 250000 set(125000) 253.41 ns/op 256.03 ns/op 0.99
MutableVector 250000 toArray() 2.7112 ms/op 2.6775 ms/op 1.01
MutableVector 250000 iterate all - toArray() + loop 2.8314 ms/op 2.8243 ms/op 1.00
MutableVector 250000 iterate all - get(i) 1.5681 ms/op 1.4781 ms/op 1.06
Array 250000 create 2.4220 ms/op 2.7719 ms/op 0.87
Array 250000 clone - spread 1.1428 ms/op 1.0997 ms/op 1.04
Array 250000 get(125000) 0.59200 ns/op 0.52900 ns/op 1.12
Array 250000 set(125000) 0.66200 ns/op 0.61600 ns/op 1.07
Array 250000 iterate all - loop 82.359 us/op 108.44 us/op 0.76
effectiveBalanceIncrements clone Uint8Array 300000 25.630 us/op 23.983 us/op 1.07
effectiveBalanceIncrements clone MutableVector 300000 370.00 ns/op 335.00 ns/op 1.10
effectiveBalanceIncrements rw all Uint8Array 300000 171.10 us/op 167.48 us/op 1.02
effectiveBalanceIncrements rw all MutableVector 300000 81.015 ms/op 76.710 ms/op 1.06
phase0 afterProcessEpoch - 250000 vs - 7PWei 111.90 ms/op 111.06 ms/op 1.01
phase0 beforeProcessEpoch - 250000 vs - 7PWei 33.689 ms/op 42.615 ms/op 0.79
altair processEpoch - mainnet_e81889 327.29 ms/op 299.39 ms/op 1.09
mainnet_e81889 - altair beforeProcessEpoch 66.831 ms/op 47.870 ms/op 1.40
mainnet_e81889 - altair processJustificationAndFinalization 17.111 us/op 16.444 us/op 1.04
mainnet_e81889 - altair processInactivityUpdates 5.4090 ms/op 5.2723 ms/op 1.03
mainnet_e81889 - altair processRewardsAndPenalties 66.400 ms/op 66.817 ms/op 0.99
mainnet_e81889 - altair processRegistryUpdates 2.7290 us/op 2.3260 us/op 1.17
mainnet_e81889 - altair processSlashings 468.00 ns/op 535.00 ns/op 0.87
mainnet_e81889 - altair processEth1DataReset 591.00 ns/op 541.00 ns/op 1.09
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.1959 ms/op 1.2298 ms/op 0.97
mainnet_e81889 - altair processSlashingsReset 4.7630 us/op 4.7810 us/op 1.00
mainnet_e81889 - altair processRandaoMixesReset 4.6490 us/op 4.2820 us/op 1.09
mainnet_e81889 - altair processHistoricalRootsUpdate 656.00 ns/op 644.00 ns/op 1.02
mainnet_e81889 - altair processParticipationFlagUpdates 2.4920 us/op 2.4280 us/op 1.03
mainnet_e81889 - altair processSyncCommitteeUpdates 747.00 ns/op 646.00 ns/op 1.16
mainnet_e81889 - altair afterProcessEpoch 127.55 ms/op 117.78 ms/op 1.08
phase0 processEpoch - mainnet_e58758 335.92 ms/op 310.34 ms/op 1.08
mainnet_e58758 - phase0 beforeProcessEpoch 117.31 ms/op 119.70 ms/op 0.98
mainnet_e58758 - phase0 processJustificationAndFinalization 17.612 us/op 16.019 us/op 1.10
mainnet_e58758 - phase0 processRewardsAndPenalties 54.814 ms/op 56.078 ms/op 0.98
mainnet_e58758 - phase0 processRegistryUpdates 7.6250 us/op 7.6620 us/op 1.00
mainnet_e58758 - phase0 processSlashings 507.00 ns/op 509.00 ns/op 1.00
mainnet_e58758 - phase0 processEth1DataReset 535.00 ns/op 524.00 ns/op 1.02
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 942.43 us/op 968.17 us/op 0.97
mainnet_e58758 - phase0 processSlashingsReset 4.4510 us/op 3.3060 us/op 1.35
mainnet_e58758 - phase0 processRandaoMixesReset 4.2720 us/op 4.7090 us/op 0.91
mainnet_e58758 - phase0 processHistoricalRootsUpdate 624.00 ns/op 601.00 ns/op 1.04
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.9730 us/op 3.8010 us/op 1.05
mainnet_e58758 - phase0 afterProcessEpoch 96.477 ms/op 95.080 ms/op 1.01
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2306 ms/op 1.2062 ms/op 1.02
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.5555 ms/op 1.4407 ms/op 1.08
altair processInactivityUpdates - 250000 normalcase 22.609 ms/op 23.544 ms/op 0.96
altair processInactivityUpdates - 250000 worstcase 22.035 ms/op 20.475 ms/op 1.08
phase0 processRegistryUpdates - 250000 normalcase 6.9810 us/op 6.2910 us/op 1.11
phase0 processRegistryUpdates - 250000 badcase_full_deposits 237.95 us/op 239.39 us/op 0.99
phase0 processRegistryUpdates - 250000 worstcase 0.5 110.81 ms/op 111.66 ms/op 0.99
altair processRewardsAndPenalties - 250000 normalcase 59.618 ms/op 51.939 ms/op 1.15
altair processRewardsAndPenalties - 250000 worstcase 59.740 ms/op 60.646 ms/op 0.99
phase0 getAttestationDeltas - 250000 normalcase 6.3341 ms/op 6.2305 ms/op 1.02
phase0 getAttestationDeltas - 250000 worstcase 6.4336 ms/op 6.3208 ms/op 1.02
phase0 processSlashings - 250000 worstcase 3.5172 ms/op 3.4424 ms/op 1.02
altair processSyncCommitteeUpdates - 250000 173.38 ms/op 172.58 ms/op 1.00
BeaconState.hashTreeRoot - No change 250.00 ns/op 257.00 ns/op 0.97
BeaconState.hashTreeRoot - 1 full validator 52.649 us/op 50.512 us/op 1.04
BeaconState.hashTreeRoot - 32 full validator 509.00 us/op 512.21 us/op 0.99
BeaconState.hashTreeRoot - 512 full validator 4.8776 ms/op 4.9410 ms/op 0.99
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 60.614 us/op 60.809 us/op 1.00
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 865.10 us/op 928.49 us/op 0.93
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.069 ms/op 11.062 ms/op 1.00
BeaconState.hashTreeRoot - 1 balances 50.546 us/op 48.884 us/op 1.03
BeaconState.hashTreeRoot - 32 balances 436.49 us/op 439.76 us/op 0.99
BeaconState.hashTreeRoot - 512 balances 4.2360 ms/op 4.2475 ms/op 1.00
BeaconState.hashTreeRoot - 250000 balances 73.198 ms/op 75.768 ms/op 0.97
aggregationBits - 2048 els - zipIndexesInBitList 15.876 us/op 16.131 us/op 0.98
regular array get 100000 times 40.649 us/op 32.740 us/op 1.24
wrappedArray get 100000 times 31.822 us/op 32.657 us/op 0.97
arrayWithProxy get 100000 times 15.783 ms/op 16.036 ms/op 0.98
ssz.Root.equals 524.00 ns/op 543.00 ns/op 0.97
byteArrayEquals 519.00 ns/op 539.00 ns/op 0.96
shuffle list - 16384 els 6.8254 ms/op 6.8081 ms/op 1.00
shuffle list - 250000 els 99.981 ms/op 98.044 ms/op 1.02
processSlot - 1 slots 8.7080 us/op 8.4500 us/op 1.03
processSlot - 32 slots 1.3087 ms/op 1.2989 ms/op 1.01
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 33.848 ms/op 34.201 ms/op 0.99
getCommitteeAssignments - req 1 vs - 250000 vc 2.8412 ms/op 2.8550 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 4.0326 ms/op 4.0577 ms/op 0.99
getCommitteeAssignments - req 1000 vs - 250000 vc 4.3975 ms/op 4.3951 ms/op 1.00
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.3200 ns/op 4.4400 ns/op 0.97
state getBlockRootAtSlot - 250000 vs - 7PWei 918.70 ns/op 935.35 ns/op 0.98
computeProposers - vc 250000 10.264 ms/op 10.805 ms/op 0.95
computeEpochShuffling - vc 250000 100.31 ms/op 100.98 ms/op 0.99
getNextSyncCommittee - vc 250000 167.89 ms/op 176.34 ms/op 0.95

by benchmarkbot/action

@twoeths twoeths added the status-do-not-merge Merging this issue will break the build. Do not merge! label Mar 22, 2023
@twoeths

twoeths commented Apr 4, 2023

Copy link
Copy Markdown
Member Author

replaced by #5327

@twoeths twoeths closed this Apr 4, 2023
@twoeths
twoeths deleted the tuyen/reprocess_attestations branch April 4, 2023 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

status-do-not-merge Merging this issue will break the build. Do not merge!

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@twoeths