Skip to content

Only preaggregate attestations if there are connected aggregators - #5284

Merged
twoeths merged 3 commits into
unstablefrom
tuyen/should_process_attestations
Mar 29, 2023
Merged

Only preaggregate attestations if there are connected aggregators#5284
twoeths merged 3 commits into
unstablefrom
tuyen/should_process_attestations

Conversation

@twoeths

@twoeths twoeths commented Mar 20, 2023

Copy link
Copy Markdown
Member

Motivation

  • Preaggregating attestations could take up to 8% of cpu time while a lot of them are redundant, see v1.6.0-rc.0: Increased block process time聽#5247 (comment). This PR is to preaggregate attestations if there are a connected aggregator at specified subnet/slot
  • An effort to mitigate/fix the issue we have with new gossip queues

Description

  • Right now we track committee subscriptions up until a slot, and we process all attestations up until that slot/subnet while an aggregator only has duty at a specified subnet/slot.
  • Notice that prepareBeaconCommitteeSubnet() api is called for next epoch at vc side, so this causes a lot of redundancy to preaggregate attestations
  • Instead of that track aggregator at slot/subnet and based on that to process attestations
  • Also at submitPoolAtestations api, only add attestation to pool if necessary

part of #5247

Steps to test or reproduce

@github-actions

github-actions Bot commented Mar 20, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Performance Report

鉁旓笍 no performance regression detected

Full benchmark results
Benchmark suite Current: 3419174 Previous: b861ab8 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 742.39 us/op 902.87 us/op 0.82
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 47.047 us/op 46.746 us/op 1.01
BLS verify - blst-native 1.2494 ms/op 1.2268 ms/op 1.02
BLS verifyMultipleSignatures 3 - blst-native 2.5377 ms/op 2.4862 ms/op 1.02
BLS verifyMultipleSignatures 8 - blst-native 5.3762 ms/op 5.3377 ms/op 1.01
BLS verifyMultipleSignatures 32 - blst-native 19.491 ms/op 19.364 ms/op 1.01
BLS aggregatePubkeys 32 - blst-native 25.941 us/op 26.459 us/op 0.98
BLS aggregatePubkeys 128 - blst-native 101.48 us/op 101.51 us/op 1.00
getAttestationsForBlock 58.546 ms/op 52.723 ms/op 1.11
isKnown best case - 1 super set check 263.00 ns/op 246.00 ns/op 1.07
isKnown normal case - 2 super set checks 263.00 ns/op 240.00 ns/op 1.10
isKnown worse case - 16 super set checks 255.00 ns/op 244.00 ns/op 1.05
CheckpointStateCache - add get delete 5.6980 us/op 4.8540 us/op 1.17
validate gossip signedAggregateAndProof - struct 2.8502 ms/op 2.7776 ms/op 1.03
validate gossip attestation - struct 1.3539 ms/op 1.3290 ms/op 1.02
pickEth1Vote - no votes 1.3886 ms/op 1.2548 ms/op 1.11
pickEth1Vote - max votes 9.9742 ms/op 11.555 ms/op 0.86
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.1561 ms/op 9.1310 ms/op 1.00
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.583 ms/op 14.489 ms/op 1.01
pickEth1Vote - Eth1Data fastSerialize value x2048 691.85 us/op 648.69 us/op 1.07
pickEth1Vote - Eth1Data fastSerialize tree x2048 8.9733 ms/op 7.7747 ms/op 1.15
bytes32 toHexString 538.00 ns/op 478.00 ns/op 1.13
bytes32 Buffer.toString(hex) 375.00 ns/op 343.00 ns/op 1.09
bytes32 Buffer.toString(hex) from Uint8Array 578.00 ns/op 531.00 ns/op 1.09
bytes32 Buffer.toString(hex) + 0x 370.00 ns/op 333.00 ns/op 1.11
Object access 1 prop 0.17400 ns/op 0.16400 ns/op 1.06
Map access 1 prop 0.17200 ns/op 0.15600 ns/op 1.10
Object get x1000 6.9730 ns/op 6.4860 ns/op 1.08
Map get x1000 0.52500 ns/op 0.56000 ns/op 0.94
Object set x1000 54.369 ns/op 49.833 ns/op 1.09
Map set x1000 45.598 ns/op 41.887 ns/op 1.09
Return object 10000 times 0.24170 ns/op 0.22860 ns/op 1.06
Throw Error 10000 times 4.2784 us/op 4.0222 us/op 1.06
fastMsgIdFn sha256 / 200 bytes 3.5970 us/op 3.3280 us/op 1.08
fastMsgIdFn h32 xxhash / 200 bytes 308.00 ns/op 270.00 ns/op 1.14
fastMsgIdFn h64 xxhash / 200 bytes 479.00 ns/op 378.00 ns/op 1.27
fastMsgIdFn sha256 / 1000 bytes 12.181 us/op 11.310 us/op 1.08
fastMsgIdFn h32 xxhash / 1000 bytes 463.00 ns/op 396.00 ns/op 1.17
fastMsgIdFn h64 xxhash / 1000 bytes 548.00 ns/op 448.00 ns/op 1.22
fastMsgIdFn sha256 / 10000 bytes 107.05 us/op 100.95 us/op 1.06
fastMsgIdFn h32 xxhash / 10000 bytes 2.0390 us/op 1.8620 us/op 1.10
fastMsgIdFn h64 xxhash / 10000 bytes 1.5220 us/op 1.3250 us/op 1.15
enrSubnets - fastDeserialize 64 bits 1.4720 us/op 1.2530 us/op 1.17
enrSubnets - ssz BitVector 64 bits 617.00 ns/op 462.00 ns/op 1.34
enrSubnets - fastDeserialize 4 bits 200.00 ns/op 165.00 ns/op 1.21
enrSubnets - ssz BitVector 4 bits 571.00 ns/op 468.00 ns/op 1.22
prioritizePeers score -10:0 att 32-0.1 sync 2-0 128.42 us/op 104.63 us/op 1.23
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 160.07 us/op 129.97 us/op 1.23
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 195.30 us/op 171.67 us/op 1.14
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 350.75 us/op 302.50 us/op 1.16
prioritizePeers score 0:0 att 64-1 sync 4-1 421.87 us/op 364.82 us/op 1.16
array of 16000 items push then shift 1.7016 us/op 1.6274 us/op 1.05
LinkedList of 16000 items push then shift 9.2200 ns/op 8.8590 ns/op 1.04
array of 16000 items push then pop 116.99 ns/op 78.641 ns/op 1.49
LinkedList of 16000 items push then pop 9.0380 ns/op 8.6300 ns/op 1.05
array of 24000 items push then shift 2.4048 us/op 2.3913 us/op 1.01
LinkedList of 24000 items push then shift 9.1410 ns/op 8.8530 ns/op 1.03
array of 24000 items push then pop 85.137 ns/op 74.127 ns/op 1.15
LinkedList of 24000 items push then pop 8.8000 ns/op 8.3720 ns/op 1.05
intersect bitArray bitLen 8 13.447 ns/op 13.167 ns/op 1.02
intersect array and set length 8 83.299 ns/op 76.914 ns/op 1.08
intersect bitArray bitLen 128 44.439 ns/op 43.571 ns/op 1.02
intersect array and set length 128 1.1438 us/op 1.0424 us/op 1.10
Buffer.concat 32 items 2.8360 us/op 2.5760 us/op 1.10
Uint8Array.set 32 items 2.6440 us/op 2.9030 us/op 0.91
pass gossip attestations to forkchoice per slot 3.5370 ms/op 3.1369 ms/op 1.13
computeDeltas 2.9104 ms/op 3.3523 ms/op 0.87
computeProposerBoostScoreFromBalances 1.7912 ms/op 1.7936 ms/op 1.00
altair processAttestation - 250000 vs - 7PWei normalcase 2.0991 ms/op 2.1321 ms/op 0.98
altair processAttestation - 250000 vs - 7PWei worstcase 3.3508 ms/op 3.3587 ms/op 1.00
altair processAttestation - setStatus - 1/6 committees join 140.34 us/op 142.07 us/op 0.99
altair processAttestation - setStatus - 1/3 committees join 274.79 us/op 279.48 us/op 0.98
altair processAttestation - setStatus - 1/2 committees join 357.22 us/op 358.46 us/op 1.00
altair processAttestation - setStatus - 2/3 committees join 452.63 us/op 446.96 us/op 1.01
altair processAttestation - setStatus - 4/5 committees join 658.51 us/op 636.80 us/op 1.03
altair processAttestation - setStatus - 100% committees join 734.95 us/op 752.99 us/op 0.98
altair processBlock - 250000 vs - 7PWei normalcase 18.732 ms/op 18.435 ms/op 1.02
altair processBlock - 250000 vs - 7PWei normalcase hashState 27.571 ms/op 28.389 ms/op 0.97
altair processBlock - 250000 vs - 7PWei worstcase 46.902 ms/op 47.485 ms/op 0.99
altair processBlock - 250000 vs - 7PWei worstcase hashState 65.612 ms/op 65.453 ms/op 1.00
phase0 processBlock - 250000 vs - 7PWei normalcase 2.0275 ms/op 1.9827 ms/op 1.02
phase0 processBlock - 250000 vs - 7PWei worstcase 28.396 ms/op 26.991 ms/op 1.05
altair processEth1Data - 250000 vs - 7PWei normalcase 470.64 us/op 451.59 us/op 1.04
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 6.7930 us/op 6.7240 us/op 1.01
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 20.573 us/op 19.591 us/op 1.05
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 8.5220 us/op 8.2370 us/op 1.03
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 6.6730 us/op 6.1490 us/op 1.09
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 81.100 us/op 74.511 us/op 1.09
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 665.93 us/op 627.01 us/op 1.06
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 916.34 us/op 910.46 us/op 1.01
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 923.41 us/op 854.80 us/op 1.08
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.3234 ms/op 2.2624 ms/op 1.03
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.8770 ms/op 1.4799 ms/op 1.27
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.8605 ms/op 3.7144 ms/op 1.04
Tree 40 250000 create 300.85 ms/op 294.90 ms/op 1.02
Tree 40 250000 get(125000) 183.55 ns/op 177.52 ns/op 1.03
Tree 40 250000 set(125000) 930.26 ns/op 841.54 ns/op 1.11
Tree 40 250000 toArray() 17.357 ms/op 16.070 ms/op 1.08
Tree 40 250000 iterate all - toArray() + loop 17.191 ms/op 16.193 ms/op 1.06
Tree 40 250000 iterate all - get(i) 67.445 ms/op 63.605 ms/op 1.06
MutableVector 250000 create 10.564 ms/op 9.9340 ms/op 1.06
MutableVector 250000 get(125000) 6.0490 ns/op 6.0540 ns/op 1.00
MutableVector 250000 set(125000) 245.84 ns/op 248.56 ns/op 0.99
MutableVector 250000 toArray() 2.6866 ms/op 2.7310 ms/op 0.98
MutableVector 250000 iterate all - toArray() + loop 2.8266 ms/op 2.8652 ms/op 0.99
MutableVector 250000 iterate all - get(i) 1.4909 ms/op 1.4489 ms/op 1.03
Array 250000 create 2.7785 ms/op 2.4904 ms/op 1.12
Array 250000 clone - spread 1.0959 ms/op 1.2991 ms/op 0.84
Array 250000 get(125000) 0.52900 ns/op 0.60300 ns/op 0.88
Array 250000 set(125000) 0.61600 ns/op 0.67100 ns/op 0.92
Array 250000 iterate all - loop 82.655 us/op 82.726 us/op 1.00
effectiveBalanceIncrements clone Uint8Array 300000 24.108 us/op 27.322 us/op 0.88
effectiveBalanceIncrements clone MutableVector 300000 330.00 ns/op 404.00 ns/op 0.82
effectiveBalanceIncrements rw all Uint8Array 300000 168.84 us/op 168.43 us/op 1.00
effectiveBalanceIncrements rw all MutableVector 300000 75.806 ms/op 82.996 ms/op 0.91
phase0 afterProcessEpoch - 250000 vs - 7PWei 107.03 ms/op 113.59 ms/op 0.94
phase0 beforeProcessEpoch - 250000 vs - 7PWei 43.834 ms/op 42.867 ms/op 1.02
altair processEpoch - mainnet_e81889 317.95 ms/op 305.02 ms/op 1.04
mainnet_e81889 - altair beforeProcessEpoch 46.695 ms/op 63.387 ms/op 0.74
mainnet_e81889 - altair processJustificationAndFinalization 16.703 us/op 17.679 us/op 0.94
mainnet_e81889 - altair processInactivityUpdates 5.2897 ms/op 5.3227 ms/op 0.99
mainnet_e81889 - altair processRewardsAndPenalties 66.897 ms/op 53.641 ms/op 1.25
mainnet_e81889 - altair processRegistryUpdates 3.0790 us/op 2.6770 us/op 1.15
mainnet_e81889 - altair processSlashings 471.00 ns/op 612.00 ns/op 0.77
mainnet_e81889 - altair processEth1DataReset 500.00 ns/op 786.00 ns/op 0.64
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2354 ms/op 1.3085 ms/op 0.94
mainnet_e81889 - altair processSlashingsReset 4.8480 us/op 5.5330 us/op 0.88
mainnet_e81889 - altair processRandaoMixesReset 4.7430 us/op 4.8890 us/op 0.97
mainnet_e81889 - altair processHistoricalRootsUpdate 693.00 ns/op 665.00 ns/op 1.04
mainnet_e81889 - altair processParticipationFlagUpdates 2.2010 us/op 2.3090 us/op 0.95
mainnet_e81889 - altair processSyncCommitteeUpdates 504.00 ns/op 612.00 ns/op 0.82
mainnet_e81889 - altair afterProcessEpoch 114.95 ms/op 123.92 ms/op 0.93
phase0 processEpoch - mainnet_e58758 313.88 ms/op 319.33 ms/op 0.98
mainnet_e58758 - phase0 beforeProcessEpoch 120.43 ms/op 124.57 ms/op 0.97
mainnet_e58758 - phase0 processJustificationAndFinalization 15.382 us/op 15.435 us/op 1.00
mainnet_e58758 - phase0 processRewardsAndPenalties 53.378 ms/op 55.397 ms/op 0.96
mainnet_e58758 - phase0 processRegistryUpdates 7.8710 us/op 8.9250 us/op 0.88
mainnet_e58758 - phase0 processSlashings 520.00 ns/op 535.00 ns/op 0.97
mainnet_e58758 - phase0 processEth1DataReset 504.00 ns/op 599.00 ns/op 0.84
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 993.61 us/op 994.36 us/op 1.00
mainnet_e58758 - phase0 processSlashingsReset 2.7910 us/op 3.3820 us/op 0.83
mainnet_e58758 - phase0 processRandaoMixesReset 4.7200 us/op 4.5490 us/op 1.04
mainnet_e58758 - phase0 processHistoricalRootsUpdate 641.00 ns/op 883.00 ns/op 0.73
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.9750 us/op 5.6430 us/op 0.70
mainnet_e58758 - phase0 afterProcessEpoch 94.998 ms/op 96.639 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2199 ms/op 1.2713 ms/op 0.96
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4458 ms/op 1.5035 ms/op 0.96
altair processInactivityUpdates - 250000 normalcase 19.355 ms/op 26.604 ms/op 0.73
altair processInactivityUpdates - 250000 worstcase 22.254 ms/op 37.941 ms/op 0.59
phase0 processRegistryUpdates - 250000 normalcase 6.6370 us/op 20.655 us/op 0.32
phase0 processRegistryUpdates - 250000 badcase_full_deposits 256.15 us/op 538.22 us/op 0.48
phase0 processRegistryUpdates - 250000 worstcase 0.5 121.42 ms/op 208.78 ms/op 0.58
altair processRewardsAndPenalties - 250000 normalcase 51.997 ms/op 100.51 ms/op 0.52
altair processRewardsAndPenalties - 250000 worstcase 59.109 ms/op 105.94 ms/op 0.56
phase0 getAttestationDeltas - 250000 normalcase 6.4336 ms/op 12.856 ms/op 0.50
phase0 getAttestationDeltas - 250000 worstcase 6.4554 ms/op 13.070 ms/op 0.49
phase0 processSlashings - 250000 worstcase 3.5522 ms/op 7.5046 ms/op 0.47
altair processSyncCommitteeUpdates - 250000 168.36 ms/op 220.84 ms/op 0.76
BeaconState.hashTreeRoot - No change 260.00 ns/op 354.00 ns/op 0.73
BeaconState.hashTreeRoot - 1 full validator 51.288 us/op 71.761 us/op 0.71
BeaconState.hashTreeRoot - 32 full validator 517.57 us/op 678.42 us/op 0.76
BeaconState.hashTreeRoot - 512 full validator 5.1504 ms/op 7.1447 ms/op 0.72
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 61.152 us/op 66.046 us/op 0.93
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 944.44 us/op 1.0266 ms/op 0.92
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 10.959 ms/op 14.820 ms/op 0.74
BeaconState.hashTreeRoot - 1 balances 50.441 us/op 54.563 us/op 0.92
BeaconState.hashTreeRoot - 32 balances 461.46 us/op 580.07 us/op 0.80
BeaconState.hashTreeRoot - 512 balances 4.3676 ms/op 5.9385 ms/op 0.74
BeaconState.hashTreeRoot - 250000 balances 75.098 ms/op 105.57 ms/op 0.71
aggregationBits - 2048 els - zipIndexesInBitList 16.173 us/op 39.111 us/op 0.41
regular array get 100000 times 32.443 us/op 41.625 us/op 0.78
wrappedArray get 100000 times 32.436 us/op 36.893 us/op 0.88
arrayWithProxy get 100000 times 15.614 ms/op 16.780 ms/op 0.93
ssz.Root.equals 537.00 ns/op 720.00 ns/op 0.75
byteArrayEquals 530.00 ns/op 810.00 ns/op 0.65
shuffle list - 16384 els 6.8144 ms/op 7.7617 ms/op 0.88
shuffle list - 250000 els 99.222 ms/op 126.50 ms/op 0.78
processSlot - 1 slots 9.0070 us/op 13.581 us/op 0.66
processSlot - 32 slots 1.3467 ms/op 1.6638 ms/op 0.81
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 34.954 ms/op 39.744 ms/op 0.88
getCommitteeAssignments - req 1 vs - 250000 vc 2.7757 ms/op 2.9538 ms/op 0.94
getCommitteeAssignments - req 100 vs - 250000 vc 3.9311 ms/op 4.1995 ms/op 0.94
getCommitteeAssignments - req 1000 vs - 250000 vc 4.2981 ms/op 4.4752 ms/op 0.96
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.2900 ns/op 4.7400 ns/op 0.91
state getBlockRootAtSlot - 250000 vs - 7PWei 839.32 ns/op 1.0465 us/op 0.80
computeProposers - vc 250000 10.344 ms/op 12.489 ms/op 0.83
computeEpochShuffling - vc 250000 101.51 ms/op 111.58 ms/op 0.91
getNextSyncCommittee - vc 250000 167.52 ms/op 200.11 ms/op 0.84

by benchmarkbot/action

@twoeths

twoeths commented Mar 20, 2023

Copy link
Copy Markdown
Member Author

Attestation Pool Size is reduced 95%

Screenshot 2023-03-20 at 17 03 19

Similar to Preaggregation per slot
Screenshot 2023-03-20 at 17 09 02

@twoeths
twoeths marked this pull request as ready for review March 20, 2023 10:10
@twoeths
twoeths requested a review from a team as a code owner March 20, 2023 10:10
@twoeths
twoeths force-pushed the tuyen/should_process_attestations branch from c3301cd to 8171ff7 Compare March 24, 2023 07:21
@twoeths
twoeths force-pushed the tuyen/should_process_attestations branch from 8171ff7 to 11d5142 Compare March 28, 2023 02:35
@twoeths twoeths mentioned this pull request Mar 28, 2023
Closed
@wemeetagain wemeetagain changed the title Only process attestations if there are connected aggregators Only preaggregate attestations if there are connected aggregators Mar 29, 2023
dapplion
dapplion approved these changes Mar 29, 2023
View reviewed changes

@dapplion dapplion left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@twoeths
twoeths merged commit d360e62 into unstable Mar 29, 2023
@twoeths
twoeths deleted the tuyen/should_process_attestations branch March 29, 2023 03:35
twoeths added a commit that referenced this pull request Mar 29, 2023
@twoeths
Only preaggregate attestations if there are connected aggregators (#5284
97f5360 
)

* Only process attestations if there are connected aggregators

* Track aggregator slot subnet count

* Fix deneb sim test
@wemeetagain

wemeetagain commented May 4, 2023

Copy link
Copy Markdown
Member

馃帀 This PR is included in v1.8.0 馃帀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@dapplion dapplion dapplion approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@twoeths @wemeetagain @dapplion