Skip to content

DOS protection based on message slot - #5394

Merged
twoeths merged 3 commits into
unstablefrom
tuyen/dos_protection_based_on_message_slot
Apr 21, 2023
Merged

DOS protection based on message slot#5394
twoeths merged 3 commits into
unstablefrom
tuyen/dos_protection_based_on_message_slot

Conversation

@twoeths

@twoeths twoeths commented Apr 21, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Motivation

  • There are a lot of very old gossiped blocks sent to our node randomly on mainnet, it took our time to deserialize them before we ignore them (since they're known in forkchoice)
  • This contributes to our I/O lag issue
  • It happened in v1.7 as well but likely more painful in unstable

Description

  • Check if gossiped blocks are > 32 slots from clock slot, ignore them
  • Apply for beacon_attestation and aggregate_and_proof topic as well (don't apply for sync committee metrics for now since there are not a lot of them and message size is small)
    • For attestation topics, the number of 32 is respective to ATTESTATION_PROPAGATION_SLOT_RANGE per spec
    • For gossiped blocks, we only to want to receive gossip blocks in the last 32 slot and have gossipsub seenCache handle the duplicate, also 32 slots = gossip seenCache TTL per spec . Otherwise it'll be handled by our sync services
  • Remove gossipValidationErrorTooManySkippedSlots metric because we have gossipValidationError metric already
  • Refactor the api to extract slot from gossiped block serialized data

part of #5393

Test

Was able to caught the new GOSSIP_ERROR_PAST_SLOT error in the last 3h (this is on a mainnet node)

Screenshot 2023-04-21 at 18 36 12

@github-actions

github-actions Bot commented Apr 21, 2023

Copy link
Copy Markdown
Contributor

Performance Report

鉁旓笍 no performance regression detected

Full benchmark results
Benchmark suite Current: 988d927 Previous: fb192bf Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 839.77 us/op 721.78 us/op 1.16
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 59.239 us/op 48.913 us/op 1.21
BLS verify - blst-native 1.2289 ms/op 1.2447 ms/op 0.99
BLS verifyMultipleSignatures 3 - blst-native 2.4974 ms/op 2.5650 ms/op 0.97
BLS verifyMultipleSignatures 8 - blst-native 5.4344 ms/op 5.5203 ms/op 0.98
BLS verifyMultipleSignatures 32 - blst-native 19.539 ms/op 19.837 ms/op 0.99
BLS aggregatePubkeys 32 - blst-native 26.008 us/op 26.935 us/op 0.97
BLS aggregatePubkeys 128 - blst-native 102.04 us/op 103.38 us/op 0.99
getAttestationsForBlock 55.677 ms/op 64.471 ms/op 0.86
isKnown best case - 1 super set check 259.00 ns/op 279.00 ns/op 0.93
isKnown normal case - 2 super set checks 251.00 ns/op 275.00 ns/op 0.91
isKnown worse case - 16 super set checks 253.00 ns/op 273.00 ns/op 0.93
CheckpointStateCache - add get delete 5.0930 us/op 6.2390 us/op 0.82
validate gossip signedAggregateAndProof - struct 2.8136 ms/op 2.8864 ms/op 0.97
validate gossip attestation - struct 1.3521 ms/op 1.3822 ms/op 0.98
pickEth1Vote - no votes 1.3783 ms/op 1.4212 ms/op 0.97
pickEth1Vote - max votes 11.587 ms/op 10.615 ms/op 1.09
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.7904 ms/op 9.4887 ms/op 1.03
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 16.795 ms/op 17.723 ms/op 0.95
pickEth1Vote - Eth1Data fastSerialize value x2048 775.64 us/op 804.53 us/op 0.96
pickEth1Vote - Eth1Data fastSerialize tree x2048 5.6890 ms/op 9.3120 ms/op 0.61
bytes32 toHexString 712.00 ns/op 952.00 ns/op 0.75
bytes32 Buffer.toString(hex) 438.00 ns/op 461.00 ns/op 0.95
bytes32 Buffer.toString(hex) from Uint8Array 647.00 ns/op 697.00 ns/op 0.93
bytes32 Buffer.toString(hex) + 0x 438.00 ns/op 476.00 ns/op 0.92
Object access 1 prop 0.19300 ns/op 0.24000 ns/op 0.80
Map access 1 prop 0.16600 ns/op 0.17700 ns/op 0.94
Object get x1000 7.0600 ns/op 8.3250 ns/op 0.85
Map get x1000 0.58600 ns/op 0.73300 ns/op 0.80
Object set x1000 62.037 ns/op 86.286 ns/op 0.72
Map set x1000 54.605 ns/op 64.053 ns/op 0.85
Return object 10000 times 0.24450 ns/op 0.30460 ns/op 0.80
Throw Error 10000 times 4.3220 us/op 4.7827 us/op 0.90
fastMsgIdFn sha256 / 200 bytes 3.6240 us/op 3.8580 us/op 0.94
fastMsgIdFn h32 xxhash / 200 bytes 311.00 ns/op 358.00 ns/op 0.87
fastMsgIdFn h64 xxhash / 200 bytes 469.00 ns/op 546.00 ns/op 0.86
fastMsgIdFn sha256 / 1000 bytes 11.881 us/op 12.280 us/op 0.97
fastMsgIdFn h32 xxhash / 1000 bytes 440.00 ns/op 515.00 ns/op 0.85
fastMsgIdFn h64 xxhash / 1000 bytes 549.00 ns/op 601.00 ns/op 0.91
fastMsgIdFn sha256 / 10000 bytes 104.51 us/op 108.73 us/op 0.96
fastMsgIdFn h32 xxhash / 10000 bytes 1.9990 us/op 2.1120 us/op 0.95
fastMsgIdFn h64 xxhash / 10000 bytes 1.4180 us/op 1.5380 us/op 0.92
enrSubnets - fastDeserialize 64 bits 1.5600 us/op 1.9040 us/op 0.82
enrSubnets - ssz BitVector 64 bits 734.00 ns/op 696.00 ns/op 1.05
enrSubnets - fastDeserialize 4 bits 245.00 ns/op 243.00 ns/op 1.01
enrSubnets - ssz BitVector 4 bits 727.00 ns/op 707.00 ns/op 1.03
prioritizePeers score -10:0 att 32-0.1 sync 2-0 210.96 us/op 129.23 us/op 1.63
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 270.76 us/op 181.94 us/op 1.49
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 285.87 us/op 239.46 us/op 1.19
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 636.38 us/op 374.13 us/op 1.70
prioritizePeers score 0:0 att 64-1 sync 4-1 649.36 us/op 435.76 us/op 1.49
array of 16000 items push then shift 2.8870 us/op 1.7801 us/op 1.62
LinkedList of 16000 items push then shift 19.207 ns/op 10.026 ns/op 1.92
array of 16000 items push then pop 197.03 ns/op 114.63 ns/op 1.72
LinkedList of 16000 items push then pop 14.897 ns/op 9.5000 ns/op 1.57
array of 24000 items push then shift 4.2848 us/op 2.4945 us/op 1.72
LinkedList of 24000 items push then shift 12.253 ns/op 10.233 ns/op 1.20
array of 24000 items push then pop 111.75 ns/op 92.094 ns/op 1.21
LinkedList of 24000 items push then pop 12.486 ns/op 10.441 ns/op 1.20
intersect bitArray bitLen 8 24.737 ns/op 32.375 ns/op 0.76
intersect array and set length 8 108.29 ns/op 102.72 ns/op 1.05
intersect bitArray bitLen 128 50.667 ns/op 70.377 ns/op 0.72
intersect array and set length 128 1.2834 us/op 1.3294 us/op 0.97
Buffer.concat 32 items 3.0770 us/op 3.2410 us/op 0.95
Uint8Array.set 32 items 2.4390 us/op 2.7410 us/op 0.89
pass gossip attestations to forkchoice per slot 2.9640 ms/op 3.4681 ms/op 0.85
computeDeltas 3.2825 ms/op 3.4194 ms/op 0.96
computeProposerBoostScoreFromBalances 2.3514 ms/op 1.8748 ms/op 1.25
altair processAttestation - 250000 vs - 7PWei normalcase 4.0252 ms/op 3.4719 ms/op 1.16
altair processAttestation - 250000 vs - 7PWei worstcase 5.4882 ms/op 4.7791 ms/op 1.15
altair processAttestation - setStatus - 1/6 committees join 170.52 us/op 150.73 us/op 1.13
altair processAttestation - setStatus - 1/3 committees join 324.79 us/op 284.46 us/op 1.14
altair processAttestation - setStatus - 1/2 committees join 412.50 us/op 394.67 us/op 1.05
altair processAttestation - setStatus - 2/3 committees join 494.46 us/op 478.49 us/op 1.03
altair processAttestation - setStatus - 4/5 committees join 775.30 us/op 687.19 us/op 1.13
altair processAttestation - setStatus - 100% committees join 879.47 us/op 786.03 us/op 1.12
altair processBlock - 250000 vs - 7PWei normalcase 19.986 ms/op 19.866 ms/op 1.01
altair processBlock - 250000 vs - 7PWei normalcase hashState 34.583 ms/op 29.611 ms/op 1.17
altair processBlock - 250000 vs - 7PWei worstcase 58.728 ms/op 57.298 ms/op 1.02
altair processBlock - 250000 vs - 7PWei worstcase hashState 79.789 ms/op 78.047 ms/op 1.02
phase0 processBlock - 250000 vs - 7PWei normalcase 3.0918 ms/op 2.4653 ms/op 1.25
phase0 processBlock - 250000 vs - 7PWei worstcase 35.956 ms/op 31.087 ms/op 1.16
altair processEth1Data - 250000 vs - 7PWei normalcase 710.17 us/op 643.19 us/op 1.10
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 12.672 us/op 10.827 us/op 1.17
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 36.704 us/op 32.762 us/op 1.12
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 16.884 us/op 15.159 us/op 1.11
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 14.269 us/op 13.689 us/op 1.04
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 125.30 us/op 113.98 us/op 1.10
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 852.05 us/op 778.15 us/op 1.09
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.2323 ms/op 1.0975 ms/op 1.12
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.2228 ms/op 941.57 us/op 1.30
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 3.6414 ms/op 3.1660 ms/op 1.15
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 2.1653 ms/op 1.7494 ms/op 1.24
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 6.7412 ms/op 4.4430 ms/op 1.52
Tree 40 250000 create 398.55 ms/op 582.73 ms/op 0.68
Tree 40 250000 get(125000) 188.02 ns/op 203.38 ns/op 0.92
Tree 40 250000 set(125000) 946.69 ns/op 1.3292 us/op 0.71
Tree 40 250000 toArray() 18.980 ms/op 29.882 ms/op 0.64
Tree 40 250000 iterate all - toArray() + loop 19.328 ms/op 28.727 ms/op 0.67
Tree 40 250000 iterate all - get(i) 74.112 ms/op 80.826 ms/op 0.92
MutableVector 250000 create 11.362 ms/op 11.643 ms/op 0.98
MutableVector 250000 get(125000) 6.4200 ns/op 7.1360 ns/op 0.90
MutableVector 250000 set(125000) 259.05 ns/op 521.70 ns/op 0.50
MutableVector 250000 toArray() 2.9937 ms/op 4.4363 ms/op 0.67
MutableVector 250000 iterate all - toArray() + loop 3.3476 ms/op 4.8298 ms/op 0.69
MutableVector 250000 iterate all - get(i) 1.5327 ms/op 1.6995 ms/op 0.90
Array 250000 create 2.9168 ms/op 4.3538 ms/op 0.67
Array 250000 clone - spread 1.1429 ms/op 1.4609 ms/op 0.78
Array 250000 get(125000) 0.53000 ns/op 0.91300 ns/op 0.58
Array 250000 set(125000) 0.62600 ns/op 1.1750 ns/op 0.53
Array 250000 iterate all - loop 84.855 us/op 113.18 us/op 0.75
effectiveBalanceIncrements clone Uint8Array 300000 26.399 us/op 69.333 us/op 0.38
effectiveBalanceIncrements clone MutableVector 300000 321.00 ns/op 490.00 ns/op 0.66
effectiveBalanceIncrements rw all Uint8Array 300000 167.50 us/op 186.87 us/op 0.90
effectiveBalanceIncrements rw all MutableVector 300000 84.710 ms/op 122.13 ms/op 0.69
phase0 afterProcessEpoch - 250000 vs - 7PWei 115.57 ms/op 130.81 ms/op 0.88
phase0 beforeProcessEpoch - 250000 vs - 7PWei 35.651 ms/op 47.385 ms/op 0.75
altair processEpoch - mainnet_e81889 332.55 ms/op 335.21 ms/op 0.99
mainnet_e81889 - altair beforeProcessEpoch 53.775 ms/op 73.165 ms/op 0.73
mainnet_e81889 - altair processJustificationAndFinalization 19.913 us/op 24.808 us/op 0.80
mainnet_e81889 - altair processInactivityUpdates 6.5695 ms/op 6.7402 ms/op 0.97
mainnet_e81889 - altair processRewardsAndPenalties 80.289 ms/op 57.333 ms/op 1.40
mainnet_e81889 - altair processRegistryUpdates 8.4600 us/op 3.0200 us/op 2.80
mainnet_e81889 - altair processSlashings 2.0570 us/op 614.00 ns/op 3.35
mainnet_e81889 - altair processEth1DataReset 1.7730 us/op 1.1780 us/op 1.51
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2830 ms/op 1.3751 ms/op 0.93
mainnet_e81889 - altair processSlashingsReset 7.0870 us/op 6.8620 us/op 1.03
mainnet_e81889 - altair processRandaoMixesReset 11.054 us/op 9.4580 us/op 1.17
mainnet_e81889 - altair processHistoricalRootsUpdate 722.00 ns/op 1.3070 us/op 0.55
mainnet_e81889 - altair processParticipationFlagUpdates 3.4120 us/op 4.6160 us/op 0.74
mainnet_e81889 - altair processSyncCommitteeUpdates 1.0410 us/op 1.0110 us/op 1.03
mainnet_e81889 - altair afterProcessEpoch 129.09 ms/op 132.70 ms/op 0.97
phase0 processEpoch - mainnet_e58758 398.31 ms/op 405.91 ms/op 0.98
mainnet_e58758 - phase0 beforeProcessEpoch 159.43 ms/op 166.15 ms/op 0.96
mainnet_e58758 - phase0 processJustificationAndFinalization 37.759 us/op 22.853 us/op 1.65
mainnet_e58758 - phase0 processRewardsAndPenalties 92.157 ms/op 68.801 ms/op 1.34
mainnet_e58758 - phase0 processRegistryUpdates 17.510 us/op 11.140 us/op 1.57
mainnet_e58758 - phase0 processSlashings 1.4460 us/op 652.00 ns/op 2.22
mainnet_e58758 - phase0 processEth1DataReset 947.00 ns/op 652.00 ns/op 1.45
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.2438 ms/op 1.2026 ms/op 1.03
mainnet_e58758 - phase0 processSlashingsReset 10.484 us/op 7.6120 us/op 1.38
mainnet_e58758 - phase0 processRandaoMixesReset 7.7090 us/op 5.1890 us/op 1.49
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.0360 us/op 599.00 ns/op 1.73
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.7560 us/op 4.4810 us/op 1.06
mainnet_e58758 - phase0 afterProcessEpoch 102.01 ms/op 101.59 ms/op 1.00
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.8208 ms/op 1.2813 ms/op 1.42
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.9463 ms/op 1.4880 ms/op 1.98
altair processInactivityUpdates - 250000 normalcase 36.250 ms/op 26.792 ms/op 1.35
altair processInactivityUpdates - 250000 worstcase 28.405 ms/op 29.661 ms/op 0.96
phase0 processRegistryUpdates - 250000 normalcase 14.620 us/op 11.689 us/op 1.25
phase0 processRegistryUpdates - 250000 badcase_full_deposits 287.99 us/op 286.11 us/op 1.01
phase0 processRegistryUpdates - 250000 worstcase 0.5 130.66 ms/op 142.73 ms/op 0.92
altair processRewardsAndPenalties - 250000 normalcase 64.627 ms/op 71.428 ms/op 0.90
altair processRewardsAndPenalties - 250000 worstcase 71.463 ms/op 81.181 ms/op 0.88
phase0 getAttestationDeltas - 250000 normalcase 7.1777 ms/op 8.7636 ms/op 0.82
phase0 getAttestationDeltas - 250000 worstcase 7.7927 ms/op 8.6102 ms/op 0.91
phase0 processSlashings - 250000 worstcase 3.4587 ms/op 3.8264 ms/op 0.90
altair processSyncCommitteeUpdates - 250000 182.02 ms/op 194.82 ms/op 0.93
BeaconState.hashTreeRoot - No change 357.00 ns/op 294.00 ns/op 1.21
BeaconState.hashTreeRoot - 1 full validator 52.240 us/op 56.637 us/op 0.92
BeaconState.hashTreeRoot - 32 full validator 541.20 us/op 621.98 us/op 0.87
BeaconState.hashTreeRoot - 512 full validator 6.0245 ms/op 5.8540 ms/op 1.03
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 64.620 us/op 68.699 us/op 0.94
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 921.95 us/op 949.64 us/op 0.97
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 12.779 ms/op 13.139 ms/op 0.97
BeaconState.hashTreeRoot - 1 balances 50.544 us/op 54.735 us/op 0.92
BeaconState.hashTreeRoot - 32 balances 509.79 us/op 469.46 us/op 1.09
BeaconState.hashTreeRoot - 512 balances 4.4277 ms/op 4.6164 ms/op 0.96
BeaconState.hashTreeRoot - 250000 balances 75.713 ms/op 74.579 ms/op 1.02
aggregationBits - 2048 els - zipIndexesInBitList 17.966 us/op 18.266 us/op 0.98
regular array get 100000 times 37.044 us/op 35.059 us/op 1.06
wrappedArray get 100000 times 33.659 us/op 39.886 us/op 0.84
arrayWithProxy get 100000 times 17.208 ms/op 15.968 ms/op 1.08
ssz.Root.equals 583.00 ns/op 601.00 ns/op 0.97
byteArrayEquals 585.00 ns/op 600.00 ns/op 0.97
shuffle list - 16384 els 7.0371 ms/op 7.1355 ms/op 0.99
shuffle list - 250000 els 103.41 ms/op 104.35 ms/op 0.99
processSlot - 1 slots 9.0910 us/op 9.3710 us/op 0.97
processSlot - 32 slots 1.4063 ms/op 1.3898 ms/op 1.01
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 38.295 ms/op 38.503 ms/op 0.99
getCommitteeAssignments - req 1 vs - 250000 vc 3.0433 ms/op 2.9863 ms/op 1.02
getCommitteeAssignments - req 100 vs - 250000 vc 4.3834 ms/op 4.3075 ms/op 1.02
getCommitteeAssignments - req 1000 vs - 250000 vc 4.7737 ms/op 4.7006 ms/op 1.02
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.2200 ns/op 5.0700 ns/op 1.03
state getBlockRootAtSlot - 250000 vs - 7PWei 737.86 ns/op 1.1022 us/op 0.67
computeProposers - vc 250000 11.545 ms/op 11.485 ms/op 1.01
computeEpochShuffling - vc 250000 106.22 ms/op 105.82 ms/op 1.00
getNextSyncCommittee - vc 250000 195.55 ms/op 195.01 ms/op 1.00
computeSigningRoot for AttestationData 14.907 us/op 15.162 us/op 0.98
hash AttestationData serialized data then Buffer.toString(base64) 2.5508 us/op 2.5687 us/op 0.99
toHexString serialized data 1.3224 us/op 1.8395 us/op 0.72
Buffer.toString(base64) 367.59 ns/op 386.75 ns/op 0.95

by benchmarkbot/action

@twoeths
twoeths marked this pull request as ready for review April 21, 2023 12:41
@twoeths
twoeths requested a review from a team as a code owner April 21, 2023 12:41
wemeetagain
wemeetagain approved these changes Apr 21, 2023
View reviewed changes

@wemeetagain wemeetagain left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@twoeths
twoeths merged commit e746f9c into unstable Apr 21, 2023
@twoeths
twoeths deleted the tuyen/dos_protection_based_on_message_slot branch April 21, 2023 13:49
@twoeths

twoeths commented Apr 21, 2023

Copy link
Copy Markdown
Member Author

just merged in order to reassess our performance on unstable, feel free to drop some comments if you see anything to revise @dapplion

@philknows philknows mentioned this pull request Apr 23, 2023
Closed
@wemeetagain

wemeetagain commented May 4, 2023

Copy link
Copy Markdown
Member

馃帀 This PR is included in v1.8.0 馃帀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@twoeths @wemeetagain