Skip to content

Add support for the dial only protocols - #5325

Merged
wemeetagain merged 6 commits into
unstablefrom
nh/dialonly-reqresp
Apr 3, 2023
Merged

Add support for the dial only protocols#5325
wemeetagain merged 6 commits into
unstablefrom
nh/dialonly-reqresp

Conversation

@nazarhussain

@nazarhussain nazarhussain commented Mar 30, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation

Add support for the dial only protocol in the reqresp package.

Description

While building Lightclient P2P transport for the Prover, came upon a scenario to use dial only protocols. Current reqresp package does not support it. So had to refactor the reqresp package.

Ground work for #4706

Note: The usage of this refactor will come in the upcoming PRs. Didn't wanted to blow up the scope of that prover PR, so splitted tasks in different small PRs.

Steps to test or reproduce

  • Run all tests.

@nazarhussain
nazarhussain requested a review from a team as a code owner March 30, 2023 16:09
@nazarhussain nazarhussain self-assigned this Mar 30, 2023
@github-actions

github-actions Bot commented Mar 30, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Performance Report

鉁旓笍 no performance regression detected

Full benchmark results
Benchmark suite Current: 2b655a0 Previous: 4f2d811 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.3148 ms/op 588.85 us/op 2.23
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 95.139 us/op 48.510 us/op 1.96
BLS verify - blst-native 2.2487 ms/op 1.2180 ms/op 1.85
BLS verifyMultipleSignatures 3 - blst-native 4.6114 ms/op 2.4668 ms/op 1.87
BLS verifyMultipleSignatures 8 - blst-native 10.190 ms/op 5.2735 ms/op 1.93
BLS verifyMultipleSignatures 32 - blst-native 38.584 ms/op 18.842 ms/op 2.05
BLS aggregatePubkeys 32 - blst-native 46.724 us/op 25.909 us/op 1.80
BLS aggregatePubkeys 128 - blst-native 179.51 us/op 100.71 us/op 1.78
getAttestationsForBlock 115.60 ms/op 61.163 ms/op 1.89
isKnown best case - 1 super set check 540.00 ns/op 266.00 ns/op 2.03
isKnown normal case - 2 super set checks 480.00 ns/op 258.00 ns/op 1.86
isKnown worse case - 16 super set checks 490.00 ns/op 257.00 ns/op 1.91
CheckpointStateCache - add get delete 9.9880 us/op 5.1720 us/op 1.93
validate gossip signedAggregateAndProof - struct 5.2402 ms/op 2.7743 ms/op 1.89
validate gossip attestation - struct 2.5409 ms/op 1.3221 ms/op 1.92
pickEth1Vote - no votes 2.8563 ms/op 1.2378 ms/op 2.31
pickEth1Vote - max votes 19.514 ms/op 9.6116 ms/op 2.03
pickEth1Vote - Eth1Data hashTreeRoot value x2048 19.329 ms/op 8.9216 ms/op 2.17
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 33.541 ms/op 17.079 ms/op 1.96
pickEth1Vote - Eth1Data fastSerialize value x2048 1.4472 ms/op 725.74 us/op 1.99
pickEth1Vote - Eth1Data fastSerialize tree x2048 12.143 ms/op 5.2241 ms/op 2.32
bytes32 toHexString 1.2160 us/op 518.00 ns/op 2.35
bytes32 Buffer.toString(hex) 793.00 ns/op 381.00 ns/op 2.08
bytes32 Buffer.toString(hex) from Uint8Array 1.1870 us/op 583.00 ns/op 2.04
bytes32 Buffer.toString(hex) + 0x 749.00 ns/op 385.00 ns/op 1.95
Object access 1 prop 0.37400 ns/op 0.17800 ns/op 2.10
Map access 1 prop 0.32900 ns/op 0.15800 ns/op 2.08
Object get x1000 12.750 ns/op 6.7590 ns/op 1.89
Map get x1000 1.3370 ns/op 0.66500 ns/op 2.01
Object set x1000 111.65 ns/op 73.084 ns/op 1.53
Map set x1000 98.117 ns/op 57.096 ns/op 1.72
Return object 10000 times 0.48450 ns/op 0.24430 ns/op 1.98
Throw Error 10000 times 9.8864 us/op 4.2177 us/op 2.34
fastMsgIdFn sha256 / 200 bytes 7.0450 us/op 3.4440 us/op 2.05
fastMsgIdFn h32 xxhash / 200 bytes 707.00 ns/op 297.00 ns/op 2.38
fastMsgIdFn h64 xxhash / 200 bytes 1.0400 us/op 402.00 ns/op 2.59
fastMsgIdFn sha256 / 1000 bytes 26.643 us/op 11.752 us/op 2.27
fastMsgIdFn h32 xxhash / 1000 bytes 1.0640 us/op 430.00 ns/op 2.47
fastMsgIdFn h64 xxhash / 1000 bytes 1.2010 us/op 534.00 ns/op 2.25
fastMsgIdFn sha256 / 10000 bytes 223.07 us/op 103.90 us/op 2.15
fastMsgIdFn h32 xxhash / 10000 bytes 4.0970 us/op 1.9920 us/op 2.06
fastMsgIdFn h64 xxhash / 10000 bytes 2.8700 us/op 1.4770 us/op 1.94
enrSubnets - fastDeserialize 64 bits 3.2890 us/op 1.3990 us/op 2.35
enrSubnets - ssz BitVector 64 bits 1.0390 us/op 515.00 ns/op 2.02
enrSubnets - fastDeserialize 4 bits 361.00 ns/op 181.00 ns/op 1.99
enrSubnets - ssz BitVector 4 bits 1.0120 us/op 513.00 ns/op 1.97
prioritizePeers score -10:0 att 32-0.1 sync 2-0 233.34 us/op 130.67 us/op 1.79
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 273.02 us/op 155.00 us/op 1.76
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 304.55 us/op 192.55 us/op 1.58
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 648.29 us/op 339.63 us/op 1.91
prioritizePeers score 0:0 att 64-1 sync 4-1 610.99 us/op 457.26 us/op 1.34
array of 16000 items push then shift 2.7395 us/op 1.6649 us/op 1.65
LinkedList of 16000 items push then shift 14.212 ns/op 9.0960 ns/op 1.56
array of 16000 items push then pop 152.11 ns/op 113.33 ns/op 1.34
LinkedList of 16000 items push then pop 14.314 ns/op 9.3180 ns/op 1.54
array of 24000 items push then shift 4.4490 us/op 2.3493 us/op 1.89
LinkedList of 24000 items push then shift 14.989 ns/op 8.8690 ns/op 1.69
array of 24000 items push then pop 142.20 ns/op 81.872 ns/op 1.74
LinkedList of 24000 items push then pop 15.528 ns/op 8.9200 ns/op 1.74
intersect bitArray bitLen 8 22.433 ns/op 13.556 ns/op 1.65
intersect array and set length 8 172.52 ns/op 94.894 ns/op 1.82
intersect bitArray bitLen 128 90.229 ns/op 44.874 ns/op 2.01
intersect array and set length 128 2.5016 us/op 1.2716 us/op 1.97
Buffer.concat 32 items 5.7950 us/op 2.9760 us/op 1.95
Uint8Array.set 32 items 5.5380 us/op 2.3140 us/op 2.39
pass gossip attestations to forkchoice per slot 6.3090 ms/op 2.8810 ms/op 2.19
computeDeltas 5.6837 ms/op 2.9073 ms/op 1.95
computeProposerBoostScoreFromBalances 3.0161 ms/op 1.8995 ms/op 1.59
altair processAttestation - 250000 vs - 7PWei normalcase 6.2840 ms/op 2.7053 ms/op 2.32
altair processAttestation - 250000 vs - 7PWei worstcase 8.3270 ms/op 3.8612 ms/op 2.16
altair processAttestation - setStatus - 1/6 committees join 292.87 us/op 153.25 us/op 1.91
altair processAttestation - setStatus - 1/3 committees join 530.39 us/op 287.27 us/op 1.85
altair processAttestation - setStatus - 1/2 committees join 806.48 us/op 396.54 us/op 2.03
altair processAttestation - setStatus - 2/3 committees join 971.88 us/op 480.61 us/op 2.02
altair processAttestation - setStatus - 4/5 committees join 1.0916 ms/op 674.39 us/op 1.62
altair processAttestation - setStatus - 100% committees join 1.0879 ms/op 803.38 us/op 1.35
altair processBlock - 250000 vs - 7PWei normalcase 21.233 ms/op 20.414 ms/op 1.04
altair processBlock - 250000 vs - 7PWei normalcase hashState 43.874 ms/op 27.500 ms/op 1.60
altair processBlock - 250000 vs - 7PWei worstcase 77.887 ms/op 55.704 ms/op 1.40
altair processBlock - 250000 vs - 7PWei worstcase hashState 103.19 ms/op 75.012 ms/op 1.38
phase0 processBlock - 250000 vs - 7PWei normalcase 3.4019 ms/op 2.4086 ms/op 1.41
phase0 processBlock - 250000 vs - 7PWei worstcase 35.986 ms/op 30.176 ms/op 1.19
altair processEth1Data - 250000 vs - 7PWei normalcase 653.36 us/op 526.67 us/op 1.24
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 7.4780 us/op 10.439 us/op 0.72
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 21.379 us/op 28.085 us/op 0.76
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 10.849 us/op 10.944 us/op 0.99
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 7.1790 us/op 10.408 us/op 0.69
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 82.016 us/op 124.03 us/op 0.66
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 937.92 us/op 777.14 us/op 1.21
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.2283 ms/op 1.0184 ms/op 1.21
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.1799 ms/op 1.0278 ms/op 1.15
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 3.2551 ms/op 2.4357 ms/op 1.34
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 2.4457 ms/op 1.8889 ms/op 1.29
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 6.9587 ms/op 4.4022 ms/op 1.58
Tree 40 250000 create 568.00 ms/op 354.91 ms/op 1.60
Tree 40 250000 get(125000) 267.33 ns/op 199.19 ns/op 1.34
Tree 40 250000 set(125000) 1.4043 us/op 981.21 ns/op 1.43
Tree 40 250000 toArray() 27.553 ms/op 21.993 ms/op 1.25
Tree 40 250000 iterate all - toArray() + loop 28.297 ms/op 20.371 ms/op 1.39
Tree 40 250000 iterate all - get(i) 105.52 ms/op 70.629 ms/op 1.49
MutableVector 250000 create 14.973 ms/op 13.188 ms/op 1.14
MutableVector 250000 get(125000) 11.270 ns/op 6.4740 ns/op 1.74
MutableVector 250000 set(125000) 576.55 ns/op 252.96 ns/op 2.28
MutableVector 250000 toArray() 5.1364 ms/op 2.8382 ms/op 1.81
MutableVector 250000 iterate all - toArray() + loop 5.5302 ms/op 3.0291 ms/op 1.83
MutableVector 250000 iterate all - get(i) 2.8350 ms/op 1.5503 ms/op 1.83
Array 250000 create 4.8820 ms/op 2.6948 ms/op 1.81
Array 250000 clone - spread 1.5380 ms/op 1.1887 ms/op 1.29
Array 250000 get(125000) 0.92000 ns/op 0.58600 ns/op 1.57
Array 250000 set(125000) 1.0030 ns/op 0.66600 ns/op 1.51
Array 250000 iterate all - loop 124.38 us/op 111.17 us/op 1.12
effectiveBalanceIncrements clone Uint8Array 300000 46.467 us/op 34.774 us/op 1.34
effectiveBalanceIncrements clone MutableVector 300000 501.00 ns/op 371.00 ns/op 1.35
effectiveBalanceIncrements rw all Uint8Array 300000 312.31 us/op 172.06 us/op 1.82
effectiveBalanceIncrements rw all MutableVector 300000 102.30 ms/op 83.180 ms/op 1.23
phase0 afterProcessEpoch - 250000 vs - 7PWei 154.56 ms/op 118.03 ms/op 1.31
phase0 beforeProcessEpoch - 250000 vs - 7PWei 46.403 ms/op 47.544 ms/op 0.98
altair processEpoch - mainnet_e81889 370.93 ms/op 338.35 ms/op 1.10
mainnet_e81889 - altair beforeProcessEpoch 68.207 ms/op 50.797 ms/op 1.34
mainnet_e81889 - altair processJustificationAndFinalization 24.930 us/op 24.042 us/op 1.04
mainnet_e81889 - altair processInactivityUpdates 5.9893 ms/op 6.5554 ms/op 0.91
mainnet_e81889 - altair processRewardsAndPenalties 48.800 ms/op 61.645 ms/op 0.79
mainnet_e81889 - altair processRegistryUpdates 3.4990 us/op 3.3160 us/op 1.06
mainnet_e81889 - altair processSlashings 546.00 ns/op 570.00 ns/op 0.96
mainnet_e81889 - altair processEth1DataReset 519.00 ns/op 660.00 ns/op 0.79
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2951 ms/op 1.7060 ms/op 0.76
mainnet_e81889 - altair processSlashingsReset 4.4420 us/op 4.8320 us/op 0.92
mainnet_e81889 - altair processRandaoMixesReset 4.1600 us/op 7.7980 us/op 0.53
mainnet_e81889 - altair processHistoricalRootsUpdate 588.00 ns/op 2.4370 us/op 0.24
mainnet_e81889 - altair processParticipationFlagUpdates 2.4060 us/op 6.7050 us/op 0.36
mainnet_e81889 - altair processSyncCommitteeUpdates 520.00 ns/op 911.00 ns/op 0.57
mainnet_e81889 - altair afterProcessEpoch 127.91 ms/op 130.87 ms/op 0.98
phase0 processEpoch - mainnet_e58758 371.82 ms/op 439.73 ms/op 0.85
mainnet_e58758 - phase0 beforeProcessEpoch 142.24 ms/op 152.22 ms/op 0.93
mainnet_e58758 - phase0 processJustificationAndFinalization 19.860 us/op 24.527 us/op 0.81
mainnet_e58758 - phase0 processRewardsAndPenalties 60.354 ms/op 68.182 ms/op 0.89
mainnet_e58758 - phase0 processRegistryUpdates 7.5220 us/op 12.314 us/op 0.61
mainnet_e58758 - phase0 processSlashings 559.00 ns/op 1.0310 us/op 0.54
mainnet_e58758 - phase0 processEth1DataReset 650.00 ns/op 1.0230 us/op 0.64
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.2622 ms/op 1.3584 ms/op 0.93
mainnet_e58758 - phase0 processSlashingsReset 3.5980 us/op 5.0260 us/op 0.72
mainnet_e58758 - phase0 processRandaoMixesReset 5.7770 us/op 4.9140 us/op 1.18
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.1970 us/op 1.2830 us/op 0.93
mainnet_e58758 - phase0 processParticipationRecordUpdates 6.1950 us/op 6.0390 us/op 1.03
mainnet_e58758 - phase0 afterProcessEpoch 116.84 ms/op 109.38 ms/op 1.07
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.4808 ms/op 1.2999 ms/op 1.14
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.8501 ms/op 2.1135 ms/op 0.88
altair processInactivityUpdates - 250000 normalcase 30.251 ms/op 25.528 ms/op 1.19
altair processInactivityUpdates - 250000 worstcase 30.627 ms/op 27.745 ms/op 1.10
phase0 processRegistryUpdates - 250000 normalcase 7.9220 us/op 9.8140 us/op 0.81
phase0 processRegistryUpdates - 250000 badcase_full_deposits 272.68 us/op 372.91 us/op 0.73
phase0 processRegistryUpdates - 250000 worstcase 0.5 154.36 ms/op 144.92 ms/op 1.07
altair processRewardsAndPenalties - 250000 normalcase 77.774 ms/op 72.092 ms/op 1.08
altair processRewardsAndPenalties - 250000 worstcase 74.996 ms/op 71.236 ms/op 1.05
phase0 getAttestationDeltas - 250000 normalcase 7.7830 ms/op 7.4170 ms/op 1.05
phase0 getAttestationDeltas - 250000 worstcase 8.8341 ms/op 8.4700 ms/op 1.04
phase0 processSlashings - 250000 worstcase 4.0009 ms/op 4.0751 ms/op 0.98
altair processSyncCommitteeUpdates - 250000 230.21 ms/op 187.39 ms/op 1.23
BeaconState.hashTreeRoot - No change 332.00 ns/op 380.00 ns/op 0.87
BeaconState.hashTreeRoot - 1 full validator 65.758 us/op 58.072 us/op 1.13
BeaconState.hashTreeRoot - 32 full validator 727.83 us/op 669.27 us/op 1.09
BeaconState.hashTreeRoot - 512 full validator 7.0043 ms/op 6.9645 ms/op 1.01
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 79.443 us/op 73.072 us/op 1.09
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.1363 ms/op 987.01 us/op 1.15
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 14.529 ms/op 13.257 ms/op 1.10
BeaconState.hashTreeRoot - 1 balances 66.426 us/op 60.673 us/op 1.09
BeaconState.hashTreeRoot - 32 balances 561.61 us/op 517.86 us/op 1.08
BeaconState.hashTreeRoot - 512 balances 5.3598 ms/op 5.1498 ms/op 1.04
BeaconState.hashTreeRoot - 250000 balances 98.885 ms/op 82.747 ms/op 1.20
aggregationBits - 2048 els - zipIndexesInBitList 21.007 us/op 19.788 us/op 1.06
regular array get 100000 times 42.786 us/op 46.504 us/op 0.92
wrappedArray get 100000 times 39.299 us/op 41.900 us/op 0.94
arrayWithProxy get 100000 times 21.737 ms/op 16.710 ms/op 1.30
ssz.Root.equals 700.00 ns/op 593.00 ns/op 1.18
byteArrayEquals 702.00 ns/op 595.00 ns/op 1.18
shuffle list - 16384 els 8.5896 ms/op 7.0990 ms/op 1.21
shuffle list - 250000 els 130.49 ms/op 105.58 ms/op 1.24
processSlot - 1 slots 11.870 us/op 10.013 us/op 1.19
processSlot - 32 slots 1.7535 ms/op 1.4258 ms/op 1.23
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 43.111 ms/op 40.240 ms/op 1.07
getCommitteeAssignments - req 1 vs - 250000 vc 4.1598 ms/op 2.9702 ms/op 1.40
getCommitteeAssignments - req 100 vs - 250000 vc 5.5580 ms/op 4.2763 ms/op 1.30
getCommitteeAssignments - req 1000 vs - 250000 vc 6.1093 ms/op 4.5780 ms/op 1.33
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.9700 ns/op 5.2400 ns/op 1.14
state getBlockRootAtSlot - 250000 vs - 7PWei 1.0244 us/op 953.70 ns/op 1.07
computeProposers - vc 250000 14.187 ms/op 12.167 ms/op 1.17
computeEpochShuffling - vc 250000 132.60 ms/op 111.28 ms/op 1.19
getNextSyncCommittee - vc 250000 241.48 ms/op 196.63 ms/op 1.23

by benchmarkbot/action

@dapplion

dapplion commented Mar 31, 2023

Copy link
Copy Markdown
Contributor

Since DuplexProtocolDefinition is the most used, why not keep it named ProtocolDefinition and then use a special name for dial only protocols?

@nazarhussain

nazarhussain commented Mar 31, 2023

Copy link
Copy Markdown
Contributor Author

Since DuplexProtocolDefinition is the most used, why not keep it named ProtocolDefinition and then use a special name for dial only protocols?

That could be confusing to name the union of both. What about using MixedProtocolDefinition for the union?

wemeetagain
wemeetagain reviewed Apr 3, 2023
View reviewed changes
Comment thread packages/reqresp/src/types.ts
renderRequestBody?: never;
}

export type MixedProtocolDefinition<Req = unknown, Resp = unknown> =

@wemeetagain wemeetagain Apr 3, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add jsdoc comments for these types / interfaces

@nazarhussain nazarhussain Apr 3, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wemeetagain done

@nazarhussain
nazarhussain requested a review from wemeetagain April 3, 2023 19:18
@wemeetagain
wemeetagain merged commit 9bf2ce1 into unstable Apr 3, 2023
@wemeetagain
wemeetagain deleted the nh/dialonly-reqresp branch April 3, 2023 19:51
@wemeetagain

wemeetagain commented May 4, 2023

Copy link
Copy Markdown
Member

馃帀 This PR is included in v1.8.0 馃帀

@nflaig nflaig mentioned this pull request Feb 19, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees

@nazarhussain nazarhussain

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nazarhussain @dapplion @wemeetagain