Skip to content

Enable builder api spec tests - #4496

Merged
g11tech merged 2 commits into
unstablefrom
dapplion/spec-builder
Sep 2, 2022
Merged

Enable builder api spec tests#4496
g11tech merged 2 commits into
unstablefrom
dapplion/spec-builder

Conversation

@dapplion

@dapplion dapplion commented Sep 1, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation

We have no programmatic guarantee that Lodestar follows beacon APIs spec. In multiple instances it has been found that Lodestar had formatting issues, breaking interoperability.

Description

This PR, ensure that Lodestar follows a specific builder APIs spec version having correct:

  • URL + HTTP method
  • Request format, case and properties (params, query, body)
  • Response format, case and properties (request body on 200 response)

Base automatically changed from dapplion/api-spec-test to unstable September 1, 2022 15:04
@dapplion
dapplion force-pushed the dapplion/spec-builder branch from 5db3e1e to 90506c8 Compare September 1, 2022 18:44
@dapplion
dapplion force-pushed the dapplion/spec-builder branch from 90506c8 to 33e738c Compare September 1, 2022 18:50
@dapplion
dapplion marked this pull request as ready for review September 1, 2022 18:51
@dapplion
dapplion requested a review from a team as a code owner September 1, 2022 18:51
@github-actions

github-actions Bot commented Sep 1, 2022

Copy link
Copy Markdown
Contributor

Performance Report

鉁旓笍 no performance regression detected

Full benchmark results
Benchmark suite Current: bd70255 Previous: f447743 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.6104 ms/op 1.7149 ms/op 1.52
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 91.390 us/op 62.985 us/op 1.45
BLS verify - blst-native 2.2277 ms/op 1.6424 ms/op 1.36
BLS verifyMultipleSignatures 3 - blst-native 4.5630 ms/op 3.3412 ms/op 1.37
BLS verifyMultipleSignatures 8 - blst-native 9.8221 ms/op 7.2044 ms/op 1.36
BLS verifyMultipleSignatures 32 - blst-native 35.566 ms/op 26.144 ms/op 1.36
BLS aggregatePubkeys 32 - blst-native 47.515 us/op 34.631 us/op 1.37
BLS aggregatePubkeys 128 - blst-native 183.60 us/op 134.94 us/op 1.36
getAttestationsForBlock 209.07 ms/op 152.03 ms/op 1.38
isKnown best case - 1 super set check 531.00 ns/op 370.00 ns/op 1.44
isKnown normal case - 2 super set checks 524.00 ns/op 365.00 ns/op 1.44
isKnown worse case - 16 super set checks 525.00 ns/op 369.00 ns/op 1.42
CheckpointStateCache - add get delete 10.975 us/op 8.4870 us/op 1.29
validate gossip signedAggregateAndProof - struct 5.1490 ms/op 4.2703 ms/op 1.21
validate gossip attestation - struct 2.4348 ms/op 2.0413 ms/op 1.19
pickEth1Vote - no votes 2.6029 ms/op 2.1268 ms/op 1.22
pickEth1Vote - max votes 25.929 ms/op 18.610 ms/op 1.39
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.690 ms/op 10.764 ms/op 1.27
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 26.819 ms/op 19.445 ms/op 1.38
pickEth1Vote - Eth1Data fastSerialize value x2048 1.8314 ms/op 1.4848 ms/op 1.23
pickEth1Vote - Eth1Data fastSerialize tree x2048 16.649 ms/op 12.067 ms/op 1.38
bytes32 toHexString 1.4050 us/op 1.0210 us/op 1.38
bytes32 Buffer.toString(hex) 884.00 ns/op 696.00 ns/op 1.27
bytes32 Buffer.toString(hex) from Uint8Array 1.2130 us/op 951.00 ns/op 1.28
bytes32 Buffer.toString(hex) + 0x 882.00 ns/op 704.00 ns/op 1.25
Object access 1 prop 0.48100 ns/op 0.34100 ns/op 1.41
Map access 1 prop 0.35600 ns/op 0.29100 ns/op 1.22
Object get x1000 21.522 ns/op 18.578 ns/op 1.16
Map get x1000 1.2240 ns/op 0.98400 ns/op 1.24
Object set x1000 148.22 ns/op 117.48 ns/op 1.26
Map set x1000 91.172 ns/op 68.746 ns/op 1.33
Return object 10000 times 0.45040 ns/op 0.37610 ns/op 1.20
Throw Error 10000 times 7.3127 us/op 5.8786 us/op 1.24
enrSubnets - fastDeserialize 64 bits 3.4330 us/op 2.5200 us/op 1.36
enrSubnets - ssz BitVector 64 bits 971.00 ns/op 755.00 ns/op 1.29
enrSubnets - fastDeserialize 4 bits 498.00 ns/op 366.00 ns/op 1.36
enrSubnets - ssz BitVector 4 bits 964.00 ns/op 746.00 ns/op 1.29
prioritizePeers score -10:0 att 32-0.1 sync 2-0 117.22 us/op 81.146 us/op 1.44
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 148.16 us/op 117.42 us/op 1.26
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 270.16 us/op 195.26 us/op 1.38
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 620.34 us/op 350.39 us/op 1.77
prioritizePeers score 0:0 att 64-1 sync 4-1 575.25 us/op 409.78 us/op 1.40
RateTracker 1000000 limit, 1 obj count per request 241.81 ns/op 173.06 ns/op 1.40
RateTracker 1000000 limit, 2 obj count per request 182.99 ns/op 131.91 ns/op 1.39
RateTracker 1000000 limit, 4 obj count per request 147.22 ns/op 111.87 ns/op 1.32
RateTracker 1000000 limit, 8 obj count per request 134.39 ns/op 102.89 ns/op 1.31
RateTracker with prune 5.6150 us/op 3.6380 us/op 1.54
array of 16000 items push then shift 3.7682 us/op 3.0298 us/op 1.24
LinkedList of 16000 items push then shift 20.672 ns/op 17.401 ns/op 1.19
array of 16000 items push then pop 291.28 ns/op 238.17 ns/op 1.22
LinkedList of 16000 items push then pop 19.563 ns/op 16.548 ns/op 1.18
array of 24000 items push then shift 5.4826 us/op 4.1993 us/op 1.31
LinkedList of 24000 items push then shift 24.362 ns/op 19.798 ns/op 1.23
array of 24000 items push then pop 268.98 ns/op 202.31 ns/op 1.33
LinkedList of 24000 items push then pop 21.706 ns/op 16.312 ns/op 1.33
intersect bitArray bitLen 8 14.446 ns/op 10.382 ns/op 1.39
intersect array and set length 8 224.04 ns/op 141.97 ns/op 1.58
intersect bitArray bitLen 128 86.452 ns/op 54.522 ns/op 1.59
intersect array and set length 128 2.7710 us/op 1.9215 us/op 1.44
Buffer.concat 32 items 2.7450 ns/op 1.7100 ns/op 1.61
pass gossip attestations to forkchoice per slot 3.9343 ms/op 4.8330 ms/op 0.81
computeDeltas 4.3261 ms/op 3.4869 ms/op 1.24
computeProposerBoostScoreFromBalances 1.1059 ms/op 883.98 us/op 1.25
altair processAttestation - 250000 vs - 7PWei normalcase 4.6036 ms/op 3.5103 ms/op 1.31
altair processAttestation - 250000 vs - 7PWei worstcase 7.7302 ms/op 5.5636 ms/op 1.39
altair processAttestation - setStatus - 1/6 committees join 245.26 us/op 208.69 us/op 1.18
altair processAttestation - setStatus - 1/3 committees join 486.08 us/op 398.80 us/op 1.22
altair processAttestation - setStatus - 1/2 committees join 700.07 us/op 558.86 us/op 1.25
altair processAttestation - setStatus - 2/3 committees join 852.08 us/op 640.60 us/op 1.33
altair processAttestation - setStatus - 4/5 committees join 1.2245 ms/op 1.0025 ms/op 1.22
altair processAttestation - setStatus - 100% committees join 1.4834 ms/op 1.1827 ms/op 1.25
altair processBlock - 250000 vs - 7PWei normalcase 31.506 ms/op 26.661 ms/op 1.18
altair processBlock - 250000 vs - 7PWei normalcase hashState 44.068 ms/op 38.277 ms/op 1.15
altair processBlock - 250000 vs - 7PWei worstcase 107.32 ms/op 78.783 ms/op 1.36
altair processBlock - 250000 vs - 7PWei worstcase hashState 116.98 ms/op 97.805 ms/op 1.20
phase0 processBlock - 250000 vs - 7PWei normalcase 4.8121 ms/op 3.6422 ms/op 1.32
phase0 processBlock - 250000 vs - 7PWei worstcase 56.390 ms/op 47.474 ms/op 1.19
altair processEth1Data - 250000 vs - 7PWei normalcase 1.0199 ms/op 684.50 us/op 1.49
Tree 40 250000 create 999.73 ms/op 721.44 ms/op 1.39
Tree 40 250000 get(125000) 342.81 ns/op 252.91 ns/op 1.36
Tree 40 250000 set(125000) 2.9613 us/op 2.4401 us/op 1.21
Tree 40 250000 toArray() 37.295 ms/op 29.950 ms/op 1.25
Tree 40 250000 iterate all - toArray() + loop 37.748 ms/op 30.210 ms/op 1.25
Tree 40 250000 iterate all - get(i) 133.94 ms/op 100.47 ms/op 1.33
MutableVector 250000 create 19.769 ms/op 14.414 ms/op 1.37
MutableVector 250000 get(125000) 15.911 ns/op 13.247 ns/op 1.20
MutableVector 250000 set(125000) 833.06 ns/op 601.06 ns/op 1.39
MutableVector 250000 toArray() 9.3486 ms/op 7.5646 ms/op 1.24
MutableVector 250000 iterate all - toArray() + loop 9.5336 ms/op 7.8383 ms/op 1.22
MutableVector 250000 iterate all - get(i) 4.2048 ms/op 3.2834 ms/op 1.28
Array 250000 create 8.4024 ms/op 6.9479 ms/op 1.21
Array 250000 clone - spread 5.0182 ms/op 4.0650 ms/op 1.23
Array 250000 get(125000) 1.9830 ns/op 1.6840 ns/op 1.18
Array 250000 set(125000) 2.0110 ns/op 1.6790 ns/op 1.20
Array 250000 iterate all - loop 201.48 us/op 170.43 us/op 1.18
effectiveBalanceIncrements clone Uint8Array 300000 115.48 us/op 81.204 us/op 1.42
effectiveBalanceIncrements clone MutableVector 300000 1.5040 us/op 792.00 ns/op 1.90
effectiveBalanceIncrements rw all Uint8Array 300000 303.21 us/op 252.56 us/op 1.20
effectiveBalanceIncrements rw all MutableVector 300000 252.35 ms/op 164.63 ms/op 1.53
phase0 afterProcessEpoch - 250000 vs - 7PWei 243.36 ms/op 163.72 ms/op 1.49
phase0 beforeProcessEpoch - 250000 vs - 7PWei 79.801 ms/op 70.191 ms/op 1.14
altair processEpoch - mainnet_e81889 732.96 ms/op 542.87 ms/op 1.35
mainnet_e81889 - altair beforeProcessEpoch 112.48 ms/op 138.20 ms/op 0.81
mainnet_e81889 - altair processJustificationAndFinalization 38.033 us/op 32.860 us/op 1.16
mainnet_e81889 - altair processInactivityUpdates 13.825 ms/op 10.966 ms/op 1.26
mainnet_e81889 - altair processRewardsAndPenalties 186.43 ms/op 88.576 ms/op 2.10
mainnet_e81889 - altair processRegistryUpdates 7.9680 us/op 5.8230 us/op 1.37
mainnet_e81889 - altair processSlashings 1.4230 us/op 1.0290 us/op 1.38
mainnet_e81889 - altair processEth1DataReset 1.7120 us/op 1.0640 us/op 1.61
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.9760 ms/op 2.4015 ms/op 1.24
mainnet_e81889 - altair processSlashingsReset 13.174 us/op 9.1500 us/op 1.44
mainnet_e81889 - altair processRandaoMixesReset 12.367 us/op 9.3490 us/op 1.32
mainnet_e81889 - altair processHistoricalRootsUpdate 1.8110 us/op 1.6520 us/op 1.10
mainnet_e81889 - altair processParticipationFlagUpdates 6.6990 us/op 3.9490 us/op 1.70
mainnet_e81889 - altair processSyncCommitteeUpdates 1.8410 us/op 1.1230 us/op 1.64
mainnet_e81889 - altair afterProcessEpoch 232.68 ms/op 181.63 ms/op 1.28
phase0 processEpoch - mainnet_e58758 639.04 ms/op 504.60 ms/op 1.27
mainnet_e58758 - phase0 beforeProcessEpoch 310.54 ms/op 228.17 ms/op 1.36
mainnet_e58758 - phase0 processJustificationAndFinalization 35.053 us/op 28.378 us/op 1.24
mainnet_e58758 - phase0 processRewardsAndPenalties 94.639 ms/op 131.89 ms/op 0.72
mainnet_e58758 - phase0 processRegistryUpdates 17.965 us/op 16.720 us/op 1.07
mainnet_e58758 - phase0 processSlashings 1.8760 us/op 1.4200 us/op 1.32
mainnet_e58758 - phase0 processEth1DataReset 2.1590 us/op 1.4210 us/op 1.52
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.3825 ms/op 2.0956 ms/op 1.14
mainnet_e58758 - phase0 processSlashingsReset 9.7850 us/op 6.8660 us/op 1.43
mainnet_e58758 - phase0 processRandaoMixesReset 11.905 us/op 11.880 us/op 1.00
mainnet_e58758 - phase0 processHistoricalRootsUpdate 2.2410 us/op 1.3470 us/op 1.66
mainnet_e58758 - phase0 processParticipationRecordUpdates 13.110 us/op 5.9910 us/op 2.19
mainnet_e58758 - phase0 afterProcessEpoch 207.62 ms/op 151.67 ms/op 1.37
phase0 processEffectiveBalanceUpdates - 250000 normalcase 3.0512 ms/op 2.6252 ms/op 1.16
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 4.1840 ms/op 2.7357 ms/op 1.53
altair processInactivityUpdates - 250000 normalcase 49.777 ms/op 49.103 ms/op 1.01
altair processInactivityUpdates - 250000 worstcase 58.884 ms/op 39.545 ms/op 1.49
phase0 processRegistryUpdates - 250000 normalcase 15.099 us/op 12.769 us/op 1.18
phase0 processRegistryUpdates - 250000 badcase_full_deposits 507.17 us/op 549.94 us/op 0.92
phase0 processRegistryUpdates - 250000 worstcase 0.5 264.54 ms/op 219.47 ms/op 1.21
altair processRewardsAndPenalties - 250000 normalcase 166.02 ms/op 125.08 ms/op 1.33
altair processRewardsAndPenalties - 250000 worstcase 102.26 ms/op 84.445 ms/op 1.21
phase0 getAttestationDeltas - 250000 normalcase 18.147 ms/op 13.196 ms/op 1.38
phase0 getAttestationDeltas - 250000 worstcase 17.201 ms/op 13.431 ms/op 1.28
phase0 processSlashings - 250000 worstcase 6.4330 ms/op 5.4058 ms/op 1.19
altair processSyncCommitteeUpdates - 250000 345.75 ms/op 265.53 ms/op 1.30
BeaconState.hashTreeRoot - No change 572.00 ns/op 520.00 ns/op 1.10
BeaconState.hashTreeRoot - 1 full validator 72.870 us/op 55.893 us/op 1.30
BeaconState.hashTreeRoot - 32 full validator 760.37 us/op 595.28 us/op 1.28
BeaconState.hashTreeRoot - 512 full validator 7.6750 ms/op 7.1498 ms/op 1.07
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 103.52 us/op 73.393 us/op 1.41
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.4158 ms/op 1.0822 ms/op 1.31
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 19.095 ms/op 14.957 ms/op 1.28
BeaconState.hashTreeRoot - 1 balances 69.343 us/op 59.788 us/op 1.16
BeaconState.hashTreeRoot - 32 balances 700.31 us/op 580.99 us/op 1.21
BeaconState.hashTreeRoot - 512 balances 7.4116 ms/op 5.3622 ms/op 1.38
BeaconState.hashTreeRoot - 250000 balances 122.28 ms/op 87.238 ms/op 1.40
aggregationBits - 2048 els - zipIndexesInBitList 35.603 us/op 30.581 us/op 1.16
regular array get 100000 times 80.877 us/op 67.394 us/op 1.20
wrappedArray get 100000 times 80.713 us/op 67.390 us/op 1.20
arrayWithProxy get 100000 times 34.724 ms/op 29.312 ms/op 1.18
ssz.Root.equals 576.00 ns/op 479.00 ns/op 1.20
byteArrayEquals 554.00 ns/op 454.00 ns/op 1.22
shuffle list - 16384 els 13.272 ms/op 11.241 ms/op 1.18
shuffle list - 250000 els 193.97 ms/op 165.02 ms/op 1.18
processSlot - 1 slots 14.619 us/op 11.188 us/op 1.31
processSlot - 32 slots 2.0561 ms/op 1.7239 ms/op 1.19
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 472.11 us/op 435.07 us/op 1.09
getCommitteeAssignments - req 1 vs - 250000 vc 6.3269 ms/op 5.3365 ms/op 1.19
getCommitteeAssignments - req 100 vs - 250000 vc 9.0347 ms/op 7.3097 ms/op 1.24
getCommitteeAssignments - req 1000 vs - 250000 vc 9.4213 ms/op 7.7630 ms/op 1.21
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 11.790 ns/op 10.560 ns/op 1.12
state getBlockRootAtSlot - 250000 vs - 7PWei 1.4559 us/op 1.1220 us/op 1.30
computeProposers - vc 250000 20.314 ms/op 14.469 ms/op 1.40
computeEpochShuffling - vc 250000 204.82 ms/op 169.98 ms/op 1.20
getNextSyncCommittee - vc 250000 361.29 ms/op 265.63 ms/op 1.36

by benchmarkbot/action

g11tech
g11tech approved these changes Sep 2, 2022
View reviewed changes

@g11tech g11tech left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@g11tech
g11tech merged commit 04030be into unstable Sep 2, 2022
@g11tech
g11tech deleted the dapplion/spec-builder branch September 2, 2022 11:51
@snyk-bot snyk-bot mentioned this pull request Feb 11, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes
+1 more reviewer
@g11tech g11tech g11tech approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dapplion @wemeetagain @g11tech