Skip to content

Check API definitions against OpenAPI spec - #4493

Merged
wemeetagain merged 20 commits into
unstablefrom
dapplion/api-spec-test
Sep 1, 2022
Merged

Check API definitions against OpenAPI spec#4493
wemeetagain merged 20 commits into
unstablefrom
dapplion/api-spec-test

Conversation

@dapplion

@dapplion dapplion commented Aug 30, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation

We have no programmatic guarantee that Lodestar follows beacon APIs spec. In multiple instances it has been found that Lodestar had formatting issues, breaking interoperability.

Description

This PR, ensure that Lodestar follows a specific beacon APIs spec version having correct:

  • URL + HTTP method
  • Request format, case and properties (params, query, body)
  • Response format, case and properties (request body on 200 response)

Closes #4403
Closes #4063

@dapplion
dapplion requested a review from a team as a code owner August 30, 2022 16:33
@dapplion
dapplion marked this pull request as draft August 30, 2022 16:33
@github-actions

github-actions Bot commented Aug 31, 2022

Copy link
Copy Markdown
Contributor

Performance Report

鉁旓笍 no performance regression detected

Full benchmark results
Benchmark suite Current: 053921d Previous: 4e73138 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.9715 ms/op 1.7252 ms/op 1.72
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 90.813 us/op 64.641 us/op 1.40
BLS verify - blst-native 2.3592 ms/op 2.1643 ms/op 1.09
BLS verifyMultipleSignatures 3 - blst-native 4.9753 ms/op 4.4695 ms/op 1.11
BLS verifyMultipleSignatures 8 - blst-native 10.653 ms/op 9.6614 ms/op 1.10
BLS verifyMultipleSignatures 32 - blst-native 38.285 ms/op 35.127 ms/op 1.09
BLS aggregatePubkeys 32 - blst-native 50.455 us/op 46.565 us/op 1.08
BLS aggregatePubkeys 128 - blst-native 197.52 us/op 182.44 us/op 1.08
getAttestationsForBlock 230.19 ms/op 144.44 ms/op 1.59
isKnown best case - 1 super set check 545.00 ns/op 498.00 ns/op 1.09
isKnown normal case - 2 super set checks 535.00 ns/op 485.00 ns/op 1.10
isKnown worse case - 16 super set checks 542.00 ns/op 482.00 ns/op 1.12
CheckpointStateCache - add get delete 11.814 us/op 8.7720 us/op 1.35
validate gossip signedAggregateAndProof - struct 5.7918 ms/op 5.0235 ms/op 1.15
validate gossip attestation - struct 2.6346 ms/op 2.3762 ms/op 1.11
altair verifyImport mainnet_s3766816:31 5.8993 s/op 4.7644 s/op 1.24
pickEth1Vote - no votes 2.7113 ms/op 2.2322 ms/op 1.21
pickEth1Vote - max votes 30.841 ms/op 22.796 ms/op 1.35
pickEth1Vote - Eth1Data hashTreeRoot value x2048 15.540 ms/op 12.888 ms/op 1.21
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 32.840 ms/op 19.735 ms/op 1.66
pickEth1Vote - Eth1Data fastSerialize value x2048 1.9541 ms/op 1.4095 ms/op 1.39
pickEth1Vote - Eth1Data fastSerialize tree x2048 17.352 ms/op 12.611 ms/op 1.38
bytes32 toHexString 1.2920 us/op 874.00 ns/op 1.48
bytes32 Buffer.toString(hex) 817.00 ns/op 719.00 ns/op 1.14
bytes32 Buffer.toString(hex) from Uint8Array 1.1520 us/op 984.00 ns/op 1.17
bytes32 Buffer.toString(hex) + 0x 815.00 ns/op 735.00 ns/op 1.11
Object access 1 prop 0.44600 ns/op 0.32800 ns/op 1.36
Map access 1 prop 0.35600 ns/op 0.29600 ns/op 1.20
Object get x1000 17.948 ns/op 10.853 ns/op 1.65
Map get x1000 1.1210 ns/op 0.95200 ns/op 1.18
Object set x1000 127.02 ns/op 69.414 ns/op 1.83
Map set x1000 85.235 ns/op 45.791 ns/op 1.86
Return object 10000 times 0.44560 ns/op 0.43800 ns/op 1.02
Throw Error 10000 times 8.6227 us/op 6.1657 us/op 1.40
enrSubnets - fastDeserialize 64 bits 3.2980 us/op 2.3890 us/op 1.38
enrSubnets - ssz BitVector 64 bits 967.00 ns/op 755.00 ns/op 1.28
enrSubnets - fastDeserialize 4 bits 472.00 ns/op 369.00 ns/op 1.28
enrSubnets - ssz BitVector 4 bits 936.00 ns/op 803.00 ns/op 1.17
prioritizePeers score -10:0 att 32-0.1 sync 2-0 124.75 us/op 79.863 us/op 1.56
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 169.43 us/op 115.10 us/op 1.47
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 302.79 us/op 191.70 us/op 1.58
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 681.22 us/op 382.02 us/op 1.78
prioritizePeers score 0:0 att 64-1 sync 4-1 632.47 us/op 407.21 us/op 1.55
RateTracker 1000000 limit, 1 obj count per request 214.59 ns/op 176.29 ns/op 1.22
RateTracker 1000000 limit, 2 obj count per request 160.77 ns/op 127.18 ns/op 1.26
RateTracker 1000000 limit, 4 obj count per request 134.09 ns/op 102.78 ns/op 1.30
RateTracker 1000000 limit, 8 obj count per request 119.76 ns/op 92.353 ns/op 1.30
RateTracker with prune 5.7660 us/op 3.5260 us/op 1.64
array of 16000 items push then shift 5.9612 us/op 51.585 us/op 0.12
LinkedList of 16000 items push then shift 41.035 ns/op 15.116 ns/op 2.71
array of 16000 items push then pop 279.57 ns/op 207.72 ns/op 1.35
LinkedList of 16000 items push then pop 30.573 ns/op 14.124 ns/op 2.16
array of 24000 items push then shift 9.1627 us/op 77.357 us/op 0.12
LinkedList of 24000 items push then shift 41.076 ns/op 18.034 ns/op 2.28
array of 24000 items push then pop 271.76 ns/op 195.51 ns/op 1.39
LinkedList of 24000 items push then pop 30.989 ns/op 15.087 ns/op 2.05
intersect bitArray bitLen 8 13.619 ns/op 10.703 ns/op 1.27
intersect array and set length 8 206.31 ns/op 124.00 ns/op 1.66
intersect bitArray bitLen 128 78.310 ns/op 57.741 ns/op 1.36
intersect array and set length 128 2.6002 us/op 1.7092 us/op 1.52
Buffer.concat 32 items 2.4900 ns/op 1.7540 ns/op 1.42
pass gossip attestations to forkchoice per slot 5.8238 ms/op 2.8805 ms/op 2.02
computeDeltas 4.9947 ms/op 3.5140 ms/op 1.42
computeProposerBoostScoreFromBalances 939.79 us/op 814.25 us/op 1.15
altair processAttestation - 250000 vs - 7PWei normalcase 6.3265 ms/op 3.4943 ms/op 1.81
altair processAttestation - 250000 vs - 7PWei worstcase 11.984 ms/op 5.3029 ms/op 2.26
altair processAttestation - setStatus - 1/6 committees join 269.21 us/op 176.44 us/op 1.53
altair processAttestation - setStatus - 1/3 committees join 511.06 us/op 339.34 us/op 1.51
altair processAttestation - setStatus - 1/2 committees join 742.45 us/op 486.46 us/op 1.53
altair processAttestation - setStatus - 2/3 committees join 976.87 us/op 634.64 us/op 1.54
altair processAttestation - setStatus - 4/5 committees join 1.4313 ms/op 894.99 us/op 1.60
altair processAttestation - setStatus - 100% committees join 1.6764 ms/op 1.0689 ms/op 1.57
altair processBlock - 250000 vs - 7PWei normalcase 38.888 ms/op 25.423 ms/op 1.53
altair processBlock - 250000 vs - 7PWei normalcase hashState 47.746 ms/op 34.526 ms/op 1.38
altair processBlock - 250000 vs - 7PWei worstcase 119.32 ms/op 92.633 ms/op 1.29
altair processBlock - 250000 vs - 7PWei worstcase hashState 137.92 ms/op 109.84 ms/op 1.26
phase0 processBlock - 250000 vs - 7PWei normalcase 4.8273 ms/op 4.5496 ms/op 1.06
phase0 processBlock - 250000 vs - 7PWei worstcase 61.000 ms/op 52.222 ms/op 1.17
altair processEth1Data - 250000 vs - 7PWei normalcase 1.7617 ms/op 1.0311 ms/op 1.71
Tree 40 250000 create 1.1627 s/op 676.22 ms/op 1.72
Tree 40 250000 get(125000) 353.75 ns/op 256.70 ns/op 1.38
Tree 40 250000 set(125000) 3.4496 us/op 1.8822 us/op 1.83
Tree 40 250000 toArray() 42.471 ms/op 32.287 ms/op 1.32
Tree 40 250000 iterate all - toArray() + loop 43.200 ms/op 27.780 ms/op 1.56
Tree 40 250000 iterate all - get(i) 154.95 ms/op 113.92 ms/op 1.36
MutableVector 250000 create 27.991 ms/op 19.973 ms/op 1.40
MutableVector 250000 get(125000) 15.086 ns/op 10.630 ns/op 1.42
MutableVector 250000 set(125000) 829.43 ns/op 427.09 ns/op 1.94
MutableVector 250000 toArray() 12.067 ms/op 5.4909 ms/op 2.20
MutableVector 250000 iterate all - toArray() + loop 8.2352 ms/op 5.6792 ms/op 1.45
MutableVector 250000 iterate all - get(i) 3.6836 ms/op 2.7051 ms/op 1.36
Array 250000 create 7.6604 ms/op 5.4202 ms/op 1.41
Array 250000 clone - spread 3.0551 ms/op 2.4286 ms/op 1.26
Array 250000 get(125000) 1.3600 ns/op 1.1810 ns/op 1.15
Array 250000 set(125000) 1.3480 ns/op 1.1520 ns/op 1.17
Array 250000 iterate all - loop 153.71 us/op 153.07 us/op 1.00
effectiveBalanceIncrements clone Uint8Array 300000 112.67 us/op 92.055 us/op 1.22
effectiveBalanceIncrements clone MutableVector 300000 897.00 ns/op 632.00 ns/op 1.42
effectiveBalanceIncrements rw all Uint8Array 300000 307.86 us/op 247.14 us/op 1.25
effectiveBalanceIncrements rw all MutableVector 300000 257.96 ms/op 133.04 ms/op 1.94
phase0 afterProcessEpoch - 250000 vs - 7PWei 211.28 ms/op 187.61 ms/op 1.13
phase0 beforeProcessEpoch - 250000 vs - 7PWei 176.89 ms/op 113.87 ms/op 1.55
altair processEpoch - mainnet_e81889 877.44 ms/op 643.28 ms/op 1.36
mainnet_e81889 - altair beforeProcessEpoch 261.60 ms/op 170.28 ms/op 1.54
mainnet_e81889 - altair processJustificationAndFinalization 72.957 us/op 23.810 us/op 3.06
mainnet_e81889 - altair processInactivityUpdates 12.668 ms/op 8.6876 ms/op 1.46
mainnet_e81889 - altair processRewardsAndPenalties 231.65 ms/op 169.56 ms/op 1.37
mainnet_e81889 - altair processRegistryUpdates 14.962 us/op 4.1850 us/op 3.58
mainnet_e81889 - altair processSlashings 3.9080 us/op 967.00 ns/op 4.04
mainnet_e81889 - altair processEth1DataReset 4.2340 us/op 1.0090 us/op 4.20
mainnet_e81889 - altair processEffectiveBalanceUpdates 3.7094 ms/op 2.0679 ms/op 1.79
mainnet_e81889 - altair processSlashingsReset 30.580 us/op 5.5580 us/op 5.50
mainnet_e81889 - altair processRandaoMixesReset 26.285 us/op 5.9290 us/op 4.43
mainnet_e81889 - altair processHistoricalRootsUpdate 4.7240 us/op 792.00 ns/op 5.96
mainnet_e81889 - altair processParticipationFlagUpdates 15.566 us/op 2.7640 us/op 5.63
mainnet_e81889 - altair processSyncCommitteeUpdates 3.4310 us/op 650.00 ns/op 5.28
mainnet_e81889 - altair afterProcessEpoch 224.38 ms/op 197.89 ms/op 1.13
phase0 processEpoch - mainnet_e58758 878.68 ms/op 612.63 ms/op 1.43
mainnet_e58758 - phase0 beforeProcessEpoch 385.45 ms/op 220.07 ms/op 1.75
mainnet_e58758 - phase0 processJustificationAndFinalization 61.139 us/op 19.748 us/op 3.10
mainnet_e58758 - phase0 processRewardsAndPenalties 196.53 ms/op 124.18 ms/op 1.58
mainnet_e58758 - phase0 processRegistryUpdates 32.631 us/op 7.8640 us/op 4.15
mainnet_e58758 - phase0 processSlashings 3.0710 us/op 721.00 ns/op 4.26
mainnet_e58758 - phase0 processEth1DataReset 3.3550 us/op 804.00 ns/op 4.17
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.4267 ms/op 1.8135 ms/op 1.34
mainnet_e58758 - phase0 processSlashingsReset 18.603 us/op 6.0500 us/op 3.07
mainnet_e58758 - phase0 processRandaoMixesReset 24.640 us/op 6.3720 us/op 3.87
mainnet_e58758 - phase0 processHistoricalRootsUpdate 4.7110 us/op 1.0940 us/op 4.31
mainnet_e58758 - phase0 processParticipationRecordUpdates 26.160 us/op 5.5510 us/op 4.71
mainnet_e58758 - phase0 afterProcessEpoch 183.38 ms/op 162.89 ms/op 1.13
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.7098 ms/op 1.9354 ms/op 1.40
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.8948 ms/op 2.2244 ms/op 1.30
altair processInactivityUpdates - 250000 normalcase 95.444 ms/op 55.907 ms/op 1.71
altair processInactivityUpdates - 250000 worstcase 91.304 ms/op 53.227 ms/op 1.72
phase0 processRegistryUpdates - 250000 normalcase 28.521 us/op 9.3920 us/op 3.04
phase0 processRegistryUpdates - 250000 badcase_full_deposits 545.00 us/op 478.55 us/op 1.14
phase0 processRegistryUpdates - 250000 worstcase 0.5 336.54 ms/op 215.16 ms/op 1.56
altair processRewardsAndPenalties - 250000 normalcase 199.97 ms/op 131.78 ms/op 1.52
altair processRewardsAndPenalties - 250000 worstcase 204.19 ms/op 136.62 ms/op 1.49
phase0 getAttestationDeltas - 250000 normalcase 15.018 ms/op 11.669 ms/op 1.29
phase0 getAttestationDeltas - 250000 worstcase 15.705 ms/op 11.842 ms/op 1.33
phase0 processSlashings - 250000 worstcase 6.9101 ms/op 5.0874 ms/op 1.36
altair processSyncCommitteeUpdates - 250000 353.46 ms/op 292.52 ms/op 1.21
BeaconState.hashTreeRoot - No change 609.00 ns/op 505.00 ns/op 1.21
BeaconState.hashTreeRoot - 1 full validator 83.338 us/op 71.551 us/op 1.16
BeaconState.hashTreeRoot - 32 full validator 1.0575 ms/op 811.74 us/op 1.30
BeaconState.hashTreeRoot - 512 full validator 10.024 ms/op 6.8471 ms/op 1.46
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 105.88 us/op 99.981 us/op 1.06
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.6230 ms/op 1.2960 ms/op 1.25
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 24.301 ms/op 18.924 ms/op 1.28
BeaconState.hashTreeRoot - 1 balances 80.558 us/op 68.372 us/op 1.18
BeaconState.hashTreeRoot - 32 balances 934.92 us/op 727.87 us/op 1.28
BeaconState.hashTreeRoot - 512 balances 7.7839 ms/op 7.1291 ms/op 1.09
BeaconState.hashTreeRoot - 250000 balances 107.57 ms/op 117.49 ms/op 0.92
aggregationBits - 2048 els - zipIndexesInBitList 37.481 us/op 24.297 us/op 1.54
regular array get 100000 times 61.409 us/op 61.926 us/op 0.99
wrappedArray get 100000 times 62.403 us/op 60.568 us/op 1.03
arrayWithProxy get 100000 times 43.549 ms/op 27.057 ms/op 1.61
ssz.Root.equals 718.00 ns/op 427.00 ns/op 1.68
byteArrayEquals 587.00 ns/op 425.00 ns/op 1.38
shuffle list - 16384 els 12.839 ms/op 11.381 ms/op 1.13
shuffle list - 250000 els 188.41 ms/op 168.93 ms/op 1.12
processSlot - 1 slots 20.174 us/op 14.582 us/op 1.38
processSlot - 32 slots 2.4558 ms/op 1.9513 ms/op 1.26
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 405.97 us/op 403.08 us/op 1.01
getCommitteeAssignments - req 1 vs - 250000 vc 5.6925 ms/op 5.3718 ms/op 1.06
getCommitteeAssignments - req 100 vs - 250000 vc 8.1858 ms/op 7.8697 ms/op 1.04
getCommitteeAssignments - req 1000 vs - 250000 vc 9.1564 ms/op 8.4469 ms/op 1.08
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 10.970 ns/op 7.8300 ns/op 1.40
state getBlockRootAtSlot - 250000 vs - 7PWei 1.5350 us/op 1.1307 us/op 1.36
computeProposers - vc 250000 21.520 ms/op 17.634 ms/op 1.22
computeEpochShuffling - vc 250000 191.71 ms/op 171.68 ms/op 1.12
getNextSyncCommittee - vc 250000 367.06 ms/op 290.54 ms/op 1.26

by benchmarkbot/action

@dapplion dapplion mentioned this pull request Sep 1, 2022
Closed
@dapplion
dapplion marked this pull request as ready for review September 1, 2022 12:22
This was referenced Sep 1, 2022
Closed
Merged
Closed
wemeetagain
wemeetagain approved these changes Sep 1, 2022
View reviewed changes

@wemeetagain wemeetagain left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really nice, good job :)

@wemeetagain
wemeetagain merged commit 9194e21 into unstable Sep 1, 2022
@wemeetagain
wemeetagain deleted the dapplion/api-spec-test branch September 1, 2022 15:04
@dapplion dapplion mentioned this pull request Sep 20, 2022
Merged
@dapplion dapplion mentioned this pull request Dec 24, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Invalid validators index value while connected to nimbus beacon node BeaconAPI Spec Conformance

2 participants

@dapplion @wemeetagain