fix: revert level update

[twoeths]

Motivation

classic-level was upgraded from 1.2.0 to 1.4.1 in #6795 and it caused a lot of gc in docker deployment

This PR is created on behalf of @matthewkeil

Description

• The issue only happen in docker deployment probably due to this step

  lodestar/Dockerfile

  Line 34 in 8b6ecc4

  RUN cd node_modules/classic-level && yarn rebuild

• I'm not sure if it's too much to revert this PR, looks like the issue it resolves Replace leveldown with classic-level in db tests #6794 is optional for v1.19.0 so we can add it back with classic-level v1.20.0 later then create another issue for classic-level v1.4.1

closes #6826
TODOs

The above hypothesis is not confirmed until the below step:

• reproduce in a service deployment node with extra step as mentioned above
• test this branch with docker deployment

[jeluard]

At first glance looks like setting classic-level version to 1.2.0 would do it? This would give us more control on what version is used and prevent accidental upgrades.

Not a lot of changes with 1.4.1 but some are related to napi refactorings.
Given that this lib doesn't appear super active and our surface area with leveldb is thin, might be a potential future candidate for some rust based binding.

[matthewkeil]

Thanks @twoeths !! I was looking at metrics of fea1 and feat3 which both have about the same code (node 20 and all other 1.19 commits) and only feat1 shows that weird db pattern. feat3 and feat4 (the branch without the level update) look the same with a "normal" pattern of db activity. Not entirely convinced this is the root cause we are hunting for sadly....

feat1

feat3

feat4

[matthewkeil]

At first glance looks like setting classic-level version to 1.2.0 would do it? This would give us more control on what version is used and prevent accidental upgrades.

Not a lot of changes with 1.4.1 but some are related to napi refactorings. Given that this lib doesn't appear super active and our surface area with leveldb is thin, might be a potential future candidate for some rust based binding.

I'm also curious if the base c implementation had any updates. Will take a peek. We had a memory leak in batch processing that came out of the native layer so could be that its not in the bindings or js layer.

Update: Last LevelDB update was quite a while ago... so you are def right @jeluard if level proves to be the issue we are hunting for

https://github.com/google/leveldb/releases

[github-actions]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: 7a09af8	Previous: 8b6ecc4	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	950.42 us/op	938.75 us/op	1.01
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	46.179 us/op	48.287 us/op	0.96
BLS verify - blst-native	1.0460 ms/op	1.1825 ms/op	0.88
BLS verifyMultipleSignatures 3 - blst-native	2.1805 ms/op	2.4792 ms/op	0.88
BLS verifyMultipleSignatures 8 - blst-native	4.8223 ms/op	5.3882 ms/op	0.89
BLS verifyMultipleSignatures 32 - blst-native	17.398 ms/op	19.694 ms/op	0.88
BLS verifyMultipleSignatures 64 - blst-native	34.378 ms/op	38.632 ms/op	0.89
BLS verifyMultipleSignatures 128 - blst-native	68.345 ms/op	75.759 ms/op	0.90
BLS deserializing 10000 signatures	771.51 ms/op	860.86 ms/op	0.90
BLS deserializing 100000 signatures	7.6410 s/op	8.8004 s/op	0.87
BLS verifyMultipleSignatures - same message - 3 - blst-native	1.0916 ms/op	1.2651 ms/op	0.86
BLS verifyMultipleSignatures - same message - 8 - blst-native	1.2539 ms/op	1.4332 ms/op	0.87
BLS verifyMultipleSignatures - same message - 32 - blst-native	1.9724 ms/op	2.2299 ms/op	0.88
BLS verifyMultipleSignatures - same message - 64 - blst-native	2.9320 ms/op	3.3273 ms/op	0.88
BLS verifyMultipleSignatures - same message - 128 - blst-native	4.9508 ms/op	5.4543 ms/op	0.91
BLS aggregatePubkeys 32 - blst-native	24.306 us/op	27.016 us/op	0.90
BLS aggregatePubkeys 128 - blst-native	93.775 us/op	105.64 us/op	0.89
notSeenSlots=1 numMissedVotes=1 numBadVotes=10	49.942 ms/op	50.345 ms/op	0.99
notSeenSlots=1 numMissedVotes=0 numBadVotes=4	50.100 ms/op	48.830 ms/op	1.03
notSeenSlots=2 numMissedVotes=1 numBadVotes=10	30.904 ms/op	29.092 ms/op	1.06
getSlashingsAndExits - default max	114.74 us/op	89.072 us/op	1.29
getSlashingsAndExits - 2k	355.87 us/op	258.60 us/op	1.38
proposeBlockBody type=full, size=empty	4.6223 ms/op	5.1887 ms/op	0.89
isKnown best case - 1 super set check	584.00 ns/op	298.00 ns/op	1.96
isKnown normal case - 2 super set checks	516.00 ns/op	288.00 ns/op	1.79
isKnown worse case - 16 super set checks	499.00 ns/op	301.00 ns/op	1.66
InMemoryCheckpointStateCache - add get delete	4.4620 us/op	4.3630 us/op	1.02
validate api signedAggregateAndProof - struct	2.2434 ms/op	2.4355 ms/op	0.92
validate gossip signedAggregateAndProof - struct	2.2666 ms/op	2.4335 ms/op	0.93
validate gossip attestation - vc 640000	1.0869 ms/op	1.1844 ms/op	0.92
batch validate gossip attestation - vc 640000 - chunk 32	137.09 us/op	152.42 us/op	0.90
batch validate gossip attestation - vc 640000 - chunk 64	123.46 us/op	134.15 us/op	0.92
batch validate gossip attestation - vc 640000 - chunk 128	109.33 us/op	135.13 us/op	0.81
batch validate gossip attestation - vc 640000 - chunk 256	114.76 us/op	135.43 us/op	0.85
pickEth1Vote - no votes	829.89 us/op	1.1063 ms/op	0.75
pickEth1Vote - max votes	8.3045 ms/op	9.7620 ms/op	0.85
pickEth1Vote - Eth1Data hashTreeRoot value x2048	17.568 ms/op	20.370 ms/op	0.86
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	21.987 ms/op	19.802 ms/op	1.11
pickEth1Vote - Eth1Data fastSerialize value x2048	438.09 us/op	513.50 us/op	0.85
pickEth1Vote - Eth1Data fastSerialize tree x2048	5.9184 ms/op	7.5258 ms/op	0.79
bytes32 toHexString	705.00 ns/op	533.00 ns/op	1.32
bytes32 Buffer.toString(hex)	451.00 ns/op	249.00 ns/op	1.81
bytes32 Buffer.toString(hex) from Uint8Array	615.00 ns/op	380.00 ns/op	1.62
bytes32 Buffer.toString(hex) + 0x	450.00 ns/op	266.00 ns/op	1.69
Object access 1 prop	0.33600 ns/op	0.14700 ns/op	2.29
Map access 1 prop	0.32000 ns/op	0.13300 ns/op	2.41
Object get x1000	5.0410 ns/op	6.0840 ns/op	0.83
Map get x1000	5.7030 ns/op	7.0500 ns/op	0.81
Object set x1000	24.775 ns/op	35.508 ns/op	0.70
Map set x1000	20.291 ns/op	24.734 ns/op	0.82
Return object 10000 times	0.30710 ns/op	0.29970 ns/op	1.02
Throw Error 10000 times	2.6689 us/op	3.5739 us/op	0.75
fastMsgIdFn sha256 / 200 bytes	2.0420 us/op	2.2180 us/op	0.92
fastMsgIdFn h32 xxhash / 200 bytes	466.00 ns/op	233.00 ns/op	2.00
fastMsgIdFn h64 xxhash / 200 bytes	456.00 ns/op	265.00 ns/op	1.72
fastMsgIdFn sha256 / 1000 bytes	6.0410 us/op	7.6520 us/op	0.79
fastMsgIdFn h32 xxhash / 1000 bytes	566.00 ns/op	391.00 ns/op	1.45
fastMsgIdFn h64 xxhash / 1000 bytes	520.00 ns/op	349.00 ns/op	1.49
fastMsgIdFn sha256 / 10000 bytes	50.209 us/op	66.804 us/op	0.75
fastMsgIdFn h32 xxhash / 10000 bytes	1.9280 us/op	1.9370 us/op	1.00
fastMsgIdFn h64 xxhash / 10000 bytes	1.3350 us/op	1.2560 us/op	1.06
send data - 1000 256B messages	10.294 ms/op	13.367 ms/op	0.77
send data - 1000 512B messages	13.646 ms/op	19.066 ms/op	0.72
send data - 1000 1024B messages	21.874 ms/op	27.968 ms/op	0.78
send data - 1000 1200B messages	23.629 ms/op	28.272 ms/op	0.84
send data - 1000 2048B messages	29.265 ms/op	32.347 ms/op	0.90
send data - 1000 4096B messages	26.213 ms/op	33.821 ms/op	0.78
send data - 1000 16384B messages	63.504 ms/op	78.672 ms/op	0.81
send data - 1000 65536B messages	248.66 ms/op	221.38 ms/op	1.12
enrSubnets - fastDeserialize 64 bits	1.0780 us/op	1.1220 us/op	0.96
enrSubnets - ssz BitVector 64 bits	527.00 ns/op	351.00 ns/op	1.50
enrSubnets - fastDeserialize 4 bits	328.00 ns/op	151.00 ns/op	2.17
enrSubnets - ssz BitVector 4 bits	524.00 ns/op	375.00 ns/op	1.40
prioritizePeers score -10:0 att 32-0.1 sync 2-0	119.16 us/op	157.00 us/op	0.76
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	146.13 us/op	152.62 us/op	0.96
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	227.70 us/op	271.88 us/op	0.84
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	545.51 us/op	404.83 us/op	1.35
prioritizePeers score 0:0 att 64-1 sync 4-1	660.52 us/op	602.36 us/op	1.10
array of 16000 items push then shift	1.2475 us/op	1.7163 us/op	0.73
LinkedList of 16000 items push then shift	6.6710 ns/op	7.5930 ns/op	0.88
array of 16000 items push then pop	79.965 ns/op	122.57 ns/op	0.65
LinkedList of 16000 items push then pop	6.3580 ns/op	7.1560 ns/op	0.89
array of 24000 items push then shift	1.9059 us/op	2.4802 us/op	0.77
LinkedList of 24000 items push then shift	6.4780 ns/op	7.3270 ns/op	0.88
array of 24000 items push then pop	110.38 ns/op	137.18 ns/op	0.80
LinkedList of 24000 items push then pop	6.3800 ns/op	7.6490 ns/op	0.83
intersect bitArray bitLen 8	5.5760 ns/op	6.7810 ns/op	0.82
intersect array and set length 8	41.300 ns/op	48.769 ns/op	0.85
intersect bitArray bitLen 128	26.786 ns/op	31.000 ns/op	0.86
intersect array and set length 128	597.15 ns/op	738.23 ns/op	0.81
bitArray.getTrueBitIndexes() bitLen 128	2.5740 us/op	1.9830 us/op	1.30
bitArray.getTrueBitIndexes() bitLen 248	3.1170 us/op	3.8110 us/op	0.82
bitArray.getTrueBitIndexes() bitLen 512	6.3590 us/op	7.7020 us/op	0.83
Buffer.concat 32 items	1.1380 us/op	1.0040 us/op	1.13
Uint8Array.set 32 items	2.0440 us/op	1.4950 us/op	1.37
Buffer.copy	2.2320 us/op	1.6940 us/op	1.32
Uint8Array.set - with subarray	2.4040 us/op	2.9030 us/op	0.83
Uint8Array.set - without subarray	1.5200 us/op	1.4940 us/op	1.02
Set add up to 64 items then delete first	1.7888 us/op	2.6886 us/op	0.67
OrderedSet add up to 64 items then delete first	2.7586 us/op	3.8843 us/op	0.71
Set add up to 64 items then delete last	1.9702 us/op	3.0223 us/op	0.65
OrderedSet add up to 64 items then delete last	2.9862 us/op	3.9977 us/op	0.75
Set add up to 64 items then delete middle	1.9640 us/op	2.5951 us/op	0.76
OrderedSet add up to 64 items then delete middle	4.3472 us/op	5.5474 us/op	0.78
Set add up to 128 items then delete first	3.8845 us/op	5.2599 us/op	0.74
OrderedSet add up to 128 items then delete first	6.1525 us/op	7.6838 us/op	0.80
Set add up to 128 items then delete last	3.7498 us/op	5.1576 us/op	0.73
OrderedSet add up to 128 items then delete last	5.7408 us/op	7.9401 us/op	0.72
Set add up to 128 items then delete middle	3.7579 us/op	5.2871 us/op	0.71
OrderedSet add up to 128 items then delete middle	11.447 us/op	14.662 us/op	0.78
Set add up to 256 items then delete first	7.6067 us/op	10.655 us/op	0.71
OrderedSet add up to 256 items then delete first	12.157 us/op	16.337 us/op	0.74
Set add up to 256 items then delete last	7.3250 us/op	9.7729 us/op	0.75
OrderedSet add up to 256 items then delete last	11.525 us/op	14.910 us/op	0.77
Set add up to 256 items then delete middle	7.3044 us/op	9.9254 us/op	0.74
OrderedSet add up to 256 items then delete middle	33.070 us/op	42.476 us/op	0.78
transfer serialized Status (84 B)	1.4900 us/op	1.5230 us/op	0.98
copy serialized Status (84 B)	1.2880 us/op	1.2640 us/op	1.02
transfer serialized SignedVoluntaryExit (112 B)	1.6070 us/op	1.6770 us/op	0.96
copy serialized SignedVoluntaryExit (112 B)	1.3420 us/op	1.3630 us/op	0.98
transfer serialized ProposerSlashing (416 B)	2.1380 us/op	2.3350 us/op	0.92
copy serialized ProposerSlashing (416 B)	1.7320 us/op	2.3290 us/op	0.74
transfer serialized Attestation (485 B)	1.5810 us/op	1.7670 us/op	0.89
copy serialized Attestation (485 B)	1.4090 us/op	1.6200 us/op	0.87
transfer serialized AttesterSlashing (33232 B)	1.7030 us/op	1.7960 us/op	0.95
copy serialized AttesterSlashing (33232 B)	4.9210 us/op	5.3770 us/op	0.92
transfer serialized Small SignedBeaconBlock (128000 B)	2.2710 us/op	2.5500 us/op	0.89
copy serialized Small SignedBeaconBlock (128000 B)	10.504 us/op	17.098 us/op	0.61
transfer serialized Avg SignedBeaconBlock (200000 B)	2.9180 us/op	3.1060 us/op	0.94
copy serialized Avg SignedBeaconBlock (200000 B)	21.838 us/op	22.921 us/op	0.95
transfer serialized BlobsSidecar (524380 B)	2.6270 us/op	3.1200 us/op	0.84
copy serialized BlobsSidecar (524380 B)	137.46 us/op	81.207 us/op	1.69
transfer serialized Big SignedBeaconBlock (1000000 B)	3.3910 us/op	2.8550 us/op	1.19
copy serialized Big SignedBeaconBlock (1000000 B)	150.21 us/op	157.49 us/op	0.95
pass gossip attestations to forkchoice per slot	2.6452 ms/op	3.1808 ms/op	0.83
forkChoice updateHead vc 100000 bc 64 eq 0	390.08 us/op	507.21 us/op	0.77
forkChoice updateHead vc 600000 bc 64 eq 0	2.6916 ms/op	3.1880 ms/op	0.84
forkChoice updateHead vc 1000000 bc 64 eq 0	4.9083 ms/op	5.4507 ms/op	0.90
forkChoice updateHead vc 600000 bc 320 eq 0	2.6588 ms/op	3.0998 ms/op	0.86
forkChoice updateHead vc 600000 bc 1200 eq 0	2.7962 ms/op	3.3082 ms/op	0.85
forkChoice updateHead vc 600000 bc 7200 eq 0	3.9494 ms/op	3.9695 ms/op	0.99
forkChoice updateHead vc 600000 bc 64 eq 1000	10.068 ms/op	11.389 ms/op	0.88
forkChoice updateHead vc 600000 bc 64 eq 10000	10.084 ms/op	11.522 ms/op	0.88
forkChoice updateHead vc 600000 bc 64 eq 300000	17.845 ms/op	15.355 ms/op	1.16
computeDeltas 500000 validators 300 proto nodes	3.0784 ms/op	3.7209 ms/op	0.83
computeDeltas 500000 validators 1200 proto nodes	3.0183 ms/op	3.8218 ms/op	0.79
computeDeltas 500000 validators 7200 proto nodes	2.9701 ms/op	4.1058 ms/op	0.72
computeDeltas 750000 validators 300 proto nodes	4.4899 ms/op	6.5951 ms/op	0.68
computeDeltas 750000 validators 1200 proto nodes	4.5703 ms/op	6.1981 ms/op	0.74
computeDeltas 750000 validators 7200 proto nodes	4.4296 ms/op	5.9281 ms/op	0.75
computeDeltas 1400000 validators 300 proto nodes	8.4489 ms/op	10.811 ms/op	0.78
computeDeltas 1400000 validators 1200 proto nodes	8.3202 ms/op	10.407 ms/op	0.80
computeDeltas 1400000 validators 7200 proto nodes	8.2665 ms/op	10.135 ms/op	0.82
computeDeltas 2100000 validators 300 proto nodes	12.939 ms/op	14.836 ms/op	0.87
computeDeltas 2100000 validators 1200 proto nodes	12.309 ms/op	14.403 ms/op	0.85
computeDeltas 2100000 validators 7200 proto nodes	12.508 ms/op	14.908 ms/op	0.84
altair processAttestation - 250000 vs - 7PWei normalcase	1.4483 ms/op	1.7732 ms/op	0.82
altair processAttestation - 250000 vs - 7PWei worstcase	2.1048 ms/op	2.6951 ms/op	0.78
altair processAttestation - setStatus - 1/6 committees join	98.544 us/op	88.826 us/op	1.11
altair processAttestation - setStatus - 1/3 committees join	170.72 us/op	184.30 us/op	0.93
altair processAttestation - setStatus - 1/2 committees join	264.19 us/op	246.23 us/op	1.07
altair processAttestation - setStatus - 2/3 committees join	338.12 us/op	334.35 us/op	1.01
altair processAttestation - setStatus - 4/5 committees join	480.23 us/op	469.32 us/op	1.02
altair processAttestation - setStatus - 100% committees join	575.27 us/op	567.62 us/op	1.01
altair processBlock - 250000 vs - 7PWei normalcase	3.4754 ms/op	4.3318 ms/op	0.80
altair processBlock - 250000 vs - 7PWei normalcase hashState	19.952 ms/op	26.430 ms/op	0.75
altair processBlock - 250000 vs - 7PWei worstcase	33.833 ms/op	39.800 ms/op	0.85
altair processBlock - 250000 vs - 7PWei worstcase hashState	67.942 ms/op	85.064 ms/op	0.80
phase0 processBlock - 250000 vs - 7PWei normalcase	1.3275 ms/op	2.2540 ms/op	0.59
phase0 processBlock - 250000 vs - 7PWei worstcase	23.068 ms/op	26.556 ms/op	0.87
altair processEth1Data - 250000 vs - 7PWei normalcase	257.94 us/op	409.92 us/op	0.63
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	5.0680 us/op	6.5370 us/op	0.78
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	19.297 us/op	24.866 us/op	0.78
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	7.5300 us/op	9.0260 us/op	0.83
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	5.3350 us/op	6.1080 us/op	0.87
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	82.621 us/op	118.29 us/op	0.70
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	796.45 us/op	706.45 us/op	1.13
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	657.05 us/op	954.76 us/op	0.69
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	709.72 us/op	977.66 us/op	0.73
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	1.7822 ms/op	2.2173 ms/op	0.80
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	1.1581 ms/op	1.5585 ms/op	0.74
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	2.7155 ms/op	3.6733 ms/op	0.74
Tree 40 250000 create	192.81 ms/op	265.84 ms/op	0.73
Tree 40 250000 get(125000)	120.49 ns/op	158.84 ns/op	0.76
Tree 40 250000 set(125000)	540.70 ns/op	760.55 ns/op	0.71
Tree 40 250000 toArray()	18.365 ms/op	21.602 ms/op	0.85
Tree 40 250000 iterate all - toArray() + loop	13.044 ms/op	22.924 ms/op	0.57
Tree 40 250000 iterate all - get(i)	42.746 ms/op	58.627 ms/op	0.73
MutableVector 250000 create	7.9617 ms/op	10.938 ms/op	0.73
MutableVector 250000 get(125000)	6.0390 ns/op	6.3670 ns/op	0.95
MutableVector 250000 set(125000)	173.74 ns/op	220.73 ns/op	0.79
MutableVector 250000 toArray()	2.6870 ms/op	3.8434 ms/op	0.70
MutableVector 250000 iterate all - toArray() + loop	2.8129 ms/op	4.4589 ms/op	0.63
MutableVector 250000 iterate all - get(i)	1.4407 ms/op	1.6040 ms/op	0.90
Array 250000 create	2.3699 ms/op	3.7210 ms/op	0.64
Array 250000 clone - spread	1.2713 ms/op	1.4920 ms/op	0.85
Array 250000 get(125000)	0.58700 ns/op	0.43800 ns/op	1.34
Array 250000 set(125000)	0.60000 ns/op	0.46200 ns/op	1.30
Array 250000 iterate all - loop	76.603 us/op	95.067 us/op	0.81
effectiveBalanceIncrements clone Uint8Array 300000	14.348 us/op	39.069 us/op	0.37
effectiveBalanceIncrements clone MutableVector 300000	321.00 ns/op	132.00 ns/op	2.43
effectiveBalanceIncrements rw all Uint8Array 300000	167.10 us/op	200.81 us/op	0.83
effectiveBalanceIncrements rw all MutableVector 300000	56.978 ms/op	71.587 ms/op	0.80
phase0 afterProcessEpoch - 250000 vs - 7PWei	75.046 ms/op	91.639 ms/op	0.82
phase0 beforeProcessEpoch - 250000 vs - 7PWei	35.143 ms/op	44.203 ms/op	0.80
altair processEpoch - mainnet_e81889	411.42 ms/op	426.31 ms/op	0.97
mainnet_e81889 - altair beforeProcessEpoch	55.441 ms/op	71.217 ms/op	0.78
mainnet_e81889 - altair processJustificationAndFinalization	17.199 us/op	16.760 us/op	1.03
mainnet_e81889 - altair processInactivityUpdates	7.7871 ms/op	6.7589 ms/op	1.15
mainnet_e81889 - altair processRewardsAndPenalties	43.004 ms/op	44.483 ms/op	0.97
mainnet_e81889 - altair processRegistryUpdates	3.9200 us/op	2.9110 us/op	1.35
mainnet_e81889 - altair processSlashings	1.1530 us/op	472.00 ns/op	2.44
mainnet_e81889 - altair processEth1DataReset	713.00 ns/op	484.00 ns/op	1.47
mainnet_e81889 - altair processEffectiveBalanceUpdates	1.2981 ms/op	1.1177 ms/op	1.16
mainnet_e81889 - altair processSlashingsReset	2.2690 us/op	3.8280 us/op	0.59
mainnet_e81889 - altair processRandaoMixesReset	3.1600 us/op	5.7230 us/op	0.55
mainnet_e81889 - altair processHistoricalRootsUpdate	705.00 ns/op	1.1660 us/op	0.60
mainnet_e81889 - altair processParticipationFlagUpdates	2.7650 us/op	3.4650 us/op	0.80
mainnet_e81889 - altair processSyncCommitteeUpdates	820.00 ns/op	770.00 ns/op	1.06
mainnet_e81889 - altair afterProcessEpoch	85.964 ms/op	100.10 ms/op	0.86
capella processEpoch - mainnet_e217614	1.3435 s/op	2.0809 s/op	0.65
mainnet_e217614 - capella beforeProcessEpoch	244.49 ms/op	337.65 ms/op	0.72
mainnet_e217614 - capella processJustificationAndFinalization	7.1740 us/op	42.337 us/op	0.17
mainnet_e217614 - capella processInactivityUpdates	14.045 ms/op	35.901 ms/op	0.39
mainnet_e217614 - capella processRewardsAndPenalties	248.22 ms/op	289.96 ms/op	0.86
mainnet_e217614 - capella processRegistryUpdates	14.462 us/op	14.420 us/op	1.00
mainnet_e217614 - capella processSlashings	717.00 ns/op	735.00 ns/op	0.98
mainnet_e217614 - capella processEth1DataReset	703.00 ns/op	467.00 ns/op	1.51
mainnet_e217614 - capella processEffectiveBalanceUpdates	13.420 ms/op	3.8323 ms/op	3.50
mainnet_e217614 - capella processSlashingsReset	3.0170 us/op	5.3130 us/op	0.57
mainnet_e217614 - capella processRandaoMixesReset	3.8430 us/op	10.147 us/op	0.38
mainnet_e217614 - capella processHistoricalRootsUpdate	693.00 ns/op	1.3510 us/op	0.51
mainnet_e217614 - capella processParticipationFlagUpdates	3.9260 us/op	4.7550 us/op	0.83
mainnet_e217614 - capella afterProcessEpoch	249.44 ms/op	284.38 ms/op	0.88
phase0 processEpoch - mainnet_e58758	397.59 ms/op	426.22 ms/op	0.93
mainnet_e58758 - phase0 beforeProcessEpoch	126.04 ms/op	117.85 ms/op	1.07
mainnet_e58758 - phase0 processJustificationAndFinalization	11.245 us/op	13.725 us/op	0.82
mainnet_e58758 - phase0 processRewardsAndPenalties	24.780 ms/op	21.140 ms/op	1.17
mainnet_e58758 - phase0 processRegistryUpdates	6.8280 us/op	7.3030 us/op	0.93
mainnet_e58758 - phase0 processSlashings	681.00 ns/op	327.00 ns/op	2.08
mainnet_e58758 - phase0 processEth1DataReset	673.00 ns/op	459.00 ns/op	1.47
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.0856 ms/op	938.17 us/op	1.16
mainnet_e58758 - phase0 processSlashingsReset	3.3090 us/op	3.7790 us/op	0.88
mainnet_e58758 - phase0 processRandaoMixesReset	3.3750 us/op	5.2320 us/op	0.65
mainnet_e58758 - phase0 processHistoricalRootsUpdate	681.00 ns/op	624.00 ns/op	1.09
mainnet_e58758 - phase0 processParticipationRecordUpdates	3.0920 us/op	3.8340 us/op	0.81
mainnet_e58758 - phase0 afterProcessEpoch	67.826 ms/op	84.475 ms/op	0.80
phase0 processEffectiveBalanceUpdates - 250000 normalcase	811.54 us/op	1.1501 ms/op	0.71
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.2870 ms/op	2.0175 ms/op	0.64
altair processInactivityUpdates - 250000 normalcase	15.784 ms/op	17.222 ms/op	0.92
altair processInactivityUpdates - 250000 worstcase	16.892 ms/op	17.050 ms/op	0.99
phase0 processRegistryUpdates - 250000 normalcase	6.1170 us/op	7.4830 us/op	0.82
phase0 processRegistryUpdates - 250000 badcase_full_deposits	306.43 us/op	293.97 us/op	1.04
phase0 processRegistryUpdates - 250000 worstcase 0.5	108.55 ms/op	137.08 ms/op	0.79
altair processRewardsAndPenalties - 250000 normalcase	30.352 ms/op	37.847 ms/op	0.80
altair processRewardsAndPenalties - 250000 worstcase	31.651 ms/op	37.862 ms/op	0.84
phase0 getAttestationDeltas - 250000 normalcase	5.7536 ms/op	8.1059 ms/op	0.71
phase0 getAttestationDeltas - 250000 worstcase	6.0504 ms/op	10.539 ms/op	0.57
phase0 processSlashings - 250000 worstcase	90.956 us/op	114.35 us/op	0.80
altair processSyncCommitteeUpdates - 250000	101.81 ms/op	145.04 ms/op	0.70
BeaconState.hashTreeRoot - No change	550.00 ns/op	329.00 ns/op	1.67
BeaconState.hashTreeRoot - 1 full validator	103.99 us/op	101.07 us/op	1.03
BeaconState.hashTreeRoot - 32 full validator	739.60 us/op	1.0721 ms/op	0.69
BeaconState.hashTreeRoot - 512 full validator	7.5398 ms/op	14.460 ms/op	0.52
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	88.933 us/op	161.74 us/op	0.55
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	1.3117 ms/op	2.4683 ms/op	0.53
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	18.029 ms/op	25.247 ms/op	0.71
BeaconState.hashTreeRoot - 1 balances	107.59 us/op	103.76 us/op	1.04
BeaconState.hashTreeRoot - 32 balances	824.45 us/op	1.4518 ms/op	0.57
BeaconState.hashTreeRoot - 512 balances	8.7085 ms/op	9.8096 ms/op	0.89
BeaconState.hashTreeRoot - 250000 balances	177.46 ms/op	186.78 ms/op	0.95
aggregationBits - 2048 els - zipIndexesInBitList	19.015 us/op	31.803 us/op	0.60
byteArrayEquals 32	48.088 ns/op	58.200 ns/op	0.83
Buffer.compare 32	38.869 ns/op	52.358 ns/op	0.74
byteArrayEquals 1024	1.2565 us/op	1.7325 us/op	0.73
Buffer.compare 1024	48.337 ns/op	62.227 ns/op	0.78
byteArrayEquals 16384	20.098 us/op	27.618 us/op	0.73
Buffer.compare 16384	231.41 ns/op	252.49 ns/op	0.92
byteArrayEquals 123687377	151.06 ms/op	196.27 ms/op	0.77
Buffer.compare 123687377	6.1042 ms/op	8.0961 ms/op	0.75
byteArrayEquals 32 - diff last byte	46.192 ns/op	53.400 ns/op	0.87
Buffer.compare 32 - diff last byte	37.298 ns/op	47.230 ns/op	0.79
byteArrayEquals 1024 - diff last byte	1.2361 us/op	1.6079 us/op	0.77
Buffer.compare 1024 - diff last byte	44.687 ns/op	56.020 ns/op	0.80
byteArrayEquals 16384 - diff last byte	19.594 us/op	25.853 us/op	0.76
Buffer.compare 16384 - diff last byte	230.26 ns/op	250.56 ns/op	0.92
byteArrayEquals 123687377 - diff last byte	149.38 ms/op	193.15 ms/op	0.77
Buffer.compare 123687377 - diff last byte	4.6388 ms/op	7.3294 ms/op	0.63
byteArrayEquals 32 - random bytes	5.2830 ns/op	5.4360 ns/op	0.97
Buffer.compare 32 - random bytes	40.536 ns/op	48.696 ns/op	0.83
byteArrayEquals 1024 - random bytes	4.8710 ns/op	5.2320 ns/op	0.93
Buffer.compare 1024 - random bytes	39.807 ns/op	47.178 ns/op	0.84
byteArrayEquals 16384 - random bytes	4.9120 ns/op	5.1980 ns/op	0.94
Buffer.compare 16384 - random bytes	39.432 ns/op	46.831 ns/op	0.84
byteArrayEquals 123687377 - random bytes	7.9500 ns/op	6.5500 ns/op	1.21
Buffer.compare 123687377 - random bytes	44.390 ns/op	48.330 ns/op	0.92
regular array get 100000 times	30.011 us/op	34.529 us/op	0.87
wrappedArray get 100000 times	29.591 us/op	33.163 us/op	0.89
arrayWithProxy get 100000 times	10.710 ms/op	13.856 ms/op	0.77
ssz.Root.equals	46.026 ns/op	46.477 ns/op	0.99
byteArrayEquals	41.918 ns/op	46.021 ns/op	0.91
Buffer.compare	8.8240 ns/op	10.719 ns/op	0.82
shuffle list - 16384 els	5.5046 ms/op	6.6723 ms/op	0.82
shuffle list - 250000 els	80.903 ms/op	97.445 ms/op	0.83
processSlot - 1 slots	10.339 us/op	17.303 us/op	0.60
processSlot - 32 slots	2.1451 ms/op	2.8729 ms/op	0.75
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	38.195 ms/op	34.746 ms/op	1.10
getCommitteeAssignments - req 1 vs - 250000 vc	1.8220 ms/op	2.2612 ms/op	0.81
getCommitteeAssignments - req 100 vs - 250000 vc	3.6024 ms/op	4.3363 ms/op	0.83
getCommitteeAssignments - req 1000 vs - 250000 vc	3.9594 ms/op	4.5972 ms/op	0.86
findModifiedValidators - 10000 modified validators	314.28 ms/op	281.96 ms/op	1.11
findModifiedValidators - 1000 modified validators	165.33 ms/op	177.21 ms/op	0.93
findModifiedValidators - 100 modified validators	159.26 ms/op	193.84 ms/op	0.82
findModifiedValidators - 10 modified validators	140.31 ms/op	176.08 ms/op	0.80
findModifiedValidators - 1 modified validators	165.09 ms/op	188.97 ms/op	0.87
findModifiedValidators - no difference	163.22 ms/op	165.16 ms/op	0.99
compare ViewDUs	3.0922 s/op	3.0163 s/op	1.03
compare each validator Uint8Array	1.4580 s/op	1.2934 s/op	1.13
compare ViewDU to Uint8Array	740.14 ms/op	1.2452 s/op	0.59
migrate state 1000000 validators, 24 modified, 0 new	544.87 ms/op	559.07 ms/op	0.97
migrate state 1000000 validators, 1700 modified, 1000 new	786.61 ms/op	862.84 ms/op	0.91
migrate state 1000000 validators, 3400 modified, 2000 new	950.98 ms/op	919.95 ms/op	1.03
migrate state 1500000 validators, 24 modified, 0 new	609.58 ms/op	577.46 ms/op	1.06
migrate state 1500000 validators, 1700 modified, 1000 new	787.43 ms/op	819.79 ms/op	0.96
migrate state 1500000 validators, 3400 modified, 2000 new	1.0406 s/op	971.28 ms/op	1.07
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	5.9800 ns/op	4.9400 ns/op	1.21
state getBlockRootAtSlot - 250000 vs - 7PWei	546.91 ns/op	929.97 ns/op	0.59
computeProposers - vc 250000	5.3132 ms/op	8.3208 ms/op	0.64
computeEpochShuffling - vc 250000	80.601 ms/op	94.436 ms/op	0.85
getNextSyncCommittee - vc 250000	94.882 ms/op	148.82 ms/op	0.64
computeSigningRoot for AttestationData	13.576 us/op	23.739 us/op	0.57
hash AttestationData serialized data then Buffer.toString(base64)	1.1711 us/op	1.5761 us/op	0.74
toHexString serialized data	779.88 ns/op	1.4616 us/op	0.53
Buffer.toString(base64)	141.69 ns/op	277.38 ns/op	0.51

by benchmarkbot/action

[twoeths]

At first glance looks like setting classic-level version to 1.2.0 would do it? This would give us more control on what version is used and prevent accidental upgrades.

yes I wanted to only downgrade classic-level but still got v1.4.1 in the end maybe because v1.4.1 was already in yarn.lock so got Matt's branch as a quick try. Feel free to go for it if you can find a solution @jeluard

need to confirm the hypothesis through, testing in progress

[twoeths]

this is not the cause, added the classic-level rebuild to a service deployment and it's still fine

while this PR itself still have bad performance

[twoeths]

while this PR itself still have bad performance

it was a docker deployment