feat: synchronize keys between validator client and external signer by nflaig · Pull Request #6672 · ChainSafe/lodestar

nflaig · 2024-04-15T22:25:50Z

Motivation

Synchronize keys between validator client and external signer #6624

Description

Adds new service that is responsible for keeping the keys managed by the connected external signer and the validator client in sync by adding newly discovered keys and removing no longer present keys on external signer from the validator store.

Only enabled if --externalSigner.url and --externalSigner.fetch is set.

The interval can be configured via --externalSigner.fetchInterval flag and is set to once per epoch by default.

(see referenced issue for more details)

Closes #6624

codecov · 2024-04-15T22:37:39Z

Codecov Report

Merging #6672 (fa78e38) into unstable (eb84b49) will increase coverage by 0.07%.
Report is 1 commits behind head on unstable.
The diff coverage is 94.84%.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #6672      +/-   ##
============================================
+ Coverage     61.80%   61.87%   +0.07%     
============================================
  Files           556      557       +1     
  Lines         59103    59200      +97     
  Branches       1898     1915      +17     
============================================
+ Hits          36526    36628     +102     
+ Misses        22534    22529       -5     
  Partials         43       43

github-actions · 2024-04-15T23:34:07Z

Performance Report

✔️ no performance regression detected

🚀🚀 Significant benchmark improvement detected

Benchmark suite	Current: `a114b51`	Previous: `f5148b2`	Ratio
forkChoice updateHead vc 600000 bc 64 eq 300000	16.092 ms/op	54.969 ms/op	0.29
phase0 processBlock - 250000 vs - 7PWei normalcase	2.2576 ms/op	6.9290 ms/op	0.33
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	42.726 us/op	148.47 us/op	0.29
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	3.8492 ms/op	12.393 ms/op	0.31
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	5.7169 ms/op	19.883 ms/op	0.29

Full benchmark results

Benchmark suite	Current: `a114b51`	Previous: `f5148b2`	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	685.97 us/op	566.38 us/op	1.21
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	84.028 us/op	105.04 us/op	0.80
BLS verify - blst-native	1.2810 ms/op	1.3367 ms/op	0.96
BLS verifyMultipleSignatures 3 - blst-native	2.6963 ms/op	2.8140 ms/op	0.96
BLS verifyMultipleSignatures 8 - blst-native	5.9250 ms/op	6.1794 ms/op	0.96
BLS verifyMultipleSignatures 32 - blst-native	21.675 ms/op	22.211 ms/op	0.98
BLS verifyMultipleSignatures 64 - blst-native	42.634 ms/op	44.382 ms/op	0.96
BLS verifyMultipleSignatures 128 - blst-native	84.387 ms/op	86.316 ms/op	0.98
BLS deserializing 10000 signatures	870.50 ms/op	906.25 ms/op	0.96
BLS deserializing 100000 signatures	9.2393 s/op	9.2687 s/op	1.00
BLS verifyMultipleSignatures - same message - 3 - blst-native	1.3215 ms/op	1.3740 ms/op	0.96
BLS verifyMultipleSignatures - same message - 8 - blst-native	1.6564 ms/op	1.5319 ms/op	1.08
BLS verifyMultipleSignatures - same message - 32 - blst-native	2.6451 ms/op	2.4246 ms/op	1.09
BLS verifyMultipleSignatures - same message - 64 - blst-native	4.2170 ms/op	3.5064 ms/op	1.20
BLS verifyMultipleSignatures - same message - 128 - blst-native	6.7552 ms/op	5.8519 ms/op	1.15
BLS aggregatePubkeys 32 - blst-native	26.055 us/op	25.941 us/op	1.00
BLS aggregatePubkeys 128 - blst-native	104.55 us/op	101.38 us/op	1.03
notSeenSlots=1 numMissedVotes=1 numBadVotes=10	65.934 ms/op	63.597 ms/op	1.04
notSeenSlots=1 numMissedVotes=0 numBadVotes=4	66.568 ms/op	80.511 ms/op	0.83
notSeenSlots=2 numMissedVotes=1 numBadVotes=10	36.616 ms/op	49.355 ms/op	0.74
getSlashingsAndExits - default max	146.74 us/op	221.03 us/op	0.66
getSlashingsAndExits - 2k	446.27 us/op	472.31 us/op	0.94
proposeBlockBody type=full, size=empty	5.5040 ms/op	6.8600 ms/op	0.80
isKnown best case - 1 super set check	356.00 ns/op	466.00 ns/op	0.76
isKnown normal case - 2 super set checks	305.00 ns/op	524.00 ns/op	0.58
isKnown worse case - 16 super set checks	300.00 ns/op	439.00 ns/op	0.68
InMemoryCheckpointStateCache - add get delete	5.9320 us/op	7.0400 us/op	0.84
validate api signedAggregateAndProof - struct	3.0035 ms/op	2.9167 ms/op	1.03
validate gossip signedAggregateAndProof - struct	2.7405 ms/op	2.8513 ms/op	0.96
validate gossip attestation - vc 640000	1.3411 ms/op	1.4036 ms/op	0.96
batch validate gossip attestation - vc 640000 - chunk 32	163.10 us/op	176.99 us/op	0.92
batch validate gossip attestation - vc 640000 - chunk 64	145.06 us/op	159.14 us/op	0.91
batch validate gossip attestation - vc 640000 - chunk 128	134.57 us/op	154.55 us/op	0.87
batch validate gossip attestation - vc 640000 - chunk 256	130.28 us/op	150.83 us/op	0.86
pickEth1Vote - no votes	1.2496 ms/op	1.5095 ms/op	0.83
pickEth1Vote - max votes	12.539 ms/op	11.698 ms/op	1.07
pickEth1Vote - Eth1Data hashTreeRoot value x2048	16.089 ms/op	21.620 ms/op	0.74
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	21.608 ms/op	33.432 ms/op	0.65
pickEth1Vote - Eth1Data fastSerialize value x2048	588.84 us/op	828.77 us/op	0.71
pickEth1Vote - Eth1Data fastSerialize tree x2048	4.2052 ms/op	6.4259 ms/op	0.65
bytes32 toHexString	513.00 ns/op	819.00 ns/op	0.63
bytes32 Buffer.toString(hex)	301.00 ns/op	315.00 ns/op	0.96
bytes32 Buffer.toString(hex) from Uint8Array	425.00 ns/op	614.00 ns/op	0.69
bytes32 Buffer.toString(hex) + 0x	304.00 ns/op	310.00 ns/op	0.98
Object access 1 prop	0.16800 ns/op	0.24400 ns/op	0.69
Map access 1 prop	0.14900 ns/op	0.16400 ns/op	0.91
Object get x1000	7.7660 ns/op	8.1800 ns/op	0.95
Map get x1000	0.78500 ns/op	0.95900 ns/op	0.82
Object set x1000	54.057 ns/op	84.287 ns/op	0.64
Map set x1000	44.423 ns/op	61.462 ns/op	0.72
Return object 10000 times	0.24510 ns/op	0.28510 ns/op	0.86
Throw Error 10000 times	3.8498 us/op	4.2165 us/op	0.91
fastMsgIdFn sha256 / 200 bytes	3.3810 us/op	3.7510 us/op	0.90
fastMsgIdFn h32 xxhash / 200 bytes	336.00 ns/op	413.00 ns/op	0.81
fastMsgIdFn h64 xxhash / 200 bytes	375.00 ns/op	435.00 ns/op	0.86
fastMsgIdFn sha256 / 1000 bytes	11.131 us/op	12.280 us/op	0.91
fastMsgIdFn h32 xxhash / 1000 bytes	421.00 ns/op	531.00 ns/op	0.79
fastMsgIdFn h64 xxhash / 1000 bytes	424.00 ns/op	547.00 ns/op	0.78
fastMsgIdFn sha256 / 10000 bytes	100.35 us/op	114.83 us/op	0.87
fastMsgIdFn h32 xxhash / 10000 bytes	1.8970 us/op	2.1830 us/op	0.87
fastMsgIdFn h64 xxhash / 10000 bytes	1.3090 us/op	1.5450 us/op	0.85
send data - 1000 256B messages	17.114 ms/op	27.359 ms/op	0.63
send data - 1000 512B messages	23.396 ms/op	32.180 ms/op	0.73
send data - 1000 1024B messages	39.781 ms/op	48.050 ms/op	0.83
send data - 1000 1200B messages	38.592 ms/op	45.378 ms/op	0.85
send data - 1000 2048B messages	49.847 ms/op	62.919 ms/op	0.79
send data - 1000 4096B messages	42.182 ms/op	56.302 ms/op	0.75
send data - 1000 16384B messages	113.97 ms/op	150.72 ms/op	0.76
send data - 1000 65536B messages	434.95 ms/op	573.16 ms/op	0.76
enrSubnets - fastDeserialize 64 bits	1.2300 us/op	1.7930 us/op	0.69
enrSubnets - ssz BitVector 64 bits	438.00 ns/op	592.00 ns/op	0.74
enrSubnets - fastDeserialize 4 bits	169.00 ns/op	252.00 ns/op	0.67
enrSubnets - ssz BitVector 4 bits	432.00 ns/op	570.00 ns/op	0.76
prioritizePeers score -10:0 att 32-0.1 sync 2-0	205.26 us/op	238.81 us/op	0.86
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	262.70 us/op	369.90 us/op	0.71
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	332.85 us/op	453.47 us/op	0.73
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	561.37 us/op	717.26 us/op	0.78
prioritizePeers score 0:0 att 64-1 sync 4-1	697.59 us/op	900.22 us/op	0.77
array of 16000 items push then shift	1.5576 us/op	1.7660 us/op	0.88
LinkedList of 16000 items push then shift	8.7680 ns/op	11.094 ns/op	0.79
array of 16000 items push then pop	57.044 ns/op	129.29 ns/op	0.44
LinkedList of 16000 items push then pop	8.7140 ns/op	9.7640 ns/op	0.89
array of 24000 items push then shift	2.3909 us/op	2.6920 us/op	0.89
LinkedList of 24000 items push then shift	8.7340 ns/op	11.155 ns/op	0.78
array of 24000 items push then pop	110.50 ns/op	188.23 ns/op	0.59
LinkedList of 24000 items push then pop	8.4970 ns/op	9.8030 ns/op	0.87
intersect bitArray bitLen 8	5.6160 ns/op	6.7700 ns/op	0.83
intersect array and set length 8	62.665 ns/op	114.83 ns/op	0.55
intersect bitArray bitLen 128	34.993 ns/op	42.991 ns/op	0.81
intersect array and set length 128	887.17 ns/op	1.2180 us/op	0.73
bitArray.getTrueBitIndexes() bitLen 128	1.3870 us/op	2.2200 us/op	0.62
bitArray.getTrueBitIndexes() bitLen 248	2.5740 us/op	3.6100 us/op	0.71
bitArray.getTrueBitIndexes() bitLen 512	4.7440 us/op	8.3180 us/op	0.57
Buffer.concat 32 items	948.00 ns/op	1.2730 us/op	0.74
Uint8Array.set 32 items	1.5820 us/op	2.3180 us/op	0.68
Set add up to 64 items then delete first	4.3029 us/op	5.5707 us/op	0.77
OrderedSet add up to 64 items then delete first	5.3714 us/op	7.6495 us/op	0.70
Set add up to 64 items then delete last	4.6095 us/op	6.0255 us/op	0.77
OrderedSet add up to 64 items then delete last	5.6841 us/op	7.7540 us/op	0.73
Set add up to 64 items then delete middle	4.5408 us/op	5.7392 us/op	0.79
OrderedSet add up to 64 items then delete middle	7.1544 us/op	9.2501 us/op	0.77
Set add up to 128 items then delete first	9.2415 us/op	12.065 us/op	0.77
OrderedSet add up to 128 items then delete first	12.267 us/op	16.048 us/op	0.76
Set add up to 128 items then delete last	9.4415 us/op	12.202 us/op	0.77
OrderedSet add up to 128 items then delete last	12.110 us/op	15.219 us/op	0.80
Set add up to 128 items then delete middle	9.7800 us/op	11.376 us/op	0.86
OrderedSet add up to 128 items then delete middle	17.965 us/op	21.905 us/op	0.82
Set add up to 256 items then delete first	19.576 us/op	23.504 us/op	0.83
OrderedSet add up to 256 items then delete first	25.598 us/op	34.963 us/op	0.73
Set add up to 256 items then delete last	19.184 us/op	22.940 us/op	0.84
OrderedSet add up to 256 items then delete last	27.284 us/op	30.891 us/op	0.88
Set add up to 256 items then delete middle	18.356 us/op	22.976 us/op	0.80
OrderedSet add up to 256 items then delete middle	45.292 us/op	55.380 us/op	0.82
transfer serialized Status (84 B)	1.6860 us/op	2.2720 us/op	0.74
copy serialized Status (84 B)	1.2860 us/op	1.6690 us/op	0.77
transfer serialized SignedVoluntaryExit (112 B)	1.8050 us/op	2.3460 us/op	0.77
copy serialized SignedVoluntaryExit (112 B)	1.3710 us/op	1.7720 us/op	0.77
transfer serialized ProposerSlashing (416 B)	2.0280 us/op	3.6030 us/op	0.56
copy serialized ProposerSlashing (416 B)	1.7910 us/op	3.5310 us/op	0.51
transfer serialized Attestation (485 B)	1.9230 us/op	3.7960 us/op	0.51
copy serialized Attestation (485 B)	1.8410 us/op	3.0380 us/op	0.61
transfer serialized AttesterSlashing (33232 B)	2.3390 us/op	3.7480 us/op	0.62
copy serialized AttesterSlashing (33232 B)	5.5870 us/op	20.864 us/op	0.27
transfer serialized Small SignedBeaconBlock (128000 B)	2.5610 us/op	5.1360 us/op	0.50
copy serialized Small SignedBeaconBlock (128000 B)	14.532 us/op	36.207 us/op	0.40
transfer serialized Avg SignedBeaconBlock (200000 B)	2.6420 us/op	5.1810 us/op	0.51
copy serialized Avg SignedBeaconBlock (200000 B)	22.755 us/op	60.548 us/op	0.38
transfer serialized BlobsSidecar (524380 B)	2.9270 us/op	6.0200 us/op	0.49
copy serialized BlobsSidecar (524380 B)	126.78 us/op	159.42 us/op	0.80
transfer serialized Big SignedBeaconBlock (1000000 B)	2.8040 us/op	5.8760 us/op	0.48
copy serialized Big SignedBeaconBlock (1000000 B)	314.70 us/op	307.28 us/op	1.02
pass gossip attestations to forkchoice per slot	3.6464 ms/op	5.0232 ms/op	0.73
forkChoice updateHead vc 100000 bc 64 eq 0	672.12 us/op	714.60 us/op	0.94
forkChoice updateHead vc 600000 bc 64 eq 0	4.0784 ms/op	6.8959 ms/op	0.59
forkChoice updateHead vc 1000000 bc 64 eq 0	7.2025 ms/op	7.9774 ms/op	0.90
forkChoice updateHead vc 600000 bc 320 eq 0	4.1416 ms/op	4.6475 ms/op	0.89
forkChoice updateHead vc 600000 bc 1200 eq 0	4.2371 ms/op	4.9942 ms/op	0.85
forkChoice updateHead vc 600000 bc 7200 eq 0	5.7486 ms/op	6.6299 ms/op	0.87
forkChoice updateHead vc 600000 bc 64 eq 1000	10.737 ms/op	12.220 ms/op	0.88
forkChoice updateHead vc 600000 bc 64 eq 10000	11.880 ms/op	14.463 ms/op	0.82
forkChoice updateHead vc 600000 bc 64 eq 300000	16.092 ms/op	54.969 ms/op	0.29
computeDeltas 500000 validators 300 proto nodes	6.6589 ms/op	7.6026 ms/op	0.88
computeDeltas 500000 validators 1200 proto nodes	6.4582 ms/op	7.3888 ms/op	0.87
computeDeltas 500000 validators 7200 proto nodes	6.3729 ms/op	6.9354 ms/op	0.92
computeDeltas 750000 validators 300 proto nodes	9.7661 ms/op	10.657 ms/op	0.92
computeDeltas 750000 validators 1200 proto nodes	9.5473 ms/op	10.698 ms/op	0.89
computeDeltas 750000 validators 7200 proto nodes	9.5503 ms/op	10.418 ms/op	0.92
computeDeltas 1400000 validators 300 proto nodes	18.236 ms/op	21.535 ms/op	0.85
computeDeltas 1400000 validators 1200 proto nodes	18.065 ms/op	20.608 ms/op	0.88
computeDeltas 1400000 validators 7200 proto nodes	18.243 ms/op	20.142 ms/op	0.91
computeDeltas 2100000 validators 300 proto nodes	27.646 ms/op	30.415 ms/op	0.91
computeDeltas 2100000 validators 1200 proto nodes	28.246 ms/op	31.930 ms/op	0.88
computeDeltas 2100000 validators 7200 proto nodes	28.203 ms/op	33.406 ms/op	0.84
altair processAttestation - 250000 vs - 7PWei normalcase	4.1760 ms/op	7.9320 ms/op	0.53
altair processAttestation - 250000 vs - 7PWei worstcase	5.0600 ms/op	10.812 ms/op	0.47
altair processAttestation - setStatus - 1/6 committees join	293.13 us/op	412.80 us/op	0.71
altair processAttestation - setStatus - 1/3 committees join	525.03 us/op	812.36 us/op	0.65
altair processAttestation - setStatus - 1/2 committees join	717.87 us/op	1.0978 ms/op	0.65
altair processAttestation - setStatus - 2/3 committees join	828.31 us/op	1.3693 ms/op	0.60
altair processAttestation - setStatus - 4/5 committees join	1.1617 ms/op	1.7185 ms/op	0.68
altair processAttestation - setStatus - 100% committees join	1.3516 ms/op	2.0279 ms/op	0.67
altair processBlock - 250000 vs - 7PWei normalcase	9.6363 ms/op	19.435 ms/op	0.50
altair processBlock - 250000 vs - 7PWei normalcase hashState	35.883 ms/op	89.449 ms/op	0.40
altair processBlock - 250000 vs - 7PWei worstcase	44.377 ms/op	77.257 ms/op	0.57
altair processBlock - 250000 vs - 7PWei worstcase hashState	87.782 ms/op	196.50 ms/op	0.45
phase0 processBlock - 250000 vs - 7PWei normalcase	2.2576 ms/op	6.9290 ms/op	0.33
phase0 processBlock - 250000 vs - 7PWei worstcase	32.379 ms/op	58.098 ms/op	0.56
altair processEth1Data - 250000 vs - 7PWei normalcase	455.62 us/op	1.2087 ms/op	0.38
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	13.578 us/op	35.117 us/op	0.39
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	42.726 us/op	148.47 us/op	0.29
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	29.456 us/op	59.735 us/op	0.49
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	15.087 us/op	40.492 us/op	0.37
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	180.94 us/op	321.66 us/op	0.56
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	1.2654 ms/op	2.7972 ms/op	0.45
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	1.5583 ms/op	3.9335 ms/op	0.40
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	2.0166 ms/op	3.3347 ms/op	0.60
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	3.8492 ms/op	12.393 ms/op	0.31
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	2.3684 ms/op	6.0161 ms/op	0.39
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	5.7169 ms/op	19.883 ms/op	0.29
Tree 40 250000 create	353.88 ms/op	1.6332 s/op	0.22
Tree 40 250000 get(125000)	201.41 ns/op	387.26 ns/op	0.52
Tree 40 250000 set(125000)	1.0300 us/op	5.8199 us/op	0.18
Tree 40 250000 toArray()	19.404 ms/op	59.792 ms/op	0.32
Tree 40 250000 iterate all - toArray() + loop	20.910 ms/op	65.092 ms/op	0.32
Tree 40 250000 iterate all - get(i)	70.963 ms/op	163.02 ms/op	0.44
MutableVector 250000 create	17.015 ms/op	27.365 ms/op	0.62
MutableVector 250000 get(125000)	6.5160 ns/op	11.670 ns/op	0.56
MutableVector 250000 set(125000)	284.21 ns/op	1.0745 us/op	0.26
MutableVector 250000 toArray()	2.9520 ms/op	7.7227 ms/op	0.38
MutableVector 250000 iterate all - toArray() + loop	3.0695 ms/op	8.2208 ms/op	0.37
MutableVector 250000 iterate all - get(i)	1.5965 ms/op	3.0926 ms/op	0.52
Array 250000 create	2.6445 ms/op	6.8597 ms/op	0.39
Array 250000 clone - spread	1.2864 ms/op	7.3420 ms/op	0.18
Array 250000 get(125000)	1.0920 ns/op	4.0960 ns/op	0.27
Array 250000 set(125000)	4.1460 ns/op	7.0830 ns/op	0.59
Array 250000 iterate all - loop	165.37 us/op	218.46 us/op	0.76
effectiveBalanceIncrements clone Uint8Array 300000	31.678 us/op	121.64 us/op	0.26
effectiveBalanceIncrements clone MutableVector 300000	405.00 ns/op	2.6890 us/op	0.15
effectiveBalanceIncrements rw all Uint8Array 300000	205.97 us/op	310.20 us/op	0.66
effectiveBalanceIncrements rw all MutableVector 300000	102.82 ms/op	323.01 ms/op	0.32
phase0 afterProcessEpoch - 250000 vs - 7PWei	114.49 ms/op	176.60 ms/op	0.65
phase0 beforeProcessEpoch - 250000 vs - 7PWei	55.867 ms/op	134.76 ms/op	0.41
altair processEpoch - mainnet_e81889	471.99 ms/op	1.1120 s/op	0.42
mainnet_e81889 - altair beforeProcessEpoch	83.276 ms/op	227.70 ms/op	0.37
mainnet_e81889 - altair processJustificationAndFinalization	15.602 us/op	45.629 us/op	0.34
mainnet_e81889 - altair processInactivityUpdates	6.1051 ms/op	19.414 ms/op	0.31
mainnet_e81889 - altair processRewardsAndPenalties	54.814 ms/op	96.297 ms/op	0.57
mainnet_e81889 - altair processRegistryUpdates	2.3650 us/op	10.440 us/op	0.23
mainnet_e81889 - altair processSlashings	439.00 ns/op	1.8030 us/op	0.24
mainnet_e81889 - altair processEth1DataReset	512.00 ns/op	2.2380 us/op	0.23
mainnet_e81889 - altair processEffectiveBalanceUpdates	1.3454 ms/op	2.7041 ms/op	0.50
mainnet_e81889 - altair processSlashingsReset	3.5040 us/op	11.812 us/op	0.30
mainnet_e81889 - altair processRandaoMixesReset	8.2820 us/op	15.698 us/op	0.53
mainnet_e81889 - altair processHistoricalRootsUpdate	698.00 ns/op	3.1170 us/op	0.22
mainnet_e81889 - altair processParticipationFlagUpdates	3.0970 us/op	7.2460 us/op	0.43
mainnet_e81889 - altair processSyncCommitteeUpdates	723.00 ns/op	2.7940 us/op	0.26
mainnet_e81889 - altair afterProcessEpoch	119.82 ms/op	183.83 ms/op	0.65
capella processEpoch - mainnet_e217614	2.4166 s/op	4.8386 s/op	0.50
mainnet_e217614 - capella beforeProcessEpoch	538.06 ms/op	1.0608 s/op	0.51
mainnet_e217614 - capella processJustificationAndFinalization	15.002 us/op	45.954 us/op	0.33
mainnet_e217614 - capella processInactivityUpdates	19.713 ms/op	37.480 ms/op	0.53
mainnet_e217614 - capella processRewardsAndPenalties	558.39 ms/op	984.94 ms/op	0.57
mainnet_e217614 - capella processRegistryUpdates	24.468 us/op	50.644 us/op	0.48
mainnet_e217614 - capella processSlashings	1.0340 us/op	1.7950 us/op	0.58
mainnet_e217614 - capella processEth1DataReset	467.00 ns/op	2.1750 us/op	0.21
mainnet_e217614 - capella processEffectiveBalanceUpdates	4.3283 ms/op	22.927 ms/op	0.19
mainnet_e217614 - capella processSlashingsReset	3.3840 us/op	10.773 us/op	0.31
mainnet_e217614 - capella processRandaoMixesReset	4.7110 us/op	13.542 us/op	0.35
mainnet_e217614 - capella processHistoricalRootsUpdate	552.00 ns/op	2.9160 us/op	0.19
mainnet_e217614 - capella processParticipationFlagUpdates	1.4710 us/op	5.2580 us/op	0.28
mainnet_e217614 - capella afterProcessEpoch	302.98 ms/op	411.99 ms/op	0.74
phase0 processEpoch - mainnet_e58758	462.04 ms/op	681.65 ms/op	0.68
mainnet_e58758 - phase0 beforeProcessEpoch	152.79 ms/op	194.96 ms/op	0.78
mainnet_e58758 - phase0 processJustificationAndFinalization	15.623 us/op	29.893 us/op	0.52
mainnet_e58758 - phase0 processRewardsAndPenalties	65.962 ms/op	72.289 ms/op	0.91
mainnet_e58758 - phase0 processRegistryUpdates	9.5790 us/op	16.124 us/op	0.59
mainnet_e58758 - phase0 processSlashings	640.00 ns/op	1.0100 us/op	0.63
mainnet_e58758 - phase0 processEth1DataReset	419.00 ns/op	622.00 ns/op	0.67
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.1464 ms/op	1.2906 ms/op	0.89
mainnet_e58758 - phase0 processSlashingsReset	3.0210 us/op	3.7600 us/op	0.80
mainnet_e58758 - phase0 processRandaoMixesReset	5.5950 us/op	6.3060 us/op	0.89
mainnet_e58758 - phase0 processHistoricalRootsUpdate	665.00 ns/op	588.00 ns/op	1.13
mainnet_e58758 - phase0 processParticipationRecordUpdates	5.1970 us/op	4.6940 us/op	1.11
mainnet_e58758 - phase0 afterProcessEpoch	102.17 ms/op	106.40 ms/op	0.96
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.4447 ms/op	1.4257 ms/op	1.01
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.4674 ms/op	2.1938 ms/op	0.67
altair processInactivityUpdates - 250000 normalcase	30.665 ms/op	38.465 ms/op	0.80
altair processInactivityUpdates - 250000 worstcase	26.973 ms/op	31.807 ms/op	0.85
phase0 processRegistryUpdates - 250000 normalcase	11.772 us/op	20.381 us/op	0.58
phase0 processRegistryUpdates - 250000 badcase_full_deposits	394.61 us/op	574.55 us/op	0.69
phase0 processRegistryUpdates - 250000 worstcase 0.5	140.63 ms/op	234.79 ms/op	0.60
altair processRewardsAndPenalties - 250000 normalcase	55.998 ms/op	65.764 ms/op	0.85
altair processRewardsAndPenalties - 250000 worstcase	70.621 ms/op	103.72 ms/op	0.68
phase0 getAttestationDeltas - 250000 normalcase	11.991 ms/op	12.271 ms/op	0.98
phase0 getAttestationDeltas - 250000 worstcase	9.1325 ms/op	13.195 ms/op	0.69
phase0 processSlashings - 250000 worstcase	83.260 us/op	134.19 us/op	0.62
altair processSyncCommitteeUpdates - 250000	161.84 ms/op	189.30 ms/op	0.85
BeaconState.hashTreeRoot - No change	335.00 ns/op	761.00 ns/op	0.44
BeaconState.hashTreeRoot - 1 full validator	153.71 us/op	184.91 us/op	0.83
BeaconState.hashTreeRoot - 32 full validator	1.5425 ms/op	1.7553 ms/op	0.88
BeaconState.hashTreeRoot - 512 full validator	17.484 ms/op	19.088 ms/op	0.92
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	152.99 us/op	188.49 us/op	0.81
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	2.5747 ms/op	2.4198 ms/op	1.06
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	28.374 ms/op	32.610 ms/op	0.87
BeaconState.hashTreeRoot - 1 balances	132.29 us/op	150.81 us/op	0.88
BeaconState.hashTreeRoot - 32 balances	1.1615 ms/op	1.3751 ms/op	0.84
BeaconState.hashTreeRoot - 512 balances	10.732 ms/op	13.476 ms/op	0.80
BeaconState.hashTreeRoot - 250000 balances	196.88 ms/op	240.96 ms/op	0.82
aggregationBits - 2048 els - zipIndexesInBitList	56.836 us/op	45.848 us/op	1.24
byteArrayEquals 32	75.625 ns/op	84.308 ns/op	0.90
Buffer.compare 32	58.028 ns/op	59.218 ns/op	0.98
byteArrayEquals 1024	2.0682 us/op	2.4922 us/op	0.83
Buffer.compare 1024	75.147 ns/op	80.018 ns/op	0.94
byteArrayEquals 16384	32.425 us/op	34.557 us/op	0.94
Buffer.compare 16384	271.65 ns/op	287.23 ns/op	0.95
byteArrayEquals 123687377	243.41 ms/op	303.31 ms/op	0.80
Buffer.compare 123687377	6.1020 ms/op	12.570 ms/op	0.49
byteArrayEquals 32 - diff last byte	73.389 ns/op	98.967 ns/op	0.74
Buffer.compare 32 - diff last byte	58.268 ns/op	75.029 ns/op	0.78
byteArrayEquals 1024 - diff last byte	2.0225 us/op	2.5268 us/op	0.80
Buffer.compare 1024 - diff last byte	73.885 ns/op	84.220 ns/op	0.88
byteArrayEquals 16384 - diff last byte	32.284 us/op	40.096 us/op	0.81
Buffer.compare 16384 - diff last byte	279.72 ns/op	294.85 ns/op	0.95
byteArrayEquals 123687377 - diff last byte	245.04 ms/op	296.95 ms/op	0.83
Buffer.compare 123687377 - diff last byte	8.2305 ms/op	12.590 ms/op	0.65
byteArrayEquals 32 - random bytes	5.6090 ns/op	7.6330 ns/op	0.73
Buffer.compare 32 - random bytes	65.318 ns/op	67.310 ns/op	0.97
byteArrayEquals 1024 - random bytes	6.2070 ns/op	7.0850 ns/op	0.88
Buffer.compare 1024 - random bytes	65.387 ns/op	71.807 ns/op	0.91
byteArrayEquals 16384 - random bytes	5.8380 ns/op	7.7170 ns/op	0.76
Buffer.compare 16384 - random bytes	63.102 ns/op	78.324 ns/op	0.81
byteArrayEquals 123687377 - random bytes	8.7000 ns/op	17.140 ns/op	0.51
Buffer.compare 123687377 - random bytes	69.750 ns/op	96.990 ns/op	0.72
regular array get 100000 times	45.192 us/op	56.159 us/op	0.80
wrappedArray get 100000 times	45.712 us/op	58.899 us/op	0.78
arrayWithProxy get 100000 times	14.824 ms/op	18.004 ms/op	0.82
ssz.Root.equals	53.628 ns/op	62.096 ns/op	0.86
byteArrayEquals	52.206 ns/op	59.580 ns/op	0.88
Buffer.compare	10.783 ns/op	13.009 ns/op	0.83
shuffle list - 16384 els	8.4468 ms/op	9.5796 ms/op	0.88
shuffle list - 250000 els	126.14 ms/op	148.97 ms/op	0.85
processSlot - 1 slots	17.841 us/op	24.986 us/op	0.71
processSlot - 32 slots	4.5943 ms/op	4.8993 ms/op	0.94
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	61.574 ms/op	75.280 ms/op	0.82
getCommitteeAssignments - req 1 vs - 250000 vc	2.7073 ms/op	3.0088 ms/op	0.90
getCommitteeAssignments - req 100 vs - 250000 vc	4.0464 ms/op	4.2934 ms/op	0.94
getCommitteeAssignments - req 1000 vs - 250000 vc	4.3304 ms/op	4.7927 ms/op	0.90
findModifiedValidators - 10000 modified validators	342.01 ms/op	431.38 ms/op	0.79
findModifiedValidators - 1000 modified validators	199.41 ms/op	259.87 ms/op	0.77
findModifiedValidators - 100 modified validators	169.82 ms/op	210.14 ms/op	0.81
findModifiedValidators - 10 modified validators	205.57 ms/op	214.07 ms/op	0.96
findModifiedValidators - 1 modified validators	195.03 ms/op	224.56 ms/op	0.87
findModifiedValidators - no difference	198.64 ms/op	230.81 ms/op	0.86
compare ViewDUs	4.5018 s/op	7.4018 s/op	0.61
compare each validator Uint8Array	1.5207 s/op	2.7749 s/op	0.55
compare ViewDU to Uint8Array	1.2568 s/op	1.7809 s/op	0.71
migrate state 1000000 validators, 24 modified, 0 new	731.56 ms/op	1.2691 s/op	0.58
migrate state 1000000 validators, 1700 modified, 1000 new	1.0738 s/op	1.7469 s/op	0.61
migrate state 1000000 validators, 3400 modified, 2000 new	1.2510 s/op	1.4233 s/op	0.88
migrate state 1500000 validators, 24 modified, 0 new	751.78 ms/op	1.0225 s/op	0.74
migrate state 1500000 validators, 1700 modified, 1000 new	1.5821 s/op	1.3702 s/op	1.15
migrate state 1500000 validators, 3400 modified, 2000 new	1.6961 s/op	1.4702 s/op	1.15
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	6.2200 ns/op	4.5000 ns/op	1.38
state getBlockRootAtSlot - 250000 vs - 7PWei	1.3510 us/op	527.42 ns/op	2.56
computeProposers - vc 250000	18.415 ms/op	9.5596 ms/op	1.93
computeEpochShuffling - vc 250000	159.46 ms/op	127.28 ms/op	1.25
getNextSyncCommittee - vc 250000	207.62 ms/op	167.16 ms/op	1.24
computeSigningRoot for AttestationData	30.686 us/op	25.338 us/op	1.21
hash AttestationData serialized data then Buffer.toString(base64)	2.4421 us/op	2.4253 us/op	1.01
toHexString serialized data	1.2909 us/op	1.2787 us/op	1.01
Buffer.toString(base64)	247.22 ns/op	250.39 ns/op	0.99

by benchmarkbot/action

nflaig · 2024-04-17T17:28:26Z

  },

-  // Remote signer
+  // External signer


I tried to make the wording a bit more consistent within the code, and also few logs. We generally use the term "external signer" to describe an external/(remote) server that handles the signing operations for a specific public key. Whereas the term "remote signer" is also used to describe a remote signer in the local validator store.

nflaig · 2024-04-17T19:17:48Z

-  // Remote signer
+  // External signer

  "externalSigner.url": {


We could consider supporting multiple external signers in the future, our implementation can easily be adjusted to support this but I think it's better to not include it in this PR. Opened an issue to keep track of it and for further discussion #6679.

g11tech

lgtm

wemeetagain · 2024-04-23T19:03:29Z

🎉 This PR is included in v1.18.0 🎉

nflaig changed the title ~~feat: synchronize keys between validator client and remote signer~~ feat: synchronize keys between validator client and external signer Apr 15, 2024

Synchronize keys between validator client and external signer

d63adce

nflaig force-pushed the nflaig/external-signer-sync branch from e3b5466 to d63adce Compare April 15, 2024 22:53

nflaig added 4 commits April 17, 2024 13:07

More consistent logs and naming

a422e59

Improve consistency of using external vs. remote signer wording

d89d2ce

Simplify function parameters

fe66111

Add unit tests for external signer sync

0091ead

nflaig commented Apr 17, 2024

View reviewed changes

nflaig marked this pull request as ready for review April 17, 2024 17:37

nflaig requested a review from a team as a code owner April 17, 2024 17:37

nflaig added 2 commits April 17, 2024 18:57

Update no keys found error message

99112ab

Ensure fetched pubkeys are properly validated

70b92ff

nflaig added this to the v1.18.0 milestone Apr 17, 2024

nflaig added 2 commits April 17, 2024 19:50

Use arrow functions to define tests

b4dccd9

Update logs for added / removed public keys

064d561

nflaig commented Apr 17, 2024

View reviewed changes

wemeetagain reviewed Apr 17, 2024

View reviewed changes

Comment thread packages/cli/src/cmds/validator/options.ts

nflaig added 8 commits April 18, 2024 00:01

Properly define external signer options type

bbd695b

Simplify added / removed info logs

ec59efb

Move no signers warning logs / errors out of handler

018c514

Update comment wording

c6be4bf

Add external signer docs page

836c875

Fix spellcheck

7cda41d

Merge branch 'unstable' into nflaig/external-signer-sync

28e90d0

Clarify to use pubkeys flag instead of fetch

fa78e38

wemeetagain reviewed Apr 18, 2024

View reviewed changes

Comment thread docs/pages/validator-management/external-signer.md

g11tech approved these changes Apr 19, 2024

View reviewed changes

wemeetagain approved these changes Apr 19, 2024

View reviewed changes

wemeetagain merged commit 05fe0ac into unstable Apr 19, 2024

wemeetagain deleted the nflaig/external-signer-sync branch April 19, 2024 14:12

Navigation Menu

Uh oh!

Uh oh!

feat: synchronize keys between validator client and external signer - #6672

feat: synchronize keys between validator client and external signer#6672
wemeetagain merged 17 commits into
unstablefrom
nflaig/external-signer-sync

nflaig commented Apr 15, 2024

Uh oh!

codecov Bot commented Apr 15, 2024 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 15, 2024 •

edited

Loading

Uh oh!

nflaig Apr 17, 2024 •

edited

Loading

Uh oh!

nflaig Apr 17, 2024

Uh oh!

Uh oh!

Uh oh!

g11tech left a comment

Uh oh!

wemeetagain commented Apr 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

nflaig commented Apr 15, 2024

Uh oh!

codecov Bot commented Apr 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Apr 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Report

Uh oh!

nflaig Apr 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nflaig Apr 17, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

g11tech left a comment

Choose a reason for hiding this comment

Uh oh!

wemeetagain commented Apr 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Apr 15, 2024 •

edited

Loading

github-actions Bot commented Apr 15, 2024 •

edited

Loading

nflaig Apr 17, 2024 •

edited

Loading