Skip to content

Use more deterministic keystore cache path - #5197

Merged
nazarhussain merged 3 commits into
unstablefrom
nh/5141-cache-path
Feb 23, 2023
Merged

Use more deterministic keystore cache path #5197
nazarhussain merged 3 commits into
unstablefrom
nh/5141-cache-path

Conversation

@nazarhussain

Copy link
Copy Markdown
Contributor

Motivation

Use more deterministic keystore cache.

Description

Create a new cache directory and use the keystore cache stored in there.

Closes #5141

Steps to test or reproduce

Ru all tests.

@nazarhussain
nazarhussain requested a review from a team as a code owner February 22, 2023 14:57
@github-actions

github-actions Bot commented Feb 22, 2023

Copy link
Copy Markdown
Contributor

Performance Report

鉁旓笍 no performance regression detected

Full benchmark results
Benchmark suite Current: ef2368b Previous: 960b8c5 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 660.25 us/op 562.82 us/op 1.17
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 66.299 us/op 48.552 us/op 1.37
BLS verify - blst-native 1.3024 ms/op 1.2425 ms/op 1.05
BLS verifyMultipleSignatures 3 - blst-native 2.6172 ms/op 2.5587 ms/op 1.02
BLS verifyMultipleSignatures 8 - blst-native 5.6624 ms/op 5.5324 ms/op 1.02
BLS verifyMultipleSignatures 32 - blst-native 22.243 ms/op 19.900 ms/op 1.12
BLS aggregatePubkeys 32 - blst-native 29.111 us/op 26.297 us/op 1.11
BLS aggregatePubkeys 128 - blst-native 113.98 us/op 102.54 us/op 1.11
getAttestationsForBlock 85.737 ms/op 62.333 ms/op 1.38
isKnown best case - 1 super set check 303.00 ns/op 284.00 ns/op 1.07
isKnown normal case - 2 super set checks 293.00 ns/op 270.00 ns/op 1.09
isKnown worse case - 16 super set checks 280.00 ns/op 270.00 ns/op 1.04
CheckpointStateCache - add get delete 6.7520 us/op 5.7660 us/op 1.17
validate gossip signedAggregateAndProof - struct 2.9511 ms/op 2.8605 ms/op 1.03
validate gossip attestation - struct 1.4250 ms/op 1.3571 ms/op 1.05
pickEth1Vote - no votes 1.6020 ms/op 1.3628 ms/op 1.18
pickEth1Vote - max votes 13.828 ms/op 12.590 ms/op 1.10
pickEth1Vote - Eth1Data hashTreeRoot value x2048 10.904 ms/op 10.231 ms/op 1.07
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 21.284 ms/op 19.934 ms/op 1.07
pickEth1Vote - Eth1Data fastSerialize value x2048 938.66 us/op 819.78 us/op 1.15
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.6083 ms/op 9.0167 ms/op 0.73
bytes32 toHexString 887.00 ns/op 723.00 ns/op 1.23
bytes32 Buffer.toString(hex) 454.00 ns/op 421.00 ns/op 1.08
bytes32 Buffer.toString(hex) from Uint8Array 698.00 ns/op 657.00 ns/op 1.06
bytes32 Buffer.toString(hex) + 0x 450.00 ns/op 449.00 ns/op 1.00
Object access 1 prop 0.21600 ns/op 0.20400 ns/op 1.06
Map access 1 prop 0.18600 ns/op 0.19200 ns/op 0.97
Object get x1000 8.0270 ns/op 7.6850 ns/op 1.04
Map get x1000 0.81000 ns/op 0.60300 ns/op 1.34
Object set x1000 90.453 ns/op 77.210 ns/op 1.17
Map set x1000 65.190 ns/op 56.308 ns/op 1.16
Return object 10000 times 0.33930 ns/op 0.25910 ns/op 1.31
Throw Error 10000 times 4.5956 us/op 4.4641 us/op 1.03
fastMsgIdFn sha256 / 200 bytes 3.9260 us/op 3.7260 us/op 1.05
fastMsgIdFn h32 xxhash / 200 bytes 351.00 ns/op 314.00 ns/op 1.12
fastMsgIdFn h64 xxhash / 200 bytes 508.00 ns/op 472.00 ns/op 1.08
fastMsgIdFn sha256 / 1000 bytes 12.478 us/op 12.291 us/op 1.02
fastMsgIdFn h32 xxhash / 1000 bytes 493.00 ns/op 469.00 ns/op 1.05
fastMsgIdFn h64 xxhash / 1000 bytes 579.00 ns/op 558.00 ns/op 1.04
fastMsgIdFn sha256 / 10000 bytes 113.23 us/op 109.66 us/op 1.03
fastMsgIdFn h32 xxhash / 10000 bytes 2.2090 us/op 2.0950 us/op 1.05
fastMsgIdFn h64 xxhash / 10000 bytes 1.5760 us/op 1.5260 us/op 1.03
enrSubnets - fastDeserialize 64 bits 2.0750 us/op 2.0130 us/op 1.03
enrSubnets - ssz BitVector 64 bits 645.00 ns/op 658.00 ns/op 0.98
enrSubnets - fastDeserialize 4 bits 241.00 ns/op 217.00 ns/op 1.11
enrSubnets - ssz BitVector 4 bits 657.00 ns/op 635.00 ns/op 1.03
prioritizePeers score -10:0 att 32-0.1 sync 2-0 127.81 us/op 113.94 us/op 1.12
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 164.78 us/op 143.23 us/op 1.15
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 224.62 us/op 214.28 us/op 1.05
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 392.92 us/op 351.70 us/op 1.12
prioritizePeers score 0:0 att 64-1 sync 4-1 413.27 us/op 392.72 us/op 1.05
array of 16000 items push then shift 1.9009 us/op 1.7637 us/op 1.08
LinkedList of 16000 items push then shift 11.861 ns/op 9.5070 ns/op 1.25
array of 16000 items push then pop 132.60 ns/op 117.21 ns/op 1.13
LinkedList of 16000 items push then pop 12.846 ns/op 9.2550 ns/op 1.39
array of 24000 items push then shift 2.7421 us/op 2.5721 us/op 1.07
LinkedList of 24000 items push then shift 12.041 ns/op 10.507 ns/op 1.15
array of 24000 items push then pop 106.85 ns/op 86.830 ns/op 1.23
LinkedList of 24000 items push then pop 12.826 ns/op 9.2940 ns/op 1.38
intersect bitArray bitLen 8 19.724 ns/op 14.503 ns/op 1.36
intersect array and set length 8 154.21 ns/op 94.345 ns/op 1.63
intersect bitArray bitLen 128 67.468 ns/op 49.869 ns/op 1.35
intersect array and set length 128 1.7842 us/op 1.2482 us/op 1.43
Buffer.concat 32 items 4.3360 us/op 3.0290 us/op 1.43
Uint8Array.set 32 items 2.9450 us/op 2.9550 us/op 1.00
pass gossip attestations to forkchoice per slot 3.2388 ms/op 2.5869 ms/op 1.25
computeDeltas 4.9842 ms/op 3.2905 ms/op 1.51
computeProposerBoostScoreFromBalances 2.0374 ms/op 1.8832 ms/op 1.08
altair processAttestation - 250000 vs - 7PWei normalcase 3.6764 ms/op 3.1906 ms/op 1.15
altair processAttestation - 250000 vs - 7PWei worstcase 5.5780 ms/op 5.2901 ms/op 1.05
altair processAttestation - setStatus - 1/6 committees join 152.11 us/op 148.53 us/op 1.02
altair processAttestation - setStatus - 1/3 committees join 304.42 us/op 286.32 us/op 1.06
altair processAttestation - setStatus - 1/2 committees join 388.74 us/op 385.35 us/op 1.01
altair processAttestation - setStatus - 2/3 committees join 486.11 us/op 480.46 us/op 1.01
altair processAttestation - setStatus - 4/5 committees join 681.50 us/op 696.75 us/op 0.98
altair processAttestation - setStatus - 100% committees join 790.41 us/op 849.83 us/op 0.93
altair processBlock - 250000 vs - 7PWei normalcase 18.778 ms/op 20.244 ms/op 0.93
altair processBlock - 250000 vs - 7PWei normalcase hashState 31.191 ms/op 31.036 ms/op 1.00
altair processBlock - 250000 vs - 7PWei worstcase 52.559 ms/op 57.579 ms/op 0.91
altair processBlock - 250000 vs - 7PWei worstcase hashState 74.396 ms/op 81.982 ms/op 0.91
phase0 processBlock - 250000 vs - 7PWei normalcase 2.6388 ms/op 2.3359 ms/op 1.13
phase0 processBlock - 250000 vs - 7PWei worstcase 34.020 ms/op 32.284 ms/op 1.05
altair processEth1Data - 250000 vs - 7PWei normalcase 513.30 us/op 604.78 us/op 0.85
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 11.495 us/op 13.144 us/op 0.87
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 35.937 us/op 40.299 us/op 0.89
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 16.912 us/op 16.226 us/op 1.04
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 11.637 us/op 14.273 us/op 0.82
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 116.74 us/op 133.75 us/op 0.87
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 774.32 us/op 683.82 us/op 1.13
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 933.66 us/op 916.60 us/op 1.02
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 947.10 us/op 913.12 us/op 1.04
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 3.0562 ms/op 2.5492 ms/op 1.20
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 2.1098 ms/op 1.5453 ms/op 1.37
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 6.0373 ms/op 4.1465 ms/op 1.46
Tree 40 250000 create 470.49 ms/op 320.67 ms/op 1.47
Tree 40 250000 get(125000) 199.51 ns/op 187.50 ns/op 1.06
Tree 40 250000 set(125000) 1.4631 us/op 969.04 ns/op 1.51
Tree 40 250000 toArray() 28.694 ms/op 21.177 ms/op 1.35
Tree 40 250000 iterate all - toArray() + loop 27.776 ms/op 21.301 ms/op 1.30
Tree 40 250000 iterate all - get(i) 81.434 ms/op 71.682 ms/op 1.14
MutableVector 250000 create 12.480 ms/op 10.972 ms/op 1.14
MutableVector 250000 get(125000) 7.0830 ns/op 6.6100 ns/op 1.07
MutableVector 250000 set(125000) 475.05 ns/op 277.59 ns/op 1.71
MutableVector 250000 toArray() 4.9405 ms/op 2.9426 ms/op 1.68
MutableVector 250000 iterate all - toArray() + loop 4.4864 ms/op 3.0028 ms/op 1.49
MutableVector 250000 iterate all - get(i) 1.5793 ms/op 1.5669 ms/op 1.01
Array 250000 create 3.8222 ms/op 3.0969 ms/op 1.23
Array 250000 clone - spread 1.3721 ms/op 1.0874 ms/op 1.26
Array 250000 get(125000) 0.64100 ns/op 0.58700 ns/op 1.09
Array 250000 set(125000) 0.67700 ns/op 0.62700 ns/op 1.08
Array 250000 iterate all - loop 85.093 us/op 112.57 us/op 0.76
effectiveBalanceIncrements clone Uint8Array 300000 40.947 us/op 29.012 us/op 1.41
effectiveBalanceIncrements clone MutableVector 300000 364.00 ns/op 319.00 ns/op 1.14
effectiveBalanceIncrements rw all Uint8Array 300000 171.27 us/op 171.98 us/op 1.00
effectiveBalanceIncrements rw all MutableVector 300000 89.030 ms/op 82.846 ms/op 1.07
phase0 afterProcessEpoch - 250000 vs - 7PWei 118.22 ms/op 115.98 ms/op 1.02
phase0 beforeProcessEpoch - 250000 vs - 7PWei 51.459 ms/op 36.756 ms/op 1.40
altair processEpoch - mainnet_e81889 351.58 ms/op 331.62 ms/op 1.06
mainnet_e81889 - altair beforeProcessEpoch 60.259 ms/op 68.938 ms/op 0.87
mainnet_e81889 - altair processJustificationAndFinalization 20.327 us/op 19.063 us/op 1.07
mainnet_e81889 - altair processInactivityUpdates 7.1036 ms/op 6.0771 ms/op 1.17
mainnet_e81889 - altair processRewardsAndPenalties 52.599 ms/op 69.454 ms/op 0.76
mainnet_e81889 - altair processRegistryUpdates 4.3200 us/op 2.9800 us/op 1.45
mainnet_e81889 - altair processSlashings 551.00 ns/op 414.00 ns/op 1.33
mainnet_e81889 - altair processEth1DataReset 565.00 ns/op 500.00 ns/op 1.13
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2905 ms/op 1.2795 ms/op 1.01
mainnet_e81889 - altair processSlashingsReset 6.0980 us/op 3.9910 us/op 1.53
mainnet_e81889 - altair processRandaoMixesReset 6.5680 us/op 5.8340 us/op 1.13
mainnet_e81889 - altair processHistoricalRootsUpdate 1.0600 us/op 926.00 ns/op 1.14
mainnet_e81889 - altair processParticipationFlagUpdates 4.1000 us/op 4.5000 us/op 0.91
mainnet_e81889 - altair processSyncCommitteeUpdates 766.00 ns/op 1.3430 us/op 0.57
mainnet_e81889 - altair afterProcessEpoch 130.45 ms/op 133.62 ms/op 0.98
phase0 processEpoch - mainnet_e58758 440.32 ms/op 374.47 ms/op 1.18
mainnet_e58758 - phase0 beforeProcessEpoch 166.05 ms/op 142.73 ms/op 1.16
mainnet_e58758 - phase0 processJustificationAndFinalization 31.015 us/op 18.556 us/op 1.67
mainnet_e58758 - phase0 processRewardsAndPenalties 71.697 ms/op 64.917 ms/op 1.10
mainnet_e58758 - phase0 processRegistryUpdates 12.178 us/op 8.2410 us/op 1.48
mainnet_e58758 - phase0 processSlashings 628.00 ns/op 483.00 ns/op 1.30
mainnet_e58758 - phase0 processEth1DataReset 586.00 ns/op 492.00 ns/op 1.19
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.0684 ms/op 1.0327 ms/op 1.03
mainnet_e58758 - phase0 processSlashingsReset 5.0570 us/op 3.8600 us/op 1.31
mainnet_e58758 - phase0 processRandaoMixesReset 7.1350 us/op 5.6020 us/op 1.27
mainnet_e58758 - phase0 processHistoricalRootsUpdate 769.00 ns/op 554.00 ns/op 1.39
mainnet_e58758 - phase0 processParticipationRecordUpdates 6.4100 us/op 4.1800 us/op 1.53
mainnet_e58758 - phase0 afterProcessEpoch 104.35 ms/op 101.44 ms/op 1.03
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.3474 ms/op 1.2717 ms/op 1.06
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.6416 ms/op 1.5382 ms/op 1.07
altair processInactivityUpdates - 250000 normalcase 28.336 ms/op 18.408 ms/op 1.54
altair processInactivityUpdates - 250000 worstcase 29.290 ms/op 24.649 ms/op 1.19
phase0 processRegistryUpdates - 250000 normalcase 8.5350 us/op 6.0660 us/op 1.41
phase0 processRegistryUpdates - 250000 badcase_full_deposits 356.45 us/op 262.80 us/op 1.36
phase0 processRegistryUpdates - 250000 worstcase 0.5 141.73 ms/op 118.96 ms/op 1.19
altair processRewardsAndPenalties - 250000 normalcase 71.623 ms/op 62.680 ms/op 1.14
altair processRewardsAndPenalties - 250000 worstcase 67.088 ms/op 64.165 ms/op 1.05
phase0 getAttestationDeltas - 250000 normalcase 7.2732 ms/op 7.9350 ms/op 0.92
phase0 getAttestationDeltas - 250000 worstcase 7.0610 ms/op 7.1474 ms/op 0.99
phase0 processSlashings - 250000 worstcase 3.6895 ms/op 3.7115 ms/op 0.99
altair processSyncCommitteeUpdates - 250000 191.48 ms/op 187.35 ms/op 1.02
BeaconState.hashTreeRoot - No change 275.00 ns/op 266.00 ns/op 1.03
BeaconState.hashTreeRoot - 1 full validator 57.390 us/op 52.336 us/op 1.10
BeaconState.hashTreeRoot - 32 full validator 572.72 us/op 675.82 us/op 0.85
BeaconState.hashTreeRoot - 512 full validator 5.2011 ms/op 5.1980 ms/op 1.00
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 65.243 us/op 65.259 us/op 1.00
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 941.69 us/op 1.0596 ms/op 0.89
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 13.540 ms/op 11.999 ms/op 1.13
BeaconState.hashTreeRoot - 1 balances 51.138 us/op 50.152 us/op 1.02
BeaconState.hashTreeRoot - 32 balances 487.38 us/op 457.70 us/op 1.06
BeaconState.hashTreeRoot - 512 balances 4.9799 ms/op 4.6391 ms/op 1.07
BeaconState.hashTreeRoot - 250000 balances 80.677 ms/op 80.532 ms/op 1.00
aggregationBits - 2048 els - zipIndexesInBitList 18.413 us/op 18.707 us/op 0.98
regular array get 100000 times 34.807 us/op 34.514 us/op 1.01
wrappedArray get 100000 times 33.658 us/op 36.897 us/op 0.91
arrayWithProxy get 100000 times 15.401 ms/op 16.514 ms/op 0.93
ssz.Root.equals 577.00 ns/op 588.00 ns/op 0.98
byteArrayEquals 571.00 ns/op 583.00 ns/op 0.98
shuffle list - 16384 els 7.0273 ms/op 7.1690 ms/op 0.98
shuffle list - 250000 els 104.44 ms/op 103.86 ms/op 1.01
processSlot - 1 slots 10.191 us/op 9.2420 us/op 1.10
processSlot - 32 slots 1.4176 ms/op 1.4494 ms/op 0.98
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 214.41 us/op 217.17 us/op 0.99
getCommitteeAssignments - req 1 vs - 250000 vc 3.0125 ms/op 3.0168 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 4.2575 ms/op 4.2964 ms/op 0.99
getCommitteeAssignments - req 1000 vs - 250000 vc 4.5947 ms/op 4.6852 ms/op 0.98
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.2400 ns/op 5.2900 ns/op 0.99
state getBlockRootAtSlot - 250000 vs - 7PWei 646.03 ns/op 642.91 ns/op 1.00
computeProposers - vc 250000 11.127 ms/op 11.735 ms/op 0.95
computeEpochShuffling - vc 250000 106.64 ms/op 108.27 ms/op 0.98
getNextSyncCommittee - vc 250000 178.70 ms/op 188.76 ms/op 0.95

by benchmarkbot/action

@nflaig nflaig left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! This should address the issue noted in #5043 (comment)

...args,
onDecrypt: needle,
cacheFilePath: `${args.importKeystores[0]}.cache`,
cacheFilePath: join(accountPaths.cacheDir, "imported_keystores.cache"),

@nflaig nflaig Feb 22, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

highly opinionated (feel free to ignore) but I really don't like just having join here, someone has to hover the function to know it from path library or scroll up to the imports to see where this is coming from

since path uses such general names for it's functions I generally like to do path., to make it more obvious we are joining paths here

Suggested change
cacheFilePath: join(accountPaths.cacheDir, "imported_keystores.cache"),
cacheFilePath: path.join(accountPaths.cacheDir, "imported_keystores.cache"),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use destructuring convention all over the code, so seems fine to me. In my preference, I always tend to relate a function to standard library function. If I see a function join I will consider it path.join. If we define a user function which joins string, I would declare it as joinStr to avoid confusion with standard library.

@nflaig nflaig Feb 22, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use destructuring convention all over the code, so seems fine to me.

I would disagree on when an import should be destructured or not. I do think this should not be based on convention in the code but rather on the design of the library, meaning how did the author/s want you to use it.

In the case of most of the native nodejs libraries those are designed to not use destructuring, one indicator of this are the ambiguous names of functions like for example path has things like delimiter, normalize(path) or parse(path). If we just take parse as an example, this could mean anything, parse a URL, a encoded value, something else? If the authors would have written the library to be more destructuring friendly they would have called this function parsePath.

And if you just look at the docs, for any nodejs library it becomes also clear how they want the libraries to be used, e.g. path, crypto, os

If I see a function join I will consider it path.join.

That might be fine for you as you might know all of the nodejs APIs but for a more unexperienced developer it might not. Functions should clearly convey their purpose without having to think or consider anything and if it requires the code to be more verbose to achieve this it is fine to me.

so seems fine to me.

it definitely is fine in this case, especially since the property right before is called cacheFilePath and the variable is called accountPaths.cacheDir is should be clear that we are joining paths here

This is just more general thing how I think native libraries should be used due to their ambiguous names.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like to see path.join over just join too

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I mentioned in some earlier discussion, if any style/pattern can't be enforced by linting then we should keep it flexible.

I am creating a separate issue to enforce not to use the destructuring for the some module imports e.g. path. Once we have that rule, the same style will be enforced everywhere.

@nazarhussain nazarhussain Feb 23, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the issue #5201

@nazarhussain
nazarhussain merged commit 474cfbe into unstable Feb 23, 2023
@nazarhussain
nazarhussain deleted the nh/5141-cache-path branch February 23, 2023 19:50
@wemeetagain

Copy link
Copy Markdown
Member

馃帀 This PR is included in v1.6.0 馃帀

twoeths added a commit that referenced this pull request May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

validator_keys.cache is stored in the same directory as my validator_keys

4 participants