Skip to content

fix: recompute head before proposer boost dependent root check - #9813

Closed
ensi321 wants to merge 2 commits into
unstablefrom
nc/proposer-boost-stale-head
Closed

fix: recompute head before proposer boost dependent root check#9813
ensi321 wants to merge 2 commits into
unstablefrom
nc/proposer-boost-stale-head

Conversation

@ensi321

@ensi321 ensi321 commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Nightly compliance tests have been failing. This is due to lodestar uses the head cache which can potentially be stale when doing dependent root check for proposer boost.

Call updateHead to make sure we use the most up-to-date head

Evidence that this works: https://github.com/ChainSafe/lodestar/actions/runs/31605213102

@ensi321
fix: recompute head before proposer boost dependent root check
74a44de 
The spec computes `head = get_head(store)` inside `on_block`, immediately
before `update_proposer_boost_root(store, head.root, block_root)` compares the
head's proposer-shuffling dependent root against the imported block's. We were
passing the cached `this.head` instead, which can be stale: an epoch transition
tick pulls up the justified checkpoint, which moves the head, but nothing
recomputes it before the next block is imported.

Surfaced by the alpha.13 compliance vectors (nightly comptests, red since
2026-08-04):

    FAIL electra|fulu/fork_choice_compliance/block_tree_test/pyspec_tests/
         block_tree_test_15_397678783_1
    Error: Invalid proposer boost root at step 136
      Expected: "0xa27134108a78f561433773014c169ddc29fd5ae46905aba26b9c7c31b8842eda"
      Received: "0x0000000000000000000000000000000000000000000000000000000000000000"

In that case the `tick 192` step crosses into epoch 4 and pulls justified from
epoch 2 to 3, moving the head off `0xcc0a54e2` (slot 24). The block imported at
slot 32 shares its dependent root with the new head but not the stale one, so
the boost was withheld.

Split out `isProposerBoostCandidate` so `updateHead()` stays off the hot path:
it now runs only for the first timely block of a slot, bounding the extra
`get_head()` to at most once per slot rather than once per block import.

Measured that cost while reviewing this: `updateHead()` is ~4.6ms at 1M
validators, and flat regardless of vote churn since `computeDeltas()` scans
every validator either way. Refresh the stale "400ms / run as of Aug 2021"
claim on its docblock, which is off by ~80x and is the figure a reader would
otherwise use to judge this change.
@github-actions

github-actions Bot commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

鈿狅笍 Performance Alert 鈿狅笍

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: fe9fff0 Previous: 1fd2d8b Ratio
proposeBlockBody type=full, size=empty 18.191 ms/op 727.89 us/op 24.99
Full benchmark results
Benchmark suite Current: fe9fff0 Previous: 1fd2d8b Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.4662 ms/op 2.0523 ms/op 1.20
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 42.418 us/op 42.065 us/op 1.01
BLS verify - blst 669.76 us/op 659.82 us/op 1.02
BLS verifyMultipleSignatures 3 - blst 1.0625 ms/op 1.0292 ms/op 1.03
BLS verifyMultipleSignatures 8 - blst 1.9892 ms/op 1.5587 ms/op 1.28
BLS verifyMultipleSignatures 32 - blst 8.2331 ms/op 4.5475 ms/op 1.81
BLS verifyMultipleSignatures 64 - blst 11.462 ms/op 8.9763 ms/op 1.28
BLS verifyMultipleSignatures 128 - blst 22.251 ms/op 16.707 ms/op 1.33
BLS deserializing 10000 signatures 824.63 ms/op 655.96 ms/op 1.26
BLS deserializing 100000 signatures 8.4490 s/op 7.0288 s/op 1.20
BLS verifyMultipleSignatures - same message - 3 - blst 750.51 us/op 763.92 us/op 0.98
BLS verifyMultipleSignatures - same message - 8 - blst 888.49 us/op 937.17 us/op 0.95
BLS verifyMultipleSignatures - same message - 32 - blst 2.0640 ms/op 1.5579 ms/op 1.32
BLS verifyMultipleSignatures - same message - 64 - blst 3.2377 ms/op 2.4126 ms/op 1.34
BLS verifyMultipleSignatures - same message - 128 - blst 5.4321 ms/op 4.2482 ms/op 1.28
BLS aggregatePubkeys 32 - blst 18.796 us/op 17.786 us/op 1.06
BLS aggregatePubkeys 128 - blst 65.938 us/op 64.111 us/op 1.03
getSlashingsAndExits - default max 74.305 us/op 68.649 us/op 1.08
getSlashingsAndExits - 2k 505.55 us/op 458.19 us/op 1.10
proposeBlockBody type=full, size=empty 18.191 ms/op 727.89 us/op 24.99
isKnown best case - 1 super set check 175.00 ns/op 165.00 ns/op 1.06
isKnown normal case - 2 super set checks 167.00 ns/op 168.00 ns/op 0.99
isKnown worse case - 16 super set checks 173.00 ns/op 163.00 ns/op 1.06
validate api signedAggregateAndProof - struct 1.2842 ms/op 1.2143 ms/op 1.06
validate gossip signedAggregateAndProof - struct 1.2946 ms/op 1.2188 ms/op 1.06
batch validate gossip attestation - vc 640000 - chunk 32 169.47 us/op 113.20 us/op 1.50
batch validate gossip attestation - vc 640000 - chunk 64 138.62 us/op 101.96 us/op 1.36
batch validate gossip attestation - vc 640000 - chunk 128 133.82 us/op 116.20 us/op 1.15
batch validate gossip attestation - vc 640000 - chunk 256 125.70 us/op 110.96 us/op 1.13
bytes32 toHexString 305.00 ns/op 296.00 ns/op 1.03
bytes32 Buffer.toString(hex) 183.00 ns/op 175.00 ns/op 1.05
bytes32 Buffer.toString(hex) from Uint8Array 244.00 ns/op 230.00 ns/op 1.06
bytes32 Buffer.toString(hex) + 0x 184.00 ns/op 176.00 ns/op 1.05
Return object 10000 times 0.22070 ns/op 0.22000 ns/op 1.00
Throw Error 10000 times 4.4680 us/op 5.0698 us/op 0.88
toHex 98.082 ns/op 100.13 ns/op 0.98
Buffer.from 91.786 ns/op 92.455 ns/op 0.99
shared Buffer 60.327 ns/op 64.282 ns/op 0.94
fastMsgIdFn sha256 / 200 bytes 1.5220 us/op 1.5260 us/op 1.00
fastMsgIdFn h32 xxhash / 200 bytes 171.00 ns/op 163.00 ns/op 1.05
fastMsgIdFn h64 xxhash / 200 bytes 214.00 ns/op 215.00 ns/op 1.00
fastMsgIdFn sha256 / 1000 bytes 4.9340 us/op 4.8900 us/op 1.01
fastMsgIdFn h32 xxhash / 1000 bytes 254.00 ns/op 247.00 ns/op 1.03
fastMsgIdFn h64 xxhash / 1000 bytes 274.00 ns/op 266.00 ns/op 1.03
fastMsgIdFn sha256 / 10000 bytes 43.569 us/op 43.373 us/op 1.00
fastMsgIdFn h32 xxhash / 10000 bytes 1.3460 us/op 1.3240 us/op 1.02
fastMsgIdFn h64 xxhash / 10000 bytes 882.00 ns/op 873.00 ns/op 1.01
send data - 1000 256B messages 5.2762 ms/op 5.6801 ms/op 0.93
send data - 1000 512B messages 7.0402 ms/op 6.5202 ms/op 1.08
send data - 1000 1024B messages 7.3965 ms/op 7.9920 ms/op 0.93
send data - 1000 1200B messages 8.8871 ms/op 10.117 ms/op 0.88
send data - 1000 2048B messages 15.233 ms/op 22.992 ms/op 0.66
send data - 1000 4096B messages 150.93 ms/op 128.08 ms/op 1.18
send data - 1000 16384B messages 569.87 ms/op 483.59 ms/op 1.18
send data - 1000 65536B messages 2.5274 s/op 2.0775 s/op 1.22
enrSubnets - fastDeserialize 64 bits 799.00 ns/op 780.00 ns/op 1.02
enrSubnets - ssz BitVector 64 bits 273.00 ns/op 270.00 ns/op 1.01
enrSubnets - fastDeserialize 4 bits 106.00 ns/op 102.00 ns/op 1.04
enrSubnets - ssz BitVector 4 bits 283.00 ns/op 266.00 ns/op 1.06
prioritizePeers score -10:0 att 32-0.1 sync 2-0 217.93 us/op 220.70 us/op 0.99
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 246.71 us/op 256.25 us/op 0.96
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 354.21 us/op 388.08 us/op 0.91
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 633.44 us/op 711.00 us/op 0.89
prioritizePeers score 0:0 att 64-1 sync 4-1 749.05 us/op 905.52 us/op 0.83
array of 16000 items push then shift 1.4125 us/op 1.8132 us/op 0.78
LinkedList of 16000 items push then shift 8.2790 ns/op 8.0650 ns/op 1.03
array of 16000 items push then pop 116.19 ns/op 121.16 ns/op 0.96
LinkedList of 16000 items push then pop 6.5330 ns/op 6.4590 ns/op 1.01
array of 24000 items push then shift 2.6714 us/op 3.9474 us/op 0.68
LinkedList of 24000 items push then shift 8.1960 ns/op 7.8410 ns/op 1.05
array of 24000 items push then pop 171.42 ns/op 162.28 ns/op 1.06
LinkedList of 24000 items push then pop 6.4750 ns/op 6.4970 ns/op 1.00
intersect bitArray bitLen 8 4.0240 ns/op 4.0410 ns/op 1.00
intersect array and set length 8 31.280 ns/op 31.138 ns/op 1.00
intersect bitArray bitLen 128 24.301 ns/op 24.643 ns/op 0.99
intersect array and set length 128 519.39 ns/op 559.52 ns/op 0.93
bitArray.getTrueBitIndexes() bitLen 128 998.00 ns/op 1.0170 us/op 0.98
bitArray.getTrueBitIndexes() bitLen 248 1.7920 us/op 1.8160 us/op 0.99
bitArray.getTrueBitIndexes() bitLen 512 3.7820 us/op 3.7810 us/op 1.00
Full columns - reconstruct all 6 blobs 116.80 us/op 419.33 us/op 0.28
Full columns - reconstruct half of the blobs out of 6 65.703 us/op 60.068 us/op 1.09
Full columns - reconstruct single blob out of 6 29.110 us/op 26.057 us/op 1.12
Half columns - reconstruct all 6 blobs 490.18 ms/op 431.18 ms/op 1.14
Half columns - reconstruct half of the blobs out of 6 244.63 ms/op 210.72 ms/op 1.16
Half columns - reconstruct single blob out of 6 85.886 ms/op 77.572 ms/op 1.11
Set add up to 64 items then delete first 3.1967 us/op 2.1999 us/op 1.45
OrderedSet add up to 64 items then delete first 4.8806 us/op 3.6521 us/op 1.34
Set add up to 64 items then delete last 3.3375 us/op 2.2236 us/op 1.50
OrderedSet add up to 64 items then delete last 4.8193 us/op 4.1721 us/op 1.16
Set add up to 64 items then delete middle 3.0350 us/op 2.8032 us/op 1.08
OrderedSet add up to 64 items then delete middle 7.2818 us/op 6.6083 us/op 1.10
Set add up to 128 items then delete first 5.7215 us/op 6.3025 us/op 0.91
OrderedSet add up to 128 items then delete first 8.8671 us/op 9.9558 us/op 0.89
Set add up to 128 items then delete last 5.7621 us/op 5.5161 us/op 1.04
OrderedSet add up to 128 items then delete last 8.4898 us/op 8.3525 us/op 1.02
Set add up to 128 items then delete middle 5.4873 us/op 5.6410 us/op 0.97
OrderedSet add up to 128 items then delete middle 18.233 us/op 17.183 us/op 1.06
Set add up to 256 items then delete first 11.639 us/op 12.420 us/op 0.94
OrderedSet add up to 256 items then delete first 17.217 us/op 17.168 us/op 1.00
Set add up to 256 items then delete last 11.189 us/op 11.053 us/op 1.01
OrderedSet add up to 256 items then delete last 17.502 us/op 17.432 us/op 1.00
Set add up to 256 items then delete middle 11.466 us/op 10.334 us/op 1.11
OrderedSet add up to 256 items then delete middle 58.030 us/op 43.123 us/op 1.35
runFastConfirmationRules vc:100000 bc:96 eq:0 7.6655 ms/op 6.5504 ms/op 1.17
runFastConfirmationRules vc:600000 bc:96 eq:0 57.558 ms/op 57.661 ms/op 1.00
runFastConfirmationRules vc:1000000 bc:96 eq:0 94.490 ms/op 96.588 ms/op 0.98
runFastConfirmationRules vc:600000 bc:320 eq:0 51.410 ms/op 51.773 ms/op 0.99
runFastConfirmationRules vc:100000 bc:96 eq:1000 1.7662 s/op 2.1349 s/op 0.83
pass gossip attestations to forkchoice per slot 3.8127 ms/op 4.8898 ms/op 0.78
forkChoice updateHead vc 100000 bc 64 eq 0 434.54 us/op 611.38 us/op 0.71
forkChoice updateHead vc 600000 bc 64 eq 0 3.4161 ms/op 4.6172 ms/op 0.74
forkChoice updateHead vc 1000000 bc 64 eq 0 4.3249 ms/op 6.4793 ms/op 0.67
forkChoice updateHead vc 600000 bc 320 eq 0 3.2537 ms/op 3.3646 ms/op 0.97
forkChoice updateHead vc 600000 bc 1200 eq 0 3.6003 ms/op 3.4138 ms/op 1.05
forkChoice updateHead vc 600000 bc 7200 eq 0 6.4062 ms/op 6.2009 ms/op 1.03
forkChoice updateHead vc 600000 bc 64 eq 1000 3.8412 ms/op 3.0910 ms/op 1.24
forkChoice updateHead vc 600000 bc 64 eq 10000 4.7110 ms/op 2.8265 ms/op 1.67
forkChoice updateHead vc 600000 bc 64 eq 300000 9.6916 ms/op 8.7707 ms/op 1.10
computeDeltas 1400000 validators 0% inactive 14.593 ms/op 13.770 ms/op 1.06
computeDeltas 1400000 validators 10% inactive 16.263 ms/op 12.834 ms/op 1.27
computeDeltas 1400000 validators 20% inactive 14.390 ms/op 12.060 ms/op 1.19
computeDeltas 1400000 validators 50% inactive 11.466 ms/op 9.5163 ms/op 1.20
computeDeltas 2100000 validators 0% inactive 31.139 ms/op 20.693 ms/op 1.50
computeDeltas 2100000 validators 10% inactive 30.502 ms/op 19.535 ms/op 1.56
computeDeltas 2100000 validators 20% inactive 31.745 ms/op 18.205 ms/op 1.74
computeDeltas 2100000 validators 50% inactive 16.069 ms/op 12.186 ms/op 1.32
altair processAttestation - 250000 vs - 7PWei normalcase 2.6363 ms/op 1.7765 ms/op 1.48
altair processAttestation - 250000 vs - 7PWei worstcase 4.9733 ms/op 2.5584 ms/op 1.94
altair processAttestation - setStatus - 1/6 committees join 124.15 us/op 107.06 us/op 1.16
altair processAttestation - setStatus - 1/3 committees join 202.15 us/op 212.68 us/op 0.95
altair processAttestation - setStatus - 1/2 committees join 303.39 us/op 297.67 us/op 1.02
altair processAttestation - setStatus - 2/3 committees join 390.41 us/op 384.03 us/op 1.02
altair processAttestation - setStatus - 4/5 committees join 606.22 us/op 560.55 us/op 1.08
altair processAttestation - setStatus - 100% committees join 721.91 us/op 649.98 us/op 1.11
altair processBlock - 250000 vs - 7PWei normalcase 3.6982 ms/op 3.1994 ms/op 1.16
altair processBlock - 250000 vs - 7PWei normalcase hashState 28.915 ms/op 19.719 ms/op 1.47
altair processBlock - 250000 vs - 7PWei worstcase 43.897 ms/op 19.109 ms/op 2.30
altair processBlock - 250000 vs - 7PWei worstcase hashState 82.132 ms/op 53.461 ms/op 1.54
phase0 processBlock - 250000 vs - 7PWei normalcase 2.0642 ms/op 1.3601 ms/op 1.52
phase0 processBlock - 250000 vs - 7PWei worstcase 25.254 ms/op 18.656 ms/op 1.35
altair processEth1Data - 250000 vs - 7PWei normalcase 296.12 us/op 315.91 us/op 0.94
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:16 5.5740 us/op 4.6590 us/op 1.20
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:220 23.807 us/op 28.098 us/op 0.85
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:43 10.442 us/op 10.844 us/op 0.96
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:19 8.0450 us/op 7.8490 us/op 1.02
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1021 97.548 us/op 108.72 us/op 0.90
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11778 2.4435 ms/op 1.5940 ms/op 1.53
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 2.1055 ms/op 2.0413 ms/op 1.03
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 4.4345 ms/op 2.1183 ms/op 2.09
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 4.1429 ms/op 5.7119 ms/op 0.73
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 2.2750 ms/op 2.7427 ms/op 0.83
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 4.2930 ms/op 4.7231 ms/op 0.91
Tree 40 250000 create 374.83 ms/op 469.85 ms/op 0.80
Tree 40 250000 get(125000) 93.840 ns/op 98.841 ns/op 0.95
Tree 40 250000 set(125000) 1.0274 us/op 2.0196 us/op 0.51
Tree 40 250000 toArray() 11.609 ms/op 21.537 ms/op 0.54
Tree 40 250000 iterate all - toArray() + loop 12.307 ms/op 23.814 ms/op 0.52
Tree 40 250000 iterate all - get(i) 36.842 ms/op 67.389 ms/op 0.55
Array 250000 create 3.1051 ms/op 5.6579 ms/op 0.55
Array 250000 clone - spread 841.14 us/op 858.73 us/op 0.98
Array 250000 get(125000) 0.29700 ns/op 0.30200 ns/op 0.98
Array 250000 set(125000) 0.29600 ns/op 0.30000 ns/op 0.99
Array 250000 iterate all - loop 57.606 us/op 59.181 us/op 0.97
phase0 afterProcessEpoch - 250000 vs - 7PWei 57.115 ms/op 63.369 ms/op 0.90
Array.fill - length 1000000 5.2418 ms/op 16.269 ms/op 0.32
Array push - length 1000000 20.955 ms/op 36.656 ms/op 0.57
Array.get 0.21001 ns/op 0.21110 ns/op 0.99
Uint8Array.get 0.21155 ns/op 0.21287 ns/op 0.99
phase0 beforeProcessEpoch - 250000 vs - 7PWei 13.119 ms/op 15.526 ms/op 0.84
altair processEpoch - mainnet_e81889 351.23 ms/op 272.45 ms/op 1.29
mainnet_e81889 - altair beforeProcessEpoch 36.267 ms/op 24.268 ms/op 1.49
mainnet_e81889 - altair processJustificationAndFinalization 6.2770 us/op 6.5360 us/op 0.96
mainnet_e81889 - altair processInactivityUpdates 3.8333 ms/op 5.1480 ms/op 0.74
mainnet_e81889 - altair processRewardsAndPenalties 20.719 ms/op 22.867 ms/op 0.91
mainnet_e81889 - altair processRegistryUpdates 611.00 ns/op 574.00 ns/op 1.06
mainnet_e81889 - altair processSlashings 146.00 ns/op 153.00 ns/op 0.95
mainnet_e81889 - altair processEth1DataReset 140.00 ns/op 146.00 ns/op 0.96
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.3987 ms/op 2.3494 ms/op 0.60
mainnet_e81889 - altair processSlashingsReset 733.00 ns/op 707.00 ns/op 1.04
mainnet_e81889 - altair processRandaoMixesReset 1.4530 us/op 1.2880 us/op 1.13
mainnet_e81889 - altair processHistoricalRootsUpdate 148.00 ns/op 146.00 ns/op 1.01
mainnet_e81889 - altair processParticipationFlagUpdates 480.00 ns/op 456.00 ns/op 1.05
mainnet_e81889 - altair processSyncCommitteeUpdates 114.00 ns/op 127.00 ns/op 0.90
mainnet_e81889 - altair afterProcessEpoch 60.665 ms/op 71.798 ms/op 0.84
capella processEpoch - mainnet_e217614 1.5796 s/op 981.97 ms/op 1.61
mainnet_e217614 - capella beforeProcessEpoch 146.60 ms/op 58.265 ms/op 2.52
mainnet_e217614 - capella processJustificationAndFinalization 7.8770 us/op 5.9950 us/op 1.31
mainnet_e217614 - capella processInactivityUpdates 31.923 ms/op 11.826 ms/op 2.70
mainnet_e217614 - capella processRewardsAndPenalties 197.46 ms/op 115.03 ms/op 1.72
mainnet_e217614 - capella processRegistryUpdates 4.7620 us/op 4.6200 us/op 1.03
mainnet_e217614 - capella processSlashings 143.00 ns/op 155.00 ns/op 0.92
mainnet_e217614 - capella processEth1DataReset 143.00 ns/op 155.00 ns/op 0.92
mainnet_e217614 - capella processEffectiveBalanceUpdates 34.819 ms/op 8.1439 ms/op 4.28
mainnet_e217614 - capella processSlashingsReset 739.00 ns/op 708.00 ns/op 1.04
mainnet_e217614 - capella processRandaoMixesReset 1.5900 us/op 1.3330 us/op 1.19
mainnet_e217614 - capella processHistoricalRootsUpdate 143.00 ns/op 155.00 ns/op 0.92
mainnet_e217614 - capella processParticipationFlagUpdates 484.00 ns/op 473.00 ns/op 1.02
mainnet_e217614 - capella afterProcessEpoch 200.35 ms/op 145.80 ms/op 1.37
phase0 processEpoch - mainnet_e58758 523.62 ms/op 337.93 ms/op 1.55
mainnet_e58758 - phase0 beforeProcessEpoch 92.950 ms/op 65.296 ms/op 1.42
mainnet_e58758 - phase0 processJustificationAndFinalization 6.6880 us/op 6.6600 us/op 1.00
mainnet_e58758 - phase0 processRewardsAndPenalties 19.728 ms/op 17.965 ms/op 1.10
mainnet_e58758 - phase0 processRegistryUpdates 2.3830 us/op 2.3500 us/op 1.01
mainnet_e58758 - phase0 processSlashings 144.00 ns/op 149.00 ns/op 0.97
mainnet_e58758 - phase0 processEth1DataReset 138.00 ns/op 145.00 ns/op 0.95
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 883.68 us/op 1.2187 ms/op 0.73
mainnet_e58758 - phase0 processSlashingsReset 932.00 ns/op 875.00 ns/op 1.07
mainnet_e58758 - phase0 processRandaoMixesReset 2.0010 us/op 1.2410 us/op 1.61
mainnet_e58758 - phase0 processHistoricalRootsUpdate 141.00 ns/op 146.00 ns/op 0.97
mainnet_e58758 - phase0 processParticipationRecordUpdates 1.2510 us/op 1.1810 us/op 1.06
mainnet_e58758 - phase0 afterProcessEpoch 38.084 ms/op 40.945 ms/op 0.93
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.4386 ms/op 1.2367 ms/op 1.16
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.7530 ms/op 1.7348 ms/op 1.59
altair processInactivityUpdates - 250000 normalcase 15.917 ms/op 20.024 ms/op 0.79
altair processInactivityUpdates - 250000 worstcase 18.292 ms/op 15.179 ms/op 1.21
phase0 processRegistryUpdates - 250000 normalcase 4.8220 us/op 4.6210 us/op 1.04
phase0 processRegistryUpdates - 250000 badcase_full_deposits 153.76 us/op 163.44 us/op 0.94
phase0 processRegistryUpdates - 250000 worstcase 0.5 124.92 ms/op 67.633 ms/op 1.85
altair processRewardsAndPenalties - 250000 normalcase 16.740 ms/op 15.009 ms/op 1.12
altair processRewardsAndPenalties - 250000 worstcase 26.597 ms/op 14.842 ms/op 1.79
phase0 getAttestationDeltas - 250000 normalcase 7.8447 ms/op 6.3394 ms/op 1.24
phase0 getAttestationDeltas - 250000 worstcase 6.6100 ms/op 10.479 ms/op 0.63
phase0 processSlashings - 250000 worstcase 65.560 us/op 65.999 us/op 0.99
altair processSyncCommitteeUpdates - 250000 18.627 ms/op 10.722 ms/op 1.74
BeaconState.hashTreeRoot - No change 189.00 ns/op 181.00 ns/op 1.04
BeaconState.hashTreeRoot - 1 full validator 81.629 us/op 59.100 us/op 1.38
BeaconState.hashTreeRoot - 32 full validator 1.1534 ms/op 655.73 us/op 1.76
BeaconState.hashTreeRoot - 512 full validator 14.331 ms/op 6.2278 ms/op 2.30
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 114.75 us/op 73.873 us/op 1.55
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.3543 ms/op 1.3841 ms/op 1.70
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 30.602 ms/op 13.702 ms/op 2.23
BeaconState.hashTreeRoot - 1 balances 87.673 us/op 57.329 us/op 1.53
BeaconState.hashTreeRoot - 32 balances 1.1804 ms/op 582.06 us/op 2.03
BeaconState.hashTreeRoot - 512 balances 9.4240 ms/op 4.8566 ms/op 1.94
BeaconState.hashTreeRoot - 250000 balances 195.85 ms/op 114.44 ms/op 1.71
aggregationBits - 2048 els - zipIndexesInBitList 22.636 us/op 21.107 us/op 1.07
regular array get 100000 times 24.109 us/op 23.721 us/op 1.02
wrappedArray get 100000 times 24.114 us/op 23.672 us/op 1.02
arrayWithProxy get 100000 times 12.144 ms/op 14.892 ms/op 0.82
ssz.Root.equals 22.552 ns/op 22.064 ns/op 1.02
byteArrayEquals 22.314 ns/op 21.838 ns/op 1.02
Buffer.compare 9.2690 ns/op 9.1800 ns/op 1.01
processSlot - 1 slots 9.6320 us/op 8.6310 us/op 1.12
processSlot - 32 slots 2.5839 ms/op 1.7548 ms/op 1.47
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 8.9449 ms/op 3.1517 ms/op 2.84
getCommitteeAssignments - req 1 vs - 250000 vc 2.3328 ms/op 3.7274 ms/op 0.63
getCommitteeAssignments - req 100 vs - 250000 vc 5.4982 ms/op 7.6640 ms/op 0.72
getCommitteeAssignments - req 1000 vs - 250000 vc 4.8014 ms/op 7.2895 ms/op 0.66
findModifiedValidators - 10000 modified validators 1.2617 s/op 749.49 ms/op 1.68
findModifiedValidators - 1000 modified validators 824.52 ms/op 415.36 ms/op 1.99
findModifiedValidators - 100 modified validators 770.08 ms/op 426.69 ms/op 1.80
findModifiedValidators - 10 modified validators 737.47 ms/op 334.79 ms/op 2.20
findModifiedValidators - 1 modified validators 458.29 ms/op 284.27 ms/op 1.61
findModifiedValidators - no difference 536.71 ms/op 258.71 ms/op 2.07
migrate state 1500000 validators, 3400 modified, 2000 new 4.8667 s/op 5.0626 s/op 0.96
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 3.8700 ns/op 3.8400 ns/op 1.01
state getBlockRootAtSlot - 250000 vs - 7PWei 286.54 ns/op 387.02 ns/op 0.74
computeProposerIndex 100000 validators 1.9499 ms/op 2.2951 ms/op 0.85
getNextSyncCommitteeIndices 1000 validators 5.0406 ms/op 4.7494 ms/op 1.06
getNextSyncCommitteeIndices 10000 validators 42.751 ms/op 44.723 ms/op 0.96
getNextSyncCommitteeIndices 100000 validators 146.31 ms/op 126.89 ms/op 1.15
computeProposers - vc 250000 573.67 us/op 596.53 us/op 0.96
computeEpochShuffling - vc 250000 63.582 ms/op 55.941 ms/op 1.14
getNextSyncCommittee - vc 250000 14.179 ms/op 18.315 ms/op 0.77
nodejs block root to RootHex using toHex 102.94 ns/op 103.91 ns/op 0.99
nodejs block root to RootHex using toRootHex 63.923 ns/op 66.614 ns/op 0.96
nodejs fromHex(blob) 1.0458 ms/op 1.1135 ms/op 0.94
nodejs fromHexInto(blob) 647.16 us/op 810.85 us/op 0.80
nodejs block root to RootHex using the deprecated toHexString 594.37 ns/op 634.02 ns/op 0.94
nodejs byteArrayEquals 32 bytes (block root) 26.707 ns/op 26.995 ns/op 0.99
nodejs byteArrayEquals 48 bytes (pubkey) 38.736 ns/op 39.017 ns/op 0.99
nodejs byteArrayEquals 96 bytes (signature) 36.213 ns/op 39.289 ns/op 0.92
nodejs byteArrayEquals 1024 bytes 42.489 ns/op 48.223 ns/op 0.88
nodejs byteArrayEquals 131072 bytes (blob) 1.8174 us/op 1.8546 us/op 0.98
browser block root to RootHex using toHex 147.77 ns/op 283.75 ns/op 0.52
browser block root to RootHex using toRootHex 134.94 ns/op 136.04 ns/op 0.99
browser fromHex(blob) 1.3764 ms/op 1.9078 ms/op 0.72
browser fromHexInto(blob) 709.84 us/op 914.71 us/op 0.78
browser block root to RootHex using the deprecated toHexString 426.44 ns/op 641.79 ns/op 0.66
browser byteArrayEquals 32 bytes (block root) 29.117 ns/op 28.854 ns/op 1.01
browser byteArrayEquals 48 bytes (pubkey) 40.627 ns/op 40.644 ns/op 1.00
browser byteArrayEquals 96 bytes (signature) 76.417 ns/op 76.632 ns/op 1.00
browser byteArrayEquals 1024 bytes 837.97 ns/op 783.87 ns/op 1.07
browser byteArrayEquals 131072 bytes (blob) 104.82 us/op 133.68 us/op 0.78

by benchmarkbot/action

@ensi321
ensi321 marked this pull request as ready for review August 17, 2026 22:09
@ensi321
ensi321 requested a review from a team as a code owner August 17, 2026 22:09
@ensi321 ensi321 assigned ensi321, wemeetagain and twoeths and unassigned ensi321 Aug 17, 2026
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Aug 17, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

馃挕 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74a44de2e5

鈩癸笍 About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 馃憤.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/fork-choice/src/forkChoice/forkChoice.ts
// Cached `this.head` may be stale especially after epoch transition.
// Need to updateHead() here to get the correct head
// This code only executes when block is timely. So minimal exposure to minor regression
this.isProposerBoostSameDependentRoot(this.updateHead().blockRoot, parentRootHex);

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add regression coverage for stale cached heads

Add a targeted test where an epoch transition changes fork-choice weights while this.head still points to a branch with a different shuffling-dependent root, then verify that the first timely block receives proposer boost based on the recomputed head. This consensus-critical bug fix changes only production code, so the stale-cache scenario can regress without a focused test despite the repository requiring a failing regression test for bug fixes.

AGENTS.md reference: AGENTS.md:L377-L382

Useful? React with 馃憤聽/ 馃憥.

@ensi321 ensi321 Aug 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have already conducted test and the performance impact is minimal as noted in the code comment

@nflaig nflaig Aug 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

performance impact is minimal

@ensi321 I am surprised by this, I don't see this can be minimal, it should have a measurable impact I would expect this to affect our block processing time

image

did we run this on a live mainnet node, or how was the evaluated?

@ensi321 ensi321 mentioned this pull request Aug 18, 2026
Merged
twoeths
twoeths reviewed Aug 18, 2026
View reviewed changes

@twoeths twoeths left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is due to lodestar uses the head cache which can potentially be stale when doing dependent root check for proposer boost.

I think we need to explore a cheaper fix for this, ie figure out what was the condition that caused the cached head to be stale, detect it and run updateHead() again at slot/epoch boundary

also we should run PrepareNextSlot if needed, the whole point of it is to prepare for the next slot, we'd rather update the head at that time than importBlock. And that's the difference between the compliance test and a prod lodestar node.

importBlock is a hot time that we don't want to update head at that time, it'll affect validator's performance, and we already do updateHead once there. This is the last approach if none of the above works.

@spiral-ladder

spiral-ladder commented Aug 18, 2026

Copy link
Copy Markdown
Member

also we should run PrepareNextSlot if needed, the whole point of it is to prepare for the next slot, we'd rather update the head at that time than importBlock.

but wouldn't a head update at prepareForNextSlot might likely be stale at the point of this check?

re: the pr, the compliance test specifically checks for checkpoint advancement at epoch boundaries, so if we gate it to that condition only that could help, but seems like the long term fix is to decide again where we should updateHead. Would we not be able to do this after updateTime?

@twoeths

twoeths commented Aug 18, 2026

Copy link
Copy Markdown
Member

also we should run PrepareNextSlot if needed, the whole point of it is to prepare for the next slot, we'd rather update the head at that time than importBlock.

may not be possible to do that, then the only option is at epoch boundary, when it's really needed

@spiral-ladder

spiral-ladder commented Aug 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

also we should run PrepareNextSlot if needed, the whole point of it is to prepare for the next slot, we'd rather update the head at that time than importBlock.

may not be possible to do that, then the only option is at epoch boundary, when it's really needed

we could gate that right after updateTime probably if we just want to secure the epoch boundary, but seems like the bigger picture is IIUC the stale head did not really matter until we have this mechanism, so there might be other scenarios that a stale head prior to this check might cause correctness issues (correct me if im wrong)

@twoeths

twoeths commented Aug 19, 2026

Copy link
Copy Markdown
Member

so there might be other scenarios that a stale head prior to this check might cause correctness issues (correct me if im wrong)

I don't think there is any, the (later becomes stale) head should be correct until the epoch boundary. There is a small window between epoch boundary and the new block where the head becomes stale, I also don't see any issues with that other than this issue (the isProposerBoostSameDependentRoot() check)

nflaig
nflaig reviewed Aug 19, 2026
View reviewed changes
Comment thread packages/fork-choice/src/forkChoice/forkChoice.ts
Comment on lines +775 to +776
// This code only executes when block is timely. So minimal exposure to minor regression
this.isProposerBoostSameDependentRoot(this.updateHead().blockRoot, parentRootHex);

@nflaig nflaig Aug 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code only executes when block is timely. So minimal exposure to minor regression

this comment is really misleading, when block is timely is 99.9% of blocks on mainnet, so this is very large exposure, with this change we run fork choice computation twice every slot?

the case where isProposerBoostSameDependentRoot() would produce a incorrect result seems very narrow to me, you would need two branches (A, B) have different dependent roots, meaning those have diverged more than an epoch ago, and our head needs to be on branch A before epoch tick, but branch B after epoch processing has a newer unrealized justified checkpoint so our head should be B, then first timely block extends branch B. But our head is still on branch A when we receive that block on branch B, meaning we incorrectly deny proposer boost for the timely block on branch B.

give the above, there are several pre-conditions for this to even be relevant, we might be able to narrow this down so we only need to re-compute our head if it's really necessary

@nflaig nflaig Aug 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ensi321 ensi321 Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right comment is bad

@twoeths twoeths mentioned this pull request Aug 20, 2026
Merged
wemeetagain pushed a commit that referenced this pull request Aug 24, 2026
@twoeths @lodekeeper-z
fix: recompute head after pulling up checkpoints (#9864)
bd761ec 
**Motivation**

- after epoch boundary, the head could be stale, hence
`isProposerBoostSameDependentRoot()` could fail for the next block, see
#9813

**Description**

- after we pull up checkpoints at epoch boundary, recompute head


**AI Assistance Disclosure**

- created with the help of Claude

---------

Co-authored-by: twoeths <twoeths@users.noreply.github.com>
Co-authored-by: lodekeeper-z <258924193+lodekeeper-z@users.noreply.github.com>
This was referenced Aug 24, 2026
Open
Open
@ensi321

ensi321 commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

Superseded by #9864

@ensi321 ensi321 closed this Aug 25, 2026
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Lodestar Team Coordination Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@twoeths twoeths twoeths left review comments
@nflaig nflaig nflaig left review comments
@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

Assignees

@wemeetagain wemeetagain

@twoeths twoeths

Labels

None yet

Projects

Lodestar Team Coordination
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ensi321 @spiral-ladder @twoeths @nflaig @wemeetagain