Compute proposer shuffling only once in gossip verification

[jimmygchen]

Issue Addressed

When we perform data column gossip verification, we sometimes see multiple proposer shuffling cache miss simultaneously and this results in multiple threads computing the shuffling cache and potentially slows down the gossip verification.

Proposal here is to use a OnceCell for each shuffling key to make sure it's only computed once. I have only implemented this in data column verification as a PoC, but this can also be applied to blob and block verification

Related issues:

• Getting shuffling efficiently in gossip blobs signature verification #4447
• Data column gossip verification taking more than 4s when blob count is above 20 #7203

[jimmygchen]

I've pushed a PoC, however I think this is still problematic because we hold the lock for the entire duration of the computation.

[jimmygchen]

Pushed another reworked version to avoid lock contention mentioned above @pawanjay176

[eserilev]

Nice!

[jimmygchen]

Thanks @eserilev !

@pawanjay176 I wrote a test to measure time for each step in column verification with some numbers for each step, and comparison with unstable. This seems to work and could potentially perform better when columns are received in a staggered way.

Single column verification

This single test (7 blobs) shows that verify_proposer_and_signature is the only step taking significant amount of time ~500ms. The rest are pretty negligible.

verify_data_column_sidecar: 29.458µs
verify_index_matches_subnet: 2.125µs
verify_sidecar_not_from_future_slot: 8.041µs
verify_slot_greater_than_latest_finalized_slot: 10.5µs
verify_is_first_sidecar: 3.75µs
verify_column_inclusion_proof: 218.125µs
verify_parent_block_and_finalized_descendant: 33.083µs
verify_slot_higher_than_parent: 750ns
verify_proposer_and_signature: 519.821916ms
verify_kzg_for_data_column: 4.024208ms
observe_slashable: 54.709µs
All verifications passed successfully
verify_total_time_0: 524.22925ms

Running multiple concurrent verifications + shuffling cache miss

8 verifications triggered 100ms apart in a 8 worker threads test.
On PeerDAS network the number of concurrent verifications could be higher.

Without Optimisation (unstable)

Result:

• Majority of verify_proposer_and_signature took 500+ ms, as they all tried to load state and compute shuffling.
• 6 out of 8 columns took 500+ms total time.

verify_proposer_and_signature: 514.836333ms
verify_proposer_and_signature: 1.295792ms
verify_proposer_and_signature: 523.640709ms
verify_proposer_and_signature: 1.183375ms
verify_proposer_and_signature: 536.179417ms
verify_proposer_and_signature: 537.438958ms
verify_proposer_and_signature: 510.963667ms
verify_proposer_and_signature: 501.606625ms

verify_kzg_for_data_column: 5.755458ms
verify_kzg_for_data_column: 7.827084ms
verify_kzg_for_data_column: 18.534791ms
verify_kzg_for_data_column: 5.266708ms
verify_kzg_for_data_column: 5.164334ms
verify_kzg_for_data_column: 4.364958ms
verify_kzg_for_data_column: 4.01325ms
verify_kzg_for_data_column: 3.930458ms

verify_total_time_6: 9.262542ms
verify_total_time_0: 520.857667ms
verify_total_time_1: 542.32725ms
verify_total_time_7: 6.59375ms
verify_total_time_2: 541.493416ms
verify_total_time_3: 541.944417ms
verify_total_time_4: 515.2415ms
verify_total_time_5: 505.678333ms

After Optimisation (OnceCell)

Result:

• Only one instance of verify_proposer_and_signature took 500+ ms, and the rest are lower depending on when the call was made - state was only load once and result was reused.
• Columns that arrive later took less time to verify, as they don't try to compute shuffling cache again.

verify_proposer_and_signature: 82.829084ms
verify_proposer_and_signature: 490.811709ms
verify_proposer_and_signature: 185.32525ms
verify_proposer_and_signature: 388.889333ms
verify_proposer_and_signature: 287.246667ms
verify_proposer_and_signature: 931.125µs
verify_proposer_and_signature: 2.505083ms
verify_proposer_and_signature: 1.602709ms

verify_kzg_for_data_column: 5.510167ms
verify_kzg_for_data_column: 5.613208ms
verify_kzg_for_data_column: 5.590584ms
verify_kzg_for_data_column: 5.566792ms
verify_kzg_for_data_column: 5.530834ms
verify_kzg_for_data_column: 5.974208ms
verify_kzg_for_data_column: 13.915542ms

verify_total_time_0: 496.674875ms
verify_total_time_4: 88.488625ms
verify_total_time_3: 191.037667ms
verify_total_time_2: 292.921833ms
verify_total_time_1: 394.67475ms
verify_total_time_5: 4.596292ms
verify_total_time_6: 7.768875ms
verify_total_time_7: 17.04425ms

[jimmygchen]

May be worth adding this to block verification as well, as we're likely to receive the block + 128 columns within a small window under PeerDAS.

[dapplion]

@jimmygchen we can as well cache the proposer signature verification with a similar strategy: LRUCache + OnceCell. Have a cache LRUCache<(BlockRoot, BlockSignature), Result>, and use it in block, blob and data column proposer signature verification. In a supernode case we will amortize the 500 ms across 128 columns.

[jimmygchen]

Do you see much benefit of adding this signature cache in addition to beacon_proposer_cache?
From the numbers above, if we hit the beacon_proposer_cache, the verification time is quite minimal: 0.9 - 2.5ms. Am I missing something?

[dapplion]

From the numbers above, if we hit the beacon_proposer_cache, the verification time is quite minimal: 0.9 - 2.5ms. Am I missing something?

Right not necessary for now, we can revisit later if you want to hyper optimize

[pawanjay176]

LGTM.
I think worth considering adding the same to block and blob validation after we have tested this on peerdas devnets for some time

[jimmygchen]

Thanks all!

I've found a bug in data column verification which causes proposer shuffling cache miss on every data column, due to incorrect epoch calculation only in minimal preset, as it was hard coded to use MainnetEthSpec.

This shouldn't happen often at all. But this PR should improve the performance in case if it happens and not do 128 + 1 state load + shuffling computation.

Agree to do it on block + blob verification as well once this is proven to be stable for some time on PeerDAS.

I'll go ahead and merge this now.