Implement whisk PoC

[dapplion]

The purpose of this implementation is to guide spec decisions based on practical considerations

• Consensus specs: [WIP] Introduce consensus code for Whisk (SSLE) with Curdleproofs ethereum/consensus-specs#3205
• Ethresearch post: https://ethresear.ch/t/whisk-a-practical-shuffle-based-ssle-protocol-for-ethereum/11763
• Crypto backend: https://github.com/asn-d6/curdleproofs

Progress

• types
• state transition function
• state upgrade on fork
• new crypto
• spec tests
• block producer updates
• HTTP API updates
  • Get block submitting k
    

    lighthouse/beacon_node/http_api/src/lib.rs

    Line 1062 in b371ad1

    // GET beacon/states/{state_id}/proposer_trackers

  • Get validator duties
• VC updates
  • k creation
  • detect proposal
• gossip validation

Open questions

Initial k value has trade-offs. Current implementation ensures not proposals are missed but requires more verification complexity
Initial k value is computed from public key so validator client implementations are simpler, but does not guarantee that k commitments are unique for new deposits

lighthouse/consensus/state_processing/src/per_block_processing/process_operations.rs

Lines 458 to 459 in 14a2f88

	// TODO: The resulting k may not be unique
	let k = compute_initial_whisk_k_from_pubkey(&deposit.data.pubkey);

Should new validators submit k trackers on deposits to prevent insecure proposals for new validators?

Can the validator_client produce the opening proof to not have to submit k?

Blockers

Running test harness with mainnet preset on whisk_few_epochs test so WHISK_CANDIDATE_TRACKERS_COUNT = 16384 but VALIDATOR_COUNT = 32 so there are many repetitions in the selected trackers. If a validator first and second proposal are in the same whisk round, it will not be able to produce a valid opening proof since the tracker and commitment in the state are computed with different k values.

RUST_BACKTRACE=full FORK_NAME=deneb cargo test --release --features fork_from_env,logging/test_logger -p beacon_chain whisk_few_epochs -- --nocapture

[dapplion]

Currently fixing broken release tests

RUST_BACKTRACE=1 cargo test --workspace --release --features logging/test_logger --exclude ef_tests --exclude beacon_chain --exclude slasher interactive_tests::proposer_boost_re_org_parent_distance -- --nocapture

Whisk breaks proposer registration

Aug 09 16:51:08.736 CRIT Failed to prepare proposers after fork choice, error: BeaconStateError(NoProposerShuffleAfterWhisk), module: beacon_chain::canonical_head:1212

Beacon node doesn't not currently who is the next proposer

---

Adding a new route /eth/v2/validator/prepare_beacon_proposer where the validator declares to the beacon node that it is the proposer. Beacon chain caches that information and uses it to figure out who proposes next in beacon_chain.rs prepare_beacon_proposer().

• Q: What if multiple validators register for the same slot? Validator could submit the opening proof since it's constant
• Q: How to handle re-orgs? polling trackers should return a dependant_root, and validator should submit it, declaring I'm the proposer at slot N for this dependant_root.

[dapplion]

dapplion-whisk-CI-run-beaconchaintests-bf52a25453830d133fdf62f93bbe56a7bc6ccc6d.log

beacon-chain tests timeout, all forks pass except capella

2023-08-15T17:13:36.0795690Z env FORK_NAME=capella cargo test --release --features fork_from_env,slasher/lmdb -p beacon_chain
...
2023-08-15T17:14:35.1616347Z running 115 tests
2023-08-15T17:14:39.9984230Z test attestation_production::early_attester_cache_old_request ... ok
2023-08-15T17:15:01.8646934Z test attestation_verification::aggregated_attestation_verification_use_head_state_fork ... ok
2023-08-15T17:15:35.1626741Z test attestation_production::produces_attestations has been running for over 60 seconds
2023-08-15T17:16:01.8648048Z test attestation_verification::aggregated_gossip_verification has been running for over 60 seconds
2023-08-15T17:17:14.1329459Z test attestation_verification::aggregated_gossip_verification ... ok
2023-08-15T17:17:42.1623434Z test attestation_production::produces_attestations ... ok
2023-08-15T17:18:14.1330242Z test attestation_verification::attestation_that_skips_epochs has been running for over 60 seconds
2023-08-15T17:18:42.1623834Z test attestation_verification::attestation_to_finalized_block has been running for over 60 seconds
2023-08-15T17:19:21.8593003Z test attestation_verification::attestation_that_skips_epochs ... ok
2023-08-15T17:20:21.8593980Z test attestation_verification::attestation_validator_receive_proposer_reward_and_withdrawals has been running for over 60 seconds
2023-08-15T17:20:46.8968582Z test attestation_verification::attestation_to_finalized_block ... ok
2023-08-15T17:21:08.6384505Z test attestation_verification::attestation_verification_use_head_state_fork ... ok
2023-08-15T17:21:14.5537351Z test attestation_verification::attestation_validator_receive_proposer_reward_and_withdrawals ... ok
2023-08-15T17:22:08.6385176Z test attestation_verification::unaggregated_gossip_verification has been running for over 60 seconds
2023-08-15T17:22:14.5538059Z test attestation_verification::verify_aggregate_for_gossip_doppelganger_detection has been running for over 60 seconds
2023-08-15T17:23:22.5406232Z test attestation_verification::unaggregated_gossip_verification ... ok
2023-08-15T17:23:28.7090598Z test attestation_verification::verify_aggregate_for_gossip_doppelganger_detection ... ok
2023-08-15T17:23:29.1322712Z test block_verification::add_altair_block_to_base_chain ... ok
2023-08-15T17:23:33.3477237Z test block_verification::add_base_block_to_altair_chain ... ok
2023-08-15T17:24:22.5406704Z test attestation_verification::verify_attestation_for_gossip_doppelganger_detection has been running for over 60 seconds
2023-08-15T17:24:33.3477991Z test block_verification::block_gossip_verification has been running for over 60 seconds
2023-08-15T17:25:35.4265289Z test attestation_verification::verify_attestation_for_gossip_doppelganger_detection ... ok
2023-08-15T17:26:35.4265885Z test block_verification::chain_segment_full_segment has been running for over 60 seconds
2023-08-15T18:02:00.9132123Z test block_verification::block_gossip_verification ... FAILED
2023-08-15T18:03:00.9132522Z test block_verification::chain_segment_non_linear_parent_roots has been running for over 60 seconds
2023-08-15T18:05:14.3769369Z test block_verification::chain_segment_full_segment ... ok
2023-08-15T18:06:14.3770775Z test block_verification::chain_segment_non_linear_slots has been running for over 60 seconds
2023-08-15T18:40:24.5905963Z test block_verification::chain_segment_non_linear_parent_roots ... ok
2023-08-15T18:41:24.5906371Z test block_verification::chain_segment_varying_chunk_size has been running for over 60 seconds
2023-08-15T18:43:37.5068970Z test block_verification::chain_segment_non_linear_slots ... ok
2023-08-15T18:43:47.5197470Z test block_verification::import_duplicate_block_unrealized_justification ... ok
2023-08-15T18:44:47.5197985Z test block_verification::invalid_signature_attestation has been running for over 60 seconds
2023-08-15T19:22:08.5735436Z test block_verification::invalid_signature_attestation ... FAILED
2023-08-15T19:23:08.5735882Z test block_verification::invalid_signature_attester_slashing has been running for over 60 seconds
2023-08-15T20:00:28.1944603Z test block_verification::invalid_signature_attester_slashing ... FAILED
2023-08-15T20:01:28.1945291Z test block_verification::invalid_signature_block_proposal has been running for over 60 seconds
2023-08-15T20:41:12.5938139Z test block_verification::invalid_signature_block_proposal ... ok
2023-08-15T20:42:12.5938449Z test block_verification::invalid_signature_deposit has been running for over 60 seconds
2023-08-15T21:19:26.1659667Z test block_verification::invalid_signature_deposit ... FAILED
2023-08-15T21:20:26.1659770Z test block_verification::invalid_signature_exit has been running for over 60 seconds
2023-08-15T21:51:13.6502123Z ##[error]The operation was canceled.
2023-08-15T21:51:13.6758382Z Post job cleanup.
2023-08-15T21:51:14.0065055Z [command]/usr/bin/git version
2023-08-15T21:51:14.4796937Z git version 2.41.0
2023-08-15T21:51:14.4804091Z Temporarily overriding HOME='/home/runner/work/_temp/d1b96c56-a9a3-49eb-8f5b-3dc4fc654577' before making global git config changes
2023-08-15T21:51:14.4804697Z Adding repository directory to the temporary git global config as a safe directory
2023-08-15T21:51:14.4805353Z [command]/usr/bin/git config --global --add safe.directory /home/runner/work/lighthouse/lighthouse
2023-08-15T21:51:14.4806444Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-08-15T21:51:14.4807633Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2023-08-15T21:51:14.4809177Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-08-15T21:51:14.4809760Z http.https://github.com/.extraheader
2023-08-15T21:51:14.4810780Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2023-08-15T21:51:14.4812232Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all