[do not merge] multi-block EPM and staking (for benchmarking) #5272
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…2504) > highly WIP, opening draft PR for early feedback. This PR implements a PoV friendly, multi-block EPM to be used by the staking parachain. It is split into multiple sub-pallets for better logic and storage encapsulation and better readability. The pallet split consists of a main pallet and several other sub-pallets that implement the logic for different sub-systems, namely: - **main pallet**: - implements the `trait ElectionProvider` - `fn elect(remaining_pages)` basically fetches the queued page from the `pallet::verifier`, which keeps the valid solutions in its storage. - manages current election `Phase` in `on_initialize`. The current phase signals other sub-pallets what to do. - stores and manages the (paged) target and voter snapshots. - *note*: the staking pallet needs to return/interpret a paged fetching of the snapshot data for both voters and targes. - **signed pallet**: - implements the `trait SolutionDataProvider`, which provides a paged solution for the current best score (the `pallet::verifier` is the consumer, when trying to fetch the best queued solution to verify). - keeps track of the best solution commitments from submitters. - exposes `Call::register` for submitters to submit a commitment score for their solution. - exposes callable `Call::submit_page` for submitters to submit a page of their solution. - upon the verifier pallet finalizing the paged solution verification, it handles the submission deposit/rewards based on the reported `VerificationResult` (from `pallet::signed`). - **verifier pallet**: - implements the `trait Verifier`: verifies one solution page on-call. The inputs for the verification are provided in-place. - implements the `trait AsyncVerifier`: fetches pages from the implementor of `SolutionDataProvider` (implemented by `pallet::signed`) and verifies the paged solution. - `on_initialize`, it checks if the verification is ongoing and proceeds with it - it has it's own `VerificationStatus` which signals the current state of the verification - for each successfully verified page, add it to the `QueuedSolution` storage. - at the end of verifying a solution, it reports the results of the verification back to the implementor of `trait SolutionDataProvider` (`pallet::signed`) - **unsigned pallet**: - `on_initialize` checks if on `UnsignedPhase` and no queued solution; compute a solution with offchain Miner. - implements the off-chain unsigned (paged) miner. - implements the inherent call that processes unsigned submissions. --- ### Todo/discussion - [x] E2E multi-page election with staking and EPM-MB pallet integration. - [ ] refactor the current `on_initialize` across all pallets to make explicit calls depending on the current phase, rather than relying on the pallet's `on_initialize` and current phase to decide what to do at a given block (TBD). - [ ] remove the `Emergency` phase and instead just keep trying the election in case of failure. - [ ] refactor current `SignedValidation` phase to have enough blockspace to verify all queued signed submissions, for security purposes (ie. at least `max_num_queued_submissions * T::Pages` blocks allocated to signed verification, return early if a submission is valid and accepted). - [x] implement the paged ingestion of the election results in the staking pallet. How to convert from multiple `BoundedSupports` to `Exposures` in the staking pallet in a nice way (add integration tests). - idea: if each page contains up to `N` targets and a validator may appear only in one page, we can process the pages in serie in the staking side, keeping track of the state of `Exposures` across the pages. - [ ] allow the validator to replace the current submission if their submission has better score than the accepted queued submission. - [ ] mutations to both the target and voter lists need to "freeze" while the snapshot is being generated (now multi-block). what's the best approach? Closes: #2199 Related to: #491 Inspiration from https://github.com/paritytech/substrate/tree/kiz-multi-block-election
Contributor
Author
|
bot bench substrate-pallet --pallet=pallet_election_provider_multi_block |
|
@gpestana https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7078600 was started for your command Comment |
|
The CI pipeline was cancelled due to failure one of the required jobs. |
|
@gpestana Command |
gpestana
force-pushed
the
staking-parachain-exp
branch
from
7648ad6 to
5b81cea
Compare
Refactors unsigned and verifier code to add a `trait MinerConfig` that encapsulates types and logic required for staking miner to work, without having to access the whole pallet's config.
…t-sdk into staking-parachain-exp
26 tasks
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 15, 2025
## Multi Block Election Pallet This PR adds the first iteration of the multi-block staking pallet. From this point onwards, the staking and its election provider pallets are being customized to work in AssetHub. While usage in solo-chains is still possible, it is not longer the main focus of this pallet. For a safer usage, please fork and user an older version of this pallet. --- ## Replaces - [x] #6034 - [x] #5272 ## Related PRs: - [x] #7483 - [ ] #7357 - [ ] #7424 - [ ] paritytech/polkadot-staking-miner#955 This branch can be periodically merged into #7358 -> #6996 ## TODOs: - [x] rebase to master - Benchmarking for staking critical path - [x] snapshot - [x] election result - Benchmarking for EPMB critical path - [x] snapshot - [x] verification - [x] submission - [x] unsigned submission - [ ] election results fetching - [ ] Fix deletion weights. Either of - [ ] Garbage collector + lazy removal of all paged storage items - [ ] Confirm that deletion is small PoV footprint. - [ ] Move election prediction to be push based. @tdimitrov - [ ] integrity checks for bounds - [ ] Properly benchmark this as a part of CI -- for now I will remove them as they are too slow - [x] add try-state to all pallets - [x] Staking to allow genesis dev accounts to be created internally - [x] Decouple miner config so @niklasad1 can work on the miner 72841b7 - [x] duplicate snapshot page reported by @niklasad1 - [ ] #6520 or equivalent -- during snapshot, `VoterList` must be locked - [ ] Move target snapshot to a separate block --------- Co-authored-by: Gonçalo Pestana <[email protected]> Co-authored-by: Ankan <[email protected]> Co-authored-by: command-bot <> Co-authored-by: Guillaume Thiolliere <[email protected]> Co-authored-by: Giuseppe Re <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
clangenb
pushed a commit
to clangenb/polkadot-sdk
that referenced
this pull request
Feb 19, 2025
## Multi Block Election Pallet This PR adds the first iteration of the multi-block staking pallet. From this point onwards, the staking and its election provider pallets are being customized to work in AssetHub. While usage in solo-chains is still possible, it is not longer the main focus of this pallet. For a safer usage, please fork and user an older version of this pallet. --- ## Replaces - [x] paritytech#6034 - [x] paritytech#5272 ## Related PRs: - [x] paritytech#7483 - [ ] paritytech#7357 - [ ] paritytech#7424 - [ ] paritytech/polkadot-staking-miner#955 This branch can be periodically merged into paritytech#7358 -> paritytech#6996 ## TODOs: - [x] rebase to master - Benchmarking for staking critical path - [x] snapshot - [x] election result - Benchmarking for EPMB critical path - [x] snapshot - [x] verification - [x] submission - [x] unsigned submission - [ ] election results fetching - [ ] Fix deletion weights. Either of - [ ] Garbage collector + lazy removal of all paged storage items - [ ] Confirm that deletion is small PoV footprint. - [ ] Move election prediction to be push based. @tdimitrov - [ ] integrity checks for bounds - [ ] Properly benchmark this as a part of CI -- for now I will remove them as they are too slow - [x] add try-state to all pallets - [x] Staking to allow genesis dev accounts to be created internally - [x] Decouple miner config so @niklasad1 can work on the miner 72841b7 - [x] duplicate snapshot page reported by @niklasad1 - [ ] paritytech#6520 or equivalent -- during snapshot, `VoterList` must be locked - [ ] Move target snapshot to a separate block --------- Co-authored-by: Gonçalo Pestana <[email protected]> Co-authored-by: Ankan <[email protected]> Co-authored-by: command-bot <> Co-authored-by: Guillaume Thiolliere <[email protected]> Co-authored-by: Giuseppe Re <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds the multi block election provider core pallet and sub-pallets and makes staking multiblock for benchmarking with the CI infra.