[AHM] Multi-block election patches #7601
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… updated post election via rebag
Contributor
Author
|
/cmd help |
Contributor
|
Command "help" has started 🚀 See logs here |
Contributor
|
Command "help" has failed ❌! See logs here |
Contributor
Author
|
/cmd --help |
Contributor
Command help: |
Contributor
Author
|
/cmd bench --help |
Contributor
Command help: |
…dk into kiz-mbm-election-2
Contributor
Author
|
/cmd prdoc |
Contributor
|
Command "prdoc" has failed ❌! See logs here |
…et-staking` (#7658) This PR does: - Removes `ExposureOf` usages from `pallet-staking-ah-client` - Adds `pallet-staking-ah-client` to Westend runtime - Replaces all usages of `pallet-staking` in Westend with `pallet-staking-ah-client` --------- Co-authored-by: Kian Paimani <[email protected]> Co-authored-by: Ankan <[email protected]> Co-authored-by: Ankan <[email protected]> Co-authored-by: kianenigma <[email protected]>
|
All GitHub workflows were cancelled due to failure one of the required jobs. |
github-merge-queue bot
pushed a commit
that referenced
this pull request
Mar 24, 2025
#7939) Revert the following PRs which we are pulling from release stable2503: - #7582 - #7424 - #7282 and leave pallet-staking in its pre-AHM state. ## Context We are forking pallet-staking into `pallet-staking` (also referred as staking-classic, this is the version that will stay on RC) and `pallet-staking-next` which will live on AH post AHM. Additional context: #7858 (comment) These changes in crate `pallet-staking` will become the staking classic. The staking next version is worked in the PR #7601. ## For AHM migration The `UnappliedSlashes` storage will need to be translated from `rc::staking-classic` to `ah::staking-next`. [Bookmarking](https://github.com/paritytech/polkadot-sdk/blob/master/substrate/frame/staking/src/migrations.rs#L91) the code that can be referred for this. ## Follow ups (cc: @tdimitrov and @seadanda ) 1) Revert pallet-staking v17 migration in westend. - Update `in code storage version` of pallet-staking storage from 17 to 16 (separate PR). - Update `on chain storage version` of pallet-staking storage from 17 to 16. The storage key for pallet-staking on chain version is `0x5f3e4907f716ac89b6347d15ececedca4e7b9012096b41c4eb3aaf947f6ea429` which should be set currently to `0x1100`, and needs to be updated to `0x1000`. - After the runtime upgrade with the code from this PR is deployed on Westend, kill the following storage prefixes under the Pallet prefix `Staking`: - OffenceQueue - OffenceQueueEras - ProcessingOffence - UnappliedSlashes: This also exists in staking-classic as a storage map (one key) and in pre-revert code as double storage map (two keys). Killing with prefix `UnappliedSlashes` may kill the ones created post upgrade (but that's okay for westend). - VoterSnapshotStatus - NextElectionPage - ElectableStashes 2) Remove exposure dependency Worked in the PR: #7936. --------- Co-authored-by: Tsvetomir Dimitrov <[email protected]> Co-authored-by: Maciej <[email protected]>
Ank4n
added a commit
that referenced
this pull request
Mar 25, 2025
#7939) Revert the following PRs which we are pulling from release stable2503: - #7582 - #7424 - #7282 and leave pallet-staking in its pre-AHM state. We are forking pallet-staking into `pallet-staking` (also referred as staking-classic, this is the version that will stay on RC) and `pallet-staking-next` which will live on AH post AHM. Additional context: #7858 (comment) These changes in crate `pallet-staking` will become the staking classic. The staking next version is worked in the PR #7601. The `UnappliedSlashes` storage will need to be translated from `rc::staking-classic` to `ah::staking-next`. [Bookmarking](https://github.com/paritytech/polkadot-sdk/blob/master/substrate/frame/staking/src/migrations.rs#L91) the code that can be referred for this. (cc: @tdimitrov and @seadanda ) 1) Revert pallet-staking v17 migration in westend. - Update `in code storage version` of pallet-staking storage from 17 to 16 (separate PR). - Update `on chain storage version` of pallet-staking storage from 17 to 16. The storage key for pallet-staking on chain version is `0x5f3e4907f716ac89b6347d15ececedca4e7b9012096b41c4eb3aaf947f6ea429` which should be set currently to `0x1100`, and needs to be updated to `0x1000`. - After the runtime upgrade with the code from this PR is deployed on Westend, kill the following storage prefixes under the Pallet prefix `Staking`: - OffenceQueue - OffenceQueueEras - ProcessingOffence - UnappliedSlashes: This also exists in staking-classic as a storage map (one key) and in pre-revert code as double storage map (two keys). Killing with prefix `UnappliedSlashes` may kill the ones created post upgrade (but that's okay for westend). - VoterSnapshotStatus - NextElectionPage - ElectableStashes 2) Remove exposure dependency Worked in the PR: #7936. --------- Co-authored-by: Tsvetomir Dimitrov <[email protected]> Co-authored-by: Maciej <[email protected]>
17 tasks
Contributor
Author
|
closing in favor of #8127, please use this one. |
github-merge-queue bot
pushed a commit
that referenced
this pull request
Apr 21, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <[email protected]> Co-authored-by: Tsvetomir Dimitrov <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Apr 22, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <[email protected]> Co-authored-by: Tsvetomir Dimitrov <[email protected]>
castillax
pushed a commit
that referenced
this pull request
May 12, 2025
Moved from: #7601. Follow ups to: #7282. Closes: #8146 --- This PR is the final outcome of a multi-month development period, with a lot of background work since 2022. Its main aim is to make pallet-staking, alongside its `type ElectionProvider` compatible to be used in a parachain, and report back the validator set to a relay-chain. This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the corresponding AssetHubs hosting the staking system. While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime for testing. The following is a guide to help reviewers/auditors distinguish what has actually changed in this PR. > Additional reading: See polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate. ## Added > [This shows the partial diff](ankn/diff-staking-async...ankn/diff-staking-async-1) introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase. This PR adds the following new pallets, all of which are not used anywhere yet, with the exception of one (see `westend-runtime` changes below). #### `pallet-election-provider-multi-block` This is a set of 4 pallets, capable of implementing an async, multi-page `ElectionProvider`. This pallet is not used in any real runtime yet, and is intended to be used in `AssetHub`, next to `pallet-staking-async`. #### `pallet-staking-async` A fork of the old `pallet-staking`, with a number of key differences, making it suitable to be used in a parachain: 1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era. 2. It no longer has access to a `pallet-session`. 3. It no longer has access to a `pallet-authorship`. 5. It is capable of working with a multi-page `ElectionProvider`, aka. `pallet-election-provider-multi-block`. To compensate for the above, this pallet relies on XCM messages coming from the relay-chain, informing the pallet of: * When a new era should be activated, and how long its duration was * When an offence has happened on the relay relay-chain * When a session ends on the relay-chain, and how many reward points were accumulated for each validators during that period. #### `pallet-staking-async-ah-client` and `pallet-staking-async-rc-client` Are the two new pallets that facilitate the above communication. #### `pallet-ahm-test` A test-only crate that contains e2e rust-based unit test for all of the above. #### `pallet-staking-async-rc-runtime` and `pallet-staking-async-parachain-runtime` Forks of westend and westend-asset-hub, customized to be used for testing all of the above with Zombienet. It contains a lot of unrelated code as well. ## Changed > [This shows the partial diff](ankn/8127-diff-changed-base...ankn/8127-diff-changed-compare) that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes. #### `Identification` This mechanism, which lives on the relay-chain, is expressed by `type FullIdentification` and `type FullIdentificationOf` in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their `struct Exposure`. With the move to Asset-Hub, this is no longer possible for two reasons: 1. Relay chain no longer knows the full exposures 2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable. Instead, runtimes now move to a new `type FullIdentificationOf = DefaultExposureOf`, which will identify a validator with a `Exposure::default()`. This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old `FullIdentification`, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM. #### `westend-runtime` This runtime already has the `pallet-staking-async-ah-client`, integrated into all the places such that: 1. It handles the validator reward points 2. It handles offences 5. It is the `SessionManager` Yet, it is delegating all of the above to its `type Fallback`, which is the old `pallet-staking`. This is a preparatory step for AHM, and should not be any logical change. #### `pallet-election-provider-multi-phase` This is the old single-page `ElectionProvider`. It has been updated to work with multi-page traits, yet it only supports `page-size = 1` for now. It should not have seen any logical changes. #### `pallet-bags-list` Now has two new features. 1. It can be `Locked`, in which case all updates to it fail with an `Err(_)`, even deletion of a node. This is needed because we cannot alter any nodes in this pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same `rebag` transaction can be also be used to remove a node from the list, or add a node to the list. This is done through the `score_of` api. See the file changes and tests under `./substrate/frame/bags-list` for more info. #### RuntimeDebug -> Debug To facilitate debugging, a number of types' `RuntimeDebug` impl has been changed to `Debug`. See #3107 ## Weights Below is a summary of the weights. These are generated using `staking-async/runtimes/parachain`, which assumes 22_500 nominators divided by `32` pages for Polkadot, and 12_500 nominators divided by `16` pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account. ``` #### new: polkadot/pallet_election_provider_multi_block.rs old: kusama +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +======================================================================================================================+ | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp | 2.41MiB | 2.41MiB | -0.03 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest | 3.24MiB | 3.06MiB | -5.53 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed | 3.36MiB | 3.12MiB | -7.12 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_non_terminal | 2.12MiB | 1.32MiB | -37.60 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | export_terminal | 4.08MiB | 2.25MiB | -44.82 | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_nothing | 3.53KiB | 3.53KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_unsigned | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged | |-----------------------------------------+--------------------------------------+---------+---------+-----------------| | pallet_election_provider_multi_block.rs | manage | 0B | 0B | Unchanged | +-----------------------------------------+--------------------------------------+---------+---------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama +------------------------------------------------+----------------------+----------+----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +===============================================================================================================+ | pallet_election_provider_multi_block_signed.rs | bail | 43.61KiB | 82.74KiB | +89.72 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_eject | 46.54KiB | 85.80KiB | +84.35 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | submit_page | 6.95KiB | 6.90KiB | -0.70 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | register_not_full | 6.45KiB | 6.39KiB | -1.00 | |------------------------------------------------+----------------------+----------+----------+-----------------| | pallet_election_provider_multi_block_signed.rs | unset_page | 20.76KiB | 18.55KiB | -10.67 | +------------------------------------------------+----------------------+----------+----------+-----------------+ #### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama +--------------------------------------------------+-------------------+----------+-----------+------------------+ | File | Extrinsic | Old | New | Change [%] | +================================================================================================================+ | pallet_election_provider_multi_block_unsigned.rs | submit_unsigned | 63.56KiB | 696.00KiB | +995.01 | |--------------------------------------------------+-------------------+----------+-----------+------------------| | pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB | 3.66KiB | +102.65 | +--------------------------------------------------+-------------------+----------+-----------+------------------+ #### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ | File | Extrinsic | Old | New | Change [%] | +=================================================================================================================================+ | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal | 1.18MiB | 1.69MiB | +42.87 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal | 1.18MiB | 1.69MiB | +42.71 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB | 450.82KiB | -66.08 | |--------------------------------------------------+------------------------------------+-----------+-----------+-----------------| | pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal | 279.93KiB | 62.22KiB | -77.77 | +--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+ ``` <summary> note for PR authors <details> <br> ## TODO - [x] Finalize weights - [x] Lock voter list when snapshot being taken - [x] push based election - [x] OffchainWorker miner can now run on multiple pages - [x] Trimming is improved, all bounds are respected. - [x] clients pallets: add ID - [x] make election prolonged - [x] bring westend-next and ah-next to staking-next - [x] Test pre-migration to post-migration state in ahm-test. - [x] Offence reporting works without exposure info on RC (done but recheck). - [ ] staking-async fix tests - [ ] root offence testing (minimally done in migration test) - [ ] Run benchmarking - [x] ~~Add custom decoder for OffenceDetails~~. ## TODO before finalizing PR - [x] Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private. - [ ] Create diff with changes from staking-classic. ## Migration Notes - At the start of the AHM migration, trigger: `RC::pallet_staking_async_ah_client::on_migration_start()` - At the start of the AHM migration, trigger the following: - definitely filter `staking::bond` - RC: set `staking::Forcing` to `ForceNone`. - At the end of the AHM migration, trigger the following - `RC::pallet_staking_async_ah_client::on_migration_end()` - Set `AH::pallet_staking_async::ForceEra` to `Forcing::NotForcing`. - Set RC staking and pool min bond to be u32::max. ## Follow-up - [ ] Offence generation e2e test (zombienet) </details> </summary> --------- Co-authored-by: kianenigma <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Kian Paimani <[email protected]> Co-authored-by: Tsvetomir Dimitrov <[email protected]>
21 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Follow ups to #7282
AH Next / Westend Next
Westend Classic