[AHM] Async Staking module across AH and RC (#8127)

[xlc]

Moved from: paritytech#7601.
Follow ups to: paritytech#7282.
Closes: paritytech#8146

---

This PR is the final outcome of a multi-month development period, with a lot of background work
since 2022. Its main aim is to make pallet-staking, alongside its type ElectionProvider
compatible to be used in a parachain, and report back the validator set to a relay-chain.

This setup is intended to be used for Polkadot, Kusama and Westend relay-chains, with the
corresponding AssetHubs hosting the staking system.

While this PR is quite big, a lot of the diffs are due to adding a relay and parachain runtime
for testing. The following is a guide to help reviewers/auditors distinguish what has actually
changed in this PR.

Additional reading: See
polkadot-js/apps#11401, and the hackmd shared in there, which contains more in-depth explanation of how RC <> AH communicate.

Added

This shows the partial
diff introduced in pallet-staking-async and election-provider-multi-block relative to the existing (in master) pallet-staking and election-provider-multi-phase.

This PR adds the following new pallets, all of which are not used anywhere yet, with the
exception of one (see westend-runtime changes below).

pallet-election-provider-multi-block

This is a set of 4 pallets, capable of implementing an async, multi-page ElectionProvider.
This pallet is not used in any real runtime yet, and is intended to be used in AssetHub, next
to pallet-staking-async.

pallet-staking-async

A fork of the old pallet-staking, with a number of key differences, making it suitable to be
used in a parachain:

1. It no longer has access to a secure timestamp, previously used to calculate the duration of an era.
2. It no longer has access to a pallet-session.
3. It no longer has access to a pallet-authorship.
4. It is capable of working with a multi-page ElectionProvider, aka. pallet-election-provider-multi-block.

To compensate for the above, this pallet relies on XCM messages coming from the relay-chain,
informing the pallet of:

• When a new era should be activated, and how long its duration was
• When an offence has happened on the relay relay-chain
• When a session ends on the relay-chain, and how many reward points were accumulated for each
  validators during that period.

pallet-staking-async-ah-client and

pallet-staking-async-rc-client

Are the two new pallets that facilitate the above communication.

pallet-ahm-test

A test-only crate that contains e2e rust-based unit test for all of the above.

pallet-staking-async-rc-runtime and

pallet-staking-async-parachain-runtime

Forks of westend and westend-asset-hub, customized to be used for testing all of the above with
Zombienet. It contains a lot of unrelated code as well.

Changed

This shows the partial
diff that shows the changes to existing pallets used in prod runtimes as well as westend runtime changes.

Identification

This mechanism, which lives on the relay-chain, is expressed by type FullIdentification and type FullIdentificationOf in runtimes. It is a way to identify the full data needed to slash a validator. Historically, it was pointing to a validator, and their struct Exposure. With the move to Asset-Hub, this is no longer possible for two reasons:

1. Relay chain no longer knows the full exposures
2. Even if, the full exposures are getting bigger and bigger and relying the entirety of it is not scalable.

Instead, runtimes now move to a new type FullIdentificationOf = DefaultExposureOf, which will identify a validator with a Exposure::default(). This is suboptimal, as it forces us to still store a number of bytes. Yet, it allows any old FullIdentification, pertaining to an old slash, to be decoded. This compromise is only needed to cater for slashes that happen around the time of AHM.

westend-runtime

This runtime already has the pallet-staking-async-ah-client, integrated into all the places such that:

1. It handles the validator reward points
2. It handles offences
3. It is the SessionManager

Yet, it is delegating all of the above to its type Fallback, which is the old pallet-staking. This is a preparatory step for AHM, and should not be any logical change.

pallet-election-provider-multi-phase

This is the old single-page ElectionProvider. It has been updated to work with multi-page traits, yet it only supports page-size = 1 for now. It should not have seen any logical changes.

pallet-bags-list

Now has two new features. 1. It can be Locked, in which case all updates to it fail with an
Err(_), even deletion of a node. This is needed because we cannot alter any nodes in this
pallet during a multi-page iteration, aka. multi-page snapshot. 2. To combat this, the same
rebag transaction can be also be used to remove a node from the list, or add a node to the
list. This is done through the score_of api.

See the file changes and tests under ./substrate/frame/bags-list for more info.

RuntimeDebug -> Debug

To facilitate debugging, a number of types' RuntimeDebug impl has been changed to Debug. See
paritytech#3107

Weights

Below is a summary of the weights. These are generated using staking-async/runtimes/parachain, which assumes 22_500 nominators divided by 32 pages for Polkadot, and 12_500 nominators divided by 16 pages in Kusama, both leading to ~700 nominators snapshotted and exported per page. Doubling these parameters would easily slash the PoV weights by half, but with 10MB PoV, these numbers should be good. Also noting that with PoV clawback, we migth get even more proof_size weight back in the runtime. Although, afaik this reclaimed value does not take compression into account.

#### new: polkadot/pallet_election_provider_multi_block.rs old: kusama
+-----------------------------------------+--------------------------------------+---------+---------+-----------------+
| File                                    | Extrinsic                            | Old     | New     | Change [%]      |
+======================================================================================================================+
| pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_msp      | 2.41MiB | 2.41MiB | -0.03  |
|-----------------------------------------+--------------------------------------+---------+---------+-----------------|
| pallet_election_provider_multi_block.rs | on_initialize_into_snapshot_rest     | 3.24MiB | 3.06MiB | -5.53  |
|-----------------------------------------+--------------------------------------+---------+---------+-----------------|
| pallet_election_provider_multi_block.rs | on_initialize_into_signed            | 3.36MiB | 3.12MiB | -7.12  |
|-----------------------------------------+--------------------------------------+---------+---------+-----------------|
| pallet_election_provider_multi_block.rs | export_non_terminal                  | 2.12MiB | 1.32MiB | -37.60 |
|-----------------------------------------+--------------------------------------+---------+---------+-----------------|
| pallet_election_provider_multi_block.rs | export_terminal                      | 4.08MiB | 2.25MiB | -44.82 |
|-----------------------------------------+--------------------------------------+---------+---------+-----------------|
| pallet_election_provider_multi_block.rs | on_initialize_nothing                | 3.53KiB | 3.53KiB | Unchanged       |
|-----------------------------------------+--------------------------------------+---------+---------+-----------------|
| pallet_election_provider_multi_block.rs | on_initialize_into_unsigned          | 3.71KiB | 3.71KiB | Unchanged       |
|-----------------------------------------+--------------------------------------+---------+---------+-----------------|
| pallet_election_provider_multi_block.rs | on_initialize_into_signed_validation | 3.71KiB | 3.71KiB | Unchanged       |
|-----------------------------------------+--------------------------------------+---------+---------+-----------------|
| pallet_election_provider_multi_block.rs | manage                               | 0B      | 0B      | Unchanged       |
+-----------------------------------------+--------------------------------------+---------+---------+-----------------+
#### new: polkadot/pallet_election_provider_multi_block_signed.rs old: kusama
+------------------------------------------------+----------------------+----------+----------+-----------------+
| File                                           | Extrinsic            | Old      | New      | Change [%]      |
+===============================================================================================================+
| pallet_election_provider_multi_block_signed.rs | bail                 | 43.61KiB | 82.74KiB | +89.72 |
|------------------------------------------------+----------------------+----------+----------+-----------------|
| pallet_election_provider_multi_block_signed.rs | register_eject       | 46.54KiB | 85.80KiB | +84.35 |
|------------------------------------------------+----------------------+----------+----------+-----------------|
| pallet_election_provider_multi_block_signed.rs | clear_old_round_data | 85.23KiB | 85.17KiB | -0.06  |
|------------------------------------------------+----------------------+----------+----------+-----------------|
| pallet_election_provider_multi_block_signed.rs | submit_page          | 6.95KiB  | 6.90KiB  | -0.70  |
|------------------------------------------------+----------------------+----------+----------+-----------------|
| pallet_election_provider_multi_block_signed.rs | register_not_full    | 6.45KiB  | 6.39KiB  | -1.00  |
|------------------------------------------------+----------------------+----------+----------+-----------------|
| pallet_election_provider_multi_block_signed.rs | unset_page           | 20.76KiB | 18.55KiB | -10.67 |
+------------------------------------------------+----------------------+----------+----------+-----------------+
#### new: polkadot/pallet_election_provider_multi_block_unsigned.rs old: kusama
+--------------------------------------------------+-------------------+----------+-----------+------------------+
| File                                             | Extrinsic         | Old      | New       | Change [%]       |
+================================================================================================================+
| pallet_election_provider_multi_block_unsigned.rs | submit_unsigned   | 63.56KiB | 696.00KiB | +995.01 |
|--------------------------------------------------+-------------------+----------+-----------+------------------|
| pallet_election_provider_multi_block_unsigned.rs | validate_unsigned | 1.81KiB  | 3.66KiB   | +102.65 |
+--------------------------------------------------+-------------------+----------+-----------+------------------+
#### new: polkadot/pallet_election_provider_multi_block_verifier.rs old: kusama
+--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+
| File                                             | Extrinsic                          | Old       | New       | Change [%]      |
+=================================================================================================================================+
| pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_terminal     | 1.18MiB   | 1.69MiB   | +42.87 |
|--------------------------------------------------+------------------------------------+-----------+-----------+-----------------|
| pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_terminal       | 1.18MiB   | 1.69MiB   | +42.71 |
|--------------------------------------------------+------------------------------------+-----------+-----------+-----------------|
| pallet_election_provider_multi_block_verifier.rs | on_initialize_invalid_non_terminal | 1.30MiB   | 450.82KiB | -66.08 |
|--------------------------------------------------+------------------------------------+-----------+-----------+-----------------|
| pallet_election_provider_multi_block_verifier.rs | on_initialize_valid_non_terminal   | 279.93KiB | 62.22KiB  | -77.77 |
+--------------------------------------------------+------------------------------------+-----------+-----------+-----------------+

note for PR authors

Details

TODO

• Finalize weights
• Lock voter list when snapshot being taken
• push based election
• OffchainWorker miner can now run on multiple pages
• Trimming is improved, all bounds are respected.
• clients pallets: add ID
• make election prolonged
• bring westend-next and ah-next to staking-next
• Test pre-migration to post-migration state in ahm-test.
• Offence reporting works without exposure info on RC (done but recheck).
• staking-async fix tests
• root offence testing (minimally done in migration test)
• Run benchmarking
• Add custom decoder for OffenceDetails.

TODO before finalizing PR

• Go over again and ensure no interaction with staking-classic except by AhClient (and pallets that are going away) in Westend. Make any non used apis private.
• Create diff with changes from staking-classic.

Migration Notes

• At the start of the AHM migration, trigger: RC::pallet_staking_async_ah_client::on_migration_start()
• At the start of the AHM migration, trigger the following:
  • definitely filter staking::bond
  • RC: set staking::Forcing to ForceNone.
• At the end of the AHM migration, trigger the following
  • RC::pallet_staking_async_ah_client::on_migration_end()
  • Set AH::pallet_staking_async::ForceEra to Forcing::NotForcing.
  • Set RC staking and pool min bond to be u32::max.

Follow-up

• Offence generation e2e test (zombienet)

---

✄ -----------------------------------------------------------------------------

Thank you for your Pull Request! 🙏 Please make sure it follows the contribution guidelines outlined in this
document and fill out the
sections below. Once you're ready to submit your PR for review, please delete this section and leave only the text under
the "Description" heading.

Description

A concise description of what your PR is doing, and what potential issue it is solving. Use Github semantic
linking
to link the PR to an issue that must be closed once this is merged.

Integration

In depth notes about how this PR should be integrated by downstream projects. This part is mandatory, and should be
reviewed by reviewers, if the PR does NOT have the R0-Silent label. In case of a R0-Silent, it can be ignored.

Review Notes

In depth notes about the implementation details of your PR. This should be the main guide for reviewers to
understand your approach and effectively review it. If too long, use
<details>.

Imagine that someone who is depending on the old code wants to integrate your new code and the only information that
they get is this section. It helps to include example usage and default value here, with a diff code-block to show
possibly integration.

Include your leftover TODOs, if any, here.

Checklist

• My PR includes a detailed description as outlined in the "Description" and its two subsections above.
• My PR follows the labeling requirements of this project (at minimum one label for T required)
  • External contributors: ask maintainers to put the right label on your PR.
• I have made corresponding changes to the documentation (if applicable)
• I have added tests that prove my fix is effective or that my feature works (if applicable)

You can remove the "Checklist" section once all have been checked. Thank you for your contribution!

✄ -----------------------------------------------------------------------------

[github-actions]

Found 2 logic issues and 1 documentation issue in pallet-bags-list and election-provider-support.

---

Suggestions that couldn't be attached to a specific line

substrate/frame/bags-list/src/lib.rs:314, 325-326, 330

There's an inconsistency in error reporting for the locked state. The direct check in rebag (line 314) results in Error::Locked. However, when rebag calls on_insert or on_remove (lines 325, 330), a lock error from ensure_unlocked is converted to Error::List(ListError::Locked). This means the same condition produces two different errors. The error handling should be made consistent to always return Error::Locked when the pallet is locked.

---

substrate/frame/election-provider-support/src/lib.rs:204

The ElectionDataProvider trait now includes a generic parameter T: Config = (). This is an unusual pattern that makes the trait's signature confusing, especially since existing implementations do not use it. The purpose of this parameter should be clearly documented with an example, or it should be removed to simplify the trait.

[github-actions]

The documentation for the Lock storage item at lines 269-270 states that Call::rebag can be used when the pallet is locked. However, the implementation of rebag at line 314 explicitly checks if the pallet is unlocked. This contradiction can mislead developers. The documentation should be corrected to state that rebag is also locked.