FRAME: Register on_initialize after each pallet

[bkchr]

Before this pull request, FRAME was executing all pallets on_initialize and then register the weight, including the weight of on_runtime_upgrade. Thus, other pallets were not aware on how much weight was already used when they were executing their on_initialize code. As some pallets are doing some work in on_initialize, they need to be aware of how much weight is still left.
To register the weight after each on_initialize call, a new trait is added. This new trait is implemented for tuples of types that implement OnInitialize and then it registers the weight after each call to on_initialize.

pallet-scheduler is changed to take the remaining weight into account and to not just assume that its configured weight is always available.

[kianenigma]

A few comments/questions:

• what is the motivation behind this?
• This API is overall much better, and in-line with on_poll, which was added later and has access to an ongoing weight meter.
• That being said, I think the ultimate solution is that all parachains (incl. AH) should use on_poll, including our own pallets (minus the scheduler perhaps)

Part of our worries in #9619 also related to this. On AH, a single block might might:

• Have a lot of inherents
• scheduler on-onit
• message-queue on-init
• staking election on-init

And all of this is Mandatory, and can cause the chain to stall. Our solution for this was to move the staking part to on_poll, as it can safely happen with gaps, if weight is not available.

[sigurpol]

A few comments/questions:

* That being said, I think the ultimate solution is that all parachains (incl. AH) should use `on_poll`, including our own pallets (minus the scheduler perhaps)

Part of our worries in #9619 also related to this. On AH, a single block might might:

* Have a lot of inherents

* scheduler on-onit

* message-queue on-init

* staking election on-init

And all of this is Mandatory, and can cause the chain to stall. Our solution for this was to move the staking part to on_poll, as it can safely happen with gaps, if weight is not available.

from the discussion we had on Element, my take-aways are the following:

• on_poll is indeed still the way to go for us in staking-async (and not only for us)
• the current PR should help / make easier to improve stuff in the short term until we move what we can to on_poll

[bkchr]

• what is the motivation behind this?

The 500ms blocks, where I'm afraid someone will do something bad :) And overall I think it is just better to have there a correct weight tracking. For sure it is not perfect, but still better than the status quo.

• That being said, I think the ultimate solution is that all parachains (incl. AH) should use on_poll, including our own pallets (minus the scheduler perhaps)

Yes on_poll is the future. I'm totally in line with this. Scheduler should actually go to tasks, because this is actually much better for the use case.

• Have a lot of inherents

• scheduler on-onit

• message-queue on-init

• staking election on-init

Inherents, scheduler on-init, message-queue etc all should honor the weight much better and should be more tightly controlled when it comes to this.

[gui1117]

I think the trait doesn't need the generic BlockNumber otherwise looks good to me

[gui1117]

Fortunately we define all pallets as a single tuple like: type AllPallet = (System, Balances, Assets); and not type AllPallet = (System, (Balances, Assets));. Otherwise this will not help.

But we have to be careful not to change this, maybe we can have a test specifically written for this.

(We could also generate OnInitializeWithWeightRegistration with the macro pallet, and make this trait implemented for tuple of OnInitializeWithWeightRegistration.)

[bkchr]

Fortunately we define all pallets as a single tuple like: type AllPallet = (System, Balances, Assets); and not type AllPallet = (System, (Balances, Assets));. Otherwise this will not help.

Yeah I know. I also checked this.

But we have to be careful not to change this, maybe we can have a test specifically written for this.

I first thought about writing a integrity_test, but this is right now not supported by Executive.

[bkchr]

In the worst case we have some pallets that don't register the weight immediately.

[gui1117]

I first thought about writing a integrity_test, but this is right now not supported by Executive.

yes, I was a thinking a test that ensure construct_runtime defines AllPallets as we expect, with 2 defined pallet that ensure the weight is updated inside each on_initialize. The test you wrote in scheduler already tests for it also. So in all case it is good to me

[kianenigma]

Would you be willing to backport this to 2507 and 2509? Not sure if it has added risk for us.

If yes, we can use this in EPMB and no migrate it to poll yet. The change would be simpler there if so.

[bkchr]

@kianenigma if you approve, I can backport it. :P

[paritytech-workflow-stopper]

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/17856405890
Failed job name: cargo-clippy

[bkchr]

/cmd prdoc --audience runtime_dev --bump patch

[sigurpol]

@bkchr , @kianenigma, @ggwpez : how confident are we in backporting to 2509 for the early October deployment on Westend? Additionally, how do we feel about using 2507 in the post-KAHM and pre- or post-PAHM phases with EPMB using it?

[kianenigma]

This seems pretty innocent to me, I think we can safely backport.

But lemme see if it is will be useful for EPMB, will let you know. If I can migrate to poll already, it will be easier and done once and for all.

[kianenigma]

(this PR is basically making on-init able to 👺 as poll)

[bkchr]

I'm also fine with backporting. It doesn't change that much.

[paritytech-release-backport-bot]

Created backport PR for unstable2507:

• [unstable2507] Backport #9756 #9824 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-9756-to-unstable2507
git worktree add --checkout .worktree/backport-9756-to-unstable2507 backport-9756-to-unstable2507
cd .worktree/backport-9756-to-unstable2507
git reset --hard HEAD^
git cherry-pick -x 19320f104fc4b5cb6663cffc41f340b6b5239be8
git push --force-with-lease

[paritytech-release-backport-bot]

Created backport PR for stable2509:

• [stable2509] Backport #9756 #9825 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-9756-to-stable2509
git worktree add --checkout .worktree/backport-9756-to-stable2509 backport-9756-to-stable2509
cd .worktree/backport-9756-to-stable2509
git reset --hard HEAD^
git cherry-pick -x 19320f104fc4b5cb6663cffc41f340b6b5239be8
git push --force-with-lease

[bkchr]

@sigurpol the backports are prepared.