Allow pallet in construct_runtime to have fixed index

[gui1117]

Fix #6855
companion: paritytech/polkadot#1692

BREAKING CHANGE:

the encoding of Event, Call and Origin is changed:
Each variant of these enums are now encoded at the index of the pallet, the index of the pallet is defined similarly to rust fieldless
enum:

construct_runtime!(
...
module1: .. { .. }, // index is 0
module2: .. { .. } = 3, // index is 3
module3: .. { .. }, // index is 4
module4: .. { .. } = 1, // index is 1
module5: .. { .. }, // index is 2
)

This breaking change result in the bump of metadata to version 12 which now provides for each module its index.

Note: Origin system variant is also encoded using this index, contrary to previously where it was fixed at index 0.

Note: to help migration, user should make use of indices in order not to break Call, and make use of scheduler::migrate_origin to update the usage of OriginCaller.

Context

This PR makes pallet having a fixed index. This avoid breaking call, event or origin when pallet definition changes. And ease adding/removing pallets.
So each pallet is now defined a fixed index either implicitly or using = $index syntax similarly to rust fieldless enum.

Syntax:

(syntax is as followed as suggested by Gav but I can do as suggested by Basti in issue, it's only a matter of 10 lines of code)

construct_runtime!(
...
{
   System: system::{Call, Event, Origin}, // index is 0
   Pallet1: pallet1::{Event, Origin} = 20,
   Pallet2: pallet2::{Call, Origin}, // index is 21
   Pallet3: pallet3::{Event, Origin}, // index is 22
   Pallet4: pallet4::{} = 10,
   Pallet5: pallet5::{Call, Origin}, index is 11
}
);

The index must be a integer literal strictly less than 256, it is not allowed to be an expression as parity-scale-codec doesn't allow expression.

Metadata change

So the metadata has now this new information with a new field in module_metadata: index: u8. this index must be used when decoding call variant for corresponding module.

cc @jacogr

TODO

• polkadot/kusama migration for OriginCaller in scheduler: Handle substrate construct_runtime change: break runtime Event encoding, change some pallets error index polkadot#1692

[jacogr]

Just make sure to bump the metadata version to v12 (have not looked yet if it is done), without it there is going to be absolute chaos in the ecosystem, JS, Python, ...

... EDIT: And it is bumped, so yes, can be done by all.

[jacogr]

cc @emielvanderhoek cc @arjanz

[bkchr]

I like the = Something syntax for setting the index! :)

[shawntabrizi]

I have just seen this PR and looked over it very quickly, but I think it would be a fair requirement that to use the = index syntax, we should require the user specifies the index for all pallets, rather than letting logic determine the magic number selected for each one.

I think it would avoid future footguns, and adds little overhead. So either they all have it, or none have it.

[gui1117]

ok it seems we agree on breaking call and origin (and event but it is not important), then we can just force = index; everywhere.

At the same time we can change encoding of origin: origin: OriginCaller::System variant will have the given index.

(for me the only reason to keep this implicit syntax was to not have a breaking change for the encoding of Call and Origin)

[shawntabrizi]

@thiolliere I would also include a breaking change that if a user does not include = index format, that the enums generated are always equal length, even if the pallet does not include Call or Event or whatever... prevent future breaking things.

[apopiak]

@thiolliere I would also include a breaking change that if a user does not include = index format, that the enums generated are always equal length, even if the pallet does not include Call or Event or whatever... prevent future breaking things.

+1 for having consistent indices over Call, Event, etc.

[gui1117]

So I updated PR description:

• I allowed implicit index, because it can be handy for tests.
• all event/call/origin are now using this index, so it is consisten and future proof

[apopiak]

LGTM, some language nitpicks

[shawntabrizi]

looks good to me (only very high level review of macro changes)

[kianenigma]

given assumption that System is always there, maybe you can maybe simplify a thing or two by fn get_system() -> Option<&ModulePart> or fn is_system() -> bool but just an idea.

[gui1117]

indeed, for now system is only queried one time and filtered out only one time too, but if I would refactor such functions looks more readable.

[kianenigma]

Brief view looks good!

[gui1117]

bot merge

[ghost]

Waiting for commit status.

[apopiak]

Does not include a runtime migration itself, but runtimes including this will need to migrate.