Add democracy migration and fix contracts migration #12

apopiak · 2020-06-29T16:07:36Z

This PR migrates democracy for the Edgeware node migration.

Includes migration code inspired by:

Other notes:

Fixes the block hash migration originally from Refactor away from opaque hashes paritytech/substrate#5226 introduced by Some migrations necessary for Edgeware Node Upgrade #6 to actually do a correct migration.
Fixes and reintroduces the contracts migration that was commented out.

Related to hicommonwealth/edgeware-node#164

drewstone

Looks good!

apopiak · 2020-06-30T10:36:06Z

Getting "Corrupted State" errors:

2020-06-30 12:34:37.861 tokio-runtime-worker INFO substrate  ✨ Imported #19 (0xddd4…57f6)
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  No BlockHashes to migrate...
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  🕊️  Migrating Account Balances...
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  Balances: FreeBalance -> Account
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  Balances: ReservedBalance -> Account
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  Balances: Vesting
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  Balances: Balance::Account -> System::Account
2020-06-30 12:34:38.834 event.loop0 DEBUG runtime  🕊️  Done Account Balances.
2020-06-30 12:34:38.834 event.loop0 DEBUG runtime  🕊️  Migrating Accounts...
2020-06-30 12:34:38.834 event.loop0 DEBUG runtime  2
2020-06-30 12:34:38.834 event.loop0 DEBUG runtime  🕊️  Done Accounts.
2020-06-30 12:34:38.834 event.loop0 DEBUG runtime  🕊️  Migrating Identity.
2020-06-30 12:34:38.834 event.loop0 DEBUG runtime  🕊️  Done Identity.
2020-06-30 12:34:38.836 event.loop0 DEBUG runtime  🕊️  Migrating Finality Tracker.
2020-06-30 12:34:38.836 event.loop0 DEBUG runtime  🕊️  Done Finality Tracker.
2020-06-30 12:34:38.837 event.loop0 DEBUG runtime  ERROR: Corrupted state at [242, 121, 76, 34, 227, 83, 233, 168, 57, 241, 47, 170, 176, 58, 145, 27, 246, 137, 103, 214, 53, 100, 26, 112, 135, 229, 63, 43, 255, 30, 202, 211, 28, 127, 16, 167, 25, 164, 98, 144, 41, 73, 152, 162, 173, 105, 53, 30, 231, 158, 214, 44, 82, 207, 80, 158, 209, 13, 229, 26, 159, 65, 108, 132]
2020-06-30 12:34:38.837 event.loop0 DEBUG runtime  🕊️  Migrating Transaction Payment.
2020-06-30 12:34:38.837 event.loop0 DEBUG runtime  🕊️  Done Transaction Payment.
2020-06-30 12:34:41.111 tokio-runtime-worker INFO substrate  💤 Idle (0 peers), best: #19 (0xddd4…57f6), finalized #17 (0x956a…c384), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
2020-06-30 12:34:42.000 tokio-runtime-worker DEBUG runtime  No BlockHashes to migrate...
2020-06-30 12:34:42.000 tokio-runtime-worker DEBUG runtime  🕊️  Migrating Account Balances...
2020-06-30 12:34:42.000 tokio-runtime-worker DEBUG runtime  Balances: FreeBalance -> Account
2020-06-30 12:34:42.000 tokio-runtime-worker DEBUG runtime  Balances: ReservedBalance -> Account
2020-06-30 12:34:42.000 tokio-runtime-worker DEBUG runtime  Balances: Vesting
2020-06-30 12:34:42.000 tokio-runtime-worker DEBUG runtime  Balances: Balance::Account -> System::Account
2020-06-30 12:34:42.096 tokio-runtime-worker DEBUG runtime  🕊️  Done Account Balances.
2020-06-30 12:34:42.096 tokio-runtime-worker DEBUG runtime  🕊️  Migrating Accounts...
2020-06-30 12:34:42.096 tokio-runtime-worker DEBUG runtime  2
2020-06-30 12:34:42.096 tokio-runtime-worker DEBUG runtime  🕊️  Done Accounts.
2020-06-30 12:34:42.096 tokio-runtime-worker DEBUG runtime  🕊️  Migrating Identity.
2020-06-30 12:34:42.096 tokio-runtime-worker DEBUG runtime  🕊️  Done Identity.
2020-06-30 12:34:42.097 tokio-runtime-worker DEBUG runtime  🕊️  Migrating Finality Tracker.
2020-06-30 12:34:42.097 tokio-runtime-worker DEBUG runtime  🕊️  Done Finality Tracker.
2020-06-30 12:34:42.097 tokio-runtime-worker DEBUG runtime  ERROR: Corrupted state at [242, 121, 76, 34, 227, 83, 233, 168, 57, 241, 47, 170, 176, 58, 145, 27, 246, 137, 103, 214, 53, 100, 26, 112, 135, 229, 63, 43, 255, 30, 202, 211, 28, 127, 16, 167, 25, 164, 98, 144, 41, 73, 152, 162, 173, 105, 53, 30, 231, 158, 214, 44, 82, 207, 80, 158, 209, 13, 229, 26, 159, 65, 108, 132]
2020-06-30 12:34:42.098 tokio-runtime-worker DEBUG runtime  🕊️  Migrating Transaction Payment.
2020-06-30 12:34:42.098 tokio-runtime-worker DEBUG runtime  🕊️  Done Transaction Payment.
2020-06-30 12:34:42.111 tokio-runtime-worker INFO sc_basic_authorship::basic_authorship  🙌 Starting consensus session on top of parent 0xddd49cbd2f04ff98c23029e3a372788a2258a2173f9d0957228acc31d8b857f6
2020-06-30 12:34:42.111 tokio-blocking-driver DEBUG runtime  BlockNumber: 0
2020-06-30 12:34:42.111 tokio-blocking-driver DEBUG runtime  🕊️  No BlockHashes to migrate...
2020-06-30 12:34:42.111 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Account Balances...
2020-06-30 12:34:42.111 tokio-blocking-driver DEBUG runtime  Balances: FreeBalance -> Account
2020-06-30 12:34:42.111 tokio-blocking-driver DEBUG runtime  Balances: ReservedBalance -> Account
2020-06-30 12:34:42.111 tokio-blocking-driver DEBUG runtime  Balances: Vesting
2020-06-30 12:34:42.111 tokio-blocking-driver DEBUG runtime  Balances: Balance::Account -> System::Account
2020-06-30 12:34:42.679 tokio-blocking-driver DEBUG runtime  🕊️  Done Account Balances.
2020-06-30 12:34:42.679 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Staking...
2020-06-30 12:34:42.680 tokio-blocking-driver DEBUG runtime  🕊️  Done Staking.
2020-06-30 12:34:42.680 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Accounts...
2020-06-30 12:34:42.681 tokio-blocking-driver DEBUG runtime  2
2020-06-30 12:34:42.681 tokio-blocking-driver DEBUG runtime  🕊️  Done Accounts.
2020-06-30 12:34:42.681 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Identity.
2020-06-30 12:34:42.681 tokio-blocking-driver DEBUG runtime  🕊️  Done Identity.
2020-06-30 12:34:42.685 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Finality Tracker.
2020-06-30 12:34:42.686 tokio-blocking-driver DEBUG runtime  🕊️  Done Finality Tracker.
2020-06-30 12:34:42.686 tokio-blocking-driver DEBUG runtime  🕊️  Migrating PhragmenElection...
2020-06-30 12:34:42.686 tokio-blocking-driver DEBUG runtime  🕊️  Done PhragmenElection.
2020-06-30 12:34:42.686 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Democracy...
2020-06-30 12:34:42.686 tokio-blocking-driver DEBUG runtime  Democracy: Hasher
2020-06-30 12:34:42.686 tokio-blocking-driver DEBUG runtime  Democracy: Hasher: ReferendumInfo
2020-06-30 12:34:42.686 tokio-blocking-driver DEBUG runtime  Democracy: Hasher: PublicProps
2020-06-30 12:34:42.687 tokio-blocking-driver DEBUG runtime  Preimages key: [242, 121, 76, 34, 227, 83, 233, 168, 57, 241, 47, 170, 176, 58, 145, 27, 246, 137, 103, 214, 53, 100, 26, 112, 135, 229, 63, 43, 255, 30, 202, 211, 28, 127, 16, 167, 25, 164, 98, 144, 41, 73, 152, 162, 173, 105, 53, 30, 231, 158, 214, 44, 82, 207, 80, 158, 209, 13, 229, 26, 159, 65, 108, 132]
2020-06-30 12:34:42.687 tokio-blocking-driver DEBUG runtime  Democracy: Remove Unused
2020-06-30 12:34:42.690 tokio-blocking-driver DEBUG runtime  Democracy: ReferendumInfo
2020-06-30 12:34:42.690 tokio-blocking-driver DEBUG runtime  🕊️  Done Democracy.
2020-06-30 12:34:42.690 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Transaction Payment.
2020-06-30 12:34:42.691 tokio-blocking-driver DEBUG runtime  🕊️  Done Transaction Payment.
2020-06-30 12:34:42.698 tokio-blocking-driver DEBUG runtime  No BlockHashes to migrate...
2020-06-30 12:34:42.698 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Account Balances...
2020-06-30 12:34:42.698 tokio-blocking-driver DEBUG runtime  Balances: FreeBalance -> Account
2020-06-30 12:34:42.698 tokio-blocking-driver DEBUG runtime  Balances: ReservedBalance -> Account
2020-06-30 12:34:42.698 tokio-blocking-driver DEBUG runtime  Balances: Vesting
2020-06-30 12:34:42.698 tokio-blocking-driver DEBUG runtime  Balances: Balance::Account -> System::Account
2020-06-30 12:34:42.813 tokio-blocking-driver DEBUG runtime  🕊️  Done Account Balances.
2020-06-30 12:34:42.813 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Accounts...
2020-06-30 12:34:42.813 tokio-blocking-driver DEBUG runtime  2
2020-06-30 12:34:42.813 tokio-blocking-driver DEBUG runtime  🕊️  Done Accounts.
2020-06-30 12:34:42.813 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Identity.
2020-06-30 12:34:42.813 tokio-blocking-driver DEBUG runtime  🕊️  Done Identity.
2020-06-30 12:34:42.815 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Finality Tracker.
2020-06-30 12:34:42.815 tokio-blocking-driver DEBUG runtime  🕊️  Done Finality Tracker.
2020-06-30 12:34:42.817 tokio-blocking-driver DEBUG runtime  ERROR: Corrupted state at [242, 121, 76, 34, 227, 83, 233, 168, 57, 241, 47, 170, 176, 58, 145, 27, 246, 137, 103, 214, 53, 100, 26, 112, 135, 229, 63, 43, 255, 30, 202, 211, 28, 127, 16, 167, 25, 164, 98, 144, 41, 73, 152, 162, 173, 105, 53, 30, 231, 158, 214, 44, 82, 207, 80, 158, 209, 13, 229, 26, 159, 65, 108, 132]
2020-06-30 12:34:42.817 tokio-blocking-driver DEBUG runtime  🕊️  Migrating Transaction Payment.
2020-06-30 12:34:42.817 tokio-blocking-driver DEBUG runtime  🕊️  Done Transaction Payment.
2020-06-30 12:34:42.880 tokio-blocking-driver INFO sc_basic_authorship::basic_authorship  🎁 Prepared block for proposing at 20 [hash: 0xd9eb57ef00dbee8148842227b1ea9d5123fdd9000b5217d90bea8e96809abb81; parent_hash: 0xddd4…57f6; extrinsics (2): [0xcd6b…cacf, 0x07bd…d143]]

apopiak · 2020-07-01T10:52:11Z

Getting "Corrupted State" errors:

2020-06-30 12:34:37.861 tokio-runtime-worker INFO substrate  ✨ Imported #19 (0xddd4…57f6)
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  No BlockHashes to migrate...
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  🕊️  Migrating Account Balances...
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  Balances: FreeBalance -> Account
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  Balances: ReservedBalance -> Account
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  Balances: Vesting
2020-06-30 12:34:38.555 event.loop0 DEBUG runtime  Balances: Balance::Account -> System::Account
[...]

Solved through rebuilds 🤷

drewstone

Looks good, if every migration is using MaxBlockWeight, does it sum those? Wouldn't that make the weight too larger to go in, in a single tx?

drewstone · 2020-07-01T20:03:37Z

frame/babe/src/lib.rs

 		fn on_runtime_upgrade() -> Weight {
 			for i in 0..=SegmentIndex::get() {
 				UnderConstruction::migrate_key_from_blake(i);
 			}
-			// TODO: determine weight
-			0
+			T::MaximumBlockWeight::get()


We don't use babe so not sure how necessary this one is?

True, probably isn't. Don't think it's worth it to remove it, though.

drewstone · 2020-07-01T20:06:40Z

frame/democracy/src/migration.rs

+}
+
+pub fn migrate_hasher<T: Trait>() -> Weight {
+	// TODO: is this valid?


So far, we haven't had an vetos/cancellations. Should be fine!

drewstone · 2020-07-01T20:07:09Z

frame/democracy/src/migration.rs

+}
+
+pub fn migrate_remove_unused_storage<T: Trait>() -> Weight {
+	// TODO: is this valid?


So we delete things after we move them?

I'm not sure where VotersFor is defined or if we have it on our substrate commit that mainnet is pegged to. What are the potential reprecussions of removing these?

VotersFor seems empty on current edgeware. (Checked on https://polkadot.js.org/apps/#/chainstate with some indices. referendumCount is 8.)

So we delete things after we move them?

Nope, the ones deleted here are not moved.

The repercussions I can see are the following: This migration just drops some state that currently doesn't exist on edgeware. If people were to create proposals and vote on them while this migration is in process, some data might be lost. AFAICT this would not corrupt any state, people would just have to vote again.

apopiak · 2020-07-02T09:56:49Z

Looks good, if every migration is using MaxBlockWeight, does it sum those? Wouldn't that make the weight too larger to go in, in a single tx?

These upgrades are performed by on_runtime_upgrade and thus are not subject to the same tx limits. It does result in the block being very heavy and thus not any other (non-mandatory) transactions happening during the migration block. But that's what we want as the migration will be quite heavy with all the account migration stuff.

apopiak · 2020-07-02T09:58:15Z

So returning T::MaximumBlockWeight in on_runtime_upgrade (or on_initialize) fills the block and keeps regular transactions out.

apopiak added 7 commits June 29, 2020 12:41

log current blocknumber during migration

75f9fcf

improve phragmen migration logs and docs

e4c0323

add combined democracy migration

2b9d1a1

re-add ReferendumInfo hasher migration and fix compile issues

acc62be

add more logging for democracy migration

f851407

add migration from democracy weights PR paritytech#5828

78e23ca

log key used for PublicProps

6f57659

github-actions bot added the A0-pleasereview label Jun 29, 2020

apopiak mentioned this pull request Jun 29, 2020

Add democracy migration hicommonwealth/edgeware-node#175

Merged

apopiak requested a review from drewstone June 29, 2020 16:09

drewstone approved these changes Jun 29, 2020

View reviewed changes

apopiak added 2 commits June 30, 2020 12:03

migrate preimages

3b5b074

correct logging and rename variables

bd4e44e

apopiak added 4 commits June 30, 2020 15:47

add block number for block hash migration

ff78c84

log decode errors

2b70362

add decode error log label

01da290

remove extra logging

141c11a

apopiak changed the title ~~[WIP] Add democracy migration~~ Add democracy migration Jul 1, 2020

apopiak added 3 commits July 1, 2020 17:19

re-add contracts migration

a98f24e

remove weight todos and just assume MaximumBlockWeight

2f67ad3

fix compile errors

0144dc4

drewstone approved these changes Jul 1, 2020

View reviewed changes

replace some todos with comments

332b668

log decode error

05795c1

apopiak changed the title ~~Add democracy migration~~ Add democracy migration and fix contrac migration Jul 2, 2020

apopiak changed the title ~~Add democracy migration and fix contrac migration~~ Add democracy migration and fix contracts migration Jul 2, 2020

fix contracts migration

d3c5b72

apopiak mentioned this pull request Jul 2, 2020

Runtime migrations tracking issue hicommonwealth/edgeware-node#164

Closed

20 tasks

remove grandpa authority migration

0238e7c

drewstone merged commit 0ad4f35 into time-travel Jul 2, 2020

drewstone mentioned this pull request Jul 20, 2020

Upgrade testing tracking issue hicommonwealth/edgeware-node#186

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add democracy migration and fix contracts migration #12

Add democracy migration and fix contracts migration #12

apopiak commented Jun 29, 2020 •

edited

Loading

drewstone left a comment

apopiak commented Jun 30, 2020

apopiak commented Jul 1, 2020

drewstone left a comment

drewstone Jul 1, 2020

apopiak Jul 2, 2020

drewstone Jul 1, 2020

drewstone Jul 1, 2020

drewstone Jul 1, 2020

apopiak Jul 2, 2020

apopiak Jul 2, 2020

apopiak Jul 2, 2020

apopiak commented Jul 2, 2020

apopiak commented Jul 2, 2020

Add democracy migration and fix contracts migration #12

Add democracy migration and fix contracts migration #12

Conversation

apopiak commented Jun 29, 2020 • edited Loading

drewstone left a comment

Choose a reason for hiding this comment

apopiak commented Jun 30, 2020

apopiak commented Jul 1, 2020

drewstone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apopiak commented Jul 2, 2020

apopiak commented Jul 2, 2020

apopiak commented Jun 29, 2020 •

edited

Loading