`follow-chain` testing mode for try-runtime (and revamp CLI configs). #9788

kianenigma · 2021-09-15T13:24:55Z

closes #9027
closes #9570
somewhat related #2082

And does a major cleanup and revamp of the try-runtime cli to hopefully make it more easy to use.

polkadot companion: paritytech/polkadot#3858

And, finally, it adds a new follow-chain command to try-runtime. This will copy a live chain's state, dump it into a TestExternalities as usual, and keep applying the extrinsics of a live chain as new block come in using the current runtime. This means running this with wss://rpc.polkadot.io in substrate is nonsense since it will try and decode + execute polkadot blocks in substrate-node.

@sacha-l some arguments change in this PR. Can you make an issue in the devhub to update the docs? it should be fairly simply to update, if any.

Using this is still tricky, there are a few considerations:

Let's say you are on top of branch master of polkadot. If the block decoding has changed in any way since the live node from which you are pulling things, nothing will work.
Block decoding will work, but then you have to manually tweak your transaction/spec version to match the one on the live node, since they are part of the signature verification. I am looking into ways to actually disable signature verification in this try-runtime entirely, to make this part a bit easier.

An alternative flow to avoid the previous caveats is that you would backport your feature branch to the branch of the latest release, ensuring that almost certainly the above two variables have not changed (unless if your feature branch is literally changing the block format etc.)

add a ton more docs
allow live::at to be omitted, and replaced with the latest finalized block.
prevent execution on wrong chain.
~~mock signature verification~~. UPDATE: can mock the signature verification, but then this messes with the signed extension executions. The situation was that I would get a ExhaustResource error this time, if the spec-versions did not match. I might revisit this in the future. For future references, this is the commit that removes mocking signature verification: 7d60da9 (#9788)
clean some of the unwraps.

A different, more cumbersome, but more realistic way of doing this is:

Use fork-of-substrate and create a new chain. Use a custom runtime in there that is your feature branch + sudo or some other pallet that allows us to execute arbitrary transactions. Then run this chain in the same was as fork-of-substrate, and run binary next to it that scans a live chain, and forwards the new transactions to the "backdoor" (e.g. sudo) of the forked chain.

This is obviously too much work. Maybe the only worthwhile way of doing this is if we establish RPC connections in the fn offchain_worker, and make it all seamless. So, you'd have a feature branch. You add this new magic-testing-pallet to it, and run it, and it would mirror a real chain for you.

The current scheme also allows for rather easy scripting and testing, other than being simpler.

niklasad1 · 2021-09-16T09:21:56Z

Block decoding will work, but then you have to manually tweak your transaction/spec version to match the one on the live node, since they are part of the signature verification. I am looking into ways to actually disable signature verification in this try-runtime entirely, to make this part a bit easier.

Can't you subscribe/poll runtime_version on the remote node to detect if transaction/spec version changes?

kianenigma · 2021-09-16T10:30:16Z

Can't you subscribe/poll runtime_version on the remote node to detect if transaction/spec version changes?

sure, but what would you do about it other than stopping? that's practically equal to what I do now, which is decode.unwrap() :D

niklasad1 · 2021-09-16T11:50:54Z

yeah right, dumb question I guess it's not much to do without updating.....

frame/executive/src/lib.rs

utils/frame/try-runtime/cli/src/commands/follow_chain.rs

utils/frame/try-runtime/cli/src/lib.rs

frame/executive/src/lib.rs

utils/frame/try-runtime/cli/src/lib.rs

shawntabrizi · 2021-09-21T06:13:34Z

frame/executive/src/lib.rs

+		// don't call `final_checks`, but do finalize the block. This is critical to clear transient
+		// storage items, such as weight.
+		let _header = <frame_system::Pallet<System>>::finalize();
+


Would wonder if there is some migration that would pass all the tests written, but then actually cause a problem on final checks, but we dont see that now

unlikely, but yeah that's just a pill that we have to swallow here. Final checks does:

header digest check (don't know what it does -- not is it relevant IMO)

state root check

extrinsic root check

This actually makes me think: maybe I can re-introduce item 1 and 3, since they should work. Only the state root is guaranteed to fail, since :CODE: has changed.

utils/frame/try-runtime/cli/src/commands/on_runtime_upgrade.rs

shawntabrizi · 2021-09-21T06:27:40Z

utils/frame/try-runtime/cli/src/lib.rs

-use remote_externalities::{rpc_api, Builder, Mode, OfflineConfig, OnlineConfig, SnapshotConfig};
+//! # Try-runtime
+//!
+//! Substrate's ultimate testing framework for the power users.


kian pats himself on the back

shamelessly advertising 😅

shawntabrizi

Looks reasonable at a high level. Can scrub deeper if needed

kianenigma · 2021-09-21T15:04:15Z

bot merge

ghost · 2021-09-21T15:04:20Z

Waiting for commit status.

* companion for paritytech/substrate#9788 * fmt * Some fixes * final fixes * fix a few small things * update Substrate Co-authored-by: parity-processbot <>

…#9788) * deadlock, need to ask someone to help now * Finally it seems to be working.. at least for a few blocks * self-review * major mega revamp * some small fixes * another mega refactor * add license * Apply suggestions from code review * hack around signature verification * Some fixes * Update utils/frame/try-runtime/cli/src/lib.rs Co-authored-by: Zeke Mostov <[email protected]> * Update utils/frame/try-runtime/cli/src/lib.rs Co-authored-by: Zeke Mostov <[email protected]> * Update utils/frame/try-runtime/cli/src/lib.rs Co-authored-by: Zeke Mostov <[email protected]> * final tweaks, hopefully. * a little self-review * Add the ext root check Co-authored-by: Zeke Mostov <[email protected]>

* companion for paritytech/substrate#9788 * fmt * Some fixes * final fixes * fix a few small things * update Substrate Co-authored-by: parity-processbot <>

@kianenigma

# Membership Request Hi, I am Kian Paimani, known as @kianenigma. I have been working on Polkadot/Kusama through Parity since February 2019 and I can categorize my main contributions to Polkadot's ecosystem as follows: 1. Maintaining and developing the staking sub-system. 2. General FRAME development, especially testing and quality assurance. 3. Polkadot-native side-projects. 4. Education > My first contribution to Polkadot is also indeed related to staking: paritytech/substrate#1915 ### Staking system I joke as the Polkadot staking to be both my blessing and my curse over the years. I started working on it since the first days that I joined this ecosystem and the work [is ongoing ever since](https://github.com/orgs/paritytech/projects/33/views/9). In the past, I focused on making sure that the staking system is secure and to some extent scalable. More recently, I coordinated the (imminent) launch of Nomination Pools. Nowadays I also put an extra effort on making sure that this sub-system of Polkadot is *sustainable*, through code refactor and educating other core developers. Lastly, I have been the main author of the [Polkadot staking newsletter](https://gist.github.com/kianenigma/aa835946455b9a3f167821b9d05ba376), which is my main attempt at making the entire complexity and development of this part of the protocol transparent to the end-users. I expect myself to contribute *directly* to the staking system for at least another ~12, if not more, and afterwards having the role of an advisor. Some notable contributions: - paritytech/substrate#4517 - paritytech/substrate#7910 - paritytech/substrate#6242 - paritytech/substrate#9415 - paritytech/polkadot#3141 - paritytech/substrate#11212 - paritytech/substrate#12129 ### FRAME Historically, I have contributed a variety of domains in FRAME, namely: - Early version of the weight system paritytech/substrate#3816 paritytech/substrate#3157 - Early version of the transaction fee system - Primitive arithmetic types paritytech/substrate#3456 - Council election pallet paritytech/substrate#3364 Many of which were, admittedly, a PoC at most, if not considered "poor". I am happy that nowadays many of the above have been refactored and are being maintained by new domain experts. These days, I put most of my FRAME focus on testing and quality assurance. Through my work in the staking system, I have had to deal with the high sensitivity and liveness requirement of protocol development first hand (I believe I had to do among the [very first storage migrations](paritytech/substrate#3948) in Kusama) and consequently I felt the need to make better testing facilities, all of which have been formulated in https://forum.polkadot.network/t/testing-complex-frame-pallets-discussion-tools/356. Some relevant PRs: - paritytech/substrate#8038 - paritytech/substrate#9788 - paritytech/substrate#10174 Regardless of wearing the staking hat, I plan to remain a direct contributor to FRAME, namely because I consider it to be an important requirements of successfully delivering more features to Polkadot's ecosystem. ### Polkadot-Native Side Projects I have started multiple small, mostly non-RUST projects in the polkadot ecosystem that I am very happy about, and I plan to continue doing so. I have not yet found the time to make a "polished product" out of any of these, but I hope that I can help foster our community such that someday a team will do so. I consider my role, for the time being, to *put ideas out there* through these side projects. - https://github.com/substrate-portfolio/polkadot-portfolio/ - https://github.com/kianenigma/polkadot-basic-notification/ - https://github.com/paritytech/polkadot-scripts/ - https://github.com/paritytech/substrate-debug-kit/ ### Education Lastly, aside from having had a number of educational talks over the years (all of which [are listed](https://hello.kianenigma.nl/talks/) in my personal website), I am a big enthusiast of the newly formed Polkadot Blockchain Academy. I have [been an instructor](https://singular.app/collectibles/statemine/16/2) in the first cohort, and continue to contribute for as long and as much as I can, whilst still attending to the former 3 duties. --- With all of that being said and done, I consider myself at the beginning of the path to Dan 4, but happy to start at a lower one as well.

@kianenigma

# Membership Request Hi, I am Kian Paimani, known as @kianenigma. I have been working on Polkadot/Kusama through Parity since February 2019 and I can categorize my main contributions to Polkadot's ecosystem as follows: 1. Maintaining and developing the staking sub-system. 2. General FRAME development, especially testing and quality assurance. 3. Polkadot-native side-projects. 4. Education > My first contribution to Polkadot is also indeed related to staking: paritytech/substrate#1915 ### Staking system I joke as the Polkadot staking to be both my blessing and my curse over the years. I started working on it since the first days that I joined this ecosystem and the work [is ongoing ever since](https://github.com/orgs/paritytech/projects/33/views/9). In the past, I focused on making sure that the staking system is secure and to some extent scalable. More recently, I coordinated the (imminent) launch of Nomination Pools. Nowadays I also put an extra effort on making sure that this sub-system of Polkadot is *sustainable*, through code refactor and educating other core developers. Lastly, I have been the main author of the [Polkadot staking newsletter](https://gist.github.com/kianenigma/aa835946455b9a3f167821b9d05ba376), which is my main attempt at making the entire complexity and development of this part of the protocol transparent to the end-users. I expect myself to contribute *directly* to the staking system for at least another ~12, if not more, and afterwards having the role of an advisor. Some notable contributions: - paritytech/substrate#4517 - paritytech/substrate#7910 - paritytech/substrate#6242 - paritytech/substrate#9415 - paritytech/polkadot#3141 - paritytech/substrate#11212 - paritytech/substrate#12129 ### FRAME Historically, I have contributed a variety of domains in FRAME, namely: - Early version of the weight system paritytech/substrate#3816 paritytech/substrate#3157 - Early version of the transaction fee system - Primitive arithmetic types paritytech/substrate#3456 - Council election pallet paritytech/substrate#3364 Many of which were, admittedly, a PoC at most, if not considered "poor". I am happy that nowadays many of the above have been refactored and are being maintained by new domain experts. These days, I put most of my FRAME focus on testing and quality assurance. Through my work in the staking system, I have had to deal with the high sensitivity and liveness requirement of protocol development first hand (I believe I had to do among the [very first storage migrations](paritytech/substrate#3948) in Kusama) and consequently I felt the need to make better testing facilities, all of which have been formulated in https://forum.polkadot.network/t/testing-complex-frame-pallets-discussion-tools/356. Some relevant PRs: - paritytech/substrate#8038 - paritytech/substrate#9788 - paritytech/substrate#10174 Regardless of wearing the staking hat, I plan to remain a direct contributor to FRAME, namely because I consider it to be an important requirements of successfully delivering more features to Polkadot's ecosystem. ### Polkadot-Native Side Projects I have started multiple small, mostly non-RUST projects in the polkadot ecosystem that I am very happy about, and I plan to continue doing so. I have not yet found the time to make a "polished product" out of any of these, but I hope that I can help foster our community such that someday a team will do so. I consider my role, for the time being, to *put ideas out there* through these side projects. - https://github.com/substrate-portfolio/polkadot-portfolio/ - https://github.com/kianenigma/polkadot-basic-notification/ - https://github.com/paritytech/polkadot-scripts/ - https://github.com/paritytech/substrate-debug-kit/ ### Education Lastly, aside from having had a number of educational talks over the years (all of which [are listed](https://hello.kianenigma.nl/talks/) in my personal website), I am a big enthusiast of the newly formed Polkadot Blockchain Academy. I have [been an instructor](https://singular.app/collectibles/statemine/16/2) in the first cohort, and continue to contribute for as long and as much as I can, whilst still attending to the former 3 duties. --- With all of that being said and done, I consider myself at the beginning of the path to Dan 4, but happy to start at a lower one as well. Co-authored-by: Bastian Köcher <[email protected]>

* companion for paritytech/substrate#9788 * fmt * Some fixes * final fixes * fix a few small things * update Substrate Co-authored-by: parity-processbot <>

kianenigma added 3 commits September 15, 2021 11:55

deadlock, need to ask someone to help now

3d9123a

Finally it seems to be working.. at least for a few blocks

58e2e65

self-review

c8f9886

kianenigma added 2 commits September 15, 2021 18:48

major mega revamp

120c30f

some small fixes

9356bf3

kianenigma added a commit to paritytech/polkadot that referenced this pull request Sep 15, 2021

companion for paritytech/substrate#9788

b36dd83

This was referenced Sep 15, 2021

companion for substrate#9788 paritytech/polkadot#3858

Merged

Companion for Store voters in unsorted bags to get good stake iteration properties paritytech/polkadot#3413

Merged

another mega refactor

39e4c4b

kianenigma added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Sep 16, 2021

kianenigma requested review from emostov and jsdw September 16, 2021 14:09

kianenigma marked this pull request as ready for review September 16, 2021 14:09

kianenigma commented Sep 16, 2021

View reviewed changes

frame/executive/src/lib.rs Outdated Show resolved Hide resolved

add license

cc7a546

kianenigma commented Sep 16, 2021

View reviewed changes

utils/frame/try-runtime/cli/src/commands/follow_chain.rs Outdated Show resolved Hide resolved

kianenigma commented Sep 16, 2021

View reviewed changes

utils/frame/try-runtime/cli/src/lib.rs Outdated Show resolved Hide resolved

Apply suggestions from code review

8469695

kianenigma changed the title ~~follow-chain testing mode for try-runtime (and other minor fixes).~~ follow-chain testing mode for try-runtime (and other fixes). Sep 16, 2021

kianenigma changed the title ~~follow-chain testing mode for try-runtime (and other fixes).~~ follow-chain testing mode for try-runtime (and revamp CLI configs). Sep 16, 2021

emostov reviewed Sep 16, 2021

View reviewed changes

frame/executive/src/lib.rs Show resolved Hide resolved

emostov reviewed Sep 16, 2021

View reviewed changes

utils/frame/try-runtime/cli/src/lib.rs Outdated Show resolved Hide resolved

kianenigma added the A0-please_review Pull request needs code review. label Sep 17, 2021

sacha-l mentioned this pull request Sep 18, 2021

[docs]: Update Try Runtime article polkadot-developers/substrate-developer-hub.github.io#1119

Closed

a little self-review

b34d236

kianenigma requested a review from chevdor September 20, 2021 17:56

shawntabrizi reviewed Sep 21, 2021

View reviewed changes

utils/frame/try-runtime/cli/src/commands/on_runtime_upgrade.rs Outdated Show resolved Hide resolved

shawntabrizi reviewed Sep 21, 2021

View reviewed changes

shawntabrizi approved these changes Sep 21, 2021

View reviewed changes

Add the ext root check

474a02b

ghost merged commit 04bb4e0 into master Sep 21, 2021

ghost deleted the kiz-try-runtime-execute-block-no-check branch September 21, 2021 15:04

ghost pushed a commit to paritytech/polkadot that referenced this pull request Sep 21, 2021

companion for substrate#9788 (#3858)

481dd29

* companion for paritytech/substrate#9788 * fmt * Some fixes * final fixes * fix a few small things * update Substrate Co-authored-by: parity-processbot <>

chevdor mentioned this pull request Sep 23, 2021

Backports for 0.9.10 #9845

Merged

github-actions bot mentioned this pull request Oct 11, 2021

Update substrate/polkadot/cumulus from v0.9.10 to v0.9.11 moonbeam-foundation/moonbeam#892

Closed

ghzlatarev mentioned this pull request Nov 2, 2021

[Manta-PC] Update dependencies to v0.9.12 Manta-Network/Manta#242

Merged

19 tasks

kianenigma mentioned this pull request Nov 4, 2021

Runtime State Test + Integration with try-runtime #10174

Merged

7 tasks

kianenigma mentioned this pull request Sep 27, 2022

Member Request polkadot-fellows/seeding#14

Merged

ggwpez pushed a commit to ggwpez/runtimes that referenced this pull request Mar 10, 2023

companion for substrate#9788 (#3858)

80d5864

* companion for paritytech/substrate#9788 * fmt * Some fixes * final fixes * fix a few small things * update Substrate Co-authored-by: parity-processbot <>

ggwpez pushed a commit to ggwpez/runtimes that referenced this pull request Jul 13, 2023

companion for substrate#9788 (#3858)

6279896

* companion for paritytech/substrate#9788 * fmt * Some fixes * final fixes * fix a few small things * update Substrate Co-authored-by: parity-processbot <>

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`follow-chain` testing mode for try-runtime (and revamp CLI configs). #9788

`follow-chain` testing mode for try-runtime (and revamp CLI configs). #9788

kianenigma commented Sep 15, 2021 •

edited

Loading

niklasad1 commented Sep 16, 2021

kianenigma commented Sep 16, 2021

niklasad1 commented Sep 16, 2021

shawntabrizi Sep 21, 2021

kianenigma Sep 21, 2021

shawntabrizi Sep 21, 2021

kianenigma Sep 21, 2021

shawntabrizi left a comment

kianenigma commented Sep 21, 2021

ghost commented Sep 21, 2021

follow-chain testing mode for try-runtime (and revamp CLI configs). #9788

follow-chain testing mode for try-runtime (and revamp CLI configs). #9788

Conversation

kianenigma commented Sep 15, 2021 • edited Loading

niklasad1 commented Sep 16, 2021

kianenigma commented Sep 16, 2021

niklasad1 commented Sep 16, 2021

shawntabrizi Sep 21, 2021

Choose a reason for hiding this comment

kianenigma Sep 21, 2021

Choose a reason for hiding this comment

shawntabrizi Sep 21, 2021

Choose a reason for hiding this comment

kianenigma Sep 21, 2021

Choose a reason for hiding this comment

shawntabrizi left a comment

Choose a reason for hiding this comment

kianenigma commented Sep 21, 2021

ghost commented Sep 21, 2021

`follow-chain` testing mode for try-runtime (and revamp CLI configs). #9788

`follow-chain` testing mode for try-runtime (and revamp CLI configs). #9788

kianenigma commented Sep 15, 2021 •

edited

Loading