Refactor key management #3296

gavofyork · 2019-08-03T09:37:13Z

Supersedes #3213

This PR is turning into a major refactoring over the crypto and key management system, which seems both long overdue and somewhat inescapable.

I'll want to integrate the general changes in (but not merge) #3034 and possibly #3193 together with #3137.

The main issues that this PR aims to solve are:

Independent subsystems that happened to use the same crypto had to share the same session keys (and key-store division).
Assumption that the twin consensus systems of block production (Aura/Babe) and finality (Grandpa) each required keys (which dirtied a lot of Substrate, placing APIs like fn fg_authority_key() all the way through the codebase). Though they do now, they won't forever and it shouldn't be an assumption that we bake in to internal APIs.
Keys passed in through CLI are sticky; their existence enabled the consensus system workers and force them to use these keys alone. Rather whatever key in the key store that happens to be an authority should be used at any given time.

TODO:

Refactor ImOnline:
- Should get its own key and not piggy-back on other session keys.
Dynamic key selection:
- Enable Babe and Grandpa to be able to determine current authority key (if there is one).
- Dynamically select the correct authoring key to use in each subsystem.
- Use --validator to enable active validation process; remove any validator key-management stuff from the CLI (--key). (A second --worker arg can enable off-chain workers.)
- Remove all static-key selection code (--key all the way down to key-choice).
- Remove authority_key/fg_authority_key APIs completely.
- Make --password work (should accept a manually-entered password and use it for everything in the keystore).
- Introduce RPCs for session rotation, using code from Implement session key handover #3034 if it makes sense. There should be a single RPC author_rotateKeys which creates a new set of session keys, saves them to the key-store and returns their public keys, so that a rotation-transaction can be constructed. There should be no pollution of the APIs with session keys: the rotation API should be wholly independent of each of the individual keys. This may mean new APIs in the OpaqueKeys trait.
Refactor Client:
- Create a basic Client trait that provides all the basic APIs and has Block as an associated item. Probably a separate PR.

I would also like to solve the fact that service construction is highly monolithic and polluted by consensus-module-specific code. Really it should just be a case of specifying the block production and finality systems and letting it do the rest. Code for set-up, take-down, RPC and key-store integration should be provided through traits that the consensus modules (Babe, Aura, Grandpa and eventually Rho and PoW) implement within their module's scope. There shouldn't be any consensus-module-specific logic dirtying up service.rs.

Refactor service:
- Move out any consensus-module-specific code, and hardwired assumptions that they each have "authority keys", from service.rs into the respective consensus modules. Provide a single API endpoint (i.e. a type that implements some ConsensusSystem trait and have that as the only resource that service needs know of. Probably a separate PR.

This also adds:

Add Call type to extensible transactions.
Add swap to allow two storage items in the same map to be swapped.

Cleanup some naming

This will eventually allow for dynamic determination of authority keys and avoid having to set them directly on CLI.

…o gav-ext-tx-call

Experiment with modular consensus API.

with validator IDs

…o gav-in-progress

gnunicorn

A few remarks, but nothing major.

Note: I only glanced over the macros and superficially looked at the SRML changes, as I missing the context to understand the design behind the changes - but nothing bad jumped out.

core/application-crypto/src/ed25519.rs

gnunicorn · 2019-08-07T11:13:05Z

core/application-crypto/src/lib.rs

+			#[cfg_attr(feature = "std", derive(Debug, Hash))]
+			pub struct Public($public);
+		}
+		// TODO: needed for verify since it takes an AsRef, but should be removed once that is


No TODO without a link to the issue pls.

core/application-crypto/src/lib.rs

gnunicorn · 2019-08-07T12:06:13Z

core/primitives/src/crypto.rs

+/// A type of supported crypto.
+#[derive(Clone, Copy, PartialEq, Eq, Encode, Decode)]
+#[cfg_attr(feature = "std", derive(Debug))]
+#[repr(u32)]


any particular reason this is a u32 and not, for e.g. a u8? Are we expecting to ever have more than 255 known values in here?

core/service/src/components.rs

core/sr-primitives/src/lib.rs

gnunicorn · 2019-08-07T12:47:37Z

node-template/src/service.rs

 		AuthoritySetup = {
 			|service: Self::FullService| {
-				if let Some(key) = service.authority_key() {
+				if let Some(key) = None::<aura_primitives::sr25519::AuthorityPair> /* service.authority_key() */ {


I suppose that is not the template that we want to hand out after this PR, is it?....

gnunicorn · 2019-08-07T12:48:22Z

node/cli/src/chain_spec.rs

 	// for i in 1 2 3 4 ; do for j in session; do subkey --ed25519 inspect "$secret"//fir//$j//$i; done; done

-	let initial_authorities: Vec<(AccountId, AccountId, BabeId, GrandpaId)> = vec![(
+	// TODO: actually use sr25519 for babe keys (right now they seems to just be copies of the


Co-Authored-By: Benjamin Kampmann <[email protected]>

Also, rename traits::KeyStore* -> traits::BareCryptoStore*

…o gav-in-progress

* Add Call type to extensible transactions. Cleanup some naming * Merge Resource and BlockExhausted into just Exhausted * Fix * Another fix * Call * Some fixes * Fix srml tests. * Fix all tests. * Refactor crypto so each application of it has its own type. * Introduce new AuthorityProvider API into Aura This will eventually allow for dynamic determination of authority keys and avoid having to set them directly on CLI. * Introduce authority determinator for Babe. Experiment with modular consensus API. * Work in progress to introduce KeyTypeId and avoid polluting API with validator IDs * Finish up drafting imonline * Rework offchain workers API. * Rework API implementation. * Make it compile for wasm, simplify app_crypto. * Fix compilation of im-online. * Fix compilation of im-online. * Fix more compilation errors. * Make it compile. * Fixing tests. * Rewrite `keystore` * Fix session tests * Bring back `TryFrom`'s' * Fix `srml-grandpa` * Fix `srml-aura` * Fix consensus babe * More fixes * Make service generate keys from dev_seed * Build fixes * Remove offchain tests * More fixes and cleanups * Fixes finality grandpa * Fix `consensus-aura` * Fix cli * Fix `node-cli` * Fix chain_spec builder * Fix doc tests * Add authority getter for grandpa. * Test fix * Fixes * Make keystore accessible from the runtime * Move app crypto to its own crate * Update `Cargo.lock` * Make the crypto stuff usable from the runtime * Adds some runtime crypto tests * Use last finalized block for grandpa authority * Fix warning * Adds `SessionKeys` runtime api * Remove `FinalityPair` and `ConsensusPair` * Minor governance tweaks to get it inline with docs. * Make the governance be up to date with the docs. * Build fixes. * Generate the inital session keys * Failing keystore is a hard error * Make babe work again * Fix grandpa * Fix tests * Disable `keystore` in consensus critical stuff * Build fix. * ImOnline supports multiple authorities at once. * Update core/application-crypto/src/ed25519.rs * Merge branch 'master' into gav-in-progress * Remove unneeded code for now. * Some `session` testing * Support querying the public keys * Cleanup offchain * Remove warnings * More cleanup * Apply suggestions from code review Co-Authored-By: Benjamin Kampmann <[email protected]> * More cleanups * JSONRPC API for setting keys. Also, rename traits::KeyStore* -> traits::BareCryptoStore* * Bad merge * Fix integration tests * Fix test build * Test fix * Fixes * Warnings * Another warning * Bump version.

gavofyork and others added 30 commits July 26, 2019 10:29

Add Call type to extensible transactions.

aadeb9c

Cleanup some naming

Merge Resource and BlockExhausted into just Exhausted

ad9af9e

Merge remote-tracking branch 'origin/master' into gav-ext-tx-call

8024922

Fix

38a488f

Another fix

8206086

Call

10a9447

Some fixes

e3cc2fe

Fix srml tests.

19c1840

Fix all tests.

539e481

Refactor crypto so each application of it has its own type.

7c09750

Introduce new AuthorityProvider API into Aura

5aad05b

This will eventually allow for dynamic determination of authority keys and avoid having to set them directly on CLI.

Merge branch 'gav-ext-tx-call' of github.com:paritytech/substrate int…

a1dc9e6

…o gav-ext-tx-call

Introduce authority determinator for Babe.

c9bb7d4

Experiment with modular consensus API.

Work in progress to introduce KeyTypeId and avoid polluting API

b1eae50

with validator IDs

Finish up drafting imonline

4715da3

Merge branch 'gav-ext-tx-call' into gav-in-progress2

ff5033b

Rework offchain workers API.

ca53ad6

Rework API implementation.

ac419a4

Make it compile for wasm, simplify app_crypto.

e140212

Fix compilation of im-online.

c4d58bf

Fix compilation of im-online.

0d81c46

Fix more compilation errors.

e1d2fbc

Make it compile.

182c753

Fixing tests.

f1ecdee

Merge branch 'master' into gav-in-progress

e172788

Rewrite keystore

82cbe9e

Fix session tests

b270f24

Bring back TryFrom's'

1bd145a

Fix srml-grandpa

0a93060

Fix srml-aura

c543989

Merge branch 'gav-in-progress' of github.com:paritytech/substrate int…

a253132

…o gav-in-progress

jacogr mentioned this pull request Aug 7, 2019

Cater for key definitions rework polkadot-js/api#1269

Merged

bkchr added 2 commits August 7, 2019 13:57

Some session testing

9806232

Support querying the public keys

315caea

gnunicorn approved these changes Aug 7, 2019

View reviewed changes

bkchr and others added 7 commits August 7, 2019 16:30

Cleanup offchain

01fc842

Remove warnings

1d2da8d

Merge remote-tracking branch 'origin/master' into gav-in-progress

4745778

More cleanup

10ffc74

Apply suggestions from code review

ac05779

Co-Authored-By: Benjamin Kampmann <[email protected]>

More cleanups

ecae834

JSONRPC API for setting keys.

900d177

Also, rename traits::KeyStore* -> traits::BareCryptoStore*

folsen mentioned this pull request Aug 7, 2019

Have ability to validate with multiple keys on the same node #3324

Open

gavofyork and others added 10 commits August 7, 2019 18:56

Merge branch 'gav-in-progress' of github.com:paritytech/substrate int…

ccc0c04

…o gav-in-progress

Bad merge

301d40b

Fix integration tests

4bf2c3f

Fix test build

e0e90dd

Merge branch 'gav-in-progress' of github.com:paritytech/substrate int…

a0659b9

…o gav-in-progress

Test fix

2ab31c8

Fixes

7bd33d0

Warnings

2afad01

Another warning

f2e676c

Bump version.

c9c1b3b

gavofyork merged commit ed61b1f into master Aug 7, 2019

gavofyork deleted the gav-in-progress branch August 7, 2019 18:47

gavofyork mentioned this pull request Aug 8, 2019

Add srml/im-online integrations #3223

Closed

4 tasks

tomusdrw mentioned this pull request Aug 21, 2019

Introduce capabilities filtering for off-chain runtime calls. #3454

Merged

gnunicorn modified the milestones: 2.1, Polkadot Mar 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor key management #3296

Refactor key management #3296

Uh oh!

gavofyork commented Aug 3, 2019 •

edited

Loading

Uh oh!

gnunicorn left a comment

Uh oh!

Uh oh!

gnunicorn Aug 7, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gnunicorn Aug 7, 2019

Uh oh!

Uh oh!

Uh oh!

gnunicorn Aug 7, 2019

Uh oh!

gnunicorn Aug 7, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Refactor key management #3296

Refactor key management #3296

Uh oh!

Conversation

gavofyork commented Aug 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gnunicorn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gnunicorn Aug 7, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gnunicorn Aug 7, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gnunicorn Aug 7, 2019

Choose a reason for hiding this comment

Uh oh!

gnunicorn Aug 7, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

gavofyork commented Aug 3, 2019 •

edited

Loading