Pass registry to all codec types

CHANGELOG.md

@@ -1,3 +1,11 @@
+# 0.97.0-beta.x
+
+- **Breaking change** Add the passing on an explicit `Registry` to all calls to `createType` as well as all codec classes. If you are upgrading form an older version, use the [migration guide](UPGRADING.md) to walk through the changes required.
+- **Breaking change** The metadata classes have been moved from `@polkadot/types` into `@polkadot/metadata`. If you explicitly create `Metadata` classes, you now would need to do so with the `import Metadata from '@polkadot/metadata';` This is a decorated class, where the previous metadata is now available on `Metadata.metadata`, alongside decorated `consts.*`', `tx.*` and `query.*`.
+- `api.derive.*` now has built-in memomization
+- Various fixes and cleanups to `api.derive.*` including era length progress and vesting calculations
+- Aligned all types with the latest Polkadot & Substrate
+
 # 0.96.1
 
 - Updated types for latest Kusama

UPGRADING.md

@@ -2,7 +2,91 @@
 
 This is an upgrade guide for users of the API. It does not attempt to detail each version (the [CHANGELOG](CHANGELOG.md) has all the changes between versions), but rather tries to explain the rationale behind major breaking changes and how users of the API should handle this.
 
-While we try to keep the user-facing interfaces as stable as possible, sometimes you just need to make additions to move forward and improve things down the road, as painful as they may be. Like you, we are also users of the API, and eat our own dogfood - and as such, feel any pains introduced first.
+While we try to keep the user-facing interfaces as stable as possible, sometimes you just need to make additions to move forward and improve things down the road, as painful as they may be. Like you, we are also users of the API, and eat our own dog food - and as such, feel any pains introduced first.
+
+
+## 0.97.1 (and newer), from 0.90.1 (and older)
+
+The 0.97 series lays the groundwork to allow type registration  to be ties to a specific chain and a specific Api instance. In the past, 2 Api instances in the same process would share types, which mean that you could not connect to 2 independent chains with different types. This is very problematic for Polkadot chains, where the idea is to connect to multiple chains.
+
+When using the Api, a new `Registry` will be created on using `new Api(...)` or `Api.create(...)` and this will be transparently passed when creating types. In the cases where you create type instances explicitly or create type classes for injection, you would need to make adjustments.
+
+### Type classes
+
+In a number of instances, developers are creating classes and making these available for interacting with their chains. For instance, an example of a custom type could be -
+
+```js
+import { Struct, Text, u32 } from '@polkadot/types';
+
+export class Preferences extends Struct {
+  constructor (value?: ahy) {
+    super({
+      name: Text,
+      id: u32
+    }, value);
+  }
+
+  ...
+}
+```
+
+In the current iteration, the underlying `@polkadot/types` bases structures now require a `Registry` to be passed as the first parameter. This means that the above signature would be adjusted to -
+
+```js
+// the next import is only required for TypeScript
+import { Registry } from '@polkadot/types/types';
+import { Struct, Text, u32 } from '@polkadot/types';
+
+export class Preferences extends Struct {
+  constructor (registry: Registry, value?: ahy) {
+    super(registry, {
+      name: Text,
+      id: u32
+    }, value);
+  }
+
+  ...
+}
+```
+
+Where the type is used or returned from the API, the `Registry` will be automatically passed to class creation.
+
+### createType
+
+Previously, when creating a type instance such as `BlockNumber`, you would do `api.createType('BlockNumber', <initValue>)`, this is unchanged. In the cases where you directly import from `@polkadot/types`, the following pattern is required -
+
+```js
+import { createType } from '@polkadot/types';
+
+...
+const blockNumber = createType(api.registry, 'BlockNumber', 12345);
+```
+
+In some cases, you would want to explicitly pass a `Registry` interface to the API, instead of relying on it explicitly. This is generally applicable in the cases where you want to use the `createType` independently from the API -
+
+```js
+import { ApiPromise } from '@polkadot/api';
+import { TypeRegistry, createType } from '@polkadot/types';
+
+...
+const registry = new TypeRegistry();
+const blockNumber = createType(registry, 'BlockNumber', 12345);
+const api = await ApiPromise.create({ registry });
+```
+
+### Extrinsic metadata
+
+In some applications, the undocumented `findFunction` has been used to determine the Api has the metadata for a specific extrinsic. The has been exposed on top of `GenericCall`, and it typically used in applications such as signers. Along with the compulsory registry, the above functions have been moved to the `Registry` itself, so if you previously had -
+
+```js
+const { meta, method, section } = GenericCall.findFunction(extrinsic.callIndex);
+```
+
+You need to change it to -
+
+```js
+const { meta, method, section } = registry.findMetaCall(extrinsic.callIndex);
+```
 
 ## 0.90.1 (and newer), from 0.81.1 (and older)
 
@@ -29,10 +113,10 @@ To better align with the actual types from the metadata, and avoid (too much) co
 
 The [@polkadot/api](packages/api) has always handled the conversion of types for parameters when making calls or queries. For example, when making a transfer to `BOB` (address), any of the following is valid -
 
-- `api.tx.balances.transfer(BOB, 12345)` - value specied as a number
-- `api.tx.balances.transfer(BOB, '12345')` - value specied as a string
-- `api.tx.balances.transfer(BOB, '0x3039')` - value specied as a hex
-- `api.tx.balances.transfer(BOB, new BN(12345))` - value specied as a [BN](https://github.com/indutny/bn.js/)
+- `api.tx.balances.transfer(BOB, 12345)` - value specified as a number
+- `api.tx.balances.transfer(BOB, '12345')` - value specified as a string
+- `api.tx.balances.transfer(BOB, '0x3039')` - value specified as a hex
+- `api.tx.balances.transfer(BOB, new BN(12345))` - value specified as a [BN](https://github.com/indutny/bn.js/)
 
 Internally the API will take the input and convert the value into a `Balance`, serialize it using the SCALE codec and transfer it to the node. In some cases users would construct the `Balance` type manually, by importing the class and calling `new` on it. This last approach has now been removed, and where classes are still available (limited reach), discouraged.
 
@@ -108,7 +192,7 @@ export class MyStruct extends Struct {
 }
 ```
 
-Internally the [@polkadot/types](packages/types) package now only defines classes where there are specific encoding logic applied. For all other types, the definitions are done via a JSON-like format and then the TypeScript definitions are generated from these. (In a world where nodes inject types and the type definitions are not needed, this functionality will be useful to allow TS devs to auto-generate type definitions based on what the node defines.)
+Internally the [@polkadot/types](packages/types) package now only defines classes where there are specific encoding logic applied. For all other types, the definitions are done via a JSON-like format and then the TypeScript definitions are generated from these. (In a world where nodes inject types and the type definitions are not needed, this functionality will be useful to allow TS developers to auto-generate type definitions based on what the node defines.)
 
 ### Signing transactions (Signer interface)
 

packages/api-contract/src/Abi.spec.ts

@@ -2,16 +2,19 @@
 // This software may be modified and distributed under the terms
 // of the Apache-2.0 license. See the LICENSE file for details.
 
-import incrementerAbi from '../test/contracts/incrementer.json';
+import { TypeRegistry } from '@polkadot/types';
 
+import incrementerAbi from '../test/contracts/incrementer.json';
 import { Abi } from '.';
 
 describe('Abi', (): void => {
+  const registry = new TypeRegistry();
+
   describe('incrementer', (): void => {
     let abi: Abi;
 
     beforeEach((): void => {
-      abi = new Abi(incrementerAbi);
+      abi = new Abi(registry, incrementerAbi);
     });
 
     it('has the attached methods', (): void => {

packages/api-contract/src/Abi.ts

@@ -2,7 +2,9 @@
 // This software may be modified and distributed under the terms
 // of the Apache-2.0 license. See the LICENSE file for details.
 
+import { Registry } from '@polkadot/types/types';
 import { AbiConstructors, AbiMessages, ContractABI, ContractABIPre, ContractABIFn, InterfaceAbi } from './types';
+
 import { stringCamelCase } from '@polkadot/util';
 
 import ContractRegistry from './ContractRegistry';
@@ -14,8 +16,8 @@ export default class ContractAbi extends ContractRegistry implements InterfaceAb
 
   public readonly messages: AbiMessages;
 
-  constructor (abi: ContractABIPre) {
-    super(abi);
+  constructor (registry: Registry, abi: ContractABIPre) {
+    super(registry, abi);
     [this.abi, this.constructors, this.messages] = this.decodeAbi(abi);
   }
 

packages/api-contract/src/ContractRegistry.ts

@@ -2,28 +2,28 @@
 // This software may be modified and distributed under the terms
 // of the Apache-2.0 license. See the LICENSE file for details.
 
-import { CodecArg, Constructor, TypeDef } from '@polkadot/types/types';
+import { CodecArg, Constructor, Registry, TypeDef } from '@polkadot/types/types';
 import { ContractABIArgBasePre, ContractABIContract, ContractABIContractPre, ContractABIEvent, ContractABIEventPre, ContractABIFn, ContractABIFnArg, ContractABIMessage, ContractABIMessageBase, ContractABIMessagePre, ContractABI, ContractABIMessageCommon, ContractABIPre, ContractABIRange, ContractABIRangePre, ContractABIStorage, ContractABIStorageLayout, ContractABIStorageLayoutPre, ContractABIStoragePre, ContractABIStorageStruct, ContractABIStorageStructPre, ContractABITypePre } from './types';
-import { Compact, u32, createClass, createType, encodeType } from '@polkadot/types';
 
+import { Compact, u32, createClass, createType, encodeType } from '@polkadot/types';
 import { assert, hexToU8a, isNumber, isString, isNull, isObject, isUndefined, stringCamelCase, isHex, hexToNumber } from '@polkadot/util';
 
 import MetaRegistry from './MetaRegistry';
 
 // parse a selector, this can be a number (older) or of [<hex>, <hex>, ...]. However,
 // just catch everything (since this is now non-standard for u32 anyway)
-function parseSelector (fnname: string, input: ContractABIMessageCommon['selector']): u32 {
+function parseSelector (registry: Registry, fnname: string, input: ContractABIMessageCommon['selector']): u32 {
   if (isNumber(input)) {
-    return createType('u32', input);
+    return createType(registry, 'u32', input);
   } else if (isHex(input)) {
-    return createType('u32', hexToU8a(input));
+    return createType(registry, 'u32', hexToU8a(input));
   } else if (typeof input === 'string') {
     try {
       const array = JSON.parse(input);
 
       assert(array.length === 4, `${fnname}: Invalid selector length`);
 
-      return createType('u32', Uint8Array.from(
+      return createType(registry, 'u32', Uint8Array.from(
         // the as number[] is here to pacify TS, it doesn't quite know how to handle the cb
         (array as number[]).map((value: string | number): number =>
           isHex(value)
@@ -39,8 +39,9 @@ function parseSelector (fnname: string, input: ContractABIMessageCommon['selecto
   throw new Error(`${fnname}: Unable to parse selector`);
 }
 
-function createArgClass (args: ContractABIFnArg[], baseDef: Record<string, string>): Constructor {
+function createArgClass (registry: Registry, args: ContractABIFnArg[], baseDef: Record<string, string>): Constructor {
   return createClass(
+    registry,
     JSON.stringify(
       args.reduce((base: Record<string, any>, { name, type }): Record<string, any> => {
         base[name] = type.displayName || encodeType(type);
@@ -87,7 +88,7 @@ export default class ContractRegistry extends MetaRegistry {
       assert(isNumber(name) && isString(this.stringAt(name)), `Expected name for ${fnname}`);
       assert(isNull(returnType) || (isNumber(returnType.ty) && isObject(this.typeDefAt(returnType.ty))), `Expected return_type for ${fnname}`);
 
-      parseSelector(fnname, selector);
+      parseSelector(this.registry, fnname, selector);
       this.validateArgs(fnname, message.args);
     });
   }
@@ -113,12 +114,13 @@ export default class ContractRegistry extends MetaRegistry {
         type
       };
     });
-    const Clazz = createArgClass(args, isUndefined(message.selector) ? {} : { __selector: 'u32' });
-    const baseStruct: { [index: string]: any } = { __selector: isUndefined(message.selector) ? undefined : parseSelector(name, message.selector) };
+    const Clazz = createArgClass(this.registry, args, isUndefined(message.selector) ? {} : { __selector: 'u32' });
+    const baseStruct: { [index: string]: any } = { __selector: isUndefined(message.selector) ? undefined : parseSelector(this.registry, name, message.selector) };
     const encoder = (...params: CodecArg[]): Uint8Array => {
       assert(params.length === args.length, `Expected ${args.length} arguments to contract ${name}, found ${params.length}`);
 
       const u8a = new Clazz(
+        this.registry,
         args.reduce((mapped, { name }, index): Record<string, CodecArg> => {
           mapped[name] = params[index];
 

packages/api-contract/src/MetaRegistry.spec.ts

@@ -4,16 +4,19 @@
 
 import fs from 'fs';
 import path from 'path';
+import { TypeRegistry } from '@polkadot/types';
 
 import MetaRegistry from './MetaRegistry';
 
 import erc20Abi from '../test/contracts/Erc20.json';
 import sharedVecAbi from '../test/contracts/SharedVecV2.json';
 
 describe('MetaRegistry', (): void => {
+  const registry = new TypeRegistry();
+
   describe('construction', (): void => {
     it('initializes from a contract ABI (ERC20)', (): void => {
-      const metaRegistry = new MetaRegistry(erc20Abi);
+      const metaRegistry = new MetaRegistry(registry, erc20Abi);
       fs.writeFile(path.join(__dirname, '../test/compare/erc20.test.json'), JSON.stringify(metaRegistry.typeDefs, null, 2), function (err): void {
         if (err) throw err;
       });
@@ -22,7 +25,7 @@ describe('MetaRegistry', (): void => {
     });
 
     it('initializes from a contract ABI (SharedVec)', (): void => {
-      const metaRegistry = new MetaRegistry(sharedVecAbi);
+      const metaRegistry = new MetaRegistry(registry, sharedVecAbi);
       fs.writeFile(path.join(__dirname, '../test/compare/shared-vec.test.json'), JSON.stringify(metaRegistry.typeDefs, null, 2), function (err): void {
         if (err) throw err;
       });

packages/api-contract/src/MetaRegistry.ts

@@ -2,7 +2,7 @@
 // This software may be modified and distributed under the terms
 // of the Apache-2.0 license. See the LICENSE file for details.
 
-import { MetaRegistryItem, MetaRegistryJson, MetaTypeDefClikeEnum, MetaType, MetaTypeDefEnum, MetaTypeDefEnumVariant, MetaTypeDefEnumVariantStruct, MetaTypeDefEnumVariantTupleStruct, MetaTypeDefEnumVariantUnit, MetaTypeDefStruct, MetaTypeDefStructField, MetaTypeDefTupleStruct, MetaTypeDefUnion, MetaTypeIdCustom, MetaTypeIdVec, MetaTypeIdVecFixed, MetaTypeInfo, StringIndex, TypeDef, TypeDefInfo, TypeIndex } from '@polkadot/types/types';
+import { MetaRegistryItem, MetaRegistryJson, MetaTypeDefClikeEnum, MetaType, MetaTypeDefEnum, MetaTypeDefEnumVariant, MetaTypeDefEnumVariantStruct, MetaTypeDefEnumVariantTupleStruct, MetaTypeDefEnumVariantUnit, MetaTypeDefStruct, MetaTypeDefStructField, MetaTypeDefTupleStruct, MetaTypeDefUnion, MetaTypeIdCustom, MetaTypeIdVec, MetaTypeIdVecFixed, MetaTypeInfo, Registry, StringIndex, TypeDef, TypeDefInfo, TypeIndex } from '@polkadot/types/types';
 
 import { assert } from '@polkadot/util';
 import { displayType, withTypeString } from '@polkadot/types';
@@ -34,13 +34,16 @@ function detectedType ({ def, id }: MetaType): MetaTypeInfo {
 }
 
 class MetadataRegistryLookup {
+  public readonly registry: Registry;
+
   protected _strings: string[] = [];
 
   protected _types: MetaType[] = [];
 
   public typeDefs: TypeDef[] = [];
 
-  constructor ({ registry: { strings, types } }: MetaRegistryJson) {
+  constructor (registry: Registry, { registry: { strings, types } }: MetaRegistryJson) {
+    this.registry = registry;
     this._strings = strings;
     this._types = types;
   }
@@ -102,8 +105,8 @@ class MetadataRegistryLookup {
 }
 
 export default class MetaRegistry extends MetadataRegistryLookup {
-  constructor (json: MetaRegistryJson) {
-    super(json);
+  constructor (registry: Registry, json: MetaRegistryJson) {
+    super(registry, json);
 
     // Generate TypeDefs for each provided registry type
     this._types.forEach((_, index: number): void => this.setTypeDefAtIndex(index + 1));

packages/api-contract/src/base/Blueprint.ts

@@ -41,7 +41,7 @@ export default class Blueprint<ApiType extends ApiTypes> extends BaseWithTx<ApiT
   constructor (api: ApiObject<ApiType>, abi: ContractABIPre | Abi, decorateMethod: DecorateMethod<ApiType>, codeHash: string | Hash) {
     super(api, abi, decorateMethod);
 
-    this.codeHash = createType('Hash', codeHash);
+    this.codeHash = createType(this.registry, 'Hash', codeHash);
   }
 
   public deployContract (constructorIndex = 0, endowment: number | BN, maxGas: number | BN, ...params: any[]): BlueprintCreate<ApiType> {

packages/api-contract/src/base/Contract.ts

@@ -44,7 +44,7 @@ export default class Contract<ApiType extends ApiTypes> extends BaseWithTxAndRpc
         as === 'rpc'
           ? (account: IKeyringPair | string | AccountId | Address): ContractCallResult<'rpc'> => {
             return this.rpcContractsCall(
-              createType('ContractCallRequest', {
+              createType(this.registry, 'ContractCallRequest', {
                 origin: account,
                 dest: this.address.toString(),
                 value,
@@ -54,7 +54,7 @@ export default class Contract<ApiType extends ApiTypes> extends BaseWithTxAndRpc
             )
               .pipe(
                 map((result: ContractExecResult): ContractCallOutcome =>
-                  this.createOutcome(result, createType('AccountId', account), def, params)
+                  this.createOutcome(result, createType(this.registry, 'AccountId', account), def, params)
                 )
               );
           }
@@ -74,8 +74,8 @@ export default class Contract<ApiType extends ApiTypes> extends BaseWithTxAndRpc
       const { data } = result.asSuccess;
 
       output = message.returnType
-        ? formatData(data, message.returnType)
-        : createType('Data', data);
+        ? formatData(this.registry, data, message.returnType)
+        : createType(this.registry, 'Data', data);
     }
 
     return {
@@ -92,6 +92,6 @@ export default class Contract<ApiType extends ApiTypes> extends BaseWithTxAndRpc
   constructor (api: ApiObject<ApiType>, abi: ContractABIPre | Abi, decorateMethod: DecorateMethod<ApiType>, address: string | AccountId | Address) {
     super(api, abi, decorateMethod);
 
-    this.address = createType('Address', address);
+    this.address = createType(this.registry, 'Address', address);
   }
 }

packages/api-contract/src/base/util.ts

@@ -5,6 +5,7 @@
 import { ApiTypes, DecorateMethod, DecoratedRpc, SubmittableModuleExtrinsics } from '@polkadot/api/types';
 import { ApiObject, ContractABIMessage, ContractABIPre, ContractBase, ContractMessage } from '../types';
 import { RpcInterface } from '@polkadot/rpc-core/jsonrpc.types';
+import { Registry } from '@polkadot/types/types';
 
 import { assert, stringCamelCase } from '@polkadot/util';
 import Abi from '../Abi';
@@ -16,10 +17,13 @@ export abstract class Base<ApiType extends ApiTypes> implements ContractBase<Api
 
   public readonly decorateMethod: DecorateMethod<ApiType>;
 
+  public readonly registry: Registry;
+
   constructor (api: ApiObject<ApiType>, abi: ContractABIPre | Abi, decorateMethod: DecorateMethod<ApiType>) {
     this.abi = abi instanceof Abi
       ? abi
-      : new Abi(abi);
+      : new Abi(api.registry, abi);
+    this.registry = api.registry;
     this.api = api;
     this.decorateMethod = decorateMethod;
   }