@joystream/types - api augmentation

[Lezek123]

Implements #1165

Brief summary of PR content

• Exposed a few more missing types in the registry
• Adjusted the scripts so that now we are able to take full advantage of @polkadot/typegen and create all the auto-generated files based on current @joystream/types with just one command (yarn workspace @joystream/types generate:all). This is furhter described in the section below.
• Fixed linter errors after eslint upgrade to allow the CI chekcs to pass.

types/augment and types/augment-codec

I changed the directory structure a little bit for more clarity, now the generated augment files can be found in two folders:

• types/augment - those augments are relying on interfaces automatically generated by @polkadot/typegen based on types/augment/all/definitions.ts and can be re-generated from current @joystream/types and runtime metadata by running yarn workspace @joystream/types generate:augment (requires running local node!). Those support augmentation for api.query, api.consts, api.tx, api.createType etc.
• types/augment-codec - those augments are using our custom codec types classes instead of interfaces automatically generated by @polkadot/typegen allowing us to access custom getters/methods, ie. api.createType('SomeEnum').isOfType or api.createType('ContentId').encode(). Currently they only support api.createType. They can be re-generated with yarn workspace @joystream/types generate:codec-defs

All auto-generated files can be updated at once by running yarn workspace @joystream/types generate:all (requires running local node first in order for the script to be able to connect to it and fetch the current metadata used to decorate the api)

NOTE: All projects that previosly used "@polkadot/types/augment": ["path/to/types/src/definitions/augment-types.ts"] inside tsconfig.json can switch to "@polkadot/types/augment": ["path/to/types/augment-codec/augment-types.ts"] for full backward-compatibility

Example of using types/augment-codec:

tsconfig.json:

    "paths": {
      "@polkadot/types/augment": ["path/to/types/augment-codec/augment-types.ts"],
    }

src/someScript.ts:

/* api initialization etc... */
const contentId = api.createType(
    'ContentId',
    '0x024d7e659d98d537e11f584a411a109f823be28c2e33d5fd5bc83705459442d9'
)
console.log(contentId.encode()) // Custom method will have TS support

Example of using types/augment:

tsconfig.json:

    "paths": {
      "@polkadot/types/augment": ["path/to/types/augment/augment-types.ts"],
      "@polkadot/api/augment": ["path/to/types/augment/augment-api.ts"]
    }

src/someScript.ts:

  /* api initialization etc... */

  // Module and method will be auto-suggested, args and return type are fully type-safe (no assertions required)!
  const member = await api.query.members.membershipById(0);
  // We can access member.isEmpty and member.handle in a type-safe way
  console.log('Frist member handle: ', member.isEmpty ? 'NONE' : member.handle.toString());

  const electionStage = await api.query.councilElection.stage();
  // Election stage will have typesafe helpers like isAnnouncing or asAnnouncing etc., since they are part
  // of the interface auto-generated by @polkadot/typegen (see: types/augment/all/types.ts)
  if (electionStage.isSome) {
    const stage = electionStage.unwrap();
    console.log('Election stage is:');
    console.log(stage.isAnnouncing ? `- Announcing (ends: ${stage.asAnnouncing.toString()})` : '- Not announcing');
    console.log(stage.isVoting ? `- Voting (ends: ${stage.asVoting.toString()})` : '- Not voting');
    console.log(stage.isRevealing ? `- Revealing (ends: ${stage.asRevealing.toString()})` : '- Not revealing');
  }

Future possabilities

I think that by making a few additional adjustments to the scripts we should be able to combine the benefits of using types/augment (api agugmentation based on runtime metadata) and types/augment-codec (using our custom getters etc. in types) by generating augment-api.ts inside types/augment-codec. This will require a few additional hours of work, but could be very beneficial before we decide to fully migrate to @polkadot/typegen (sacrificing our custom getters, but making @joystream/types trivial to manage)

[mnaamani]

So far tested generating the defs, but I used latest node build which removed the contracts module from the runtime and there was some renaming of AccountInfo -> ServiceProviderRecord, so this was reflected in:

modified:   types/augment/augment-api-consts.ts
modified:   types/augment/augment-api-query.ts
modified:   types/augment/augment-api-tx.ts

Perhaps we can check-in updated versions of these types (if latest iznik branch is merged into this types-augment branch) then we can confirm matching results.

[mnaamani]

is the json file correctly parsed despite the non-standard comments in json ?

[Lezek123]

Yes, comments seem to be allowed in tsconfig.json (related PR: microsoft/TypeScript#5450) even though it's technically not a valid json.

[Lezek123]

This file is actually used during the CI checks now (it's part of check:augment script which checks the validity of auto-generated files and is ran as the last step of checks)

[mnaamani]

So this is really great, tested the types definitions working nicely. I had problem using one of our modules though because of a name clash with another substrate module, the council So when using this:

const council = await api.query.council.activeCouncil()

typescript infers general Codec type for council, because the augment type is coming from @polkadot/api/augment/query.d.ts not our custom one. So forced to keep doing this:

const council = ((await api.query.council.activeCouncil()) as unknown) as Seat[]

I can't remember off the top of my head if there are other modules with identical names to a substrate module where this might happen as well. Is there a way to change the order of which type defs are looked at first with some magic ts-config settings?

The obvious solution would be to rename our own module, but since we don't even use the substrate council module it would be nicer to find a way to drop the council defs coming from the polkadot/api/augment/query.d.ts

Edit:
This behavior appeared only in VSCode. it seems on the command line the build is still successful.

[Lezek123]

I had problem using one of our modules though because of a name clash with another substrate module, the council

I cannot reproduce this issue. I'm using a project with those paths inside tsconfig.json (@joystream/types were added as a tarball):

    "baseUrl": ".",
    "paths": {
      "@polkadot/types/augment": ["./node_modules/@joystream/types/augment/augment-types.ts"],
      "@polkadot/api/augment": ["./node_modules/@joystream/types//augment/augment-api.ts"]
    },

The right types are inferred both by VSCode and tsc.

Sometimes changes in tsconfig.json require me to close the editor and start it again in order to be processed.

[mnaamani]

Ok so just an update, I added "@polkadot/api/augment": ["../../types/augment/augment-api.ts"] to the path, but also I had to open VSCode with the working dir where the root of that tsconfig file was to get it to work correctly

[mnaamani]

Sometimes changes in tsconfig.json require me to close the editor and start it again in order to be processed.

Ah yes, I also restarted VScode and that was what resolved it after modifying tsconfig file 👍

[mnaamani]

This is a fantastic PR making coding in the IDE much more pleasant now that all type definitions are available. Nice work.