redefining responsibilities of serve.ts

[harrysolovay]

Currently, running serve.ts starts a server from which one can get request / import on-the-fly codegened TypeScript files (also implements import spec completions prescribed by the Deno LS intellisense API). Capi also houses three approaches to running dev nets:

1. run test_util/ctx.ts can be run––this allows multiple processes to share / simultaneously interact with the same dev net(s)
2. spawn the test chain as a subprocess
3. Via zombienet (this approach still needs to be integrated into the other test utils)

I propose that we blur the line between the codegen and testing servers. The standard path for development would still be to run the serve.ts of the desired Capi version. This is comparable to react-scripts in create-react-app; the user runs a single command to spin up an optimal development environment. In the case of create-react-app, one spins up a server that handles hot-module reloading, babel transpilation, etc. In our case, we spin up a server that handles launching test networks, generating chain-specific TypeScript files from a number of discovery value types (both remote and local), import completion and––of course––serving the desired version of Capi (should be the same as the version of the serve.ts in use (?)).

Running serve.ts locally means that the codegen server can interact with the user's file system, to for instance scan the file tree for chainspecs, zombienet configs, smart contract wasm+metadata, etc. capi.dev would still exist, but it would allow only a subset of discovery values and lack test chain management.

If we agree on this direction, a follow up question is: what kinds of discovery values / import specifiers do we want to support?

In the case of chain specs? Maybe...

"http://localhost:5646/chainspec/../path/to/chainspec.json/mod.ts"
"http://localhost:5646/chainspec/https://raw.githubusercontent.com/paritytech/substrate-connect/main/packages/connect/src/connector/specs/polkadot.json/mod.ts"

Do we want to allow reference to scale bytes? This would be more in line with Subxt's codegen.

"http://localhost:5646/scale_bytes/../path/to/metadata.scale/mod.ts"

How about allowing serve.ts to launch zombienets?

"http://localhost:5646/zombienet/../path/to/zombienet.toml#some_parachain/mod.ts"

How about an arbitrary bin?

"http://localhost:5646/bin/../path/to/substrate_chain_bin/mod.ts"

... and what might we want to do in the case of contracts? @kratico

Some other questions:

• Do we even need to specify the discovery value type (such as proxy), or can we infer it?
• How do we treat runtime versions, considering that some discovery values are (kind of) unversioned? (conclusion from offline conversation with @tjjfvi: chain-specific versions in URLs are important/expected)
• Should the local server read the file system to provide discovery value completions?
• Given that we'd like to decouple the codegen from the discovery value utilized at runtime, how might we sometimes (as in the case with zombienets) inject a runtime dicovery value, while leaving it empty in other cases (such as pointing to raw scale bytes)?

[kratico]

For a deployed contracts we need 3 things

• the chain where the contract is deployed
• contract address
• contract metadata

Given that it may be complex to encode many of these values in a url, perhaps we can consider having a mapping file similar to import_map.json.

Given that we'd like to decouple the codegen from the discovery value utilized at runtime, how might we sometimes (as in the case with zombienets) inject a runtime dicovery value, while leaving it empty in other cases (such as pointing to raw scale bytes)?

The zombienet .toml file can interpolate environment variables (for example, an RPC endpoint port or a node name), so we could encode variables in the import url and launch the test network configured by the import url.

[harrysolovay]

To reiterate an offline convo with @tjjfvi:

• The hosted codegen server––currently capi.dev––will be moved into a dedicated repo. Capi will be a dependency of that repo.
• This repo will still contain a server, but it will not support an import scheme that contains Capi version (as we can assume that the desired Capi version is the same as the running server).
• Capi will contain a server lib, with a provider model. This is to enable different kinds of discovery values. Ie.

const ctx = new Ctx({
  wss: new WsProvider(),
  dev: new DevProvider(),
})

This context (containing methods for controlling providers / getting codegen results) would allow one to utilize both WS proxy URLs and Polkadot CLI dev runtime names as discovery values.

Via URL:

import {} from "http://localhost:8000/wss/rpc:rpc.polkadot.io/@v0.9.36/mod.ts";

Via dev:

import {} from "http://localhost:8000/dev/polkadot/@v0.9.36/mod.ts";

The hosted codegen server will not support spinning up test networks. Hence, it's context would lack the dev provider.

const ctx = new Ctx({
  wss: new WsProvider(),
- dev: new DevProvider(),
})

Other providers will include:

SmoldotProvider

import {} from "http://localhost:8000/smoldot/path/to/chainspec.json/@v0.9.36/mod.ts";

ZombienetProvider

import {} from "http://localhost:8000/zombienet/path/to/config.toml/chainId/@v0.9.36/mod.ts";

And... perhaps...

WorkspaceProvider

import {} from "http://localhost:8000/workspace/path/to/Cargo.toml/@v0.9.36/mod.ts";

This provider could specify a Substrate workspace. Under-the-hood, the Capi codegen server could compile and run the code in dev mode (as well as restart upon changes to the Rust code... just a thought)

---

There's also the question of adding an additional subpath to specify codegen type. This may help if/when we tackle Ink-contract-related codegen.

- import {} from "http://localhost:8000/dev/polkadot/@v0.9.36/mod.ts";
+ import {} from "http://localhost:8000/frame/dev/polkadot/@v0.9.36/mod.ts";

Then, ink imports:

import { Contract } from "http://localhost:8000/ink/path/to/ink_metadata.json/mod.ts";

declare address: string;
declare code: undefined | Uint8Array

const contract = new Contract({ address, code });

[kratico]

For the smoldot provider, to connect to a parachain we also need the relay chain spec.
So the url could be something like

import {} from "http://localhost:8000/smoldot/path/to/relay-chainspec.json/@v0.x.y/path/to/para-chainspec.json/@v0.a.b/mod.ts";

[harrysolovay]

Very good point! Parachain chainspecs don't encapsulate the chainspecs of their relay chains. Your proposed import URL is great, given that the first @ can be used to determine where to split the two paths. Nice.