Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI tool: recursive derives and attributes #1379

Merged
merged 70 commits into from
Jan 22, 2024

Conversation

tadeohepperle
Copy link
Contributor

Fixes #1353

This PR introduces an optional appendix of comma separated values to the CLI arguments derive-for-type and attributes-for-type. Right now, the appendix arguments are only checked for containing "recursive", but we could add more options in the future.

I verified that these commands work with and without the recursive flag:

subxt codegen --derive-for-type xcm::v2::multilocation::MultiLocation=Clone,recursive
subxt codegen --attributes-for-type "xcm::v2::multilocation::MultiLocation=#[foofoo],recursive"

tadeohepperle and others added 30 commits November 20, 2023 10:12
* config: Add `SkipCheckIfFeeless` signed extension

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Add extra extension to the default params

Signed-off-by: Alexandru Vasile <[email protected]>

* examples: Adjust signed extension example

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Extend SkipCheckIfFeeless with inner signed extension

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Configure SkipCheck with inner signed extension params

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Implement Deafult for SkipCheckIfFeelessParams with Option

Signed-off-by: Alexandru Vasile <[email protected]>

* examples: Fix example with proper extension

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Extend <T as Config>::AssetId with EncodeAsType and Clone

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Add SkipCheck with AssetTx

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Encode as type from metadata the inner signed extensions

Signed-off-by: Alexandru Vasile <[email protected]>

* Adjust examples

Signed-off-by: Alexandru Vasile <[email protected]>

* blocks: Use `SkipCheckIfFeeless` for decoding the tip of extensions

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Decode `SkipCheckIfFeeless` with `Self`

Signed-off-by: Alexandru Vasile <[email protected]>

* tests: Adjust testing

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Descriptive errors for building `SkipCheckIfFeeless`

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Add docs for extra error types

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Add extra derives to signed extensions

Signed-off-by: Alexandru Vasile <[email protected]>

* config: Use `Default::default` to simplify type init

Signed-off-by: Alexandru Vasile <[email protected]>

---------

Signed-off-by: Alexandru Vasile <[email protected]>
* lightclient: Make `smoldot::chainID` part of the RPC requests

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Make `BackgroundTask` generic over `PlatformRef` and chain

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Construct from raw smoldot and target different chains

Signed-off-by: Alexandru Vasile <[email protected]>

* testing: Update cargo lock for wasm tests

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Reuse `new_from_client` method and removed unused imports

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Reexport smoldot client and RPC objects used in pub
interface

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Adjust `new_from_client` interface

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Extend background to poll over multiple RPC objects

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Build light client from raw and target different chains

Signed-off-by: Alexandru Vasile <[email protected]>

* artifacts: Add demo chain specs

Signed-off-by: Alexandru Vasile <[email protected]>

* artifacts: Move artifacts to dedicated folder

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Use SelectAll to drive all streams

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Fetch initial data from the target chain

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Reexport other smoldot objects

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Target chain with potentially different config

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt/rpc: Log chainID for debugging

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt/examples: Add smoldot client with parachain example

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Propagate chain ID together with rpc responses object

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Multiplex responses by request ID and chain ID

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Add raw light client builder

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Add cargo feature flag for parachains example

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Derive default for internal structure

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Guard reexports by std feature flag

Signed-off-by: Alexandru Vasile <[email protected]>

* Update subxt/src/client/light_client/mod.rs

Co-authored-by: James Wilson <[email protected]>

* lightclient: Update the builder pattern and chain targetting

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Fix documentation

Signed-off-by: Alexandru Vasile <[email protected]>

* Provide more insightful docs wrt native/wasm panics

Signed-off-by: Alexandru Vasile <[email protected]>

* examples: Adjust comment location

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Refactor UniqueChainId into the background task

Signed-off-by: Alexandru Vasile <[email protected]>

* Update lightclient/src/background.rs

Co-authored-by: Niklas Adolfsson <[email protected]>

* Update subxt/src/client/light_client/builder.rs

Co-authored-by: James Wilson <[email protected]>

* lightclient: Update docs wrt panics

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Update docs wrt to smoldot instance -> client

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Use IntoIter instead of Iterator

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Adjsut docs wrt [`Self::new_from_client`]

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Remove RawRpc from LightClient in favor of chainID

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Reexport everything under smoldot module

Signed-off-by: Alexandru Vasile <[email protected]>

* artifacts: Use stateRootHash instead of genesis.raw

Signed-off-by: Alexandru Vasile <[email protected]>

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: James Wilson <[email protected]>
Co-authored-by: Niklas Adolfsson <[email protected]>
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.28 to 0.3.29.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](rust-lang/futures-rs@0.3.28...0.3.29)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [zeroize](https://github.com/RustCrypto/utils) from 1.6.0 to 1.7.0.
- [Commits](https://github.com/RustCrypto/utils/commits)

---
updated-dependencies:
- dependency-name: zeroize
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.17 to 0.3.18.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](tokio-rs/tracing@tracing-subscriber-0.3.17...tracing-subscriber-0.3.18)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.17 to 0.3.18.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](tokio-rs/tracing@tracing-subscriber-0.3.17...tracing-subscriber-0.3.18)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [getrandom](https://github.com/rust-random/getrandom) from 0.2.10 to 0.2.11.
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md)
- [Commits](rust-random/getrandom@v0.2.10...v0.2.11)

---
updated-dependencies:
- dependency-name: getrandom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* impl RpcClientT for Arc<WsClient>

* fix grumbles: impl for Box<T> and Arc<T>

* grumbles: move RpcClientT impls
* improve `substrate-compat`

* From => Into

---------

Co-authored-by: James Wilson <[email protected]>
Bumps [proc-macro2](https://github.com/dtolnay/proc-macro2) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/dtolnay/proc-macro2/releases)
- [Commits](dtolnay/proc-macro2@1.0.69...1.0.70)

---
updated-dependencies:
- dependency-name: proc-macro2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.192 to 1.0.193.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.192...v1.0.193)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* lightclient: Close wasm socket while dropping from connecting state

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Construct one time only closures

Signed-off-by: Alexandru Vasile <[email protected]>

* testing: Enable console logs for lightclient WASM testing

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Separate wakes and check connectivity on poll_read

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Close the socket depending on internal state

Signed-off-by: Alexandru Vasile <[email protected]>

* Revert "lightclient: Separate wakes and check connectivity on poll_read"

This reverts commit 8660940.

* lightclient: Return pending if socket is opening from poll_read

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Close the socket on `poll_close`

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Reset closures on Drop to avoid recursive invokation

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Close the socket if not already closing

Signed-off-by: Alexandru Vasile <[email protected]>

---------

Signed-off-by: Alexandru Vasile <[email protected]>
* cli: Add chainSpec command

Signed-off-by: Alexandru Vasile <[email protected]>

* cli/chainSpec: Move to dedicated module

Signed-off-by: Alexandru Vasile <[email protected]>

* cli: Compute the state root hash

Signed-off-by: Alexandru Vasile <[email protected]>

* cli: Remove code substitutes

Signed-off-by: Alexandru Vasile <[email protected]>

* artifacts: Update polkadot.json

Signed-off-by: Alexandru Vasile <[email protected]>

* scripts: Generate the chain spec

Signed-off-by: Alexandru Vasile <[email protected]>

* cli: Remove testing artifacts

Signed-off-by: Alexandru Vasile <[email protected]>

* cli: Fix clippy

Signed-off-by: Alexandru Vasile <[email protected]>

* cli: Apply rustfmt

Signed-off-by: Alexandru Vasile <[email protected]>

* cli: Introduce feature flag for smoldot dependency

Signed-off-by: Alexandru Vasile <[email protected]>

* cli: Rename chain-spec to chain-spec-pruning

Signed-off-by: Alexandru Vasile <[email protected]>

* scripts: Update chain-spec command

Signed-off-by: Alexandru Vasile <[email protected]>

---------

Signed-off-by: Alexandru Vasile <[email protected]>
jsdw and others added 17 commits January 18, 2024 17:07
* rpcmethods

* followstr

* mod and else
…1352)

* github CI action cronjob

* add commit message

* fix the CI yml files

* binary crate for CI script with substrate-runner

* update the CI script

* correct the artifacts script

* remove bash script
…old (#1372)

* testing(fix): Ensure lightclient chainSpec is at least one block old

Signed-off-by: Alexandru Vasile <[email protected]>

* Revert "testing(fix): Ensure lightclient chainSpec is at least one block old"

This reverts commit 0eafcb2.

* lightclient(fix): Ensure lightclient chainSpec is at least one block old

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Link smoldot issue

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Use tokio under lightclient feature flag

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Do not sleep on errors to fetch the chainSpec

Signed-off-by: Alexandru Vasile <[email protected]>

* artifacts: Remove test file

Signed-off-by: Alexandru Vasile <[email protected]>

* lightclient: Subscribe to two finalized blocks

Signed-off-by: Alexandru Vasile <[email protected]>

* subxt: Revert cargo toml

Signed-off-by: Alexandru Vasile <[email protected]>

---------

Signed-off-by: Alexandru Vasile <[email protected]>
* ci: Reduce the light client timpeut to 15 seconds

Signed-off-by: Alexandru Vasile <[email protected]>

* ci: Use ubuntu-latest for light-client tests

Signed-off-by: Alexandru Vasile <[email protected]>

---------

Signed-off-by: Alexandru Vasile <[email protected]>
* asset hub example and book adjustment

* formatting

* recursive derives

* polkadot monitor example and book adjustments

* formatting

* adjust docs and examples, add dynamic example

* james suggestions

* fmt

* chore(subxt/src): typo fix (#1370)

* rpcmethods

* followstr

* mod and else

* Weekly Cronjob fetching artifacts and generating polkadot.rs file. (#1352)

* github CI action cronjob

* add commit message

* fix the CI yml files

* binary crate for CI script with substrate-runner

* update the CI script

* correct the artifacts script

* remove bash script

---------

Co-authored-by: James Wilson <[email protected]>
Co-authored-by: Pan chao <[email protected]>
@tadeohepperle tadeohepperle requested a review from a team as a code owner January 19, 2024 09:21
@jsdw
Copy link
Collaborator

jsdw commented Jan 19, 2024

Looks good to me, but it's branched off #1290 I think so hard to tell if I'm missing anything right now.

Probably best to avoid doing too much on top of large PRs until they merge, because it becomes a pain all of a sudden if said PRs have comments that need addressing, though in this case it's fairly small :)

@tadeohepperle
Copy link
Contributor Author

Looks good to me, but it's branched off #1290 I think so hard to tell if I'm missing anything right now.

Yeah, I did not know you guys would be so fast with the review, #1290 was so close to being merged, so I based it on it to avoid merge conflicts later :)

Copy link
Collaborator

@jsdw jsdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

str_equals_str_comma_maybe_recursive_parser is a strange name, but I don't have any great suggestions for it!

}

/// Parses a `&str` of the form `str1=str2` into `(str1, str2, false)` or `str1=str2,recursive` into `(str1, str2, true)`.
fn str_equals_str_comma_maybe_recursive_parser(src: &str) -> Option<(String, String, bool)> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably have a few unit-tests for this. Would str1=str2,a,b,c,recursive also be a valid syntax?

I'd also probably rename this to type_parser or something similar

Copy link
Collaborator

@lexnv lexnv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one! 👍

Would be good to have a few test scenarios for str_equals_str_comma_maybe_recursive_parser to make sure we only capture the intended form, overall looks good!

Comment on lines 265 to 269
assert_eq!(type_map_parser("Foo=Bar,a"), Some(("Foo", "Bar", false)));
assert_eq!(
type_map_parser("Foo=Bar,a,b,c,recursive"),
Some(("Foo", "Bar", true))
);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we reject this format? 🤔

Maybe we could also check that we expect this function to panic on these inputs since they are invalid

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, let's reject it. To keep the possibility open to add other attributes (similar to recursive) in the future I implemented a simple match over the trailing attributes:

let mut recursive = false;
for r in split_rest {
    match r {
        "recursive" => {
            recursive = true;
        }
        _ => return None,
    }
}

@tadeohepperle tadeohepperle merged commit 65dde11 into master Jan 22, 2024
11 checks passed
@tadeohepperle tadeohepperle deleted the tadeo-hepperle-cli-recursive-derives branch January 22, 2024 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CLI tool codgen command: How to apply recursive derives and attributes?
6 participants