testing: Prepare light client testing with substrate binary and add subxt-test macro #1507

lexnv · 2024-03-28T11:45:43Z

In this PR

subxt-macro is added. The macro runs the tests with a default timeout (6 minutes if not specified)
added cfg aliases for our testing with a new unstable-light-client-long-running feature flag to reuse subxt testing with the lightclient
some tests are disabled in the context of the light-client
chainHead initialized event supports the legacy format (to make it work with our outdated lightclient version)
small testing refactoring to decouple the construction of transaction from the submit operation (might still be needed in the future if we want to retry failed light-client operations)

Builds on top of: #1459 (because the CI was not running anymore)

Signed-off-by: Alexandru Vasile <[email protected]>

This reverts commit ea6f3cc.

Signed-off-by: Alexandru Vasile <[email protected]>

…testing Signed-off-by: Alexandru Vasile <[email protected]>

Signed-off-by: Alexandru Vasile <[email protected]>

…testing-macro

Signed-off-by: Alexandru Vasile <[email protected]>

jsdw · 2024-04-02T10:40:03Z

testing/integration-tests/proc-macro/Cargo.toml

@@ -0,0 +1,20 @@
+[package]
+name = "subxt-test-proc-macro"


Nit: also rename the folder to subxt-test-proc-macro (or subxt-test-macro if you prefer to shorten this crate name)?

jsdw · 2024-04-02T10:46:29Z

testing/integration-tests/src/utils/tx_retries.rs

+        }
+    }
+
+    submit().await


So, if using the light client, we try submitting a couple of additional times to handle light client specific issues?

I wonder if there's anything we can do to avoid needing to do this; does needing it imply that using the light client to submit transactions now is not that robust and will fail sometimes? Is there anything we can do before we try submitting that might help things (ie wait a few blocks), so that by the time we submit we don't need anything special? Is this just because of the legacy backend and everything works ok with the unstable backend (and in that case I think I'd only try testing the unstable backend with the light client)?

If we do need to keep this function then I think it'd be worth adding some docs to explain why it's used for future us :)

Yep that makes sense. I've removed the tx_retries for now.
Instead maybe we should report those tx::Dropped events to smoldot if they still happen after an upgrade.

Signed-off-by: Alexandru Vasile <[email protected]>

jsdw

LGTM!

Is the effect of removing that function that some tests will fail sometimes now when using the light client?

lexnv · 2024-04-03T16:23:47Z

When we run tests with the unstable-light-client-long-running we'll probably encounter more flaky failures to due TransactionEvent::Dropped. However, that is not enabled now in the CI and I would generally wait to enable that until we have: #1400.

I'll look into that issue shortly, hopefully we could handle that and create a PR in smoldot if the fix is not too involved

niklasad1 · 2024-04-04T14:26:40Z

subxt/src/backend/unstable/rpc_methods.rs

+        }
+
+        let ir = InitializedIR::deserialize(deserializer)?;
+        let finalized_block_hashes = ir


slightly weird comment; we could actually throw an error if both finalized_block_hash and finalized_block_hashes.

now, we checking finalized_block_hashes first and if that doesn't exist we fallback to finalized_block_hash which is probably fine but if one inserts both by accident it lead to confusing scenarios but fine :)

I know this only exists until we update smoldot :)

niklasad1 · 2024-04-04T14:31:26Z

testing/integration-tests/src/full_client/blocks/mod.rs

+
+#[cfg(fullclient)]
+use crate::utils::node_runtime;
+#[cfg(fullclient)]


is it possible to merge the use stuff to avoid repeating #[cfg(fullclient)]?

niklasad1 · 2024-04-04T14:37:19Z

testing/integration-tests/subxt-test-macro/src/lib.rs

+const SUBXT_TEST_TIMEOUT: &str = "SUBXT_TEST_TIMEOUT";
+
+/// Default timeout for the test.
+const DEFAULT_TIMEOUT: u64 = 60 * 6;


would be nicer/more readable to use Duration here IMO

We would still need to pass std::time::Duration to the quote! below, which does not impl toTokens , I've renamed this to illustrate better that we are counting seconds, thanks!

niklasad1

awesome, I really like this

Signed-off-by: Alexandru Vasile <[email protected]>

lexnv added 30 commits February 29, 2024 15:24

testing: Add long running light client flag and cfg aliases

34e9222

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Expose clients depending on feature flags

518aa8e

Signed-off-by: Alexandru Vasile <[email protected]>

subxt: Use unstable backend for light client

ea6f3cc

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Disable flaky lightclient tests

ca36ef6

Signed-off-by: Alexandru Vasile <[email protected]>

ci: Add long runnnig step

8fe7e86

Signed-off-by: Alexandru Vasile <[email protected]>

Revert "subxt: Use unstable backend for light client"

593fbeb

This reverts commit ea6f3cc.

ci: Long running tests for 60 mins

1fc1bb4

Signed-off-by: Alexandru Vasile <[email protected]>

ci: Use 16 cores for light-client testing

ca60c31

Signed-off-by: Alexandru Vasile <[email protected]>

ci: Isolate light-client testing to save CI minutes

3c58c5e

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Retry on Tx::Dropped for lightclinet only

0248343

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Wait for more blocks for the lightclient init

8400a0e

Signed-off-by: Alexandru Vasile <[email protected]>

subxt: Use unstable backend for light client

d671664

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Disable legacy RPC tests

318ba5c

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Disable sudo and contracts tests

1a6f1af

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Retry constructing lightclient on read-proof errors

332fa72

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Disable tx dynamic test

88ccc54

Signed-off-by: Alexandru Vasile <[email protected]>

proc-macro: Timeout for tests

b6c98c7

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Add timeout 800 seconds

20e75bc

Signed-off-by: Alexandru Vasile <[email protected]>

proc-macro/tests: Adjust subxt-test proc-macro

c5c7972

Signed-off-by: Alexandru Vasile <[email protected]>

proc-macro: Rename crate to subxt-test-proc-macro

d3923c8

Signed-off-by: Alexandru Vasile <[email protected]>

Use default subxt-proc-macro timeout

1a64750

Signed-off-by: Alexandru Vasile <[email protected]>

light-client: Remove println

4c3226e

Signed-off-by: Alexandru Vasile <[email protected]>

subxt: Remove tokio as dependency, use it only for testing

fbe2a74

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Chagne default timeout to 6 seconds

30073d3

Signed-off-by: Alexandru Vasile <[email protected]>

proc-macro: Add env timeout variable

aa55cea

Signed-off-by: Alexandru Vasile <[email protected]>

ci: Add subxt env var for controling test timeouts

09277fe

Signed-off-by: Alexandru Vasile <[email protected]>

tests/tx-retries: Retry on Non node available error

5039f1c

Signed-off-by: Alexandru Vasile <[email protected]>

Merge remote-tracking branch 'origin/master' into lenxv/light-client-…

410aaed

…testing Signed-off-by: Alexandru Vasile <[email protected]>

testing: Use unstable backend for testing lightclient

d81c70c

Signed-off-by: Alexandru Vasile <[email protected]>

testing: Remove old lightclient object

00c4b78

Signed-off-by: Alexandru Vasile <[email protected]>

lexnv added 3 commits March 26, 2024 18:18

testing: Adjust for the new interface

03da904

Signed-off-by: Alexandru Vasile <[email protected]>

backend/rpc: Allow older version of the initialized event

292f971

Signed-off-by: Alexandru Vasile <[email protected]>

rpc/tests: Check initialized decodes correctly

2bc8947

Signed-off-by: Alexandru Vasile <[email protected]>

lexnv requested review from a team as code owners March 28, 2024 11:45

lexnv added 6 commits March 28, 2024 13:47

ci: Reset workflow

811f70c

Signed-off-by: Alexandru Vasile <[email protected]>

Apply cargo fmt

2bb8acb

Signed-off-by: Alexandru Vasile <[email protected]>

Remove unused dep

1fdd734

Signed-off-by: Alexandru Vasile <[email protected]>

Merge remote-tracking branch 'origin/master' into lexnv/light-client-…

1bc6eb9

…testing-macro

Remove gitmerge old file

5692703

Signed-off-by: Alexandru Vasile <[email protected]>

Remove unused dep

fd47be1

Signed-off-by: Alexandru Vasile <[email protected]>

jsdw reviewed Apr 2, 2024

View reviewed changes

lexnv added 3 commits April 2, 2024 16:11

rename proc-macro to subxt-test-macro

be4b34c

Signed-off-by: Alexandru Vasile <[email protected]>

tests: Remove txretries for lightclient

7270da8

Signed-off-by: Alexandru Vasile <[email protected]>

tests: Wait for 5 blocks for the lightclient full testing suite

a62ceb7

Signed-off-by: Alexandru Vasile <[email protected]>

lexnv changed the title ~~testing: Test light client with substrate binary~~ testing: Prepare light client testing with substrate binary and add subxt-test macro Apr 3, 2024

jsdw approved these changes Apr 3, 2024

View reviewed changes

niklasad1 reviewed Apr 4, 2024

View reviewed changes

niklasad1 approved these changes Apr 4, 2024

View reviewed changes

lexnv added 2 commits April 8, 2024 10:58

tests: Group imports

7664b56

Signed-off-by: Alexandru Vasile <[email protected]>

macro: Rename const value

510578f

Signed-off-by: Alexandru Vasile <[email protected]>

lexnv merged commit b31131d into master Apr 8, 2024
13 checks passed

lexnv deleted the lexnv/light-client-testing-macro branch April 8, 2024 08:34

jsdw mentioned this pull request May 16, 2024

Prep for 0.36.0 release #1587

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

testing: Prepare light client testing with substrate binary and add subxt-test macro #1507

testing: Prepare light client testing with substrate binary and add subxt-test macro #1507

lexnv commented Mar 28, 2024 •

edited

Loading

jsdw Apr 2, 2024 •

edited

Loading

jsdw Apr 2, 2024 •

edited

Loading

lexnv Apr 3, 2024

jsdw left a comment •

edited

Loading

lexnv commented Apr 3, 2024

niklasad1 Apr 4, 2024

niklasad1 Apr 4, 2024

niklasad1 Apr 4, 2024

niklasad1 Apr 4, 2024 •

edited

Loading

lexnv Apr 8, 2024

niklasad1 left a comment •

edited

Loading

testing: Prepare light client testing with substrate binary and add subxt-test macro #1507

testing: Prepare light client testing with substrate binary and add subxt-test macro #1507

Conversation

lexnv commented Mar 28, 2024 • edited Loading

jsdw Apr 2, 2024 • edited Loading

Choose a reason for hiding this comment

jsdw Apr 2, 2024 • edited Loading

Choose a reason for hiding this comment

lexnv Apr 3, 2024

Choose a reason for hiding this comment

jsdw left a comment • edited Loading

Choose a reason for hiding this comment

lexnv commented Apr 3, 2024

niklasad1 Apr 4, 2024

Choose a reason for hiding this comment

niklasad1 Apr 4, 2024

Choose a reason for hiding this comment

niklasad1 Apr 4, 2024

Choose a reason for hiding this comment

niklasad1 Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

lexnv Apr 8, 2024

Choose a reason for hiding this comment

niklasad1 left a comment • edited Loading

Choose a reason for hiding this comment

lexnv commented Mar 28, 2024 •

edited

Loading

jsdw Apr 2, 2024 •

edited

Loading

jsdw Apr 2, 2024 •

edited

Loading

jsdw left a comment •

edited

Loading

niklasad1 Apr 4, 2024 •

edited

Loading

niklasad1 left a comment •

edited

Loading