Multiple threads using the same client, transaction outdated

[th7nder]

Context

We're building a system parachain for data storage and heavily utilizing subxt.
We have a off-chain service, which is running a pipeline and RPC at the same time.
Pipeline periodically performs a operation on-chain (and waits for finalization), and RPC as well.
We just put a OnlineClient behind an and are using it in multiple places (threads).
E.g:

1. pipeline when it calls an extrinsic (ref)
2. RPC when it also calls an extrinsic (ref)

Problem

There is a moment, where pipeline (1.) waits for the extrinsic to be finalized and RPC (2.) wants to submit the extrinsic, using the same client.
It ends up being an error:

Error: RpcClient(Call(ErrorObject { code: InternalError, message: "RPC error: ErrorObject { code: ServerError(1014), message: \"Priority is too low: (17955 vs 11737)\", data: Some(RawValue(\"The transaction has too low priority to replace another transaction already in the pool.\")) }", data: None }))

I think this is because, while we waiting for the extrinsic to be finalized, the Account Nonce is the same for both of the calls. The second call tries to override it (has a lower priority and it fails)

Discussion

Potential solutions from our side are:

• put the client behind a Mutex and unlock only after finalization (tragically slow, DoS attacks, etc.)
• create some kind of queue that will be processing the chain request in-order (simpler, but not that fast)
• keep track of the nonce in a shared state and increment it (complex, but fast)

However, I wanna discuss this with you guys. Is this the right approach? Should we handle it on our side, should the client be resilient on its own for this kind of behaviours?

Looking for your feedback, thanks!

P.S couldn't find the better title, feel free to change it to something more fitting.

[niklasad1]

Hey hey, if you are using the default settings of ExtrinsicParams in subxt such as sign_and_submit_then_watch_default then subxt will fetch the latest nonce for the given account when the extrinsic is constructed and signed. If you are doing that concurrently what could happen is that you may use the same nonce more than if the transaction hasn't been sent yet.

It's likely that the RPC layer is slower then constructing extrinsic that causes that I guess..

keep track of the nonce in a shared state and increment it (complex, but fast)

I think this is the easiest to solution to this issue is ☝️ and just keep track of your nonce with a Arc<AtomicUsize> and increment it for every extrinsic and set it explicit in subxt. You probably need to fetch the nonce at start-up and then just use your own state (assuming that no other app is using that account)

However, you may have already come up with a better solution by now.

[th7nder]

Hey @niklasad1, thanks for your inputs.
Yeah, we ended up doing it a bit different.

The main drawback of having it in the shared state was that if an extrinsic fail, all of them later would fail too, as the sequence needs to be continuous.

We ended up:

• using system_account_next_index which gets latest nonce with the transaction pool in mind. Before submitting an extrinsic, we query for the nonce from the node.
  • https://github.com/eigerco/polka-storage/blob/7789d2bb56cb29e5f237cad7fc498e0b075d6317/storagext/lib/src/runtime/client.rs#L208
• using fork-aware tx pool, so it's working across the reorgs on the parachain
  • https://github.com/eigerco/polka-storage/blob/7789d2bb56cb29e5f237cad7fc498e0b075d6317/zombienet/local-testnet.toml#L24
  • fork-aware transaction pool added polkadot-sdk#4639