`fatxpool`: limits handling optimizations and fixes by michalkucharczyk · Pull Request #8596 · paritytech/polkadot-sdk

michalkucharczyk · 2025-05-21T13:26:24Z

This PR adds some optimization and fixes in handling limits in fork-aware transaction pool.

Notes for reviewers

Changes made (random order):

debug levels adjusted in numerous places places ( debug -> trace for tx, trace -> debug for general flow) for better readablity,
internal TxMemPool storage is now sorted. A new helper exposes methods to reduce number of transaction clones (e.g. here, see previous verions). This new structure eliminates the necessity of sorting transactions on the fly which was a not efficient, naive, first implementation.
some mutexes were migrated to tokio::sync::Mutex to avoid tokio threads locking,
sync to async message based bridge was implemented. It's purpose is mainly to support LocalTransactionPool / OffchainTransactionPool infrastucture. sync methods can be called from both non-tokio and tokio context. This requires one additional blocking task for transaction pool.
ViewStore::most_recent_view is now a reference to View.
TXMEMPOOL_TRANSACTION_LIMIT_MULTIPLIER removed, there is no point for buffering more. Initially intended to work as buffer accommodating transactions from two different full views (which could have different set of transactions), turned out to be a bottleneck in maintain function (it still is but aligning sizes reduced the impact),
bug 🙈 fixed in pre-insert actions removal
ValidateTransactionPriority was added. The goal is to allow faster processing of validation requests that were made from the maintain context. Otherwise all requests were landing in the same queue and maintain requests could be delayed. Now the processing power is evenly 50/50 split between maintain and submit+revalidate context.

Related work:

Todo:

some run should be done also for parachain,

substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool/tx_mem_pool_map.rs

substrate/client/transaction-pool/Cargo.toml

substrate/client/network/src/litep2p/shim/notification/config.rs

substrate/client/network/src/protocol/notifications/handler.rs

substrate/client/network/transactions/src/config.rs

substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs

substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs

paritytech-workflow-stopper · 2025-06-24T11:50:39Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/15849446511
Failed job name: build-rustdoc

paritytech-release-backport-bot · 2025-06-24T19:32:46Z

Successfully created backport PR for stable2506:

[stable2506] Backport #8596 #8970

This PR adds some optimization and fixes in handling limits in fork-aware transaction pool. #### Notes for reviewers Changes made (random order): - debug levels adjusted in numerous places places ( `debug -> trace` for tx, `trace -> debug` for general flow) for better readablity, - internal TxMemPool [storage](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L268-L272) is now [sorted](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool/tx_mem_pool_map.rs#L123-L132). A new helper exposes methods to reduce number of transaction clones (e.g. [here](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1404-L1410), see [previous verions](https://github.com/paritytech/polkadot-sdk/blob/2863b7a9a879935ff16987d0e95065d088dad9f8/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1352-L1359)). This new structure eliminates the necessity of sorting transactions on the fly which was a not efficient, naive, [first](https://github.com/paritytech/polkadot-sdk/blob/e44b89fb7ca3385f314803c733ad97b26cd14e9f/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L384-L400) implementation. - some _mutexes_ were migrated to `tokio::sync::Mutex` to avoid tokio threads locking, - `sync` to `async` _[message](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L743) based_ bridge was implemented. It's purpose is mainly to support `LocalTransactionPool / OffchainTransactionPool` infrastucture. `sync` methods can be called from both non-tokio and tokio context. This requires one additional [blocking](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L398-L402) task for transaction pool. - `ViewStore::most_recent_view` is now a [reference](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L171) to `View`. - `TXMEMPOOL_TRANSACTION_LIMIT_MULTIPLIER` removed, there is no point for buffering more. Initially intended to work as buffer accommodating transactions from two different full views (which could have different set of transactions), turned out to be a bottleneck in maintain function (it still is but aligning sizes reduced the impact), - [bug](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L769) :see_no_evil: fixed in pre-insert actions removal - [`ValidateTransactionPriority`](https://github.com/paritytech/polkadot-sdk/blob/045bc6d342620a02ee9b28d8de51f72ae680f06f/substrate/client/transaction-pool/src/graph/pool.rs#L64-L71) was added. The goal is to allow faster processing of validation requests that were made from the `maintain` context. Otherwise all requests were landing in the same queue and maintain requests could be delayed. Now the processing power is evenly 50/50 split between _maintain_ and _submit+revalidate_ context. Related work: - michalkucharczyk/tx-test-tool#42, - #8152, Todo: - [x] some run should be done also for parachain, --------- Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> (cherry picked from commit 6efe9f5)

[`TransactionPool::report_invalid`](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/api/src/lib.rs#L314) is now `async`, function was typically used in async context, seems right to be fully async. _Note_: expected to be merged after: #8596 --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Backport #8596 into `stable2506` from michalkucharczyk. See the [documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md) on how to use this bot.  --------- Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Egor_P <egor@parity.io>

…addressed (#2559) # Goal The goal of this PR is to upgrade polkadot version to 2506-1 Following changes took place - Host functions require some safe marshalling. - xcm and assets weights updated to match latest interface. - `RuntimeEvent` in `Config` is being made redundant ( haven't removed it but added allow deprecated for now) - Updated tomls - Suppressed a warning around `runtume_interface` that rust was warning about but it's polkadot design choice. - Update node layer with updated functions and introduce new arguments - update xcm configs in runtime with latest trait - address clippy errors Polkadot Notes on migration to [2506](https://github.com/paritytech/polkadot-sdk/blob/master/prdoc/stable2506) Closes #2521 # Discussion * Polkadot fixes towards transaction pool paritytech/polkadot-sdk#8596 # Checklist - [ ] Updated Pallet Readme? - [ ] Updated js/api-augment for Custom RPC APIs? - [ ] Design doc(s) updated? - [ ] Unit Tests added? - [ ] e2e Tests added? - [ ] Benchmarks added? - [x] Spec version incremented?

This PR adds some optimization and fixes in handling limits in fork-aware transaction pool. Changes made (random order): - debug levels adjusted in numerous places places ( `debug -> trace` for tx, `trace -> debug` for general flow) for better readablity, - internal TxMemPool [storage](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L268-L272) is now [sorted](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool/tx_mem_pool_map.rs#L123-L132). A new helper exposes methods to reduce number of transaction clones (e.g. [here](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1404-L1410), see [previous verions](https://github.com/paritytech/polkadot-sdk/blob/2863b7a9a879935ff16987d0e95065d088dad9f8/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1352-L1359)). This new structure eliminates the necessity of sorting transactions on the fly which was a not efficient, naive, [first](https://github.com/paritytech/polkadot-sdk/blob/e44b89fb7ca3385f314803c733ad97b26cd14e9f/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L384-L400) implementation. - some _mutexes_ were migrated to `tokio::sync::Mutex` to avoid tokio threads locking, - `sync` to `async` _[message](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L743) based_ bridge was implemented. It's purpose is mainly to support `LocalTransactionPool / OffchainTransactionPool` infrastucture. `sync` methods can be called from both non-tokio and tokio context. This requires one additional [blocking](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L398-L402) task for transaction pool. - `ViewStore::most_recent_view` is now a [reference](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L171) to `View`. - `TXMEMPOOL_TRANSACTION_LIMIT_MULTIPLIER` removed, there is no point for buffering more. Initially intended to work as buffer accommodating transactions from two different full views (which could have different set of transactions), turned out to be a bottleneck in maintain function (it still is but aligning sizes reduced the impact), - [bug](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L769) :see_no_evil: fixed in pre-insert actions removal - [`ValidateTransactionPriority`](https://github.com/paritytech/polkadot-sdk/blob/045bc6d342620a02ee9b28d8de51f72ae680f06f/substrate/client/transaction-pool/src/graph/pool.rs#L64-L71) was added. The goal is to allow faster processing of validation requests that were made from the `maintain` context. Otherwise all requests were landing in the same queue and maintain requests could be delayed. Now the processing power is evenly 50/50 split between _maintain_ and _submit+revalidate_ context. Related work: - michalkucharczyk/tx-test-tool#42, - paritytech#8152, Todo: - [x] some run should be done also for parachain, --------- Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This PR adds some optimization and fixes in handling limits in fork-aware transaction pool. #### Notes for reviewers Changes made (random order): - debug levels adjusted in numerous places places ( `debug -> trace` for tx, `trace -> debug` for general flow) for better readablity, - internal TxMemPool [storage](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L268-L272) is now [sorted](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool/tx_mem_pool_map.rs#L123-L132). A new helper exposes methods to reduce number of transaction clones (e.g. [here](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1404-L1410), see [previous verions](https://github.com/paritytech/polkadot-sdk/blob/2863b7a9a879935ff16987d0e95065d088dad9f8/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1352-L1359)). This new structure eliminates the necessity of sorting transactions on the fly which was a not efficient, naive, [first](https://github.com/paritytech/polkadot-sdk/blob/e44b89fb7ca3385f314803c733ad97b26cd14e9f/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L384-L400) implementation. - some _mutexes_ were migrated to `tokio::sync::Mutex` to avoid tokio threads locking, - `sync` to `async` _[message](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L743) based_ bridge was implemented. It's purpose is mainly to support `LocalTransactionPool / OffchainTransactionPool` infrastucture. `sync` methods can be called from both non-tokio and tokio context. This requires one additional [blocking](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L398-L402) task for transaction pool. - `ViewStore::most_recent_view` is now a [reference](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L171) to `View`. - `TXMEMPOOL_TRANSACTION_LIMIT_MULTIPLIER` removed, there is no point for buffering more. Initially intended to work as buffer accommodating transactions from two different full views (which could have different set of transactions), turned out to be a bottleneck in maintain function (it still is but aligning sizes reduced the impact), - [bug](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L769) :see_no_evil: fixed in pre-insert actions removal - [`ValidateTransactionPriority`](https://github.com/paritytech/polkadot-sdk/blob/045bc6d342620a02ee9b28d8de51f72ae680f06f/substrate/client/transaction-pool/src/graph/pool.rs#L64-L71) was added. The goal is to allow faster processing of validation requests that were made from the `maintain` context. Otherwise all requests were landing in the same queue and maintain requests could be delayed. Now the processing power is evenly 50/50 split between _maintain_ and _submit+revalidate_ context. Related work: - michalkucharczyk/tx-test-tool#42, - #8152, Todo: - [x] some run should be done also for parachain, --------- Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

[`TransactionPool::report_invalid`](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/api/src/lib.rs#L314) is now `async`, function was typically used in async context, seems right to be fully async. _Note_: expected to be merged after: #8596 --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

michalkucharczyk added 16 commits May 14, 2025 16:18

tokio rwlock + sync code bridge

7ee1652

wip: new async trackedmap

2153a5d

report_invalid is async now

32caa5e

clean-up + HRTB fun

2650303

debug_assert enabled

b2f6764

sync <> async bridghe + todos cleanup

d6418fb

rename

d4275bf

tracked_map: removed unused methods

d0e28a3

transaction-network: consts updated

6d5c3e9

FullChainApi: async_trait used, tokio locking primitives

a507ff8

ChainAPi: async_trait + fixes

8c71aaf

tx_mem_pool: limit adjusted to x2

f3292bb

view_store: most_recent_view is a ref to view now

7c57cd8

view_store: apply_pending_tx_replacements bug fix

d563da9

debug levels adjusted

9ead46d

fatp: most_recent_view is a view ref

74dafae