some improvements for testing pool limits#42
Merged
michalkucharczyk merged 8 commits intomasterfrom Jun 4, 2025
Merged
Conversation
michalkucharczyk
changed the title
1 task
iulianbarbu
approved these changes
Jun 1, 2025
github-merge-queue bot
pushed a commit
to paritytech/polkadot-sdk
that referenced
this pull request
Jun 24, 2025
This PR adds some optimization and fixes in handling limits in fork-aware transaction pool. #### Notes for reviewers Changes made (random order): - debug levels adjusted in numerous places places ( `debug -> trace` for tx, `trace -> debug` for general flow) for better readablity, - internal TxMemPool [storage](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L268-L272) is now [sorted](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool/tx_mem_pool_map.rs#L123-L132). A new helper exposes methods to reduce number of transaction clones (e.g. [here](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1404-L1410), see [previous verions](https://github.com/paritytech/polkadot-sdk/blob/2863b7a9a879935ff16987d0e95065d088dad9f8/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1352-L1359)). This new structure eliminates the necessity of sorting transactions on the fly which was a not efficient, naive, [first](https://github.com/paritytech/polkadot-sdk/blob/e44b89fb7ca3385f314803c733ad97b26cd14e9f/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L384-L400) implementation. - some _mutexes_ were migrated to `tokio::sync::Mutex` to avoid tokio threads locking, - `sync` to `async` _[message](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L743) based_ bridge was implemented. It's purpose is mainly to support `LocalTransactionPool / OffchainTransactionPool` infrastucture. `sync` methods can be called from both non-tokio and tokio context. This requires one additional [blocking](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L398-L402) task for transaction pool. - `ViewStore::most_recent_view` is now a [reference](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L171) to `View`. - `TXMEMPOOL_TRANSACTION_LIMIT_MULTIPLIER` removed, there is no point for buffering more. Initially intended to work as buffer accommodating transactions from two different full views (which could have different set of transactions), turned out to be a bottleneck in maintain function (it still is but aligning sizes reduced the impact), - [bug](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L769) :see_no_evil: fixed in pre-insert actions removal - [`ValidateTransactionPriority`](https://github.com/paritytech/polkadot-sdk/blob/045bc6d342620a02ee9b28d8de51f72ae680f06f/substrate/client/transaction-pool/src/graph/pool.rs#L64-L71) was added. The goal is to allow faster processing of validation requests that were made from the `maintain` context. Otherwise all requests were landing in the same queue and maintain requests could be delayed. Now the processing power is evenly 50/50 split between _maintain_ and _submit+revalidate_ context. Related work: - michalkucharczyk/tx-test-tool#42, - #8152, Todo: - [x] some run should be done also for parachain, --------- Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
paritytech-release-backport-bot bot
pushed a commit
to paritytech/polkadot-sdk
that referenced
this pull request
Jun 24, 2025
This PR adds some optimization and fixes in handling limits in fork-aware transaction pool. #### Notes for reviewers Changes made (random order): - debug levels adjusted in numerous places places ( `debug -> trace` for tx, `trace -> debug` for general flow) for better readablity, - internal TxMemPool [storage](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L268-L272) is now [sorted](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool/tx_mem_pool_map.rs#L123-L132). A new helper exposes methods to reduce number of transaction clones (e.g. [here](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1404-L1410), see [previous verions](https://github.com/paritytech/polkadot-sdk/blob/2863b7a9a879935ff16987d0e95065d088dad9f8/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1352-L1359)). This new structure eliminates the necessity of sorting transactions on the fly which was a not efficient, naive, [first](https://github.com/paritytech/polkadot-sdk/blob/e44b89fb7ca3385f314803c733ad97b26cd14e9f/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L384-L400) implementation. - some _mutexes_ were migrated to `tokio::sync::Mutex` to avoid tokio threads locking, - `sync` to `async` _[message](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L743) based_ bridge was implemented. It's purpose is mainly to support `LocalTransactionPool / OffchainTransactionPool` infrastucture. `sync` methods can be called from both non-tokio and tokio context. This requires one additional [blocking](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L398-L402) task for transaction pool. - `ViewStore::most_recent_view` is now a [reference](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L171) to `View`. - `TXMEMPOOL_TRANSACTION_LIMIT_MULTIPLIER` removed, there is no point for buffering more. Initially intended to work as buffer accommodating transactions from two different full views (which could have different set of transactions), turned out to be a bottleneck in maintain function (it still is but aligning sizes reduced the impact), - [bug](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L769) :see_no_evil: fixed in pre-insert actions removal - [`ValidateTransactionPriority`](https://github.com/paritytech/polkadot-sdk/blob/045bc6d342620a02ee9b28d8de51f72ae680f06f/substrate/client/transaction-pool/src/graph/pool.rs#L64-L71) was added. The goal is to allow faster processing of validation requests that were made from the `maintain` context. Otherwise all requests were landing in the same queue and maintain requests could be delayed. Now the processing power is evenly 50/50 split between _maintain_ and _submit+revalidate_ context. Related work: - michalkucharczyk/tx-test-tool#42, - #8152, Todo: - [x] some run should be done also for parachain, --------- Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> (cherry picked from commit 6efe9f5)
manuelmauro
added a commit
to moonbeam-foundation/polkadot-sdk
that referenced
this pull request
Oct 3, 2025
This PR adds some optimization and fixes in handling limits in fork-aware transaction pool. Changes made (random order): - debug levels adjusted in numerous places places ( `debug -> trace` for tx, `trace -> debug` for general flow) for better readablity, - internal TxMemPool [storage](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L268-L272) is now [sorted](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool/tx_mem_pool_map.rs#L123-L132). A new helper exposes methods to reduce number of transaction clones (e.g. [here](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1404-L1410), see [previous verions](https://github.com/paritytech/polkadot-sdk/blob/2863b7a9a879935ff16987d0e95065d088dad9f8/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1352-L1359)). This new structure eliminates the necessity of sorting transactions on the fly which was a not efficient, naive, [first](https://github.com/paritytech/polkadot-sdk/blob/e44b89fb7ca3385f314803c733ad97b26cd14e9f/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L384-L400) implementation. - some _mutexes_ were migrated to `tokio::sync::Mutex` to avoid tokio threads locking, - `sync` to `async` _[message](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L743) based_ bridge was implemented. It's purpose is mainly to support `LocalTransactionPool / OffchainTransactionPool` infrastucture. `sync` methods can be called from both non-tokio and tokio context. This requires one additional [blocking](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L398-L402) task for transaction pool. - `ViewStore::most_recent_view` is now a [reference](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L171) to `View`. - `TXMEMPOOL_TRANSACTION_LIMIT_MULTIPLIER` removed, there is no point for buffering more. Initially intended to work as buffer accommodating transactions from two different full views (which could have different set of transactions), turned out to be a bottleneck in maintain function (it still is but aligning sizes reduced the impact), - [bug](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L769) :see_no_evil: fixed in pre-insert actions removal - [`ValidateTransactionPriority`](https://github.com/paritytech/polkadot-sdk/blob/045bc6d342620a02ee9b28d8de51f72ae680f06f/substrate/client/transaction-pool/src/graph/pool.rs#L64-L71) was added. The goal is to allow faster processing of validation requests that were made from the `maintain` context. Otherwise all requests were landing in the same queue and maintain requests could be delayed. Now the processing power is evenly 50/50 split between _maintain_ and _submit+revalidate_ context. Related work: - michalkucharczyk/tx-test-tool#42, - paritytech#8152, Todo: - [x] some run should be done also for parachain, --------- Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
manuelmauro
added a commit
to moonbeam-foundation/polkadot-sdk
that referenced
this pull request
Oct 3, 2025
This PR adds some optimization and fixes in handling limits in fork-aware transaction pool. Changes made (random order): - debug levels adjusted in numerous places places ( `debug -> trace` for tx, `trace -> debug` for general flow) for better readablity, - internal TxMemPool [storage](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L268-L272) is now [sorted](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool/tx_mem_pool_map.rs#L123-L132). A new helper exposes methods to reduce number of transaction clones (e.g. [here](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1404-L1410), see [previous verions](https://github.com/paritytech/polkadot-sdk/blob/2863b7a9a879935ff16987d0e95065d088dad9f8/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1352-L1359)). This new structure eliminates the necessity of sorting transactions on the fly which was a not efficient, naive, [first](https://github.com/paritytech/polkadot-sdk/blob/e44b89fb7ca3385f314803c733ad97b26cd14e9f/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L384-L400) implementation. - some _mutexes_ were migrated to `tokio::sync::Mutex` to avoid tokio threads locking, - `sync` to `async` _[message](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L743) based_ bridge was implemented. It's purpose is mainly to support `LocalTransactionPool / OffchainTransactionPool` infrastucture. `sync` methods can be called from both non-tokio and tokio context. This requires one additional [blocking](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L398-L402) task for transaction pool. - `ViewStore::most_recent_view` is now a [reference](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L171) to `View`. - `TXMEMPOOL_TRANSACTION_LIMIT_MULTIPLIER` removed, there is no point for buffering more. Initially intended to work as buffer accommodating transactions from two different full views (which could have different set of transactions), turned out to be a bottleneck in maintain function (it still is but aligning sizes reduced the impact), - [bug](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L769) :see_no_evil: fixed in pre-insert actions removal - [`ValidateTransactionPriority`](https://github.com/paritytech/polkadot-sdk/blob/045bc6d342620a02ee9b28d8de51f72ae680f06f/substrate/client/transaction-pool/src/graph/pool.rs#L64-L71) was added. The goal is to allow faster processing of validation requests that were made from the `maintain` context. Otherwise all requests were landing in the same queue and maintain requests could be delayed. Now the processing power is evenly 50/50 split between _maintain_ and _submit+revalidate_ context. Related work: - michalkucharczyk/tx-test-tool#42, - paritytech#8152, Todo: - [x] some run should be done also for parachain, --------- Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
alvicsam
pushed a commit
to paritytech/polkadot-sdk
that referenced
this pull request
Oct 17, 2025
This PR adds some optimization and fixes in handling limits in fork-aware transaction pool. #### Notes for reviewers Changes made (random order): - debug levels adjusted in numerous places places ( `debug -> trace` for tx, `trace -> debug` for general flow) for better readablity, - internal TxMemPool [storage](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L268-L272) is now [sorted](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool/tx_mem_pool_map.rs#L123-L132). A new helper exposes methods to reduce number of transaction clones (e.g. [here](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1404-L1410), see [previous verions](https://github.com/paritytech/polkadot-sdk/blob/2863b7a9a879935ff16987d0e95065d088dad9f8/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1352-L1359)). This new structure eliminates the necessity of sorting transactions on the fly which was a not efficient, naive, [first](https://github.com/paritytech/polkadot-sdk/blob/e44b89fb7ca3385f314803c733ad97b26cd14e9f/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L384-L400) implementation. - some _mutexes_ were migrated to `tokio::sync::Mutex` to avoid tokio threads locking, - `sync` to `async` _[message](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L743) based_ bridge was implemented. It's purpose is mainly to support `LocalTransactionPool / OffchainTransactionPool` infrastucture. `sync` methods can be called from both non-tokio and tokio context. This requires one additional [blocking](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L398-L402) task for transaction pool. - `ViewStore::most_recent_view` is now a [reference](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L171) to `View`. - `TXMEMPOOL_TRANSACTION_LIMIT_MULTIPLIER` removed, there is no point for buffering more. Initially intended to work as buffer accommodating transactions from two different full views (which could have different set of transactions), turned out to be a bottleneck in maintain function (it still is but aligning sizes reduced the impact), - [bug](https://github.com/paritytech/polkadot-sdk/blob/74dafaee5c600fd2c8a59a280f647f94ccf0a755/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L769) :see_no_evil: fixed in pre-insert actions removal - [`ValidateTransactionPriority`](https://github.com/paritytech/polkadot-sdk/blob/045bc6d342620a02ee9b28d8de51f72ae680f06f/substrate/client/transaction-pool/src/graph/pool.rs#L64-L71) was added. The goal is to allow faster processing of validation requests that were made from the `maintain` context. Otherwise all requests were landing in the same queue and maintain requests could be delayed. Now the processing power is evenly 50/50 split between _maintain_ and _submit+revalidate_ context. Related work: - michalkucharczyk/tx-test-tool#42, - #8152, Todo: - [x] some run should be done also for parachain, --------- Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.