-
Notifications
You must be signed in to change notification settings - Fork 1.1k
client/net: Use litep2p as the default network backend #8461
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This reverts commit bcc272a. Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
The test is failing too shortly in the overloaded CI. ``` output: 2025-05-16 16:02:40 polkadot-omni-node ... 2025-05-16 16:02:49 [Parachain] 🏆 Imported #3 (0xf558…5881 → 0x3909…8e9c) // final log line ``` It is clear 10 seconds are not enough for this test and will randomly pass if the CI is not overloaded. Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
This reverts commit 6310cd5.
|
Created backport PR for
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-8461-to-stable2409
git worktree add --checkout .worktree/backport-8461-to-stable2409 backport-8461-to-stable2409
cd .worktree/backport-8461-to-stable2409
git reset --hard HEAD^
git cherry-pick -x f6d18b73c4138cd834c6979be6eed291e63c6f17
git push --force-with-lease |
|
Created backport PR for
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-8461-to-stable2503
git worktree add --checkout .worktree/backport-8461-to-stable2503 backport-8461-to-stable2503
cd .worktree/backport-8461-to-stable2503
git reset --hard HEAD^
git cherry-pick -x f6d18b73c4138cd834c6979be6eed291e63c6f17
git push --force-with-lease |
|
Created backport PR for
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-8461-to-stable2412
git worktree add --checkout .worktree/backport-8461-to-stable2412 backport-8461-to-stable2412
cd .worktree/backport-8461-to-stable2412
git reset --hard HEAD^
git cherry-pick -x f6d18b73c4138cd834c6979be6eed291e63c6f17
git push --force-with-lease |
|
Git push to origin failed for stable2503 with exitcode 1 |
…8650) This PR rejects non-reserved peers in the reserved-only mode of the litep2p notification peerset. Previously, litep2p ignored completely the reserved-only state while accepting inbound connections. However, it handled it properly during the slot allocation phase. - the main changes are in the `report_inbound_substream` function, which now propagated a `Rejected` response to litep2p on the reserved-only state - in response, litep2p should never open an inbound substream after receiving the rejected response - the state of peers is not advanced while in `Disconnected` or `Backoff` states - the opening state is moved to `Cancelled` - for consistency purposes (and fuzzing purposes), the `report_substream_opened` is more robustly handling the `Disconnected` state - while at it have replaced a panic with `debug_assert` and an instant reject ## Testing Done - started 2 nodes in Kusama and Polkadot with litep2p - added the `reserved_only_rejects_non_reserved_peers` test to ensure litep2p handles peers properly from different states This PR has been extracted from #8461 to ease the review process cc @paritytech/networking --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Dmitry Markin <[email protected]> (cherry picked from commit e82b4db)
…8650) This PR rejects non-reserved peers in the reserved-only mode of the litep2p notification peerset. Previously, litep2p ignored completely the reserved-only state while accepting inbound connections. However, it handled it properly during the slot allocation phase. - the main changes are in the `report_inbound_substream` function, which now propagated a `Rejected` response to litep2p on the reserved-only state - in response, litep2p should never open an inbound substream after receiving the rejected response - the state of peers is not advanced while in `Disconnected` or `Backoff` states - the opening state is moved to `Cancelled` - for consistency purposes (and fuzzing purposes), the `report_substream_opened` is more robustly handling the `Disconnected` state - while at it have replaced a panic with `debug_assert` and an instant reject ## Testing Done - started 2 nodes in Kusama and Polkadot with litep2p - added the `reserved_only_rejects_non_reserved_peers` test to ensure litep2p handles peers properly from different states This PR has been extracted from #8461 to ease the review process cc @paritytech/networking --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Dmitry Markin <[email protected]> (cherry picked from commit e82b4db)
* master: omni-node: fix `benchmark pallet` to work with `--runtime` (#8594) Handle and suppress "New unknown `FromSwarm` libp2p event" warning (#8731) Implement detailed logging for XCM failures (#8724) [pallet-revive] contract's nonce starts at 1 (#8734) sync/fix: Clear gap sync on known imported blocks (#8445) [PoP] Add personhood tracking pallets (#8164) client/net: Use litep2p as the default network backend (#8461) Unflake `returns_status_for_pruned_blocks` (#8709) [AHM] Report the weights of epmb pallet to expose kusama and polkadot weights (#8704) Remove all XCM dependencies from `pallet-revive` (#8584) Docker master image tag fix (#8711) Record ed as part of the storage deposit (#8718) [pallet-revive] update dry-run logic (#8662) feat: add collator peer ID to ParachainInherentData (#8708) Nest errors in pallet-xcm (#7730) pallet-assets ERC20 precompile (#8554) Broker: Introduce min price + adjust renewals to lower market. (#8630) [AHM] Staking async fixes for XCM and election planning (#8422) Staking (EPMB): Add defensive error handling to voter snapshot creation and solution verification (#8687)
…nd (#8730) Backport #8461 into `stable2503` from lexnv. See the [documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md) on how to use this bot. <!-- # To be used by other automation, do not modify: original-pr-number: #${pull_number} --> --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: Alexandru Vasile <[email protected]> Co-authored-by: Alexandru Vasile <[email protected]> Co-authored-by: Egor_P <[email protected]>
…nd (#8733) Backport #8461 into `stable2412` from lexnv. See the [documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md) on how to use this bot. <!-- # To be used by other automation, do not modify: original-pr-number: #${pull_number} --> --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: Alexandru Vasile <[email protected]> Co-authored-by: Alexandru Vasile <[email protected]>
…8650) This PR rejects non-reserved peers in the reserved-only mode of the litep2p notification peerset. Previously, litep2p ignored completely the reserved-only state while accepting inbound connections. However, it handled it properly during the slot allocation phase. - the main changes are in the `report_inbound_substream` function, which now propagated a `Rejected` response to litep2p on the reserved-only state - in response, litep2p should never open an inbound substream after receiving the rejected response - the state of peers is not advanced while in `Disconnected` or `Backoff` states - the opening state is moved to `Cancelled` - for consistency purposes (and fuzzing purposes), the `report_substream_opened` is more robustly handling the `Disconnected` state - while at it have replaced a panic with `debug_assert` and an instant reject ## Testing Done - started 2 nodes in Kusama and Polkadot with litep2p - added the `reserved_only_rejects_non_reserved_peers` test to ensure litep2p handles peers properly from different states This PR has been extracted from #8461 to ease the review process cc @paritytech/networking --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Dmitry Markin <[email protected]>
# Litep2p Becomes the Default Network Backend This PR finalizes the [litep2p](https://github.com/paritytech/litep2p) integration and makes it the default network backend for substrate-based chains. ## Litep2p Improvements After the stabilization, a forum post will follow with up to date information and more accurate measurements of the live impact of litep2p. ### CPU Usage Reduction **Litep2p consumes roughly 2x less CPU than the libp2p alternative**. This frees up resources for other usecases (subsystems) and enables running nodes on more cost-efficient hardware. This metric has been collected by the `networking::libp2p-node` metric of a live Kusama validator. This represents the CPU time spent on polling the networking task. Litep2p CPU consumption is on the left, using roughtly 1.3x CPUs, while libp2p on the right uses roughly 2.9-3x CPUs:  This metric has been collected by the NodeExporter of a live Kusama validator. Litep2p CPU consumption is on the left, using roughtly 230 CPU units, while libp2p on the right uses roughly 350 CPU units. This makes litep2p ~1.52 times more effiecient:  ### DHT Improvements and Authority Discovery Litep2p is able to discover peers faster via the Kademlia protocol than libp2p. This behavior manifests in faster discovery times for validators. For context, libp2p discovers 1K DHT records (authority records) in approximately 10 minutes, while litep2p discovers them in just 2.5 minutes (for more info see #7077 (comment)). This will improve issues we've seen with libp2p that causes validators to not receive rewards: - #8548 ### Stable Sync Peers Litep2p presents a more stable peer count in comparison with the libp2p backend. This ensures we can sync up faster than libp2p to the tip of the chain. In an older experiment, litep2p syncs to the tip of the chain in 526s, compared to 803s for libp2p. The stability of connections shows improvements for other protocols as well:  The previous image shows on the left the litep2p version and on the right the libp2p version. ### Revert Kusama Enablement This PR reverts #7866. Litep2p is now enabled by default, we don't need to selectively enable it on different chains. ### Litep2p 0.9.5 This release primarily focuses on strengthening the stability of the websocket transport. We've resolved an issue where higher-level buffering was causing the Noise protocol to fail when decoding messages. We've also significantly improved connectivity between litep2p and Smoldot (the Substrate-based light client). Empty frames are now handled correctly, preventing handshake timeouts and ensuring smoother communication. Finally, we've carried out several dependency updates to keep the library current with the latest versions of its underlying components. Fixed: - substream/fix: Allow empty payloads with 0-length frame ([#395](paritytech/litep2p#395)) - websocket: Fix connection stability on decrypt messages ([#393](paritytech/litep2p#393)) Changed: - crypto/noise: Show peerIDs that fail to decode ([#392](paritytech/litep2p#392)) - cargo: Bump yamux to 0.13.5 and tokio to 1.45.0 ([#396](paritytech/litep2p#396)) - ci: Enforce and apply clippy rules ([#388](paritytech/litep2p#388)) - build(deps): bump ring from 0.16.20 to 0.17.14 ([#389](paritytech/litep2p#389)) - Update hickory-resolver 0.24.2 -> 0.25.2 ([#386](paritytech/litep2p#386)) ### Fix peerset reserve only mode This has been move in PR: #8650 for ease of reviewing. The PR rejects non-reserved peers in the reserved-only mode of the litep2p notification peerset. --------- Signed-off-by: Alexandru Vasile <[email protected]>
|
This pull request has been mentioned on Polkadot Forum. There might be relevant details there: https://forum.polkadot.network/t/announcing-the-stabilization-of-the-litep2p-network-backend/13712/1 |
…e2412-6` instead of volatile branch to track `polkadot-sdk`) The latest release tag for `stable2412` is `polkadot-stable2412-7`. Testing has shown that updating to this version causes frequent errors in RPC calls from light clients (`smoldot`) and existing client SDK code. This may be because this version uses `litep2p` as the default library for `substrate`. watch this: paritytech/polkadot-sdk#8461
…e2412-6` instead of volatile branch to track `polkadot-sdk`) The latest release tag for `stable2412` is `polkadot-stable2412-7`. Testing has shown that updating to this version causes frequent errors in RPC calls from light clients (`smoldot`) and existing client SDK code. This may be because this version uses `litep2p` as the default library for `substrate`. watch this: paritytech/polkadot-sdk#8461
…work backend (paritytech#8730) Backport paritytech#8461 into `stable2503` from lexnv. See the [documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md) on how to use this bot. <!-- # To be used by other automation, do not modify: original-pr-number: #${pull_number} --> --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: Alexandru Vasile <[email protected]> Co-authored-by: Alexandru Vasile <[email protected]> Co-authored-by: Egor_P <[email protected]>
…e2412-6` instead of volatile branch to track `polkadot-sdk`) The latest release tag for `stable2412` is `polkadot-stable2412-7`. Testing has shown that updating to this version causes frequent errors in RPC calls from light clients (`smoldot`) and existing client SDK code. This may be because this version uses `litep2p` as the default library for `substrate`. watch this: paritytech/polkadot-sdk#8461
…e2412-6` instead of volatile branch to track `polkadot-sdk`) The latest release tag for `stable2412` is `polkadot-stable2412-7`. Testing has shown that updating to this version causes frequent errors in RPC calls from light clients (`smoldot`) and existing client SDK code. This may be because this version uses `litep2p` as the default library for `substrate`. watch this: paritytech/polkadot-sdk#8461
…8650) This PR rejects non-reserved peers in the reserved-only mode of the litep2p notification peerset. Previously, litep2p ignored completely the reserved-only state while accepting inbound connections. However, it handled it properly during the slot allocation phase. - the main changes are in the `report_inbound_substream` function, which now propagated a `Rejected` response to litep2p on the reserved-only state - in response, litep2p should never open an inbound substream after receiving the rejected response - the state of peers is not advanced while in `Disconnected` or `Backoff` states - the opening state is moved to `Cancelled` - for consistency purposes (and fuzzing purposes), the `report_substream_opened` is more robustly handling the `Disconnected` state - while at it have replaced a panic with `debug_assert` and an instant reject ## Testing Done - started 2 nodes in Kusama and Polkadot with litep2p - added the `reserved_only_rejects_non_reserved_peers` test to ensure litep2p handles peers properly from different states This PR has been extracted from #8461 to ease the review process cc @paritytech/networking --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Dmitry Markin <[email protected]>
# Litep2p Becomes the Default Network Backend This PR finalizes the [litep2p](https://github.com/paritytech/litep2p) integration and makes it the default network backend for substrate-based chains. ## Litep2p Improvements After the stabilization, a forum post will follow with up to date information and more accurate measurements of the live impact of litep2p. ### CPU Usage Reduction **Litep2p consumes roughly 2x less CPU than the libp2p alternative**. This frees up resources for other usecases (subsystems) and enables running nodes on more cost-efficient hardware. This metric has been collected by the `networking::libp2p-node` metric of a live Kusama validator. This represents the CPU time spent on polling the networking task. Litep2p CPU consumption is on the left, using roughtly 1.3x CPUs, while libp2p on the right uses roughly 2.9-3x CPUs:  This metric has been collected by the NodeExporter of a live Kusama validator. Litep2p CPU consumption is on the left, using roughtly 230 CPU units, while libp2p on the right uses roughly 350 CPU units. This makes litep2p ~1.52 times more effiecient:  ### DHT Improvements and Authority Discovery Litep2p is able to discover peers faster via the Kademlia protocol than libp2p. This behavior manifests in faster discovery times for validators. For context, libp2p discovers 1K DHT records (authority records) in approximately 10 minutes, while litep2p discovers them in just 2.5 minutes (for more info see #7077 (comment)). This will improve issues we've seen with libp2p that causes validators to not receive rewards: - #8548 ### Stable Sync Peers Litep2p presents a more stable peer count in comparison with the libp2p backend. This ensures we can sync up faster than libp2p to the tip of the chain. In an older experiment, litep2p syncs to the tip of the chain in 526s, compared to 803s for libp2p. The stability of connections shows improvements for other protocols as well:  The previous image shows on the left the litep2p version and on the right the libp2p version. ### Revert Kusama Enablement This PR reverts #7866. Litep2p is now enabled by default, we don't need to selectively enable it on different chains. ### Litep2p 0.9.5 This release primarily focuses on strengthening the stability of the websocket transport. We've resolved an issue where higher-level buffering was causing the Noise protocol to fail when decoding messages. We've also significantly improved connectivity between litep2p and Smoldot (the Substrate-based light client). Empty frames are now handled correctly, preventing handshake timeouts and ensuring smoother communication. Finally, we've carried out several dependency updates to keep the library current with the latest versions of its underlying components. Fixed: - substream/fix: Allow empty payloads with 0-length frame ([#395](paritytech/litep2p#395)) - websocket: Fix connection stability on decrypt messages ([#393](paritytech/litep2p#393)) Changed: - crypto/noise: Show peerIDs that fail to decode ([#392](paritytech/litep2p#392)) - cargo: Bump yamux to 0.13.5 and tokio to 1.45.0 ([#396](paritytech/litep2p#396)) - ci: Enforce and apply clippy rules ([#388](paritytech/litep2p#388)) - build(deps): bump ring from 0.16.20 to 0.17.14 ([#389](paritytech/litep2p#389)) - Update hickory-resolver 0.24.2 -> 0.25.2 ([#386](paritytech/litep2p#386)) ### Fix peerset reserve only mode This has been move in PR: #8650 for ease of reviewing. The PR rejects non-reserved peers in the reserved-only mode of the litep2p notification peerset. --------- Signed-off-by: Alexandru Vasile <[email protected]>
…e2412-6` instead of volatile branch to track `polkadot-sdk`) The latest release tag for `stable2412` is `polkadot-stable2412-7`. Testing has shown that updating to this version causes frequent errors in RPC calls from light clients (`smoldot`) and existing client SDK code. This may be because this version uses `litep2p` as the default library for `substrate`. watch this: paritytech/polkadot-sdk#8461
Litep2p Becomes the Default Network Backend
This PR finalizes the litep2p integration and makes it the default network backend for substrate-based chains.
Litep2p Improvements
After the stabilization, a forum post will follow with up to date information and more accurate measurements of the live impact of litep2p.
CPU Usage Reduction
Litep2p consumes roughly 2x less CPU than the libp2p alternative. This frees up resources for other usecases (subsystems) and enables running nodes on more cost-efficient hardware.
This metric has been collected by the
networking::libp2p-nodemetric of a live Kusama validator. This represents the CPU time spent on polling the networking task. Litep2p CPU consumption is on the left, using roughtly 1.3x CPUs, while libp2p on the right uses roughly 2.9-3x CPUs:This metric has been collected by the NodeExporter of a live Kusama validator. Litep2p CPU consumption is on the left, using roughtly 230 CPU units, while libp2p on the right uses roughly 350 CPU units. This makes litep2p ~1.52 times more effiecient:
DHT Improvements and Authority Discovery
Litep2p is able to discover peers faster via the Kademlia protocol than libp2p. This behavior manifests in faster discovery times for validators. For context, libp2p discovers 1K DHT records (authority records) in approximately 10 minutes, while litep2p discovers them in just 2.5 minutes (for more info see #7077 (comment)).
This will improve issues we've seen with libp2p that causes validators to not receive rewards:
Stable Sync Peers
Litep2p presents a more stable peer count in comparison with the libp2p backend. This ensures we can sync up faster than libp2p to the tip of the chain. In an older experiment, litep2p syncs to the tip of the chain in 526s, compared to 803s for libp2p. The stability of connections shows improvements for other protocols as well:
The previous image shows on the left the litep2p version and on the right the libp2p version.
Revert Kusama Enablement
This PR reverts #7866. Litep2p is now enabled by default, we don't need to selectively enable it on different chains.
Litep2p 0.9.5
This release primarily focuses on strengthening the stability of the websocket transport. We've resolved an issue where higher-level buffering was causing the Noise protocol to fail when decoding messages.
We've also significantly improved connectivity between litep2p and Smoldot (the Substrate-based light client). Empty frames are now handled correctly, preventing handshake timeouts and ensuring smoother communication.
Finally, we've carried out several dependency updates to keep the library current with the latest versions of its underlying components.
Fixed:
Changed:
Fix peerset reserve only mode
This has been move in PR: #8650 for ease of reviewing.
The PR rejects non-reserved peers in the reserved-only mode of the litep2p notification peerset.