Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
131 commits
Select commit Hold shift + click to select a range
5234b57
wip hacked warp sync to store blocks
sistemd Sep 8, 2025
4bfe85a
feat(sync): skip gap sync if block pruning is enabled
sistemd Sep 8, 2025
e9ab534
wip it's not good
sistemd Sep 8, 2025
7693e3b
seems to work but gives me stack overflow for some odd reason
sistemd Sep 8, 2025
b06500d
wip for pr
sistemd Sep 8, 2025
9e96373
remove some debugging stuff
sistemd Sep 9, 2025
4fa618c
Merge branch 'master' into sistemd/pruned-gap-sync-consensus-broadcast
sistemd Sep 18, 2025
9ce4b36
allow_missing_parent
lrubasze Nov 17, 2025
32c9824
full_node_warp_sync test: more traces
lrubasze Nov 13, 2025
c7654d4
full_node_warp_sync test: more tracec
lrubasze Nov 14, 2025
78cf02d
full_node_warp_sync test: change debug levels
lrubasze Nov 17, 2025
60fcadb
full_node_warp_sync test: use libp2p
lrubasze Nov 18, 2025
9f72ba6
warp: exclude target block when importing
lrubasze Nov 19, 2025
cdb7964
service client: more debugs
lrubasze Nov 20, 2025
c3608fd
service client: import block if gap not updated for parent block
lrubasze Nov 20, 2025
335d209
babe: revert import_block method
lrubasze Nov 20, 2025
2243786
Revert "babe: revert import_block method"
lrubasze Nov 20, 2025
0ebdf87
babe: add some TODO
lrubasze Nov 20, 2025
698b1a1
full_node_warp_sync test: use default network-backend
lrubasze Nov 25, 2025
1264039
babe: cleanup TODO
lrubasze Nov 25, 2025
00ffe9b
cleanup debugs
lrubasze Nov 25, 2025
1764c5d
full_node_warp_sync test: cleanup debug levels
lrubasze Nov 26, 2025
0b5340e
sync: debug improvements
lrubasze Nov 26, 2025
b2d1109
cumulus-test-service: enable prometheus metrics for import_queue
lrubasze Nov 26, 2025
3ef2b8f
debug if metric enabled
lrubasze Nov 26, 2025
0d572eb
sync: debug improvements
lrubasze Nov 27, 2025
f09283d
full_node_warp_sync test: assert warp sync
lrubasze Nov 27, 2025
f641b10
full_node_warp_sync test: assert gap sync
lrubasze Nov 27, 2025
0e8f38f
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Nov 27, 2025
de63ce1
Revert "debug if metric enabled"
lrubasze Nov 27, 2025
b9ce2d3
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] Nov 27, 2025
85b1fec
sp-consensus: Add WarpSync origin
lrubasze Nov 27, 2025
d6bad88
sp-consensus: Add GapSync origin
lrubasze Nov 27, 2025
90faa2e
babe: fix debug
lrubasze Nov 27, 2025
29c1c4f
import-queue: improve debug
lrubasze Nov 27, 2025
0e04d3f
temp debugs
lrubasze Nov 27, 2025
ee8add8
allow_missing_parents false
lrubasze Nov 27, 2025
9fd02dd
simplify logic - get rid of allow_missing_parent
lrubasze Nov 27, 2025
587a493
fmt
lrubasze Nov 27, 2025
573aac1
remove debug
lrubasze Nov 27, 2025
adce235
fix comment
lrubasze Nov 28, 2025
1f58c98
warp: add some comment
lrubasze Nov 28, 2025
6b21144
simplify logic - get rid of block pruning flag for
lrubasze Nov 28, 2025
238ec26
db: add remove_gap closure
lrubasze Nov 28, 2025
32b349b
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Dec 2, 2025
46c50aa
Revert "temp debugs"
lrubasze Dec 2, 2025
f027459
full_node_warp_sync test: return errors instead of panicking
lrubasze Dec 2, 2025
aef5e4d
full_node_warp_sync test: adjust debug levels
lrubasze Dec 2, 2025
9967883
Revert "cumulus-test-service: enable prometheus metrics for import_qu…
lrubasze Dec 2, 2025
1b2953f
babe: restore comment
lrubasze Dec 2, 2025
0aa8fe6
cleanup
lrubasze Dec 3, 2025
c9a1cef
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Dec 3, 2025
13908d7
warp: fix tests
lrubasze Dec 3, 2025
b08890a
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] Dec 4, 2025
45414a8
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Dec 15, 2025
f7bfe2e
babe: return before inherents check if WarpSyncorigin
lrubasze Dec 17, 2025
ba1adc4
babe: comment cleanup
lrubasze Dec 17, 2025
2507816
sync: set create_gap only for non-WarpSync origins
lrubasze Dec 17, 2025
8d287c1
db: unify conditions
lrubasze Dec 17, 2025
bb34428
sync: improve comment
lrubasze Dec 18, 2025
457efad
full_node_warp_sync test: improve the scenario
lrubasze Dec 18, 2025
38787a3
full_node_warp_sync test: automatic snapshot generation - WIP
lrubasze Dec 18, 2025
f14f5d3
full_node_warp_sync test: further tweaks
lrubasze Dec 19, 2025
9c98f09
full_node_warp_sync test: automatic snapshot generation - refactor
lrubasze Dec 19, 2025
959184a
full_node_warp_sync test: test dynamically added nodes
lrubasze Dec 19, 2025
a84c61b
full_node_warp_sync test: automatic snapshot generation cleanup
lrubasze Dec 19, 2025
ccdf112
full_node_warp_sync test: readme
lrubasze Dec 19, 2025
3a6c61a
full_node_warp_sync test: readme 2
lrubasze Dec 19, 2025
26c5096
full_node_warp_sync test: change snapshot names
lrubasze Jan 6, 2026
a99c4fd
full_node_warp_sync test: cleanup
lrubasze Jan 6, 2026
383c2c4
full_node_warp_sync test: fix readme and licences
lrubasze Jan 6, 2026
1bf996a
clippy
lrubasze Jan 6, 2026
5e9e0ee
cleanup
lrubasze Jan 6, 2026
a2950cc
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Jan 6, 2026
1009af6
full_node_warp_sync test: tweak adding nodes to the test
lrubasze Jan 6, 2026
2da5fb3
update prdoc
lrubasze Jan 7, 2026
9814c2c
update comment
lrubasze Jan 7, 2026
73b0d78
sync: add some logs
lrubasze Dec 8, 2025
a667d69
sync: do not request body when in gap sync
lrubasze Dec 8, 2025
598c889
sync: gap sync stats
lrubasze Dec 9, 2025
e00d235
Revert "sync: do not request body when in gap sync"
lrubasze Dec 9, 2025
64cd874
sync: add gap sync stats logs
lrubasze Dec 9, 2025
cc09e29
sync: do not request body when in gap sync
lrubasze Dec 8, 2025
6f1d126
sync: do not skip bodies if archive enabled
lrubasze Dec 16, 2025
c94a913
sync: some unit tests
lrubasze Dec 16, 2025
a3f32d0
sync: fix warp sync tests
lrubasze Dec 16, 2025
39a89ba
sync: fix tests
lrubasze Dec 16, 2025
df1e484
sync: gap sync tests
lrubasze Dec 17, 2025
92d3e64
sync: refactor gap sync stats
lrubasze Dec 17, 2025
219a3cf
cleanup
lrubasze Jan 8, 2026
3b8ac2f
more cleanup
lrubasze Jan 8, 2026
f62940e
full_node_warp_sync test: simplify
lrubasze Jan 9, 2026
44f1a46
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] Jan 9, 2026
b3f55da
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Jan 9, 2026
a2cc5f7
update prdoc
lrubasze Jan 9, 2026
4e0a5e1
Merge branch 'lrubasze/block-import-improvements' into lrubasze/gap-s…
lrubasze Jan 9, 2026
b5e4ff5
sync: cleanup
lrubasze Jan 12, 2026
30858ce
sync: add more comment
lrubasze Jan 12, 2026
3b4fe11
sync: more cleanup
lrubasze Jan 12, 2026
14cf41c
fmt
lrubasze Jan 12, 2026
7bd8d46
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] Jan 12, 2026
0a3c894
reformat help text
lrubasze Jan 13, 2026
9cd6506
re-reformat help text
lrubasze Jan 13, 2026
e424247
address comments
lrubasze Jan 14, 2026
ee88414
cleanup
lrubasze Jan 14, 2026
26ad1f9
address comments part 1
lrubasze Jan 14, 2026
691e3d1
address comments part 2
lrubasze Jan 14, 2026
35d07d1
full_node_warp_sync test: increase timeouts when asserting warp and g…
lrubasze Jan 14, 2026
6c70c83
improve comment
lrubasze Jan 26, 2026
4dc95a3
babe: improve early return in verify
lrubasze Jan 26, 2026
56da3df
temp debugs
lrubasze Jan 27, 2026
523b6bb
sync: set state action to skip for Warp sync blocks
lrubasze Jan 27, 2026
8984bd5
Revert "temp debugs"
lrubasze Jan 27, 2026
bd1bcba
address comments
lrubasze Jan 27, 2026
e54d3ae
sync: allow re-importing warp-synced blocks during gap sync
lrubasze Feb 4, 2026
4020a83
add comments
lrubasze Feb 4, 2026
ce0bd49
full_node_warp_sync test: assert body downloads for gap sync
lrubasze Feb 4, 2026
9aea77c
cleanup
lrubasze Feb 4, 2026
d373b82
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Feb 4, 2026
f31b9bd
Merge branch 'lrubasze/block-import-improvements' into lrubasze/gap-s…
lrubasze Feb 4, 2026
9e3b1d3
Update from github-actions[bot] running command 'fmt'
github-actions[bot] Feb 4, 2026
425a4e5
Update from github-actions[bot] running command 'fmt'
github-actions[bot] Feb 5, 2026
97cb3fd
update prdoc
lrubasze Feb 5, 2026
e3ab13e
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Feb 9, 2026
adbdedd
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Feb 11, 2026
26cbb9e
Merge branch 'lrubasze/block-import-improvements' into lrubasze/gap-s…
lrubasze Feb 11, 2026
b006a6e
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze Feb 12, 2026
86f9062
Merge branch 'lrubasze/block-import-improvements' into lrubasze/gap-s…
lrubasze Feb 12, 2026
b6c6db0
Merge branch 'master' into lrubasze/gap-sync-improvements
lrubasze Feb 13, 2026
4ea6a27
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] Feb 13, 2026
aef01ea
update prdoc
lrubasze Feb 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ pub(crate) async fn build_network_config(
)
.into(),
("--sync", "warp").into(),
("--blocks-pruning", "256").into(),
])
})
.with_node(|node| {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,9 @@ async fn assert_warp_sync(node: &NetworkNode) -> Result<(), anyhow::Error> {
Ok(())
}

// Asserting Gap sync requires at least sync=debug level
async fn assert_gap_sync(node: &NetworkNode) -> Result<(), anyhow::Error> {
let option_1_line = LogLineCountOptions::new(|n| n == 1, Duration::from_secs(20), false);
// Asserting Gap sync requires at least sync=trace level
async fn assert_gap_sync(node: &NetworkNode, is_archive: bool) -> Result<(), anyhow::Error> {
let option_1_line = LogLineCountOptions::new(|n| n == 1, Duration::from_secs(5), false);
let option_at_least_5_lines =
LogLineCountOptions::new(|n| n >= 5, Duration::from_secs(20), false);

Expand Down Expand Up @@ -117,6 +117,27 @@ async fn assert_gap_sync(node: &NetworkNode) -> Result<(), anyhow::Error> {
return Err(anyhow!("Gap sync block imports are not started"));
}

// Verify body download behavior based on archive mode:
// - Archive nodes should download bodies (body: non-zero)
// - Non-archive nodes should skip bodies (body: 0 B)
let (body_pattern, body_error) = if is_archive {
(
r"(?<!\[Parachain\] )Gap sync cumulative stats:.*body: [1-9]",
"archive node should download bodies",
)
} else {
(
r"(?<!\[Parachain\] )Gap sync cumulative stats:.*body: 0 B",
"non-archive node should not download bodies",
)
};
let result = node
.wait_log_line_count_with_timeout(body_pattern, false, option_at_least_5_lines.clone())
.await?;
if !result.success() {
return Err(anyhow!("Gap sync: {body_error}"));
}

let result = node
.wait_log_line_count_with_timeout(
r"(?<!\[Parachain\] )Imported [0-9]+ out of [0-9]+ blocks.*\(origin: GapSync\)",
Expand Down Expand Up @@ -160,9 +181,10 @@ async fn full_node_warp_sync() -> Result<(), anyhow::Error> {

// Assert warp and gap syncs only for relaychain.
// "five" is not warp syncing the relaychain
for name in ["dave", "eve", "four"] {
// dave is non-archive (--blocks-pruning 256), eve and four use default (archive-canonical)
for (name, is_archive) in [("dave", false), ("eve", true), ("four", true)] {
assert_warp_sync(network.get_node(name)?).await?;
assert_gap_sync(network.get_node(name)?).await?;
assert_gap_sync(network.get_node(name)?, is_archive).await?;
}

// Check relaychain progress
Expand Down Expand Up @@ -204,9 +226,9 @@ async fn full_node_warp_sync() -> Result<(), anyhow::Error> {
log::info!("Waiting for ferdie to be up");
network.get_node("ferdie")?.wait_until_is_up(60u64).await?;

// Assert warp and gap sync for ferdie
// Assert warp and gap sync for ferdie (uses default archive-canonical)
assert_warp_sync(network.get_node("ferdie")?).await?;
assert_gap_sync(network.get_node("ferdie")?).await?;
assert_gap_sync(network.get_node("ferdie")?, true).await?;

// Check progress for ferdie
log::info!("Checking full node ferdie is syncing");
Expand Down
20 changes: 20 additions & 0 deletions prdoc/pr_10752.prdoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
title: ' Gap Sync: Skip Body Requests for Non-Archive Nodes'
doc:
- audience: Node Operator
description: |-
### Summary
This PR optimizes gap sync bandwidth usage by skipping body requests for non-archive nodes. Bodies are unnecessary during gap sync when the node doesn't maintain full block history, while archive nodes continue to request bodies to preserve complete history.
It reduces bandwidth consumption and database size significantly for typical validator/full nodes.

Additionally added some gap sync statistics for observability:
- Introduced `GapSyncStats` to track bandwidth usage: header bytes, body bytes, justification bytes
- Logged on gap sync completion to provide visibility into bandwidth savings
crates:
- name: sc-network-sync
bump: major
- name: sc-service
bump: major
- name: sc-client-db
bump: patch
- name: sc-cli
bump: patch
3 changes: 3 additions & 0 deletions substrate/client/cli/src/arg_enums.rs
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,9 @@ pub enum SyncMode {
/// Download blocks without executing them. Download latest state without proofs.
FastUnsafe,
/// Prove finality and download the latest state.
Comment thread
bkchr marked this conversation as resolved.
/// After warp sync completes, the node will have block headers but not bodies for historical
/// blocks (unless `blocks-pruning` is set to archive mode). This saves bandwidth while still
/// allowing the node to serve as a warp sync source for other nodes.
Warp,
}

Expand Down
4 changes: 4 additions & 0 deletions substrate/client/db/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1726,6 +1726,10 @@ impl<Block: BlockT> Backend<Block> {
}

let should_check_block_gap = !header_exists_in_db || !body_exists_in_db;
debug!(
target: "db",
"should_check_block_gap = {should_check_block_gap}",
);

if should_check_block_gap {
let update_gap =
Expand Down
Loading
Loading