-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Gap Sync: Skip Body Requests for Non-Archive Nodes #10752
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
131 commits
Select commit
Hold shift + click to select a range
5234b57
wip hacked warp sync to store blocks
sistemd 4bfe85a
feat(sync): skip gap sync if block pruning is enabled
sistemd e9ab534
wip it's not good
sistemd 7693e3b
seems to work but gives me stack overflow for some odd reason
sistemd b06500d
wip for pr
sistemd 9e96373
remove some debugging stuff
sistemd 4fa618c
Merge branch 'master' into sistemd/pruned-gap-sync-consensus-broadcast
sistemd 9ce4b36
allow_missing_parent
lrubasze 32c9824
full_node_warp_sync test: more traces
lrubasze c7654d4
full_node_warp_sync test: more tracec
lrubasze 78cf02d
full_node_warp_sync test: change debug levels
lrubasze 60fcadb
full_node_warp_sync test: use libp2p
lrubasze 9f72ba6
warp: exclude target block when importing
lrubasze cdb7964
service client: more debugs
lrubasze c3608fd
service client: import block if gap not updated for parent block
lrubasze 335d209
babe: revert import_block method
lrubasze 2243786
Revert "babe: revert import_block method"
lrubasze 0ebdf87
babe: add some TODO
lrubasze 698b1a1
full_node_warp_sync test: use default network-backend
lrubasze 1264039
babe: cleanup TODO
lrubasze 00ffe9b
cleanup debugs
lrubasze 1764c5d
full_node_warp_sync test: cleanup debug levels
lrubasze 0b5340e
sync: debug improvements
lrubasze b2d1109
cumulus-test-service: enable prometheus metrics for import_queue
lrubasze 3ef2b8f
debug if metric enabled
lrubasze 0d572eb
sync: debug improvements
lrubasze f09283d
full_node_warp_sync test: assert warp sync
lrubasze f641b10
full_node_warp_sync test: assert gap sync
lrubasze 0e8f38f
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze de63ce1
Revert "debug if metric enabled"
lrubasze b9ce2d3
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] 85b1fec
sp-consensus: Add WarpSync origin
lrubasze d6bad88
sp-consensus: Add GapSync origin
lrubasze 90faa2e
babe: fix debug
lrubasze 29c1c4f
import-queue: improve debug
lrubasze 0e04d3f
temp debugs
lrubasze ee8add8
allow_missing_parents false
lrubasze 9fd02dd
simplify logic - get rid of allow_missing_parent
lrubasze 587a493
fmt
lrubasze 573aac1
remove debug
lrubasze adce235
fix comment
lrubasze 1f58c98
warp: add some comment
lrubasze 6b21144
simplify logic - get rid of block pruning flag for
lrubasze 238ec26
db: add remove_gap closure
lrubasze 32b349b
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze 46c50aa
Revert "temp debugs"
lrubasze f027459
full_node_warp_sync test: return errors instead of panicking
lrubasze aef5e4d
full_node_warp_sync test: adjust debug levels
lrubasze 9967883
Revert "cumulus-test-service: enable prometheus metrics for import_qu…
lrubasze 1b2953f
babe: restore comment
lrubasze 0aa8fe6
cleanup
lrubasze c9a1cef
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze 13908d7
warp: fix tests
lrubasze b08890a
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] 45414a8
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze f7bfe2e
babe: return before inherents check if WarpSyncorigin
lrubasze ba1adc4
babe: comment cleanup
lrubasze 2507816
sync: set create_gap only for non-WarpSync origins
lrubasze 8d287c1
db: unify conditions
lrubasze bb34428
sync: improve comment
lrubasze 457efad
full_node_warp_sync test: improve the scenario
lrubasze 38787a3
full_node_warp_sync test: automatic snapshot generation - WIP
lrubasze f14f5d3
full_node_warp_sync test: further tweaks
lrubasze 9c98f09
full_node_warp_sync test: automatic snapshot generation - refactor
lrubasze 959184a
full_node_warp_sync test: test dynamically added nodes
lrubasze a84c61b
full_node_warp_sync test: automatic snapshot generation cleanup
lrubasze ccdf112
full_node_warp_sync test: readme
lrubasze 3a6c61a
full_node_warp_sync test: readme 2
lrubasze 26c5096
full_node_warp_sync test: change snapshot names
lrubasze a99c4fd
full_node_warp_sync test: cleanup
lrubasze 383c2c4
full_node_warp_sync test: fix readme and licences
lrubasze 1bf996a
clippy
lrubasze 5e9e0ee
cleanup
lrubasze a2950cc
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze 1009af6
full_node_warp_sync test: tweak adding nodes to the test
lrubasze 2da5fb3
update prdoc
lrubasze 9814c2c
update comment
lrubasze 73b0d78
sync: add some logs
lrubasze a667d69
sync: do not request body when in gap sync
lrubasze 598c889
sync: gap sync stats
lrubasze e00d235
Revert "sync: do not request body when in gap sync"
lrubasze 64cd874
sync: add gap sync stats logs
lrubasze cc09e29
sync: do not request body when in gap sync
lrubasze 6f1d126
sync: do not skip bodies if archive enabled
lrubasze c94a913
sync: some unit tests
lrubasze a3f32d0
sync: fix warp sync tests
lrubasze 39a89ba
sync: fix tests
lrubasze df1e484
sync: gap sync tests
lrubasze 92d3e64
sync: refactor gap sync stats
lrubasze 219a3cf
cleanup
lrubasze 3b8ac2f
more cleanup
lrubasze f62940e
full_node_warp_sync test: simplify
lrubasze 44f1a46
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] b3f55da
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze a2cc5f7
update prdoc
lrubasze 4e0a5e1
Merge branch 'lrubasze/block-import-improvements' into lrubasze/gap-s…
lrubasze b5e4ff5
sync: cleanup
lrubasze 30858ce
sync: add more comment
lrubasze 3b4fe11
sync: more cleanup
lrubasze 14cf41c
fmt
lrubasze 7bd8d46
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] 0a3c894
reformat help text
lrubasze 9cd6506
re-reformat help text
lrubasze e424247
address comments
lrubasze ee88414
cleanup
lrubasze 26ad1f9
address comments part 1
lrubasze 691e3d1
address comments part 2
lrubasze 35d07d1
full_node_warp_sync test: increase timeouts when asserting warp and g…
lrubasze 6c70c83
improve comment
lrubasze 4dc95a3
babe: improve early return in verify
lrubasze 56da3df
temp debugs
lrubasze 523b6bb
sync: set state action to skip for Warp sync blocks
lrubasze 8984bd5
Revert "temp debugs"
lrubasze bd1bcba
address comments
lrubasze e54d3ae
sync: allow re-importing warp-synced blocks during gap sync
lrubasze 4020a83
add comments
lrubasze ce0bd49
full_node_warp_sync test: assert body downloads for gap sync
lrubasze 9aea77c
cleanup
lrubasze d373b82
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze f31b9bd
Merge branch 'lrubasze/block-import-improvements' into lrubasze/gap-s…
lrubasze 9e3b1d3
Update from github-actions[bot] running command 'fmt'
github-actions[bot] 425a4e5
Update from github-actions[bot] running command 'fmt'
github-actions[bot] 97cb3fd
update prdoc
lrubasze e3ab13e
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze adbdedd
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze 26cbb9e
Merge branch 'lrubasze/block-import-improvements' into lrubasze/gap-s…
lrubasze b006a6e
Merge branch 'master' into lrubasze/block-import-improvements
lrubasze 86f9062
Merge branch 'lrubasze/block-import-improvements' into lrubasze/gap-s…
lrubasze b6c6db0
Merge branch 'master' into lrubasze/gap-sync-improvements
lrubasze 4ea6a27
Update from github-actions[bot] running command 'prdoc --audience nod…
github-actions[bot] aef01ea
update prdoc
lrubasze File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| title: ' Gap Sync: Skip Body Requests for Non-Archive Nodes' | ||
| doc: | ||
| - audience: Node Operator | ||
| description: |- | ||
| ### Summary | ||
| This PR optimizes gap sync bandwidth usage by skipping body requests for non-archive nodes. Bodies are unnecessary during gap sync when the node doesn't maintain full block history, while archive nodes continue to request bodies to preserve complete history. | ||
| It reduces bandwidth consumption and database size significantly for typical validator/full nodes. | ||
|
|
||
| Additionally added some gap sync statistics for observability: | ||
| - Introduced `GapSyncStats` to track bandwidth usage: header bytes, body bytes, justification bytes | ||
| - Logged on gap sync completion to provide visibility into bandwidth savings | ||
| crates: | ||
| - name: sc-network-sync | ||
| bump: major | ||
| - name: sc-service | ||
| bump: major | ||
| - name: sc-client-db | ||
| bump: patch | ||
| - name: sc-cli | ||
| bump: patch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.