[BREAKING CHANGE] Hardcoding a Default `assume_valid_target` to Reduce the Timecost of Block Synchronization in IBD mode #4254

eval-exec · 2023-12-04T09:00:41Z

What problem does this PR solve?

Issue Number: close #4253

What's Changed:

Related changes

Set a block in 60 days ago as default assume_valid_target if user doesn't explictly specify --assume-valid-target argument.
User specified --assume-valid-target will overwrite the default hardcoded assume_valid_target
If user specify --assume-valid-target 0x0000000000000000000000000000000000000000000000000000000000000000, ckb will do full verification.

full block synchronization result

block synchronize from 0 to 12,500,000,
v0.115.0-rc2: need 30h40m
this PR: need 12h40m

This PR's performance test results show a 59% reduction in block synchronization time compared to v0.115.0-rc2.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code ci-runs-only: [ quick_checks,linters ]

Side effects

None

Release note

Title Only: Include only the PR title in the release note.

eval-exec · 2023-12-05T08:33:40Z

In the early phase of "finding block header of assume_valid_target" CKB will print many logs like:

32(0xe9b97767424dd04aa65a1f7ad562b0faf8dd0fbf2a213d1586ea7969160f5996) please wait
2023-12-04 07:51:20.256 +00:00 BlockDownload INFO ckb_sync::synchronizer  best known header number: 10890000, hash: Byte32(0x662ba941ac8dd506832704a35fc13534e1ac402aa708670936af03d27114c643), can't find assume valid target temporarily, hash: Byte32(0xe9b97767424dd04aa65a1f7ad562b0faf8dd0fbf2a213d1586ea7969160f5996) please wait
2023-12-04 07:51:22.676 +00:00 BlockDownload INFO ckb_sync::synchronizer  best known header number: 10900000, hash: Byte32(0xc793a8184a272be18a0736b4a3713f19f75ba9f8913c88ca8d65b05a03304306), can't find assume valid target temporarily, hash: Byte32(0xe9b97767424dd04aa65a1f7ad562b0faf8dd0fbf2a213d1586ea7969160f5996) please wait
2023-12-04 07:51:25.162 +00:00 BlockDownload INFO ckb_sync::synchronizer  best known header number: 10910000, hash: Byte32(0xa7d0e834174d4c0677648c66b8d74868ce4dcf8514d8f7c40c909cf6f68578ff), can't find assume valid target temporarily, hash: Byte32(0xe9b97767424dd04aa65a1f7ad562b0faf8dd0fbf2a213d1586ea7969160f5996) please wait
2023-12-04 07:51:27.096 +00:00 BlockDownload INFO ckb_sync::synchronizer  best known header number: 10920000, hash: Byte32(0xe4162e3db5b73eedffa5267595f4b6596cf765b5b6afeae7b1e1649eff057eed), can't find assume valid target temporarily, hash: Byte32(0xe9b97767424dd04aa65a1f7ad562b0faf8dd0fbf2a213d1586ea7969160f5996) please wait
2023-12-04 07:51:29.217 +00:00 BlockDownload INFO ckb_sync::synchronizer  best known header number: 10930000, hash: Byte32(0x43ce8a7e8c1d65f313d8d1b217145477d5cbc24e12a5ffc9104da2e7ab0d0a04), can't find assume valid target temporarily, hash: Byte32(0xe9b97767424dd04aa65a1f7ad562b0faf8dd0fbf2a213d1586ea7969160f5996) please wait
2023-12-04 07:51:31.193 +00:00 BlockDownload INFO ckb_sync::synchronizer  best known header number: 10940000, hash: Byte32(0x418fe666fa6c47b98f79c2f6d579b590e4c3449dc555614063f4d3450df32852), can't find assume valid target temporarily, hash: Byte32(0xe9b97767424dd04aa65a1f7ad562b0faf8dd0fbf2a213d1586ea7969160f5996) please wait
2023-12-04 07:51:33.148 +00:00 BlockDownload INFO ckb_sync::synchronizer  best known header number: 10950000, hash: Byte32(0xa17857353561c5488bb0d82f4ad60114c96a3032eb9cbca771fc40e43ecb51c0), can't find assume valid target temporarily, hash: Byte32(0xe9b97767424dd04aa65a1f7ad562b0faf8dd0fbf2a213d1586ea7969160f5996) please wait

These logs will appear about 1 hour, need to reword these into more friendly message, like:
INFO current best known header: {number}-{hash}, assume_valid_target: {number}-{hash}, CKB need {duration} to reach the target, progress {27%}

doitian · 2024-02-29T03:39:52Z

devtools/release/find_block_60_days_ago.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash


Could you let this script update the file util/constant/src/sync.rs directly? You can add start and end mark around the assume valie target config options to simplify the modification.

…r_map

…eader_map when best_shared_header is very near

…e_valid_target.rs Signed-off-by: Eval EXEC <[email protected]>

devtools/release/update_default_valid_target.sh

Co-authored-by: ian <[email protected]>

eval-exec · 2024-03-22T08:36:44Z

@doitian How about creating a template file named default_assume_valid_target.rs.template and then using envsubst to generate default_assume_valid_target.rs from this template file?

Ref: https://www.baeldung.com/linux/envsubst-command

devtools/release/update_default_valid_target.sh

doitian · 2024-03-22T08:43:22Z

@doitian How about creating a template file named default_assume_valid_target.rs.template and then using envsubst to generate default_assume_valid_target.rs from this template file?

Ref: https://www.baeldung.com/linux/envsubst-command

I think you can just use shell HEREDOC string

cat <<EOF
/// The \`mod mainnet\` and \`mod testnet\`'s codes are generated
/// by script: ./devtools/release/update_default_valid_target.sh

/// sync config related to mainnet
pub mod mainnet {
    // Default assume valid target for mainnet, expect to be a block 60 days ago.
    // Need to update when CKB's new release
    // in mainnet: the 60 days ago block is:
    // height: $MAINNET_HEIGHT
    // hash: $MAINNET_HASH
    // date: $GENERATED_AT
    // you can view this block in https://explorer.nervos.org/block/$MAINNET_HASH
    pub const DEFAULT_ASSUME_VALID_TARGET: &str =
        "$MAINNET_HASH";
}

/// sync config related to testnet
pub mod testnet {
    // Default assume valid target for testnet, expect to be a block 60 days ago.
    // Need to update when CKB's new release
    // in testnet: the 60 days ago block is:
    // height: $TESTNET_HEIGHT
    // hash: $TESTNET_HASH
    // date: $GENERATED_AT
    // you can view this block in https://pudge.explorer.nervos.org/block/$TESTNET_HASH
    pub const DEFAULT_ASSUME_VALID_TARGET: &str =
        "$TESTNET_HASH";
}
EOF

doitian · 2024-03-27T01:49:28Z

missing documentation for a module

eval-exec · 2024-03-27T08:09:39Z

https://stackoverflow.com/questions/1167746/how-to-assign-a-heredoc-value-to-a-variable-in-bash

eval-exec · 2024-03-27T09:32:52Z

I will upload the comparison chart of the full block synchronization results between this PR and v0.115.0-rc2 in a few hours.

Signed-off-by: Eval EXEC <[email protected]>

eval-exec · 2024-03-28T02:50:35Z

I will upload the comparison chart of the full block synchronization results between this PR and v0.115.0-rc2 in a few hours.

The comparison chart has been uploaded to the PR's description.

I think this PR is ready to be reviewed.

eval-exec added t:enhancement Type: Feature, refactoring. m:sync module: ckb-sync labels Dec 5, 2023

eval-exec self-assigned this Dec 12, 2023

eval-exec force-pushed the exec/develop-assume branch from 7c34485 to c64c457 Compare December 12, 2023 09:31

eval-exec changed the title ~~Hardcoding a Default assume_valid_target to Reduce the Timecost of Block Synchronization~~ [BREAKING CHANGE] Hardcoding a Default assume_valid_target to Reduce the Timecost of Block Synchronization in IBD mode Dec 12, 2023

eval-exec force-pushed the exec/develop-assume branch from 7d673c4 to 17b3960 Compare December 20, 2023 07:13

eval-exec marked this pull request as ready for review December 20, 2023 07:13

eval-exec requested a review from a team as a code owner December 20, 2023 07:13

eval-exec requested review from doitian and removed request for a team December 20, 2023 07:13

doitian reviewed Feb 29, 2024

View reviewed changes

eval-exec added 13 commits March 21, 2024 20:03

Add find_block_60-days_ago.sh to select assume_valid_target

5c9d841

Add DEFAULT_ASSUME_VALID_TARGET for mainet and testnet

55bb676

Import ckb-constant for ckb-app-config

17df3fe

Set a default assume_valid_target if user didn't specify it

50f42db

Update help message for --assume-valid-target

18178a3

Fix parse hardcoded DEFAULT_ASSUME_VALID_TARGET

6f88c13

Fix assume_valid_target log format with 0x prefix

bce1f9e

Add log message when accept_block reached assume_valid_target

856e507

Add log message when BlockFetchCMD found assume_valid_target in heade…

7983e11

…r_map

Add log message when BlockFetchCMD not found assume_valid_target in h…

402b69f

…eader_map when best_shared_header is very near

Remove trailing whitespace

13d18ec

Fix cargo clippy

aa56d9e

Improve find_block_60_days_ago.sh

69a47a1

eval-exec force-pushed the exec/develop-assume branch from 17b3960 to 69a47a1 Compare March 21, 2024 12:05

eval-exec added 3 commits March 21, 2024 20:18

Put hardcode DEFAULT_ASSUME_VALID_TARGET to a independent file

b16e57c

Update find_block_60_days_ago.sh to write block hash to default_assum…

24afe46

…e_valid_target.rs Signed-off-by: Eval EXEC <[email protected]>

Add update-default-valid-target Makefile entry

7cff6bb

doitian requested changes Mar 22, 2024

View reviewed changes

devtools/release/update_default_valid_target.sh Outdated Show resolved Hide resolved

Update devtools/release/update_default_valid_target.sh

cfcf230

Co-authored-by: ian <[email protected]>

eval-exec commented Mar 22, 2024

View reviewed changes

devtools/release/update_default_valid_target.sh Outdated Show resolved Hide resolved

doitian previously approved these changes Mar 27, 2024

View reviewed changes

eval-exec dismissed doitian’s stale review via 8c6f353 March 27, 2024 09:04

eval-exec force-pushed the exec/develop-assume branch 2 times, most recently from be79e85 to 457f87b Compare March 27, 2024 09:11

use bash HEREDOC to generate assume_valid_target config

c742a03

Signed-off-by: Eval EXEC <[email protected]>

eval-exec force-pushed the exec/develop-assume branch from 457f87b to c742a03 Compare March 28, 2024 02:04

Add log for assume_valid_target has been found in db on startup

4315e96

eval-exec requested review from zhangsoledad and driftluo March 28, 2024 02:50

eval-exec requested review from quake and chenyukang March 28, 2024 02:50

doitian approved these changes Mar 28, 2024

View reviewed changes

doitian added this pull request to the merge queue Mar 28, 2024

Merged via the queue into nervosnetwork:develop with commit 892e06c Mar 28, 2024
32 checks passed

doitian mentioned this pull request Apr 22, 2024

Release ckb v0.116.0 #4426

Closed

eval-exec mentioned this pull request Apr 28, 2024

Print duration needed to reach assume_valid_target in the Header Sync phase. #4437

Closed

eval-exec mentioned this pull request Dec 5, 2024

Extend the hardcoded assume_valid_target to an array. #4742

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BREAKING CHANGE] Hardcoding a Default `assume_valid_target` to Reduce the Timecost of Block Synchronization in IBD mode #4254

[BREAKING CHANGE] Hardcoding a Default `assume_valid_target` to Reduce the Timecost of Block Synchronization in IBD mode #4254

eval-exec commented Dec 4, 2023 •

edited

Loading

eval-exec commented Dec 5, 2023 •

edited

Loading

doitian Feb 29, 2024 •

edited

Loading

eval-exec commented Mar 22, 2024

doitian commented Mar 22, 2024 •

edited

Loading

doitian commented Mar 27, 2024

eval-exec commented Mar 27, 2024

eval-exec commented Mar 27, 2024 •

edited

Loading

eval-exec commented Mar 28, 2024

[BREAKING CHANGE] Hardcoding a Default assume_valid_target to Reduce the Timecost of Block Synchronization in IBD mode #4254

[BREAKING CHANGE] Hardcoding a Default assume_valid_target to Reduce the Timecost of Block Synchronization in IBD mode #4254

Conversation

eval-exec commented Dec 4, 2023 • edited Loading

What problem does this PR solve?

Related changes

full block synchronization result

Check List

Release note

eval-exec commented Dec 5, 2023 • edited Loading

doitian Feb 29, 2024 • edited Loading

Choose a reason for hiding this comment

eval-exec commented Mar 22, 2024

doitian commented Mar 22, 2024 • edited Loading

doitian commented Mar 27, 2024

eval-exec commented Mar 27, 2024

eval-exec commented Mar 27, 2024 • edited Loading

eval-exec commented Mar 28, 2024

[BREAKING CHANGE] Hardcoding a Default `assume_valid_target` to Reduce the Timecost of Block Synchronization in IBD mode #4254

[BREAKING CHANGE] Hardcoding a Default `assume_valid_target` to Reduce the Timecost of Block Synchronization in IBD mode #4254

eval-exec commented Dec 4, 2023 •

edited

Loading

eval-exec commented Dec 5, 2023 •

edited

Loading

doitian Feb 29, 2024 •

edited

Loading

doitian commented Mar 22, 2024 •

edited

Loading

eval-exec commented Mar 27, 2024 •

edited

Loading