Put parent_header to `PreloadUnverified`, and keep small block in memory#4461

Closed

eval-exec wants to merge 360 commits intonervosnetwork:ckb-async-downloadfrom eval-exec:exec/ckb-async-download/reduce-load-block-timecost

+4,949-2,085

This pull request is big! We're only showing the most recent 250 commits

Commits on May 13, 2024

Remove useless import orphan_block_pool in ckb-sync
eval-exec
committed
Fix blocking process block usage in benches/benches/benchmarks/always_success.rs
eval-exec
committed
Fix blocking process block usage in benches/benches/benchmarks/overall.rs
eval-exec
committed
Fix blocking process block usage in benches/benches/benchmarks/resolve.rs
eval-exec
committed
Fix blocking process block usage in benches/benches/benchmarks/secp_2in2out.rs
eval-exec
committed
let ckb-import and ckb-replay construct ChainService with SharedPackage
eval-exec
committed
Fix Unit test: ckb-light-client-protocol-server::tests/utils/chain.rs
eval-exec
committed
Fix Unit test: ckb-verification::contextual/src/tests/uncle_verifier.rs
eval-exec
committed
Fix Unit test: ckb-verification::contextual/src/tests/contextual_block_verifier.rs
eval-exec
committed
Fix Unit test: ckb-sync::src/tests/util.rs
eval-exec
committed
Fix blocking process block usage in sync/src/relayer/tests/helper.rs
eval-exec
committed
Fix Unit test of blocking process block usage in ckb-sync
eval-exec
committed
Use blocking process_block method for benches/benches/benchmarks/secp_2in2out.rs
eval-exec
committed
Benches: Construct ChainService with SharedPackage provided verify_failed_block_rx
eval-exec
committed
Modify BlockStatus by SyncShared.Shared
eval-exec
committed
Unit Test: Use SharedPackage to construct ChainService
eval-exec
committed
Unit Test: Use SharedPackage to construct ChainService in ckb-chain::find_fork.rs
eval-exec
committed
Unit test: Modify ChainService initialization to include pack.take_verify_failed_block_tx() parameter
eval-exec
committed
Unit test: Modify the new function in ChainService to include verify_failed_block_tx parameter
eval-exec
committed
Unit test: Initialization of ChainService need pack.take_verify_failed_block_tx
eval-exec
committed
Add more minor ticks to sync progress chart
eval-exec
committed
Add draft mermaid sequence diagram for develop branch
eval-exec
committed
Add draft mermaid sequence diagram for asynchronous block download
eval-exec
committed
Add unverified block info to sync_state rpc
eval-exec
committed
Upgrade Synchronizer disconnect log level from debug to info
eval-exec
committed
Add tip_hash and tip_number to sync_state rpc
eval-exec
committed
Set receive exit signal log from debug to info
eval-exec
committed
Let HeaderMap stats feature use info log
eval-exec
committed
Activate HeaderMap stats profiling feature in make profiling
eval-exec
committed
Return removed inflight blocks count when disconnect
eval-exec
committed
Remove log message time cost unit
eval-exec
committed
Move ChainService proposal_table to the param for start method
eval-exec
committed
Remove proposal_table's RWLock
eval-exec
committed
Add VerifiedBlockStatus::PreviouslyVerified
eval-exec
committed
Will move truncate process to consume_unverified_blocks
eval-exec
committed
Extract consume_orphan_blocks from ChainService
eval-exec
committed
Extract consume_unverified_blocks from ChainService
eval-exec
committed
Move out consume_unverified_blocks and consume_orphan
eval-exec
committed
Move LonelyBlock related struct to module root
eval-exec
committed
Fix LonelyBlockWithCallBack import path
eval-exec
committed
Add ChainServicesBuilder to SharedPackage
eval-exec
committed
Launcher start chain_service by ChainServicesBuilder
eval-exec
committed
Fix ckb-sync VerifyResult import path
eval-exec
committed
Fix ckb-rpc VerifyResult import path
eval-exec
committed
Fix ChainService initialize for ckb import subcommand
eval-exec
committed
Fix ChainService initialize for ckb replay subcommand
eval-exec
committed
Fix ChainService initialize for ckb run subcommand
eval-exec
committed
Extract punish_bad_peer's params to peer_id and block_hash
eval-exec
committed
Fix tell_synchronizer_to_punish_the_bad_peer's params
eval-exec
committed
Remove search_orphan_pool's param
eval-exec
committed
Add ConsumeUnverifiedBlockProcessor as child field of ConsumeUnverifiedBlocks
eval-exec
committed
Make struct ChainService private
eval-exec
committed
Make ConsumeUnverifiedBlockProcessor public for crate
eval-exec
committed
Make chain module private, re-export ChainController and ChainSerivceBuilder
eval-exec
committed
Fix find_fork related unit test use ConsumeUnverifiedBlockProcessor::find_fork
eval-exec
committed
Use pack.take_chain_services_builder to construct chain_controller for
eval-exec
committed
Fix ChainController's import path in ckb-sync
eval-exec
committed
Fix ChainController's import path in ckb-verification, ckb-sync,
eval-exec
committed
Fix start ckb-chain's services by pack.take_chain_services_builder
eval-exec
committed
Unit test: process block by blocking way
eval-exec
committed
Fix ckb-sync unit tests: Synchronizer need verify_failed_block_rx
eval-exec
committed
Rename VerifiedBlockStatus variants to a more meaningfull name
eval-exec
committed
Extract ChainServicesBuilder start method as an independent function
eval-exec
committed
Move ChainServicesBuilder to ckb_shared
eval-exec
committed
Remove ChainServicesbuilder from ckb_chain
eval-exec
committed
Re-export start_chain_services in ckb_chain
eval-exec
committed
Re-sort shared_builder import statements
eval-exec
committed
Re-export ChainServicesBuilder in ckb_shared
eval-exec
committed
Fix ckb-launcher start chain services
eval-exec
committed
Fix subcommand::replay start chain_services
eval-exec
committed
Fix subcommand::import start chain services
eval-exec
committed
Fix ChainServicesBuilder import path in unit tests
eval-exec
committed
breadcast_compact_block only when FirstSeenAndVerified
eval-exec
committed
ChainService should execute non_contextual_verify when switch is None
eval-exec
committed
Extract accept_descendants from ConsumeOrphan
eval-exec
committed
Add contains_block for OrphanBlock
eval-exec
committed
Only insert new block to orphan pool if its parent not partial stored
eval-exec
committed
ConsumeOrphan::accept_descendant expect block is not stored
eval-exec
committed
accept_descendant does not return None
eval-exec
committed
accept_descendants need not handle the case when accept_descendant return None
eval-exec
committed
Remove VerifiedBlockStatus::PreviouslySeenButNotVerified
eval-exec
committed
contains_block shoud not take ownership of self
eval-exec
committed
Orphan: InnerPool shoud borrow self
eval-exec
committed
Extract process_descendant from accept_descendant
eval-exec
committed
Move ForkChanges to chain/src/utils
eval-exec
committed
Fix ForkChanges related import path
eval-exec
committed
Move OrphanBlocksPool to chain/src/utils
eval-exec
committed
Fix OrphanBlockPool related import path
eval-exec
committed
Trace last_common_ancestor timecost
eval-exec
committed
Fix module orphan_block_pool usage
eval-exec
committed
Fix last common_ancestor's log message
eval-exec
committed
Rename FirstSeenButNotVerified to UncleBlockNotVerified
eval-exec
committed
Remove useless import statements, fix clippy
eval-exec
committed
Using ckb_chain::start_chain_services to start ckb-chain services
eval-exec
committed
Benches: Start chain services by start_chain_services
eval-exec
committed
Fix Unit test in ckb-rpc: Start chain services by start_chain_services
eval-exec
committed
Fix Unit test in ckb-chain: Start chain services by start_chain_services
eval-exec
committed
Fix Unit test in ckb-sync: Start chain services by start_chain_services
eval-exec
committed
Add blocking_insert_new_block for SyncState for Unit Test purpose
eval-exec
committed
Fix lifetime issue for Unit test in ckb-chain
eval-exec
committed
Comment out sync_shared.rs related unit test
eval-exec
committed
Remove Synchronizer::verify_failed_blocks_rx Option wrapper
eval-exec
committed
Fix Launcher construct Synchronizer
eval-exec
committed
Remove Synchronizer::process_new_block to asynchronous_process_new_block
eval-exec
committed
Add blocking_execute method for BlockProcess for the purpose of unit test
eval-exec
committed
Modify Synchronizer::blocking_process_new_block return Result<bool,Error>
eval-exec
committed
Fix ckb-sync test_internal_db_error unit test
eval-exec
committed
Fix cargo clippy warnings
eval-exec
committed
Remove orphan_block_pool from ckb-sync
eval-exec
committed
Add Warning message for threads stop notify in ckb-chain
eval-exec
committed
Fix ckb-chain consume_unveriifed_blocks stop handle
eval-exec
committed
Pause chunk_process in consume_unverified_blocks thread
eval-exec
committed
Move truncate_block_request logic to consume_unverified_blocks
eval-exec
committed
ChainService won't need truncate_block_rx anymore
eval-exec
committed
Remove ChainService::start, since it has replaced by start_process_block
eval-exec
committed
Fix ChainService stop_rx related handling
eval-exec
committed
Fix consume_unverified truncate process
eval-exec
committed
Rename chain/src/chain.rs to chain/src/chain_service.rs
eval-exec
committed
Fix cargo clippy warnings
eval-exec
committed
Modify VerifyFailedBlockInfo::message_bytes to msg_bytes
eval-exec
committed
Pass and return peer_id_with_msg_bytes in process_block
eval-exec
committed
Fix peer_id_and_msg_bytes for ckb-chain unit tests
eval-exec
committed
Extract ConsumeDescendantProcessor from ConsumeOrphan
eval-exec
committed
Attach accept_descendant method to ConsumeDescendantProcessor
eval-exec
committed
Public ConsumeDescendantProcessor struct fields
eval-exec
committed
Fix ckb-chain find_fork related Unit Test
eval-exec
committed
Update sync_state RPC document
eval-exec
committed
Add docs for ckb-shared, fix clippy docs warnings
eval-exec
committed
Add docs for ckb-chain, fix clippy warnings
eval-exec
committed
Fix cargo clippy warnings for ckb-benches
eval-exec
committed
Fix cargo clippy warnings for ckb-sync
eval-exec
committed
Broadcast compact block when VerifiedStatus is FirstSeenAndVerified or UncleBlockNotVerified
eval-exec
committed
Add init_for_test logger helper, only used by unit test
eval-exec
committed
Fix unit test for synchronizer::basic_sync
eval-exec
committed
Improve sync chart drawer more friendly
eval-exec
committed
Fix ckb workspace members crate version to 0.114.0-pre
eval-exec
committed
Fix unit test for ckb-rpc, use blocking_process_block
eval-exec
committed
Fix BlockFetcher fetch should not use unverified_tip to change last_common
eval-exec
committed
Fix unit test: test_switch_valid_fork
eval-exec
committed
Simplify ConsumeDescendantProcessor unverified_blocks_tx send code
eval-exec
committed
Fix matches result did not assert its result
eval-exec
committed
Split get_ancestor's logic to unverified_tip and tip
eval-exec
committed
BlockFetcher get ancestor with unverified_tip
eval-exec
committed
Clean expired orphan blocks
eval-exec
committed
Split ChainController out chain_service.rs
eval-exec
committed
ChainService should mark block as BLOCK_INVALID if it does not pass non_contextual_verify
eval-exec
committed
Change VerifyResult to Result<bool, Error>
eval-exec
committed
ConsumeUnverified do not need VerifiedBlockStatus enum type anymore
eval-exec
committed
Do not need VerifiedBlockStatus in ckb-rpc and ckb-sync
eval-exec
committed
Fix unit test: do not need VerifiedBlockStatus
eval-exec
committed
Remove VerifiedBlockStatus enum type and related imports statements
eval-exec
committed
Remove useless crate dependencies, fix warnings of check-cargotoml.sh
eval-exec
committed
Fix ConsumeUnverified: should return Ok(false) if it's a block which
eval-exec
committed
Fix lint, remove whitespace
eval-exec
committed
BlockFetcher calculate start and ancestor header should aware IBDState
eval-exec
committed
BlockFetcher should only return if best_known <= unverified_tip in IBD mode
eval-exec
committed
SyncShared::accept_block will mark the block as BLOCK_RECEIVED if its block status is Entry::Vacant
eval-exec
committed
Fix ckb-sync test_insert_parent_unknown_block
eval-exec
committed
ConsumeOrphan should mark the block as PARTIAL_STORED before send it to consume_unverified thread
eval-exec
committed
Add more log message for debug
eval-exec
committed
Shared provide generic version of get_block_status for Snapshot and store
eval-exec
committed
ckb-chain load get_block_status from shared.store()
eval-exec
committed
ActiveChain get_block_status from snapshot()
eval-exec
committed
Fix SyncShared's field name in BlockFetcher and HeadersProcess
eval-exec
committed
Remove BlockStatus::BLOCK_PARTIAL_STORED
eval-exec
committed
Change BLOCK_PARTIAL_STORED to BLOCK_STORED
eval-exec
committed
Improve draw chart script add more major and minor ticks on x/yaxis
eval-exec
committed
Add log for MinerRpcImpl::submit_block new_block check
eval-exec
committed
Comment out shrink_to_fit for header_map's MemoryMap
eval-exec
committed
ChainService should not punish remote peer if failed to send block to orphan pool due to channel close
eval-exec
committed
Remove meaning less TODO note in test_internal_db_error, improve draw_sync_chart.py
eval-exec
committed
test: randomly kill and restart node

contrun
authored and
eval-exec
committed
add integration test for sync with churn nodes

contrun
authored and
eval-exec
committed
test: make Node struct clonable

contrun
authored and
eval-exec
committed
test: run SyncChurn mining and restart in different threads

contrun
authored and
eval-exec
committed
test: cargo {clippy,fmt}

contrun
authored and
eval-exec
committed
test: document what SyncChurn does and its weakness

contrun
authored and
eval-exec
committed
test: make two mining nodes in SyncChurn

contrun
authored and
eval-exec
committed
test: don't mine too many blocks in SyncChurn

contrun
authored and
eval-exec
committed
Improve draw chart script, no vertial line label overcover
eval-exec
committed
Add unverified tip related metrics to ckb_metrics
eval-exec
committed
Add minstant to ckb_chain dependency
eval-exec
committed
Collect ckb_chain timecost and unverified_tip metrics
eval-exec
committed
Collect execute_callback timecost for metrics
eval-exec
committed
Unverified block in db
eval-exec
committed
UnverifiedBlockHash size to 3
eval-exec
committed
Comment MemoryMap remove shrink_to_fit
eval-exec
committed
Remove UnverifiedBlockHash since consume_unverified can load parent_header from db
eval-exec
committed
ConsumeOrphan only need pass LonelyBlockHashWithCallback to ConsumeUnverified
eval-exec
committed
Replace the Into implementation LonelyBlockWithCallback with From<LonelyBlockWithCallback> form
eval-exec
committed
Fix ConsumeUnverified thread need LonelyBlockHashWithCallback in find_fork.rs
eval-exec
committed
Consider the edge case of processing genesis block

contrun
authored and
eval-exec
committed
Copy ibd_finished field from shared to HeaderMapKernal
eval-exec
committed
HeaderMap do not shrink_to_fit in IBD mode
eval-exec
committed
test inserting block with stored but unverified parent

contrun
authored and
eval-exec
committed
ConsumeUnverified should call get_update_for_tip_hash after begin_transaction, keep coincident with develop branch
eval-exec
committed
Cargo fmt, apply check-whitespaces
eval-exec
committed
Change process duration from Gauge to Histogram, add metrics for ckb-sync proc timecost
eval-exec
committed
Collect ckb-chain and ckb-sync timecost Histogram metrics
eval-exec
committed
Add orphan blocks count metric
eval-exec
committed
Add header_map limit_memory and operation duration metric
eval-exec
committed
Collect header_map limit_memory and operations timecost
eval-exec
committed
ConsumeOrphan should not insert a block to orphan pool if its parent is invalid
eval-exec
committed
Remove ChainController's useless methods, fix method comments
eval-exec
committed
Add HeaderMap memory count and cache hit/miss count metrics
eval-exec
committed
Collect HeaderMap MemoryMap cache hit/miss count and total count
eval-exec
committed
ConsumeUnverified do not realy need pass whole UnverifiedBlock to fn verify_block
eval-exec
committed
Remove msg_bytes from LonelyBlock
eval-exec
committed
Remove msg_bytes from VerifyFailedBlockInfo
eval-exec
committed
ckb-sync do not need pass msg_bytes to ckb-chain
eval-exec
committed
ckb-sync's unit test won't need msg_bytes anymore
eval-exec
committed
blocking_process_block returns a bool, use a proper name
eval-exec
committed
Metrics: header_map hit/miss should contains contains_key
eval-exec
committed
Add struct RemoteBlock for ckb-chain, let ckb-sync and ckb-relayer use RemoteBlock
eval-exec
committed
Modify ckb-chain use RemoteBlock, drop LonelyBlockWithCallback
eval-exec
committed
ckb-sync use accept_remote_block to process remote block
eval-exec
committed
Modify unit test in ckb-sync and ckb-chain to use RemoteBlock related API
eval-exec
committed
Fix HeaderMap memory count
eval-exec
committed
Fix rebase conflicts with develop: https://github.com/nervosnetwork/ckb/tree/dfa4f3753862261818eb93c540e7a3679ef4acc9
eval-exec
committed
Fix chain service builder for test_accept_block
eval-exec
committed
add inflight count and inflight timeout count for ckb-metrics
eval-exec
committed
Add lonely_block channel and unverified_block channel length
eval-exec
committed
Add test to ensure the results of remove_blocks_by_parent are sorted

contrun
authored and
eval-exec
committed
Fix blocks_are_sorted checking logic error

contrun
authored and
eval-exec
committed
Remove tell_synchronizer_to_punish_the_bad_peer from ckb-chain
eval-exec
committed
Use verify_callbacl to handle peer punish and compact block broadcast
eval-exec
committed
Fix sync and relayer unit test by verify_callback
eval-exec
committed
Remove VerifyFailedInfo, remove verify_failed_block channel
eval-exec
committed
Remove broadcast_compact_block channel
eval-exec
committed
Add debug log for block_fetcher->get_ancestor
eval-exec
committed
chore: remove channel send error callback

quake
authored and
eval-exec
committed
chore: remove UnverifiedBlock

quake
authored and
eval-exec
committed
Find and verify unverified blocks on ckb startup
eval-exec
committed
Fix cargo clippy warnings
eval-exec
committed
Use Release to store, Acquire to read for atomic: is_verifying_unverified_blocks_on_startup
eval-exec
committed
Do not accept descendants if parent is invalid
eval-exec
committed
Update rpc docs, fix CI test
eval-exec
committed
Release ckb-async-download rc0
eval-exec
committed
Do not re-store_unverified_block if it is still verifying by ConsumeUnverified
eval-exec
committed
Apply cargo clippy suggestions
eval-exec
committed
Rebased develop, regenerate rpc docs, remove orphan pool total_size limitation
eval-exec
committed
Prestore block to db before orphan pool, use fill unverified thread to pre load unverified block
eval-exec
committed
Shared::get_block_status always read BlockExt from snapshot
eval-exec
committed
Release ckb-async-download rc1
eval-exec
committed
Apply code review suggestions
eval-exec
committed
Remove block_status_map if block is is_internal_db_error
eval-exec
committed
Delete unverified block if verify failed
eval-exec
committed
Add SyncInvalid integration test
eval-exec
committed
Fix cargo clippy for integration test
eval-exec
committed

Commits on May 15, 2024

Temporary put parent header to PreloadUnverified, put small BlockView to
eval-exec
committed