Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PIBD_IMPL] Introduce PIBD state into sync workflow #3685

Merged
merged 6 commits into from
Jan 12, 2022
Merged

[PIBD_IMPL] Introduce PIBD state into sync workflow #3685

merged 6 commits into from
Jan 12, 2022

Conversation

yeastplume
Copy link
Member

@yeastplume yeastplume commented Jan 10, 2022 •
edited
Loading

First in a continuing series of PRs into the pibd_impl branch, the overall goal of which is to create an initial usable implementation of PIBD on the test network.

Note the current state of code is unlikely to be useful to anyone except me at the moment. Will make a note in later PRs when things are ready to try.

This:

  • Adds a TxHashsetPibd state to sync status, which is
  • Defaults to attempting PIBD after header sync when running on testnet (note this will remain in this state indefinitely at present pending more implementation)
  • Modifies sync thread to call into a pibd_continue function which will ultimately check the state of PIBD and either call the desegmenter and peers with next steps or update sync state. For now it just sends a single message to a listening peer.
  • Adds p2p functions to peers, adapters and network related code to perform PIBD requests to peers
  • Fix to desegmenter PMMR size calc function
  • Fix to BitmapAccumulator -> raw croaring::Bitmap conversion function
  • Add function to chain to select archive block header without reference to anything but stored headers (essentially move code from syncer into somewhere more common.

@yeastplume yeastplume changed the title [WIP, PIBD] Test PIBD Message Integration [WIP, PIBD] PIBD Sync over Testnet Jan 10, 2022
@yeastplume yeastplume changed the base branch from master to pibd_impl January 12, 2022 12:45
@yeastplume yeastplume changed the title [WIP, PIBD] PIBD Sync over Testnet [PIBD_IMPL] Introduce PIBD state into workflow Jan 12, 2022
@yeastplume yeastplume changed the title [PIBD_IMPL] Introduce PIBD state into workflow [PIBD_IMPL] Introduce PIBD state into sync workflow Jan 12, 2022
@yeastplume yeastplume merged commit 89730b7 into mimblewimble:pibd_impl Jan 12, 2022
@yeastplume yeastplume deleted the pibd_sync_test_1 branch January 20, 2022 10:00
@yeastplume yeastplume mentioned this pull request Feb 21, 2022
Merged
26 tasks
yeastplume added a commit that referenced this pull request Oct 18, 2022
@yeastplume
[DNM] PIBD Task / Issue Tracker (#3695)
030bd0e 
* [PIBD_IMPL] Introduce PIBD state into sync workflow (#3685)

* experimental addition of pibd download state for testnet only

* fixes to bitmap number of segments calculation + conversion of bitmap accumulator to bitmap

* attempt to call a test message

* add p2p methods for receiving bitmap segment and applying to desegmenter associated with chain

* fixes to state sync

* add pibd receive messages to network, and basic calls to desegmenter from each (#3686)

* [PIBD_IMPL] PIBD Desegmenter State (#3688)

* add functions to desegmenter to report next desired segments, begin to add state to determine which segments have been requested

* add segmentidentifier type to id requested segments uniquely

* make a call on where to keep track of which PIBD segments have been requested

* move segmenttype definition, add functions to manipulate peer segment list

* remove desegmenter state enum

* change chain desegmenter function to provide rwlock

* trace, warning cleanup

* udpate to test compliation

* [PIBD_IMPL] Bitmap accumulator reconstruction + TxHashset set reconstruction (#3689)

* application of received bitmap segments to local accumulator

* add all required elements to send/receive output segment requests and responses

* testing of output sync

* add special cases to pmmr segment request

* [PIBD_IMPL] PMMR Reassembly from Segments (#3690)

* update pibd copy test to use new desgmenter structure

* begin reconstruction of output pmmr

* clean up hash/leaf insertion logic

* push pruned subtree appears to be working, now also calculates left hand hashes correctly

* factor out ordering of segment/hash order array

* refactor for pmmr application code

* test of chain copy appears to be working

* add rangeproof functions to desegmenter

* add kernel functions, attempt refactor

* small test cleanup, reconstruction of live chain working in manual copy test

* [PIBD_IMPL] PIBD tree sync via network and kill/resume functionality (#3691)

* add functions to determing latest verifiable block height for the given pibd state

* attempting to allow for pibd to resume after killing process

* fix to ensure prune list is properly flushed during pibd sync

* removal of unneeded code

* ignore test for now (fix before full merge)

* [PIBD_IMPL] Finalize PIBD download and move state to chain validation (#3692)

* investigations as to why a slight rewind is needed on startup during PIBD

* move validation code into desegmenter validation thread (for now)

* ensure genesis entries in pmmrs are removed if they're removed in the first segment

* validation all working except for verifying kernel sums

* remove unneeded pmmr rollbacks on resume now root cause was found

* updates to remove unpruned leaves from leaf set when rebuilding pmmr

* remove + 1 to segment traversal iter length

* [PIBD_IMPL] PIBD Stats + Retry on validation errors (#3694)

* start to add stats and reset chain state after errors detected

* add functions to reset prune list when resetting chain pibd state

* debug statement

* remove test function

* [PIBD_IMPL] Update number of simultaneous peer requests for segments (#3696)

* cleanup of segment request list

* allow for more simultaneous requests during state sync

* up number of simultaneous peer requests for segments

* [PIBD_IMPL] Thread simplification + More TUI Updates + Stop State Propagation (#3698)

* change pibd stat display to show progress as a percentage of downloaded leaves

* attempt some inline rp validation

* propagate shutdown state through kernel validation

* change validation loop timing

* simplify validator threading

* add more detailed tracking of kernel history validation to tui, allow stop state during

* adding more stop state + tui progress indication

* remove progressive validate

* test fix

* revert to previous method of applying segments (#3699)

* fix for deadlock issue (#3700)

* update Cargo.lock for next release

* [PIBD_IMPL] Catch-Up functionality + Fixes based on testing (#3702)

* ensure desegmenter attempts to apply correct block after a resumte

* ensure txhashset's committed implementation takes into account output bitmap for summing purposes

* remove check to de-apply outputs during segment application

* return removal of spent outputs during pibd

* remove unneeded status

* remove uneeded change to rewind function

* documentation updates + todo fixes (#3703)

* add pibd abort timeout case (#3704)

* [PIBD_IMPL] BitmapAccumulator Serialization Fix (#3705)

* fix for writing / calculating incorrect length for negative indices

* update capabilities with new version of PIBD hist

* remove incorrect comment

* fix capabilities flag, trace output

* test fix

* Merge DNSSeed scope changes into pibd impl branch (#3708)

* update Cargo.lock for next release

* visibility scope tweaks to aid seed test utilities (#3707)

* move all PIBD-related constants into pibd_params modules (#3711)

* remove potential double read lock during compaction
bayk added a commit to mwcproject/mwc-node that referenced this pull request Jun 23, 2024
@bayk
grin v5.3 (0111) [DNM] PIBD Task / Issue Tracker (mimblewimble#3695)
70b4411 
* [PIBD_IMPL] Introduce PIBD state into sync workflow (mimblewimble#3685)
* experimental addition of pibd download state for testnet only
* fixes to bitmap number of segments calculation + conversion of bitmap accumulator to bitmap
* attempt to call a test message
* add p2p methods for receiving bitmap segment and applying to desegmenter associated with chain
* fixes to state sync
* add pibd receive messages to network, and basic calls to desegmenter from each (mimblewimble#3686)
* [PIBD_IMPL] PIBD Desegmenter State (mimblewimble#3688)
* add functions to desegmenter to report next desired segments, begin to add state to determine which segments have been requested
* add segmentidentifier type to id requested segments uniquely
. . . and more . . .
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@yeastplume