Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use ChainExportRangeInternal to write snapshots directly to disk #67

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

ognots
Copy link
Collaborator

@ognots ognots commented Mar 29, 2023

we expect this to be perform much better, than using the ChainExport method and streaming it through the API.

this commit only guarantees that filecoin-chain-archiver binary builds. writing some tests and doing manual testing is still required.

we expect this to be perform much better, than using the ChainExport method and streaming it through the API.

this commit only guarantees that filecoin-chain-archiver binary builds.
writing some tests and doing manual testing is still required.
* fixes a lot of bugs, found through local testing
* now that lotus support importing zstd compressed snapshots, don't even bother with the raw snapshots
the snapshot car files are continually appended to while the ExportRangeInternal process executes.
if one uses the os.File io.Reader, it will encounter an EOF before the snapshot is complete.
our custom io.Reader returns an io.EOF when the ExportRangeInternal method completes
* do some refactoring so we can pipe the zstd encoded snapshot to file for debugging
* select statement in custom reader was blocking without setting default
initiate reader and pass it in as argument to newSnapshotReader.
also do some cleanup
@ognots ognots force-pushed the feat/chain-export-range branch 2 times, most recently from 88dcb4c to 55bb87d Compare April 25, 2023 02:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant