feat: Streaming write for PrivateFile #163
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It can always be derived from `revision_key`. Storing it will only make it possible for `revision_key` and `content_key` to get out-of-sync.
instead of an encrypted set of Cids
So: - PrivateNodeHeader gets its own block - PrivateFile and PrivateDirectory refer back to the header via a CID - PrivateRef gets its own "disambiguation pointer" content_cid - PrivateForest now resolves PrivateRefs - PrivateRefs always refer to pre-existing content, never to "open slots"
instead of `RevisionKey` and `ContentKey`, respectively.
Also, make use of `Rc::make_mut`, accordingly.
(it had some good bugfixes regarding FinalizationRegistries recently)
matheus23
changed the title
appcypher
reviewed
Feb 21, 2023
appcypher
reviewed
Feb 21, 2023
appcypher
reviewed
Feb 21, 2023
appcypher
reviewed
Feb 21, 2023
appcypher
reviewed
Feb 21, 2023
appcypher
reviewed
Feb 21, 2023
|
For the file to test in the fixture, I went with a recording of Clara Schumann's Scherzo No. 2. The recording is from Luis Sarro and I downloaded it from musopen, so should be public domain. It's a classical music piece and Clara Schumann rocks (even though we've forgotten about her nowadays and her husband took the spotlight in today's history books). Honestly, I'm up for other ideas though! If anyone knows some cool fixture file to use that is >1MB and <10MB, let me know. |
This was referenced Feb 22, 2023
Closed
Closed
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #152
This is a minimal effort PR for importing big data into WNFS without insane memory requirements by streaming it in during file creation.
The incoming stream is split into private file chunks and encrypted & serialized one-by-one. Each chunk gets written before the next chunk gets fetched, thus this should take only roughly constant memory.
In the future we should look two more things:
PrivateDirectory::write, where currently that only takes aVec<u8>of bytes. We probably need to figure out something akin toPrivateDirectory::openwhich returns aPrivateFile, which can later be used inPrivateDirectory::write(perhaps keeping the old version, too, because it's a nice shortcut for small files).PrivateFilethat implementsAsyncRead + AsyncWrite + AsyncSeek. However, to make that performant on big files we need content-defined chunking.