Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Private File Sharding #88

Merged
merged 11 commits into from
Nov 21, 2022
Merged

Private File Sharding #88

merged 11 commits into from
Nov 21, 2022

Conversation

appcypher
Copy link
Member

Summary

This PR implements sharding of file content; breaking the content up and storing them in the HAMT if they are larger than certain threshold, storing them inline otherwise.

This PR implements the following features

  • Private file sharding
  • Stream shards
  • Wasm APIs
  • Tests

Test plan (required)

  • Testing

    rs-wnfs test --all

Closing issues

Fixes #80

@appcypher appcypher requested a review from a team as a code owner November 10, 2022 14:34
@codecov
Copy link

codecov bot commented Nov 10, 2022

Codecov Report

Merging #88 (fd1412a) into main (80dbe82) will increase coverage by 1.84%.
The diff coverage is 82.24%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #88      +/-   ##
==========================================
+ Coverage   66.05%   67.89%   +1.84%     
==========================================
  Files          26       26              
  Lines        1903     2003     +100     
  Branches      486      482       -4     
==========================================
+ Hits         1257     1360     +103     
- Misses        248      253       +5     
+ Partials      398      390       -8     
Impacted Files Coverage Δ
wnfs/src/common/error.rs 0.00% <ø> (ø)
wnfs/src/common/metadata.rs 67.56% <ø> (ø)
wnfs/src/common/utils.rs 70.00% <ø> (ø)
wnfs/src/private/forest.rs 59.15% <ø> (+2.81%) ⬆️
wnfs/src/private/node.rs 68.91% <ø> (ø)
wnfs/src/private/previous.rs 72.41% <ø> (+1.14%) ⬆️
wnfs/src/public/node.rs 54.79% <ø> (-1.37%) ⬇️
wnfs/src/private/directory.rs 75.42% <78.94%> (+2.92%) ⬆️
wnfs/src/private/file.rs 79.16% <82.95%> (+10.59%) ⬆️
... and 12 more

@appcypher
Copy link
Member Author

Just made our tests slower; enabled Safari and Edge for the wasm-js-tests. We've been ignoring them for a while. I'm considering disabling Edge since we have Chrome tests.

@appcypher
Copy link
Member Author

Made it faster switching to macos-latest. Still not as fast as I'd want so removing Edge tests is still on the table.

Copy link
Member

@matheus23 matheus23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only small things except for the shard order thing :)

wnfs/src/private/file.rs Outdated Show resolved Hide resolved
wnfs/src/private/file.rs Show resolved Hide resolved
wnfs/src/private/file.rs Outdated Show resolved Hide resolved
Copy link
Member

@matheus23 matheus23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Letsgooooo
One step closer to alpha!

Excited birthday girl

wnfs-bench/Cargo.toml Show resolved Hide resolved
wnfs-bench/Cargo.toml Show resolved Hide resolved
wnfs/src/public/node.rs Show resolved Hide resolved
wnfs/src/common/utils.rs Show resolved Hide resolved
wnfs-wasm/src/fs/private/file.rs Show resolved Hide resolved
wnfs-wasm/src/fs/utils.rs Outdated Show resolved Hide resolved
wnfs-wasm/src/fs/private/file.rs Outdated Show resolved Hide resolved
wnfs-wasm/src/fs/private/file.rs Outdated Show resolved Hide resolved
- Use array in create_private_file_result
Copy link
Contributor

@zeeshanlakhani zeeshanlakhani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comments left, great work.

- Add spec reference to `MAX_BLOCK_CONTENT_SIZE` constant
@appcypher appcypher merged commit 9d3b5c2 into main Nov 21, 2022
@appcypher appcypher deleted the appcypher/private-file-sharding branch November 21, 2022 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Private File Sharding
3 participants