unixfs: feat tree building by koivunej · Pull Request #284 · rs-ipfs/rust-ipfs

koivunej · 2020-08-05T10:36:25Z

Includes a BufferingTreeBuilder which kind of follows the js-ipfs implementation: a complete tree is built from path segments and later it's converted into a dag-pb by an "iterator" which walks the tree in post order, converting (rendering) the most nested elements first. Communication back to the "parent" is done using a HashMap<u64, Vec<Option<(String, Cid, usize)>>> which relies on initial ordering provided by BufferingTreeBuilder by using BTreeMap's on each level.

unixfs/src/dir/builder.rs

http/src/v0/root_files/add.rs

unixfs/src/dir/builder/buffered.rs

this uses the rust 1.45 features to remove recursion inside the macro. great for rs-ipfs#284 which adds even more complicated async-stream(s).

288: fix: update async-stream to 0.3 r=ljedrz a=koivunej this uses the rust 1.45 features to remove recursion inside the macro. great for #284 which adds even more complicated async-stream(s). good to go if CI passes. Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>

koivunej · 2020-08-06T08:17:54Z

I'll force push once #288 is merged with the few TODO comments which are missing. I am still thinking if I should render the AddErrors as StringError as a new json line since that's the best we can do. It might even trigger an error on the js-ipfs-http-client side which would be better than hanging.

288: fix: update async-stream to 0.3 r=ljedrz a=koivunej this uses the rust 1.45 features to remove recursion inside the macro. great for #284 which adds even more complicated async-stream(s). good to go if CI passes. Co-authored-by: Joonas Koivunen <joonas@equilibrium.co>

koivunej · 2020-08-06T13:01:57Z

I think this is now ready for review. Thanks to npm for providing me focus time to try out the perf enchancements (which could be dropped off if they look bad enough) :)

http/src/v0/root_files/add.rs

koivunej · 2020-08-06T13:20:49Z

Wantlist cancellation failure: https://github.com/rs-ipfs/rust-ipfs/pull/284/checks?check_run_id=953826545#step:5:633

http/src/v0/root_files/add.rs

not sure what is the index file hash change.

unixfs/examples/ingest-tar.rs

unixfs/src/dir/builder/buffered.rs

unixfs/benches/ingest-linux-tar.rs

unixfs/src/dir/builder/iter.rs

http/src/v0/root_files/add.rs

unixfs/src/dir/builder/iter.rs

Co-authored-by: ljedrz <ljedrz@users.noreply.github.com>

koivunej · 2020-08-10T09:04:39Z

bors r+

284: unixfs: feat tree building r=koivunej a=koivunej Includes a `BufferingTreeBuilder` which kind of follows the js-ipfs implementation: a *complete* tree is built from path segments and later it's converted into a dag-pb by an "iterator" which walks the tree in post order, converting (rendering) the most nested elements first. Communication back to the "parent" is done using a `HashMap<u64, Vec<Option<(String, Cid, usize)>>>` which relies on initial ordering provided by `BufferingTreeBuilder` by using BTreeMap's on each level. Co-authored-by: Joonas Koivunen <joonas@equilibrium.co> Co-authored-by: Joonas Koivunen <joonas.koivunen@gmail.com>

koivunej · 2020-08-10T09:12:22Z

readme_doctest with a https://github.com/rs-ipfs/rust-ipfs/pull/284/checks?check_run_id=965960327#step:5:574 in wantlist_cancellation.

koivunej · 2020-08-10T09:18:29Z

Same with bors... Interesting.

bors r-

bors · 2020-08-10T09:18:30Z

Canceled.

koivunej · 2020-08-10T09:18:34Z

bors r+

bors · 2020-08-10T09:42:20Z

Build succeeded:

ci

koivunej changed the title ~~Add tree building~~ unixfs: feat tree building Aug 5, 2020

koivunej commented Aug 5, 2020

View reviewed changes

unixfs/src/dir/builder.rs Outdated Show resolved Hide resolved

koivunej commented Aug 5, 2020

View reviewed changes

unixfs/src/dir/builder.rs Outdated Show resolved Hide resolved

koivunej commented Aug 5, 2020

View reviewed changes

unixfs/src/dir/builder.rs Outdated Show resolved Hide resolved

koivunej commented Aug 5, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

koivunej commented Aug 5, 2020

View reviewed changes

http/src/v0/root_files/add.rs Show resolved Hide resolved

koivunej commented Aug 5, 2020

View reviewed changes

unixfs/src/dir/builder/buffered.rs Outdated Show resolved Hide resolved

koivunej pushed a commit to eqlabs/rust-ipfs that referenced this pull request Aug 6, 2020

fix: update async-stream to 0.3

5b6b5dd

this uses the rust 1.45 features to remove recursion inside the macro. great for rs-ipfs#284 which adds even more complicated async-stream(s).

koivunej mentioned this pull request Aug 6, 2020

fix: update async-stream to 0.3 #288

Merged

koivunej force-pushed the add_tree_building branch 3 times, most recently from b5bf673 to aab2e82 Compare August 6, 2020 12:55

koivunej marked this pull request as ready for review August 6, 2020 13:01

koivunej requested a review from ljedrz August 6, 2020 13:02

ljedrz reviewed Aug 6, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 6, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 6, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 6, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 6, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 6, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 6, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 6, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 6, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

Joonas Koivunen added 4 commits August 9, 2020 22:24

chore: cleanup, doc

28392da

fix: make the benchmark more generic

5484f3e

chore: cargo keeps changing the index file, clippy

71b02cf

not sure what is the index file hash change.

doc: minor docs and fmt

7446810

koivunej force-pushed the add_tree_building branch from a6ce6f9 to 7446810 Compare August 9, 2020 19:25

koivunej commented Aug 10, 2020

View reviewed changes

unixfs/examples/ingest-tar.rs Outdated Show resolved Hide resolved

koivunej commented Aug 10, 2020

View reviewed changes

unixfs/src/dir/builder/buffered.rs Outdated Show resolved Hide resolved

Joonas Koivunen added 3 commits August 10, 2020 09:13

refactor: remove duplicated ingest-tar example, it remains as bench

9df95ad

refactor: rename put_file => put_link

bfc1bf1

doc: comment touch ups

665c0c4

ljedrz reviewed Aug 10, 2020

View reviewed changes

unixfs/benches/ingest-linux-tar.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 10, 2020

View reviewed changes

unixfs/benches/ingest-linux-tar.rs Show resolved Hide resolved

ljedrz reviewed Aug 10, 2020

View reviewed changes

unixfs/src/dir/builder/iter.rs Outdated Show resolved Hide resolved

ljedrz reviewed Aug 10, 2020

View reviewed changes

http/src/v0/root_files/add.rs Outdated Show resolved Hide resolved

Joonas Koivunen added 2 commits August 10, 2020 11:44

bench: move buffer and fix path growing outside

14c27fd

bench: filter symlinks earlier

276e4b7

ljedrz reviewed Aug 10, 2020

View reviewed changes

unixfs/src/dir/builder/iter.rs Outdated Show resolved Hide resolved

doc: suggestions and notes from code review

aaabc37

Co-authored-by: ljedrz <ljedrz@users.noreply.github.com>

koivunej force-pushed the add_tree_building branch from bee60ae to aaabc37 Compare August 10, 2020 09:04

bors bot merged commit e00198a into rs-ipfs:master Aug 10, 2020

koivunej mentioned this pull request Aug 10, 2020

ipfs-unixfs: importing #214

Closed

6 tasks

koivunej mentioned this pull request Sep 23, 2020

Implement Ipfs::add #395

Open

koivunej deleted the add_tree_building branch September 24, 2020 12:56

Comments

Conversation

koivunej commented Aug 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

koivunej commented Aug 6, 2020

Uh oh!

koivunej commented Aug 6, 2020

Uh oh!

Uh oh!

Uh oh!

koivunej commented Aug 6, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

koivunej commented Aug 10, 2020

Uh oh!

koivunej commented Aug 10, 2020

Uh oh!

koivunej commented Aug 10, 2020

Uh oh!

bors bot commented Aug 10, 2020

Uh oh!

koivunej commented Aug 10, 2020

Uh oh!

bors bot commented Aug 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

koivunej commented Aug 5, 2020 •

edited

Loading