Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure to add more than ~1200 files #369

Open
fabricedesre opened this issue Nov 17, 2023 · 0 comments
Open

Failure to add more than ~1200 files #369

fabricedesre opened this issue Nov 17, 2023 · 0 comments

Comments

@fabricedesre
Copy link
Collaborator

While debugging a failure to import largish directories, I found that adding more than ~1200 files would always fail with a variation of "Maximum block size exceeded". I can reproduce with the test case added in main...capyloon:rs-wnfs:perf-test

You can reproduce by running cargo test --release test_perf_add_files and will get something similar to:

fabrice@framework:~/dev/capyloon/rs-wnfs/wnfs perf-test$ cargo test --release test_perf_add_files -- --nocapture
warning: some crates are on edition 2021 which defaults to `resolver = "2"`, but virtual workspaces default to `resolver = "1"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
   Compiling wnfs v0.1.26 (/home/fabrice/dev/capyloon/rs-wnfs/wnfs)
    Finished release [optimized] target(s) in 15.49s
     Running unittests src/lib.rs (/home/fabrice/dev/capyloon/rs-wnfs/target/release/deps/wnfs-85ceb7b84c448bfe)

running 1 test
#0 : dir stored in 11ms
#100 : dir stored in 793ms
#200 : dir stored in 783ms
#300 : dir stored in 798ms
#400 : dir stored in 776ms
#500 : dir stored in 757ms
#600 : dir stored in 757ms
#700 : dir stored in 792ms
#800 : dir stored in 817ms
#900 : dir stored in 820ms
#1000 : dir stored in 866ms
#1100 : dir stored in 833ms
#1200 : dir stored in 886ms
thread 'private::directory::tests::test_perf_add_files' panicked at wnfs/src/private/directory.rs:1429:63:
called `Result::unwrap()` on an `Err` value: Maximum block size exceeded: Encountered block with 263232 bytes
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test private::directory::tests::test_perf_add_files ... FAILED

failures:

failures:
    private::directory::tests::test_perf_add_files

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 72 filtered out; finished in 19.68s

There is also a degradation of the dir.as_node().store(...) performance as we add more files. Is that a known issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant