Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat: --skip-content-hash, --max-prefix-size, --max-suffix-size options #202

Merged
merged 3 commits into from
Jun 4, 2023
Merged

Feat: --skip-content-hash, --max-prefix-size, --max-suffix-size options #202

merged 3 commits into from
Jun 4, 2023

Commits on Jun 3, 2023

  1. Upgrade dirs, fallible-iterator, sysinfo 

    - dirs 4.0 -> 5.0.1
- fallible-iterator 0.2 -> 0.3
- sysinfo 0.28 -> 0.29
  - Required renaming DiskType to DiskKind in various places
    @johnpyp
    johnpyp committed Jun 3, 2023
    Configuration menu
    Copy the full SHA
    badd973 View commit details
    Browse the repository at this point in the history

  2. Add --max-prefix-size and --max-suffix-size options 

    --max-prefix-size - Configurable byte-size parameter for the max length of a file to hash for prefix checking

--max-suffix-size - Same as --max-prefix-size, but for the suffix check
    @johnpyp
    johnpyp committed Jun 3, 2023
    Configuration menu
    Copy the full SHA
    8f6a9a9 View commit details
    Browse the repository at this point in the history

  3. Add --skip-content-hash option 

    --skip-content-hash will skip the final stage, returning the results from the
previous groupings as the final result.

This can speed up the checking byorders of magnitude on large files, and alongside
--max-prefix-size and --max-suffix-size, can still provide reasonable guarantees
on whether files are duplicates.
    @johnpyp
    johnpyp committed Jun 3, 2023
    Configuration menu
    Copy the full SHA
    e29c933 View commit details
    Browse the repository at this point in the history