Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy and/or md5 evaluation #1400

Open
tslater opened this issue Oct 15, 2024 · 1 comment
Open

Lazy and/or md5 evaluation #1400

tslater opened this issue Oct 15, 2024 · 1 comment

Comments

@tslater
Copy link

tslater commented Oct 15, 2024

I have very large files (20-100GB) with somewhat slow IO. I would love to see lazy md5 evaluation. It can takes tens of minutes just to calculate the hashes before uploading starts. Combined with uploads of multipart segments, uploads would be so much faster.

@fviard
Copy link
Contributor

fviard commented Oct 28, 2024

I understand perfectly your issue.
Sadly, at the moment, calculating the MD5 of a file is the only way to know if the file is the same for a "sync".
But on my todo list, I have the idea of the feature that you will be able to use filesystem metadata to be able to not recalculate the md5 when nothing was change.

To be noted, in the current version, you can already disable md5 calculation and just rely on "file size" and "last modified date" metadata, to go faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants