Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FR: Add feature that caches downloads #137

Open
psanker opened this issue Jan 31, 2021 · 1 comment
Open

FR: Add feature that caches downloads #137

psanker opened this issue Jan 31, 2021 · 1 comment

Comments

@psanker
Copy link
Contributor

psanker commented Jan 31, 2021

It'd be great if there were a use_cache parameter in osfr::osf_download() to make downloads easier. I have several workflows built with drake that download data files from the OSF that rarely change. Because of the infrequent changes, I wrote a utility that caches the downloads based on the date_modified field.

Since I have something written up already, I'd be more than welcome to open a PR if you think this feature is worth including!

@aaronwolen
Copy link
Member

Hey @psanker, I totally agree this would be a useful addition.

The OSF API actually provides checksums for remote files, so my plan is to add an argument that enables checksum comparisons to determine whether a file should be downloaded or not.

Dates could work too and would certainly be faster than calculating checksums locally but I'm concerned about false positives/negatives if the client/server system times aren't in sync.

I'll also note that osf_download()'s current implementation is a little janky and doesn't fully recurse the way osf_upload() does. This is only an issue when download targets include directories as the entire directory is always downloaded to a temp folder and then matching files are selectively copied over to the specified destination.

So there are a few issues but I'm open to discussing further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants