Make verification data sets smaller or optional

[hdrake]

Thanks again Gael for putting so much work into this great package. I'm finally trying to get my hands into it.

I'm not sure how big the example datasets are, but they were prohibitively large for me to either clone the repo or install the package from the manager using the ship's bandwidth at sea (other packages, like Oceananigans.jl, were just fine). Would it be possible to either make downloading this data part of a second optional step, or else to use smaller example datasets?

[gaelforget]

Hi Henri

1. for now can you try wget https://github.com/JuliaClimate/IndividualDisplacements.jl/archive/refs/tags/v0.3.2.tar.gz ?

This file is 3.6M (down from 165M!) and I hope won't be prohibitive. I wonder how long downloading this file from the ship may take if 3.6M is not prohibitive.

It's actually the .git/objects/pack folder that's surprisingly big (161M / 165M). I will dig dipper but it seems that this has to do with old ipynb and png file versions. Thanks for bringing this up!!

(to self : see 18F/C2#439 for a similar issue with fixes)

2. A second thing that may help is before you call Pkg.develop from within Julia, add lazy = true at line 3 of the Artifacts.toml file in the package. This will skip downloading that artefact (should be the default in future release).

3. I need to take another look at other repos wrt aspects 1 and 2 which are dependencies of IndividualDisplacements.jl (MITgcmTools.jl in particular; could try adding lazy = true there too).

[gaelforget]

[Update on my previous post]

Following #80 the git clone of the repo should amount to 20M instead of 165M.

Can you please confirm that this is what you are experiencing?

I also made the artifact lazy, and (re)moved dependencies that could add un-necessary downloads. New release, v0.3.3, should be available to Pkg.add soon.

[hdrake]

Thanks @gaelforget! I was able to sucessfully clone the repo this morning (after a couple of failed attempts when the internet cut out)!

Cloning into 'IndividualDisplacements.jl'...
remote: Enumerating objects: 6197, done.
remote: Counting objects: 100% (3321/3321), done.
remote: Compressing objects: 100% (1614/1614), done.
remote: Total 6197 (delta 1359), reused 3286 (delta 1341), pack-reused 2876
Receiving objects: 100% (6197/6197), 15.49 MiB | 99.00 KiB/s, done.
Resolving deltas: 100% (3378/3378), done.

[gaelforget]

closing this now. One thing to note is that you might want to delete and recreate your fork if you want to use master branch in it. That's a side effect of https://rtyley.github.io/bfg-repo-cleaner/ which I used to do this. Thanks