Caching MNIST dataset for testing

[djbyrne]

• Added MNIST datset to the tests directory

• Caches dataset based off hash of the test.pt file

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
• Did you read the contributor guideline?
• Did you make sure to update the docs?
• Did you write any new necessary tests?
• If you made a notable change (that affects users), did you update the CHANGELOG?

What does this PR do?

Fixes #859 .
Caches the MNIST dataset. This will prevent the CI/CD testing from crashing when the link to the original dataset is unavailable and will speed up CI/CD as the pipeline will not have to re-download files.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[awaelchli]

is it not possible to download it once and reuse? otherwise the package will grow by 100+ MB for only the tests and most users don't want that.

[williamFalcon]

thanks for the PR! we can’t add a dataset to the framework. @Borda

[Borda]

We are not adding them to package, it is just in CI test time, e.g. running PRs... Let review the PR =)

[williamFalcon]

i see binaries in the /test folder

[williamFalcon]

https://github.com/djbyrne/pytorch-lightning/tree/enhancement/lazy_dataloader/tests/models/mnist/MNIST

[Borda]

Ok, I see now, there is no need to add the dataset to git, torchvision check if it has it in temp and if not, then it is downloaded...
This PR shall contain only changes in Github action config

[Borda]

i see binaries in the /test folder

Yeah the was probably misunderstanding when we talked about caching lol

[williamFalcon]

https://pl-public-data.s3.amazonaws.com/MNIST/processed/test.pt
https://pl-public-data.s3.amazonaws.com/MNIST/processed/training.pt

https://pl-public-data.s3.amazonaws.com/MNIST/raw/t10k-images-idx3-ubyte
https://pl-public-data.s3.amazonaws.com/MNIST/raw/t10k-images-idx3-ubyte.gz
https://pl-public-data.s3.amazonaws.com/MNIST/raw/t10k-labels-idx1-ubyte
https://pl-public-data.s3.amazonaws.com/MNIST/raw/t10k-labels-idx1-ubyte.gz
https://pl-public-data.s3.amazonaws.com/MNIST/raw/train-images-idx3-ubyte
https://pl-public-data.s3.amazonaws.com/MNIST/raw/train-images-idx3-ubyte.gz
https://pl-public-data.s3.amazonaws.com/MNIST/raw/train-labels-idx1-ubyte
https://pl-public-data.s3.amazonaws.com/MNIST/raw/train-labels-idx1-ubyte.gz

[djbyrne]

i see binaries in the /test folder

Yeah the was probably misunderstanding when we talked about caching lol

@Borda apologies, I misunderstood the request. Ill remove the MNIST files from the directory.

[djbyrne]

It looks like the caching of the pip files is only being done for Linux. Should I add in caching of the locations for Windows and MacOS as well?

[williamFalcon]

@Borda good to go?

[williamFalcon]

@djbyrne can you subclass MNIST dataset for tests only and add the urls we have?

[djbyrne]

@williamFalcon this is just to override the resources list of links to the dataset right. This is different to the request to have an MNIST class decoupled from torchvision that @Borda mentioned?

[Borda]

Pls do not remove the blank lines, then it shall be fine for Linux tests
It would be nice to have the caching also for other platforms...

[Borda]

LGTM 🚀