Downloading MNIST fails due to Cloudflare protection

[Steven-Adriaensen]

Automatically downloading MNIST currently fails (HTTP Error 403) as Cloudflare blocks requests using the default urllib headers.

Issue & solution:
https://stackoverflow.com/questions/60548000/getting-http-error-403-forbidden-error-when-download-mnist-dataset

[TheEimer]

We could potentially also just update torchvision as I at least had issues with these fixes. So we should investigate if that would cause issues within the SGD benchmark.

[maximilianreimer]

I am currently investigating this bug and there are three things that bother me:

1. With fixes currently in the code it does not seem to work for me
2. I added an alternative fix, suggested in the issue above, and it seems to work at least sometimes but not always. (see listing below)
3. Even when I tested download the dataset with the current torch and torchvision version just using this train_dataset = datasets.MNIST('../data', train=True, download=True) I got the same error.

I get the feeling that you are getting blocked if you are trying to download it too often. Maybe we should make sure that in our test environment the file is cached over multiple runs.

import urllib

opener = urllib.request.build_opener()
opener.addheaders = [("User-agent", "Mozilla/5.0")]
urllib.request.install_opener(opener)

train_dataset = datasets.MNIST('../data', train=True, download=True)

[TheEimer]

I tried both solutions and torch (which is supposed to work with the current version according to their GitHub issues), so I think trying to cache it would be best. Everything else seems like a lot of trouble that's very hard to debug.

[maximilianreimer]

Apparently, it has only been fixed in the nightly build but will be incorporated in the next minor release (See Pytorch Issue 3549. I added a hotfix for now.