"apt-get update" returns a "File has Unexpected Size" error when using cuda:10.1-cudnn7-*-ubuntu18.04 as base image (Docker build failing in CI)

[bnsblue]

1. Issue or feature description

I am trying to build a docker image using nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04 as the base.

FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
RUN apt-get update

The image builds successfully on my Mac laptop, but in Jenkins CI, it fails at apt-get update. I am attaching the error msg below.

12:05:28  Err:10 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages
12:05:28    File has unexpected size (47871 != 49498). Mirror sync in progress? [IP: 152.195.19.142 443]
12:05:28    Hashes of expected file:
12:05:28     - Filesize:49498 [weak]
12:05:28     - SHA256:332f3ee4e353b8a5e5a2bdd8fdbd47cf140c73822b82b328815f122e09e195a0
12:05:28     - SHA1:4dc8ef9a3ee3c97b3c26d46e07fdd83997e6880b [weak]
12:05:28     - MD5Sum:bbff3b9c3462257479d72521ee78ec29 [weak]
12:05:28    Release file created at: Wed, 23 Sep 2020 22:09:13 +0000

Similar

2. Steps to reproduce the issue

I am not sure how write this section because it fails only in my CI. But the docker file I am using is

FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
RUN apt update

3. Information to attach (optional if deemed irrelevant)

There have been similar issues:
#877
#1328

I tried the following:

• I added RUN apt-get clean before apt-get update but it didn't help.
• I also tried waiting for ~20 hours between builds but it gives me the same error
• I did the approach mentioned here Problem with repositories for Ubuntu 16.04 #571 (comment) -- still doesn't help
• I did this RUN apt-get -o Acquire::http::proxy=false update, but it still gives me the same error

A response in NVIDIA's developer forum indicates that it could be a CDN issue
https://forums.developer.nvidia.com/t/sudo-apt-get-update-issue/128229/25

[chandru-r]

I am hitting the same issue -

From: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/Release

49498 Packages.gz

However the file on disk is 47871.

I tried apt-get clean as well but didn't work.

[dotannn]

same here

[xf4j]

Same here for "nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04".

[nickmayn]

I'm facing the exact same issue

[SeraphicRav]

Same for nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04

[klueska]

This is not an issue for nvidia-docker itself.

All issues related to images running on top of nvidia-docker should be directed here:
https://forums.developer.nvidia.com/c/accelerated-computing/nvidia-gpu-cloud-ngc-users/25

[chandru-r]

Thanks I have filed:

https://forums.developer.nvidia.com/t/apt-get-update-fails-on-pytorch-pytorch-1-4-cuda10-1-cudnn7-devel/156259