Minikube start fails offline with docker driver since the kicbase image is not available in the cache

[eoinreilly93]

I am trying to install minikube on a CentOS 7 VM that has no access to the internet using the docker driver. To do this, I downloaded and installed minikube on a windows machine using Virtualbox with access to the internet and then I used the cache directory to install offline on the VM. However, when I run the minikube start command, it timed out as it was trying to download an image that was not located in the cache directory. This image was gcr.io/k8s-minikube/kicbase:v0.0.07.

To try and resolve this, I pulled this image locally and saved it to .tar file, and brought it across to my VM, and used docker load to add it to my locally registry, thinking it would solve the issue. This has not worked though, since the image repo digest is not saved as part of the save command. Minikube appears to be using the repo digest as a way to verify the image, and since it doesn't exist, it then tries to pull the image from the internet, which it cannot do.

1. Why is it using the image repo digest and not the image id to verify if the image is located in the local docker registry?
2. Is there a way I can get around this problem? As I see currently see it, there is no way to install minikube without an internet connection while using the docker driver. This seems like a major issue.

My environment is:
CentOS 7.71908
Docker 19.03.3
Minikube 1.8.2

Steps to reproduce the issue:

1.Login as as non-root user
2. Bring minikube installation binary onto VM
3. Run chmod +x on the binary
4. Run install minikube /usr/local/bin
5. Run minikube version to check installation is successful
6. Bring the cache files into the .minikube/cache directory
7. Load the docker images into the local registry
8. Bring the tarred kicbase image on to the VM and load it into the local registry
9. Start minikube with minikube start --driver=docker --kubernetes-version=1.16.3

Full output of failed command:

[servicereporting@SR-SVR-206 ~]$ minikube start --driver=docker --kubernetes-version=1.16.3
! minikube v1.8.2 on Centos 7.7.1908

• Using the docker driver based on user configuration
  E0406 17:25:10.412815 7184 cache.go:106] Error downloading kic artifacts: getting remote image: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 15.6.0.42:53: read udp 15.6.1.206:41122->15.6.0.42:53: i/o timeout
• Creating Kubernetes in docker container with (CPUs=2) (8 available), Memory=15600MB (62753MB available) ...

X Unable to start VM. Please investigate and run 'minikube delete' if possible: creating host: create: creating: create kic node: create container: failed args: [run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --cpus=2 --memory=15600mb --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 gcr.io/k8s-minikube/kicbase:v0.0.7@sha256:a6f288de0e5863cdeab711fa6bafa38ee7d8d285ca14216ecf84fcfb07c7d176] output: Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.7@sha256:a6f288de0e5863cdeab711fa6bafa38ee7d8d285ca14216ecf84fcfb07c7d176' locally
docker: Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
: exit status 125
*

• minikube is exiting due to an error. If the above message is not useful, open an issue:
  • https://github.com/kubernetes/minikube/issues/new/choose

Optional: Full output of minikube logs command:

[servicereporting@SR-SVR-206 ~]$ minikube logs
E0406 17:49:55.384264 8562 logs.go:174] Failed to list containers for "kube-apiserver": docker: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}: exit status 1
stdout:

stderr:
Error: No such container: minikube
E0406 17:49:55.445187 8562 logs.go:174] Failed to list containers for "coredns": docker: docker ps -a --filter=name=k8s_coredns --format={{.ID}}: exit status 1
stdout:

stderr:
Error: No such container: minikube
E0406 17:49:55.501211 8562 logs.go:174] Failed to list containers for "kube-scheduler": docker: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}: exit status 1
stdout:

stderr:
Error: No such container: minikube
E0406 17:49:55.554556 8562 logs.go:174] Failed to list containers for "kube-proxy": docker: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}: exit status 1
stdout:

stderr:
Error: No such container: minikube
E0406 17:49:55.613018 8562 logs.go:174] Failed to list containers for "kubernetes-dashboard": docker: docker ps -a --filter=name=k8s_kubernetes-dashboard --format={{.ID}}: exit status 1
stdout:

stderr:
Error: No such container: minikube
E0406 17:49:55.665198 8562 logs.go:174] Failed to list containers for "storage-provisioner": docker: docker ps -a --filter=name=k8s_storage-provisioner --format={{.ID}}: exit status 1
stdout:

stderr:
Error: No such container: minikube
E0406 17:49:55.715815 8562 logs.go:174] Failed to list containers for "kube-controller-manager": docker: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}: exit status 1
stdout:

stderr:
Error: No such container: minikube

• ==> Docker <==
  E0406 17:49:55.777566 8562 logs.go:152] command /bin/bash -c "sudo journalctl -u docker -n 60" failed with error: /bin/bash -c "sudo journalctl -u docker -n 60": exit status 1
  stdout:

stderr:
Error: No such container: minikube
output: "\n** stderr ** \nError: No such container: minikube\n\n** /stderr **"
*

• ==> container status <==
  E0406 17:49:55.844556 8562 logs.go:152] command /bin/bash -c "sudo which crictl || echo crictl ps -a || sudo docker ps -a" failed with error: /bin/bash -c "sudo which crictl || echo crictl ps -a || sudo docker ps -a": exit status 1
  stdout:

stderr:
Error: No such container: minikube
output: "\n** stderr ** \nError: No such container: minikube\n\n** /stderr **"
*

• ==> dmesg <==
  E0406 17:49:55.904185 8562 logs.go:152] command /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 60" failed with error: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 60": exit status 1
  stdout:

stderr:
Error: No such container: minikube
output: "\n** stderr ** \nError: No such container: minikube\n\n** /stderr **"
*

• ==> kernel <==
  E0406 17:49:55.953652 8562 logs.go:152] command /bin/bash -c "uptime && uname -a && grep PRETTY /etc/os-release" failed with error: /bin/bash -c "uptime && uname -a && grep PRETTY /etc/os-release": exit status 1
  stdout:

stderr:
Error: No such container: minikube
output: "\n** stderr ** \nError: No such container: minikube\n\n** /stderr **"
*

• ==> kubelet <==
  E0406 17:49:56.007246 8562 logs.go:152] command /bin/bash -c "sudo journalctl -u kubelet -n 60" failed with error: /bin/bash -c "sudo journalctl -u kubelet -n 60": exit status 1
  stdout:

stderr:
Error: No such container: minikube
output: "\n** stderr ** \nError: No such container: minikube\n\n** /stderr **"
*
X Error getting machine logs: unable to fetch logs for: Docker, container status, dmesg, kernel, kubelet
*

• minikube is exiting due to an error. If the above message is not useful, open an issue:
  • https://github.com/kubernetes/minikube/issues/new/choose

Docker images digest

REPOSITORY                    TAG                 DIGEST              IMAGE ID            CREATED             SIZE
gcr.io/k8s-minikube/kicbase   v0.0.7              <none>              7980bce73693        6 weeks ago         935MB

[afbjorklund]

As discussed on Slack:

• Why is it using the image repo digest and not the image id to verify if the image is located in the local docker registry?

The image ID is local to your daemon, so not very worthwhile to identify an image. The alternative here would be to just use the tag (since those are supposed to be unique, if not rolling like latest)

• Is there a way I can get around this problem? As I see currently see it, there is no way to install minikube without an internet connection while using the docker driver. This seems like a major issue.

You might have better luck deploying a local registry mirror, instead of trying to save and load images (which deletes all digests by design). Then you can use that mirror, instead of the usual ones ?

See --registry-mirror for minikube start

https://docs.docker.com/registry/recipes/mirror/

[medyagh]

I am going to make a PR to add alternative image repo other than GCR (github packages)
@eoinreilly93
here is the link to the binary from a PR that I think might fix this issue, do you mind trying it out ?

http://storage.googleapis.com/minikube-builds/7943/minikube-linux-amd64

[medyagh]

@eoinreilly93 also I belive the reason loading and saving the image didn't help you is because minikube checks the RepoDigestSHA and I believe your image would loose the repo digest sha.

we also have a new flag in our latest beta release that is called
minikube start --driver=docker --base-image="yourimagename"

can you try using that option, that way u can override the SHA