minikube cache add looking for remote images not local?

[artemkozlenkov]

the following is the ouput after the command minikube cache add php:api

E0728 22:02:58.007461 352660 cache.go:63] save image to file "php:api" -> "/home/artem/.minikube/cache/images/php_api" failed: nil image for php:api: GET https://index.docker.io/v2/library/php/manifests/api: MANIFEST_UNKNOWN: manifest unknown; map[Tag:api]

💣 Failed to cache and load images: save to dir: caching images: caching image "/home/artem/.minikube/cache/images/php_api": nil image for php:api: GET https://index.docker.io/v2/library/php/manifests/api: MANIFEST_UNKNOWN: manifest unknown; map[Tag:api]

minikube cache add docker.io-image:tag adds no problem any remote image, doesn't it contradict the description of minikube cache --help where it says "Add, delete, or push a local image into minikube"

it used to work fine couple of weeks ago and I could push local images to minikube node, seems like not any more?

could someone help to clarify on this please?

Best regards,
Artem.

[afbjorklund]

Possibly related to updates to the go-containerregistry library used for caching.

05459a2

We now always pull the latest image from the registry, in order to save the digest.
Apparently that broke locally created images (that don't have a digest anyway...)

Should probably go with the local "php:api", instead of "docker.io/library/php:api" ?

[afbjorklund]

Podman uses localhost/ to discriminate, but Docker doesn't have that feature.

However the code was supposed to ask the docker daemon before retrieving:

image.go:168] retrieving image: php:api
image.go:176] daemon lookup for php:api: Error response from daemon: client version 1.41 is too new. Maximum supported API version is 1.40

It fails due to this fix not being merged properly: google/go-containerregistry@72edbad

[afbjorklund]

This is the upstream bug: google/go-containerregistry#734

We need a regression test. For caching a local image, that is...

[sehz]

Bitten by same bug

[mbaeuerle]

Short question about what can be cached from a local repository.
It seems like it's only possible to use minikube cache add on images which were built locally and then pushed to the local registry.
If Minikube fetched the images remotely (during deployment) I was not able to cache the local image afterwards. This is even though the downloaded images are shown beside the locally created ones when checking via minikube ssh docker images.
This makes it hard to cache all images after a deployment in Minikube finished for the first time.

E.g (mongo:4.2 already present via minikube ssh docker images):

$ minikube --stderrthreshold=0 cache add mongo:4.2                                                                                

I1125 18:08:35.364227  263858 cache.go:92] acquiring lock: {Name:mkead6adaa1c91f5a321b5ba17524b30cb406f5a Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I1125 18:08:35.364432  263858 image.go:168] retrieving image: mongo:4.2
I1125 18:08:35.366088  263858 image.go:176] daemon lookup for mongo:4.2: Error response from daemon: reference does not exist
I1125 18:08:38.335112  263858 cache.go:134] opening:  /home/marian/.minikube/cache/images/mongo_4.2

Whereas for a locally built image it looks like this:

I1125 18:10:14.491939  265092 cache.go:92] acquiring lock: {Name:mk91390f87dee1541681e7034d3b3107fd968ada Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I1125 18:10:14.492208  265092 image.go:168] retrieving image: xyz.azurecr.io/local-image:0-SNAPSHOT
I1125 18:10:19.083560  265092 image.go:171] found xyz.azurecr.io/local-image:0-SNAPSHOT locally: &{Image:0xc00000eac0}
I1125 18:10:19.083612  265092 cache.go:134] opening:  /home/marian/.minikube/cache/images/xyz.azurecr.io/local-image_0-SNAPSHOT

[afbjorklund]

What happened later ? It is supposed to be able to fetch it now:

I1125 19:12:26.264960  486948 image.go:168] retrieving image: mongo:4.2
I1125 19:12:26.265569  486948 image.go:176] daemon lookup for mongo:4.2: Error response from daemon: reference does not exist
I1125 19:12:28.164140  486948 cache.go:134] opening:  /home/anders/.minikube/cache/images/mongo_4.2
I1125 19:12:37.227125  486948 cache.go:129] /home/anders/.minikube/cache/images/mongo_4.2 exists
I1125 19:12:37.227194  486948 cache.go:81] cache image "mongo:4.2" -> "/home/anders/.minikube/cache/images/mongo_4.2" took 10.962812489s
I1125 19:12:37.227223  486948 cache.go:66] save to tar file mongo:4.2 -> /home/anders/.minikube/cache/images/mongo_4.2 succeeded
I1125 19:12:37.227242  486948 cache.go:73] Successfully saved all images to host disk.

Note there is a slightly pause there, as it is doing the actual fetch.

[mbaeuerle]

Sorry I guess my question and example were a bit misleading.

The command above for caching the mongo image indeed does not fail.
It will download the image from the remote server even though it is already present in the local docker registry (minikube ssh docker images).

The issue with that is if the remote is requiring authorization the request will fail and therefore also the minikube cache add command. Again it could have just retrieved the image from the local docker registry but it has not.

Exiting due to MK_CACHE_LOAD: save to dir: caching images: caching image "/home/marian/.minikube/cache/images/some-image": nil image for some-image: 
GET https://index.docker.io/v2/library/some-image/manifests/some-version: 
UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/some-image Type:repository]]

Is this behaviour correct in these cases? My expectation was that the image is always pulled from the local docker registry when the requested version is available there and just query the remote server if it is not found locally.

[afbjorklund]

I think the cache command only checks the docker daemon on the host, it does not check the images on the machine.

i.e. docker images, rather than minikube ssh docker images

It will download the images from the registry to the local disk, but it should also check on the machine before loading it.

in the "needsTransfer", it will compare the image lists and digests

[mbaeuerle]

I was running eval $(minikube docker-env -u) first then both minikube ssh docker images and docker images are pointing to the same registry.
So the behaviour I observed is only images which were built locally and pushed to the local registry were actually taken from the local repository when calling minikube cache add.
Where can I find this "needsTransfer" haven't seen it in the logs?
And thanks for your answers so far!

[afbjorklund]

Where can I find this "needsTransfer" haven't seen it in the logs?

It is here:

minikube/pkg/minikube/machine/cache_images.go

Line 139 in 322e138

func needsTransfer(imgClient *client.Client, imgName string, cr cruntime.Manager) error {

Normally we don't call the docker daemon a "registry", but reserve that term for the docker registry (i.e. the server)

But the current code works by checking the server digest, and then comparing that with the machine (ImageExists)

minikube/pkg/minikube/cruntime/docker.go

Line 137 in 322e138

func (r *Docker) ImageExists(name string, sha string) bool {

[afbjorklund]

Actually it seems the behaviour has changed, after the preload was added. Now it only checks tags, not ids/digests:

Run: docker images --format {{.Repository}}:{{.Tag}}

cache_images.go:74] Images are preloaded, skipping loading

Shouldn't matter in this particular case (mongo:4.2), but only when using something like "latest" for the image tag

It will still save it from the network to the local disk first, though.

i.e. even though it is on the VM, it will save to ~/.minikube/cache