Inconvenient on-demand mirroring - Not practical on the most obvious scenario

[kikokikok]

Hi all,

First I'd like to thank you for this great project which brought amazing value to us.
I'd like though, to share with you a scenario where I feel zot could do a lot better but unfortunately falls a bit short.

In our context, we use zot as a local laptop registry for developers in their local all-batteries included k3s cluster.
Zot is configured as an on-demand proxy to a company-wide shared artifactory instance.
In this particular setup what we expect from zot is the ability for developers to build locally as if the CI was building so that their inner-dev loop is as optimised and fast as possible.
Here is the glitch:
Whenever I have a multi-stage dockerfile to build which is referencing multiple base images through zot but in reality mirrored from the real ones in artifactory, the docker build will fail each time the image layers haven't yet been retrieved and the same applied on pull scenarios from the kubelet of k3s until the image is fully retrieved.

Question: Is it on purpose that we don't maintain an open connection to the caller for any pull until we know if the image exists locally or remotely and only after:

• throw an http error OR retrieve the layers in background while maitaining the connection open and even better streaming the layers to the client as soon as they are retrieved ?

Thanks !

[rchincha]

@kikokikok thanks for trying out zot and the feedback.

You are likely referring to (as an enhancement):
#1395

We will take a look at how we can improve this in the short-term.

Meanwhile, as a workaround, maybe enable a periodic poll as well? is this viable for you?

[kikokikok]

Oh I missed this one, it would be perfect as a target.
In the meantime, if we could just implement the intermediate fix to avoid having to rely on client-side retries on exceptions until the whole image is retrieved by zot maybe.
I can try to help if you point me to the right place where I can have a look by the way.

Unfortunately the periodic poll is not an option as we never know which images to pull and mirroring all images is not an option on a laptop.

[eusebiu-constantin-petu-dbk]

Hello @kikokikok

sync on demand works like this:

1. client asks for an image (its manifest only) not present yet in zot
2. zot initiate sync, maintains an open connection with client while pulling the entire image(manifest+layers)
3. finishes pulling entire image, serves the manifest to the client
4. client having the manifest, now knows which layers it should pull
5. pull layers already present in zot

Actually we do maintain an open connection for the on demand sync, but only for the manifest, when zot finishes syncing it will respond to the manifest call, and from that point all the layers should already be synced.

I've tried to reproduce this with a multi stage Dockerfile and it works on my side.

I don't understand how your setup tries to pull layers before pulling the manifest.

The only thing that comes to my mind is that you get a timeout because zot doesn't actually send anything on the open connection,
but how it asks for layers then? how it knows what layers to ask for?

Can you give us logs from zot and also from docker. Thank you!
Also the zot config would be useful.

[kikokikok]

Hello @peusebiu,
thanks for your reply, here are a few logs.

From docker buildx build command

docker buildx build . \
--push \
--secret id=ARTIFACTORY_USERNAME,env=REPOSILITE_USER \
--secret id=ARTIFACTORY_PASSWORD,env=REPOSILITE_PASSWORD \
--progress plain \
-t zot.kyri.test/kapp-local/front:24.6.10 \
-f front.Dockerfile \
--platform linux/arm64 \
--target dev \
--build-arg OCI_REGISTRY_ROOT_ARG=zot.kyri.test \
--build-arg ARTIFACTORY_BASE_URL_ARG=reposilite.kyri.test \
--build-arg KS_VERSION_FILE_PATH=ks-versions.properties \
--build-arg BUILD_VERSION=24.6.10 \
--build-arg GRADLE_PROPERTIES_FILE=.config/gradle.properties \

#0 building with "k3easy" instance using kubernetes driver

#1 [internal] load build definition from front.Dockerfile
#1 transferring dockerfile: 6.72kB done
#1 DONE 0.0s

#2 [internal] load metadata for zot.kyri.test/dockerhub/gradle:latest
#2 ...

#3 [internal] load metadata for zot.kyri.test/kapp/redacted:redacted
#3 DONE 1.2s

**#2 [internal] load metadata for zot.kyri.test/dockerhub/gradle:latest**
**#2 ERROR: zot.kyri.test/dockerhub/gradle:latest: not found**
------
 > [internal] load metadata for zot.kyri.test/dockerhub/gradle:latest:
------
front.Dockerfile:2
--------------------
   1 |     ARG OCI_REGISTRY_ROOT_ARG=artifactory.redacted.redacted.com
   2 | >>> FROM --platform=$BUILDPLATFORM $OCI_REGISTRY_ROOT_ARG/dockerhub/gradle AS builder
   3 |     ARG KS_VERSION_FILE_PATH=ks-versions.properties
   4 |     ARG HSM_JCE_FILE=cloudhsm-client-jce-3.4.4-1.el7.x86_64.rpm
--------------------
ERROR: failed to solve: zot.kyri.test/dockerhub/gradle: failed to resolve source metadata for zot.kyri.test/dockerhub/gradle:latest: zot.kyri.test/dockerhub/gradle:latest: not found

From the buildkit pods running inside k3s local cluster

time="2024-05-07T17:34:40Z" level=info msg="trying next host - response was http.StatusNotFound" host=zot.kyri.test
time="2024-05-07T17:34:40Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/Solve returned error: rpc error: code = Unknown desc = zot.kyri.test/dockerhub/gradle: failed to resolve source metadata for zot.kyri.test/dockerhub/gradle:latest: zot.kyri.test/dockerhub/gradle:latest: not found"
time="2024-05-07T17:34:40Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = zot.kyri.test/dockerhub/gradle: failed to resolve source metadata for zot.kyri.test/dockerhub/gradle:latest: zot.kyri.test/dockerhub/gradle:latest: not found"
time="2024-05-07T17:40:04Z" level=info msg="trying next host - response was http.StatusNotFound" host=zot.kyri.test
time="2024-05-07T17:40:04Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/Solve returned error: rpc error: code = Unknown desc = zot.kyri.test/dockerhub/gradle: failed to resolve source metadata for zot.kyri.test/dockerhub/gradle:latest: zot.kyri.test/dockerhub/gradle:latest: not found"
time="2024-05-07T17:40:04Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = zot.kyri.test/dockerhub/gradle: failed to resolve source metadata for zot.kyri.test/dockerhub/gradle:latest: zot.kyri.test/dockerhub/gradle:latest: not found"

From the ingress (configured Head timeout at 720 seconds if you ask)
2024/05/07 17:40:04 [error] 1402#1402: *230631 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.0.0.194, server: zot.kyri.test, request: "HEAD /v2/dockerhub/

From zot pod

{"level":"info","module":"http","component":"session","clientIP":"10.0.0.116:56946","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.29"]},"goroutine":10019,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-05-07T22:58:31.432962102Z","message":"HTTP API"}
{"level":"info","remote":"https://artifactory.redacted.redacted.com:443","repository":"kapp/oraclexe","reference":"24.6.10","goroutine":10018,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:283","time":"2024-05-07T22:58:35.134277728Z","message":"syncing image"}
{"level":"error","error":": Path not found: /var/lib/registry/kapp/oraclexe/index.json","dir":"/var/lib/registry/kapp/oraclexe","goroutine":10018,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1464","time":"2024-05-07T22:58:36.00233334Z","message":"failed to read index.json"}
{"level":"info","remote image":"artifactory.redacted.redacted.com:443/kapp/oraclexe:24.6.10","local image":"kapp/oraclexe:24.6.10","goroutine":10018,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:438","time":"2024-05-07T22:58:36.002709188Z","message":"syncing image"}
{"level":"info","module":"http","component":"session","clientIP":"10.0.0.116:52792","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.29"]},"goroutine":10107,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-05-07T22:58:41.432706218Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"10.0.0.116:52790","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.29"]},"goroutine":10106,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-05-07T22:58:41.432706218Z","message":"HTTP API"}
{"level":"info","repository":"kapp/oraclexe","reference":"24.6.10","goroutine":7613,"caller":"zotregistry.dev/zot/pkg/api/routes.go:1913","time":"2024-05-07T22:58:41.992189958Z","message":"trying to get updated image by syncing on demand"}
{"level":"info","goroutine":10109,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:481","time":"2024-05-07T22:58:41.992232168Z","message":"getting available client"}
{"level":"info","remote":"https://artifactory.redacted.redacted.com:443","repository":"kapp/oraclexe","reference":"24.6.10","goroutine":10109,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:283","time":"2024-05-07T22:58:42.097523689Z","message":"syncing image"}
{"level":"error","error":": Path not found: /var/lib/registry/kapp/oraclexe/index.json","dir":"/var/lib/registry/kapp/oraclexe","goroutine":10109,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1464","time":"2024-05-07T22:58:42.991061796Z","message":"failed to read index.json"}
{"level":"info","remote image":"artifactory.redacted.redacted.com:443/kapp/oraclexe:24.6.10","local image":"kapp/oraclexe:24.6.10","goroutine":10109,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:438","time":"2024-05-07T22:58:42.991251553Z","message":"syncing image"}

When trying to pull from K3s (Containerd / Kubelet)
Failed to pull image "zot.kyri.test/kapp/oraclexe:24.6.10": rpc error: code = NotFound desc = failed to pull and unpack image "zot.kyri.test/kapp/oraclexe:24.6.10": failed to resolve reference "zot.kyri.test/kapp/oraclexe:24.6.10": zot.kyri.test/kapp/oraclexe:24.6.10: not found

[eusebiu-constantin-petu-dbk]

Ok, it seems that it timeouts while waiting for a server(zot) reply. It waits for ~6 seconds before killing the connection. Head timeout is not enough here, I think there are other configurable options that can be increased on the ingress configuration.

Anyway, we definately need to implement this one: #1395