Very long timeout when pushing to HTTP registries

[astefanutti]

Actual behavior

The build is taking a long time. It seems the image upload to a local Docker registry is very slow, compared to a traditional Docker push for the same image. It gets even worse as the image size increases.

This follows up #970. As mentioned by @nicolaferraro, the last log statement is about snapshotting, and there is no other information about what Kaniko is performing afterwards. So this issue attempts at providing a reproducible use case, that indicates most of the build time is taken during the image uploading.

This reproduces as well with Minikube, that has a similar setup.

Expected behavior

The image upload time should not take forever with a local registry, and not be an order of magnitude slower than a Docker push.

To Reproduce
Steps to reproduce the behavior:

$ docker network create mynet

$ docker run -d -p 5000:5000 --name registry --net mynet registry:2.7.1

# --no-push option -> 6.559s
$ time docker run -v `pwd`:/context --net mynet gcr.io/kaniko-project/executor:v0.16.0 --context=/context --dockerfile=Dockerfile --destination=registry:5000/test/kaniko --insecure --verbosity=debug --no-push    
...
INFO[0005] Skipping push to container registry due to --no-push flag 
docker run -v `pwd`:/context --net mynet  --context=/context   --insecure    0.05s user 0.13s system 2% cpu 6.559 total

# without the --no-push option -> 1m:17.91s
time docker run -v `pwd`:/context --net mynet gcr.io/kaniko-project/executor:v0.16.0 --context=/context --dockerfile=Dockerfile --destination=registry:5000/test/kaniko --insecure --verbosity=debug         
...
docker run -v `pwd`:/context --net mynet  --context=/context   --insecure   0.05s user 0.16s system 0% cpu 1:17.91 total

On the other hand:

$ docker pull adoptopenjdk/openjdk8:slim

$ docker tag adoptopenjdk/openjdk8:slim registry:5000/adoptopenjdk/openjdk8:slim

# Docker push of the same image to the same registry -> 11.023s
$ time docker push localhost:5000/adoptopenjdk/openjdk8:slim
The push refers to repository [localhost:5000/adoptopenjdk/openjdk8]
573cb29a6a5d: Pushed 
3bfa04dc9ab0: Pushed 
1c1d913391d2: Pushed 
f55aa0bd26b8: Pushed 
1d0dfb259f6a: Pushed 
21ec61b65b20: Pushed 
43c67172d1d1: Pushed 
slim: digest: sha256:6d3b60d0bda3fb686e50f27e0c0e77e87449b12b70f0a9f666f3ab2b8ab97dc9 size: 1784
docker push localhost:5000/adoptopenjdk/openjdk8:slim  0.09s user 0.10s system 1% cpu 11.023 total

Additional Information

• Dockerfile

   FROM adoptopenjdk/openjdk8:slim
  

• Build Context:

  No additional context

• Kaniko Image (fully qualified with digest)

  gcr.io/kaniko-project/executor:v0.16.0

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[haampie]

I'm experiencing the same issue pushing to a Gitlab container registry

[tejal29]

@haampie and @astefanutti kaniko uses go-container registry to push to registry.
It would be great to introducing some debug logs to mark start of push and end of push to clearly understand if its go-container registry is a bottle neck here or something fishy going on kaniko's side.

[astefanutti]

@tejal29, we've been able to pinpoint the cause at the level of the go-containerregistry library.

Thanks to @nicolaferraro, he's created google/go-containerregistry#715 that should fix this issue.