kaniko cant push to quay ent on prem #400

MansM · 2018-10-17T13:36:43Z

Actual behavior
Kaniko cant push to quay

Expected behavior
kaniko able to push to quay
To Reproduce
Steps to reproduce the behavior:
as gitlab runner

Additional Information

Dockerfile
FROM centos:7
curl -O http://somewhere/internalcacerts.rpm && rpm -ivh internalcacerts.rpm && rm -rf internalcacerts.rpm
Build Context
Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
Kaniko Image (fully qualified with digest)
gcr.io/kaniko-project/executor:debug
Digest: sha256:24c1995868cff80f67a682449c5fe61f2016f6f6c7bbbdd908b4f8b28648fe75

INFO[0014] Taking snapshot of full filesystem...        
2018/10/17 13:27:37 pushed blob sha256:b8d3e972c2e2a5ce50d644aac7c6c6a9ea301e9992ea4678a89cf49ab29a4c22
2018/10/17 13:27:39 pushed blob sha256:bfde11b989fdb1e010026dc77bbd13641c7f654a3183855d927f198d2bcc0dd9
2018/10/17 13:27:43 pushed blob sha256:aeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7
error pushing image: failed to push to destination privatequay/kaniko/testimage:latest: MANIFEST_INVALID: "manifest invalid"
ERROR: Job failed: command terminated with exit code 1

Looks like its related to: bazelbuild/rules_docker#102

The text was updated successfully, but these errors were encountered:

priyawadhwa · 2018-10-17T17:38:13Z

Hey @MansM yah it looks like quay is working on supporting schema v2-2 based on this comment, so this probably won't work until that happens. kaniko depends on a library which assumes this format for pulling and pushing images.

xoen · 2018-11-13T10:56:13Z

Hello,

I see the following error while trying to build a docker image when the base image is on quay.io:

error building image: getting stage builder for stage 0: unsupported status code 405; body: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

EDIT: To be clear, this is a public repository.

@priyawadhwa Is this caused by the lack of support for schema v2-2 as well or should I open another issue for this?

Thanks

xoen · 2018-11-13T12:38:20Z

Also, it may be worth adding a note regarding the fact kaniko doesn't play well with quay.io in the documentation/limitations section. IMHO of course.

priyawadhwa · 2018-11-16T22:37:40Z

I'm not sure if the issue is related to schema, but definitely agree that we should add some documentation around this. Are you able to push with kaniko to a registry that supports the schema?

kameshsampath · 2019-03-07T14:38:04Z

@priyawadhwa I got a v2_2 enabled on my quay repo and I still get the same issue. When I was able to push via docker cli it was able to push the v2_2 image.

vdemeester · 2019-03-07T14:39:09Z

The detailed error is a bit different though 👼

docker run -it -v /home/vincent/.docker/config.json:/root/.docker/config.json:ro -v /home/vincent/src/github.com/vdemeester/break-all-the-thing/foo/:/workspace/foo -e DOCKER_CONFIG=/root/.docker gcr.io/kaniko-project/executor:debug --cont
ext=/workspace/foo --dockerfile=/workspace/foo/Dockerfile --destination=quay.io/rhdevelopers/small-kaniko:0.0.1
INFO[0000] Downloading base image golang:alpine
INFO[0002] Error while retrieving image from cache: getting file info: stat /cache/sha256:d0b6fa6923af1fa27cd324325de44261e7bb801d5bba39cbbf3589ffe5a59293: no such file or directory
INFO[0002] Downloading base image golang:alpine
INFO[0003] Unpacking rootfs as cmd RUN go install -v ./... requires it.
INFO[0260] Taking snapshot of full filesystem...
INFO[0261] RUN go install -v ./...
INFO[0261] cmd: /bin/sh
INFO[0261] args: [-c go install -v ./...]
go: warning: "./..." matched no packages
INFO[0262] Taking snapshot of full filesystem...
INFO[0262] Adding whiteout for /root/.cache/go-build/5e/5eddca62332aaa6cb767cf6e8b4ce6b00d97ee63b3f549ac9586becdaea0f26c-d
INFO[0262] Adding whiteout for /root/.cache/go-build/e0/e0d9637a42516f6aa3defff56cc9a4d3deaf96b2ade546448835c41bd63d82bc-a
INFO[0262] Adding whiteout for /root/.cache/go-build/98/9816bb371bccde8cb7bf5a44febdb32bbc327a0a5c2e940d9b8b3a402f62e4f0-d
INFO[0262] Adding whiteout for /root/.cache/go-build/77/77a522d1008cfd60284c5bd5cc6fc283abe7ef012d61d06d0fd6b1ff03becc68-a
INFO[0262] Adding whiteout for /root/.cache/go-build/b5/b5cb5076dbba489139e99e802f57979f0f7f4f96806267201ca4e383513a312b-a
INFO[0262] Adding whiteout for /root/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d
INFO[0262] Adding whiteout for /root/.cache/go-build/c4/c483d510e0608431902a9e3aa9e162e4b88f745b790a07a1a4b45f22ef9013f4-a
INFO[0262] Adding whiteout for /root/.cache/go-build/31/311571a11a5934d4f63ba6daaa1b0a1e578ee13a1a373547131af1d9d157dbb4-d
error pushing image: failed to push to destination quay.io/rhdevelopers/small-kaniko:0.0.1: INVALID_REQUEST: "Invalid request"

kameshsampath · 2019-03-07T14:40:22Z

@vdemeester - if you make the docker-0 to be like "https://quay.io/v2" you will get a 401 error

HerrmannHinz · 2019-03-13T22:41:40Z

running into a similar issue here:
--- jenkins file:

 * This pipeline will build and deploy a Docker image with Kaniko
 * https://github.com/GoogleContainerTools/kaniko
 * without needing a Docker host
 *
 * You need to create a jenkins-docker-cfg secret with your docker config
 * as described in
 * https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-in-the-cluster-that-holds-your-authorization-token
 */

def label = "kaniko-${UUID.randomUUID().toString()}"

podTemplate(name: 'kaniko', label: label, yaml: """
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true
    volumeMounts:
      - name: jenkins-docker-cfg
        mountPath: /root
  volumes:
  - name: jenkins-docker-cfg
    projected:
      sources:
      - secret:
          name: regcred
          items:
            - key: .dockerconfigjson
              path: .docker/config.json
"""
  ) {

  node(label) {
    stage('Build with Kaniko') {
      git 'https://github.com/jenkinsci/docker-jnlp-slave.git'
      container(name: 'kaniko', shell: '/busybox/sh') {
        withEnv(['PATH+EXTRA=/busybox:/kaniko']) {
          sh '''#!/busybox/sh
          /kaniko/executor -f `pwd`/Dockerfile -c `pwd` --insecure --skip-tls-verify --cache=true --destination=quay.cicd.dev.intra.domain.io/admin/jenkins
          '''
        }
      }
    }
  }
}```

--- the log
```[36mINFO�[0m[0000] Resolved base name jenkins/slave:latest to jenkins/slave:latest 
�[36mINFO�[0m[0000] Resolved base name jenkins/slave:latest to jenkins/slave:latest 
�[36mINFO�[0m[0000] Downloading base image jenkins/slave:latest  
2019/03/13 22:41:00 No matching credentials were found, falling back on anonymous
�[36mINFO�[0m[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:5683f906bab2e28abe332619bcec340a4da8553f10c34bbcbe6b555d32f76196: no such file or directory 
�[36mINFO�[0m[0001] Downloading base image jenkins/slave:latest  
2019/03/13 22:41:01 No matching credentials were found, falling back on anonymous
�[36mINFO�[0m[0002] Built cross stage deps: map[]                
�[36mINFO�[0m[0002] Downloading base image jenkins/slave:latest  
2019/03/13 22:41:02 No matching credentials were found, falling back on anonymous
�[36mINFO�[0m[0002] Error while retrieving image from cache: getting file info: stat /cache/sha256:5683f906bab2e28abe332619bcec340a4da8553f10c34bbcbe6b555d32f76196: no such file or directory 
�[36mINFO�[0m[0002] Downloading base image jenkins/slave:latest  
2019/03/13 22:41:03 No matching credentials were found, falling back on anonymous
�[36mINFO�[0m[0003] Executing 0 build triggers                   
�[33mWARN�[0m[0003] maintainer is deprecated, skipping           
�[36mINFO�[0m[0003] Applying label Description=This is a base image, which allows connecting Jenkins agents via JNLP protocols 
�[36mINFO�[0m[0003] Applying label Vendor=Jenkins project        
�[36mINFO�[0m[0003] Applying label Version=3.27                  
�[36mINFO�[0m[0003] Using files from context: [/home/jenkins/workspace/docker-image-build-and-push-kaniko-quay/jenkins-slave] 
�[36mINFO�[0m[0003] Skipping unpacking as no commands require it. 
�[36mINFO�[0m[0003] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0003] LABEL Description="This is a base image, which allows connecting Jenkins agents via JNLP protocols" Vendor="Jenkins project" Version="3.27" 
�[36mINFO�[0m[0003] Applying label Description=This is a base image, which allows connecting Jenkins agents via JNLP protocols 
�[36mINFO�[0m[0003] Applying label Vendor=Jenkins project        
�[36mINFO�[0m[0003] Applying label Version=3.27                  
�[36mINFO�[0m[0003] No files changed in this command, skipping snapshotting. 
�[36mINFO�[0m[0003] Using files from context: [/home/jenkins/workspace/docker-image-build-and-push-kaniko-quay/jenkins-slave] 
�[36mINFO�[0m[0003] COPY jenkins-slave /usr/local/bin/jenkins-slave 
�[36mINFO�[0m[0003] Taking snapshot of files...                  
�[36mINFO�[0m[0003] ENTRYPOINT ["jenkins-slave"]                 
�[36mINFO�[0m[0003] No files changed in this command, skipping snapshotting. 
2019/03/13 22:41:04 existing blob: sha256:9da6b28682cfe6db721c143309125728417dca93643b3d8d4c22ae7fbb4eb940
2019/03/13 22:41:04 existing blob: sha256:173a06ff64cce302b24e870f0b9d5758161a5bb6f1ab8ba330305552baf530d1
2019/03/13 22:41:04 existing blob: sha256:54f7e8ac135a5f502a6ee9537ef3d64b1cd2fa570dc0a40b4d3b6f7ac81e7486
2019/03/13 22:41:04 existing blob: sha256:eaa976dc543cb2e46a89970e2d079b99ccc3ca4b2c8e6c31adf9511ce8933950
2019/03/13 22:41:04 existing blob: sha256:e02013eddffd972a7b96f084ab9c5eb3683ca9e5c886e56db8f258b24ececedd
2019/03/13 22:41:04 existing blob: sha256:28fc185aee236ed4e066945827b8d3e1e7e63bea22a2173e54f05105a62faf3b
2019/03/13 22:41:04 existing blob: sha256:ee38d9f85cf610794355dc0458445408ded4d648cbc45984cd259611a8a72cc3
2019/03/13 22:41:04 existing blob: sha256:203f1094a1e2165da6f6ec505e8cffb8853d3c72a8088f41c25218121f883b0a
2019/03/13 22:41:04 existing blob: sha256:7f692fae02b67cf2beabf3ef9ce647697740ba979ab16eaec12425ead1c1ddfd
2019/03/13 22:41:04 existing blob: sha256:cc49fe331e2ecfab35824a84df31f7857284c5c52fcf559f6451a8d923d9435f
2019/03/13 22:41:04 existing blob: sha256:087a57faf9491b1b82a83e26bc8cc90c90c30e4a4d858b57ddd5b4c2c90095f6
2019/03/13 22:41:04 existing blob: sha256:5d71636fb824265e30ff34bf20737c9cdc4f5af28b6bce86f08215c55b89bfab
2019/03/13 22:41:04 existing blob: sha256:d6341e30912f12f56e18564a3b582853f65376766f5f9d641a68a724ed6db88f
2019/03/13 22:41:04 pushed blob: sha256:8307581c6ee32b3cd86cc97109f5e46d0c321e6dd8b00fda0f93715c0f143561
2019/03/13 22:41:04 pushed blob: sha256:5099fe4311c0246345cb4ea2a742234b79d20d5ac76b6ca950fc18d0ec66adec
error pushing image: failed to push to destination quay.cicd.dev.intra.domain.io/admin/jenkins:latest: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]```

samm-git · 2019-05-12T15:04:24Z

Same for me:

DEBU[0000] Adding /var/lib to layer, because it was changed.
DEBU[0000] Adding file /var/lib to tar
2019/05/12 15:03:58 existing blob: sha256:688a776db95ffbd66dd4696263d34ca00bd330f30f39a9d39d818a07b086ed17
2019/05/12 15:03:58 existing blob: sha256:6bfc4ec4420a10145bd40caf0499a57618342f27c0ad95f1785b8a1e31090058
2019/05/12 15:03:58 existing blob: sha256:743f2d6c1f65c793009f30acb07845ba2ef968192732afdab2ecf9a475515393
2019/05/12 15:04:00 pushed blob sha256:e927e04a911a46c6b85fec28ba094ca117aa91f6eda342e2c8251c172eb80aa0
error pushing image: failed to push to destination docker.private/oleksii_samorukov/testme:latest: MANIFEST_INVALID: "manifest invalid"

vdemeester · 2019-05-12T15:33:17Z

@samm-git @HerrmannHinz the MANIFEST_INVALID error means this repository is not a schema v2 enabled repository — and kaniko/go-containerregistry does not support schema v1. You may need to update your quay instance to get support for schema v2 (Red Hat Quay 3 supports v2 — not sure if Quay 2.x does)

samm-git · 2019-05-12T18:11:19Z

@vdemeester thank you for reply. With quay io i am getting INVALID_REQUEST: "Invalid request"

Anything i can do here?

error pushing image: failed to push to destination quay.io/samm_git/kaniko-test:latest: INVALID_REQUEST: "Invalid request"

NicolasRouquette · 2019-07-28T15:32:35Z

I ran into the same error as @HerrmannHinz pushing to a Quay Enterprise v2.9.2 repository:

...
INFO[0056] CMD "/app/zeppelin/bin/zeppelin.sh"          
2019/07/28 15:17:12 existing blob: sha256:b6abafe80f63b02535fc111df2ed6b3c728469679ab654e03e482b6f347c9639
2019/07/28 15:17:12 existing blob: sha256:f910a506b6cb1dbec766725d70356f695ae2bf2bea6224dbe8c7c6ad4f3664a2
2019/07/28 15:17:12 existing blob: sha256:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10
2019/07/28 15:17:13 pushed blob: sha256:310aee8eca5b3e62f6d7bff786e99e24a5cf933cd00c44197514e0b7affe8faf
2019/07/28 15:17:13 pushed blob: sha256:19c179ca1c56c1071175d0657a16630f234b3170612914b956f5de1b3e1011b2
2019/07/28 15:17:18 pushed blob: sha256:9ddf6b971be8df6d43dcc11156e9fce1c2d826c657e2e804fa6c2f2f9fe2613c
2019/07/28 15:18:34 pushed blob: sha256:ecba3643a9b9be57afffaacffed8ef369a3fae0c52d3fa193ab2fb4d5825128c
error pushing image: failed to push to destination registry.jpl.nasa.gov/caesar/zeppelin:latest: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]

Based on @vdemeester, it seems that the sysadmins would need to deploy Quay 3.

NicolasRouquette · 2019-07-28T17:04:49Z

I was curious what the manifest looks like.
So, I changed my k8s job like this:

apiVersion: batch/v1
kind: Job
...
spec:
  ...
  template:
    ...
    spec:
      ...
      initContainers:
          ...
        - name: build-and-publish
          image: gcr.io/kaniko-project/executor:latest
          args:
            - "--verbosity=debug"
            - "--context=/data"
            - "--dockerfile=/data/Dockerfile"
            - "--destination=registry.jpl.nasa.gov/caesar/zeppelin:latest"
            # https://github.com/GoogleContainerTools/kaniko/issues/400#issuecomment-515772279
            - "--no-push"
            - "--tarPath=/data/image.tar"
          volumeMounts:
            - mountPath: /kaniko/.docker/config.json
              name: secrets
              subPath: config.json
            - mountPath: /data
              name: data
          resources:
            requests:
              cpu: "1"
              memory: 4Gi
      containers:
        - name: inspect-image
          image: opencaesar/docker-git-utilities:latest
          command:
            - /bin/bash
          args:
            - "-c"
            - "tar xf /data/image.tar manifest.json; cat manifest.json"
          volumeMounts:
            - mountPath: /data
              name: data
          resources:
            requests:
              cpu: "1"
              memory: 128Mi

And I got this:

kubectl logs -l job-name==build-and-publish-zeppelin-image | jq .
[
  {
    "Config": "sha256:5c676888e40a41bed73936071aeabf5150fda254f3a54bcb270d966f4b6437ad",
    "RepoTags": [
      "registry.jpl.nasa.gov/caesar/zeppelin:latest"
    ],
    "Layers": [
      "e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10.tar.gz",
      "f910a506b6cb1dbec766725d70356f695ae2bf2bea6224dbe8c7c6ad4f3664a2.tar.gz",
      "b6abafe80f63b02535fc111df2ed6b3c728469679ab654e03e482b6f347c9639.tar.gz",
      "6d1eceff905aba4af4996da3dfee3ad66f9ee07535d3c4d13f5e9f50dfad3afe.tar.gz",
      "ca893d739e75743cebef3abc0af7386faafea8f5b630cc17814ac4e3a3af434e.tar.gz",
      "34b5a0a5f004db98ba6dada48442d75d72cd9d4c9f0ac104ae89e2e1eb57452d.tar.gz"
    ]
  }
]

Can anyone suggest solutions for a workaround like this:

Update the manifest.json to be compatible with the v1 format that Quay v2 accepts
Republish the updated tarball

HerrmannHinz · 2019-07-28T18:20:10Z

Afaik quay 3 on prem is not available yet? For the saas version One can request to get updated to 3.x There is an issue under moby/buildkit...

HerrmannHinz · 2019-07-28T18:21:31Z

moby/buildkit#409

cvgw · 2019-11-15T23:14:26Z

Kaniko only support the v2 schema. There are no plans to support the v1 schema. Closing this as won't fix. Please re-open if this was closed in error.

priyawadhwa added the area/documentation For all bugs related to documentation label Nov 16, 2018

cvgw added the wontfix This will not be worked on label Nov 15, 2019

cvgw closed this as completed Nov 15, 2019

suzuki-shunsuke mentioned this issue Jan 24, 2020

kaniko を試してみる suzuki-shunsuke/issue#33

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kaniko cant push to quay ent on prem #400

kaniko cant push to quay ent on prem #400

MansM commented Oct 17, 2018

priyawadhwa commented Oct 17, 2018

xoen commented Nov 13, 2018 •

edited

Loading

xoen commented Nov 13, 2018

priyawadhwa commented Nov 16, 2018

kameshsampath commented Mar 7, 2019

vdemeester commented Mar 7, 2019

kameshsampath commented Mar 7, 2019

HerrmannHinz commented Mar 13, 2019 •

edited

Loading

samm-git commented May 12, 2019 •

edited

Loading

vdemeester commented May 12, 2019

samm-git commented May 12, 2019 •

edited

Loading

NicolasRouquette commented Jul 28, 2019

NicolasRouquette commented Jul 28, 2019

HerrmannHinz commented Jul 28, 2019 via email •

edited

Loading

HerrmannHinz commented Jul 28, 2019 via email •

edited

Loading

cvgw commented Nov 15, 2019

kaniko cant push to quay ent on prem #400

kaniko cant push to quay ent on prem #400

Comments

MansM commented Oct 17, 2018

priyawadhwa commented Oct 17, 2018

xoen commented Nov 13, 2018 • edited Loading

xoen commented Nov 13, 2018

priyawadhwa commented Nov 16, 2018

kameshsampath commented Mar 7, 2019

vdemeester commented Mar 7, 2019

kameshsampath commented Mar 7, 2019

HerrmannHinz commented Mar 13, 2019 • edited Loading

samm-git commented May 12, 2019 • edited Loading

vdemeester commented May 12, 2019

samm-git commented May 12, 2019 • edited Loading

NicolasRouquette commented Jul 28, 2019

NicolasRouquette commented Jul 28, 2019

HerrmannHinz commented Jul 28, 2019 via email • edited Loading

HerrmannHinz commented Jul 28, 2019 via email • edited Loading

cvgw commented Nov 15, 2019

xoen commented Nov 13, 2018 •

edited

Loading

HerrmannHinz commented Mar 13, 2019 •

edited

Loading

samm-git commented May 12, 2019 •

edited

Loading

samm-git commented May 12, 2019 •

edited

Loading

HerrmannHinz commented Jul 28, 2019 via email •

edited

Loading

HerrmannHinz commented Jul 28, 2019 via email •

edited

Loading