Docker push creates multiple images on GCR, some with creation timestamp equal to 0 epoch

[pantelis-karamolegkos]

Troubleshooting

Before submitting a bug report please read the Troubleshooting doc.

Behaviour

Steps to reproduce this issue

Build an image and push to local registry

      - name: build and push to local registry
        uses: docker/[email protected]
        with:
          context: ${{ inputs.context }}
          file: ${{ inputs.context }}/${{ inputs.dockerfile }}
          no-cache: ${{ inputs.no_cache }}
          build-args: ${{ inputs.build_args }}
          push: true
          tags: ${{ env.LOCAL_IMAGE }}
          cache-to: type=inline
          provenance: false

Use buildx to copy the image to a GCR registry

      - name: copy tagged image to sre gcr
        if: inputs.image_build == true
        shell: bash
        run: |
          docker buildx imagetools create \
            --tag "${{ steps.set-images.outputs.base }}:${{ inputs.image_tag }}" \
            ${{ env.LOCAL_IMAGE }}

Expected behaviour

Tell us what should happen

One image should be created

Actual behaviour

Instead, I am seeing this weird behaviour

Tell us what happens instead

Configuration

My workflow uses the local registry service

    services:
      registry:
        image: registry:2
        ports:
          - 5000:5000

FWIW

• Repository URL (if public):
• Build URL (if public):

# paste your YAML workflow file here and remove sensitive data

Logs

Download the log file of your build and attach it to this issue.

[mountain-hiker]

I am also seeing 3 images built and pushed, expecting one

[tonistiigi]

It is the GCR web UI that is interpreting the OCI image incorrectly and showing it as 3 separate images. See docker/buildx#1533 for details.

[pantelis-karamolegkos]

Despite the fact that I have provenance: false ?

[pantelis-karamolegkos]

Is there a way around it for the moment?

[mountain-hiker]

Per the link to this issue this worked for me:

name: Build and push
uses: docker/build-push-action@v3
with:
  push: true
  tags: [MY REGISTRY HERE]
  provenance: false

[tonistiigi]

@pantelis-karamolegkos Do you have logs to show for that build that uses provenance: false ?

[pantelis-karamolegkos]

Yes, here they are

Run docker/[email protected]
  with:
    context: ./test_image_creation
    file: ./test_image_creation/Dockerfile
    no-cache: false
    push: true
    tags: localhost:5000/name/app:latest
    provenance: false
    load: false
    pull: false
    github-token: ***

[pantelis-karamolegkos]

@tonistiigi I assume I could switch to the docker format as well, but I am stumbling upon this

[tonistiigi]

@pantelis-karamolegkos The logs. https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/using-workflow-run-logs#downloading-logs

The config in your other issue is not correct. You might be looking for type=image,oci-mediatypes=false

[jedevc]

@pantelis-karamolegkos could you share the full logs of your github ci run? If there's something confidential in there, then feel free to just include the buildx command invoked, and the final push part of the log: e.g.:

=> exporting to image                                                                                                                                                     4.1s
 => => exporting layers                                                                                                                                                    0.0s
 => => exporting manifest sha256:fcaad12161b2745bcca40e953e1e7a9cfb66b9b48c17cee2c079899eebd6fdb5                                                                          0.0s
 => => exporting config sha256:0ba2b9314db6130620130d7f7c6361d86a13573a4551a7c55a6aeda890424136                                                                            0.0s
 => => exporting attestation manifest sha256:4646f10822117652f073b326c6349e22d23f65de67c8f47a088d6e0d6984a353                                                              0.0s
 => => exporting manifest list sha256:98fffba4640144a71473fc4edd811b1d9d276f2654a98cbd420288c3a1fd2bd0                                                                     0.0s
 => => pushing layers                                                                                                                                                      2.8s
 => => pushing manifest for docker.io/test/test:latest@sha256:98fffba4640144a71473fc4edd811b1d9d276f2654a98cbd420288c3a1fd2bd0                         1.2s

[pantelis-karamolegkos]

My setup is the following

• build and push to local registry

      - name: build and push to local registry
        uses: docker/[email protected]
        with:
          context: ${{ inputs.context }}
          file: ${{ inputs.context }}/${{ inputs.dockerfile }}
          no-cache: ${{ inputs.no_cache }}
          build-args: ${{ inputs.build_args }}
          push: true
          tags: ${{ env.LOCAL_IMAGE }}
          builder: ${{ steps.setup-buildx.outputs.name }}
          provenance: false

copy image to gcr

      - name: copy tagged image to gcr
        if: inputs.image_build == true
        shell: bash
        run: |
          docker buildx imagetools create \
            --tag "${{ steps.set-images.outputs.sre_base }}:${{ inputs.image_tag }}" \
            ${{ env.LOCAL_IMAGE }}

Trying to be a little cautious for the moment and avoiding pasting full log output, not in a position to sanitise them

[crazy-max]

@pantelis-karamolegkos Output of Docker info too please:

[jedevc]

Provenance is disabled - there's no messages about it in the log output that would definitely appear if provenance was enabled.

Your logs indicate that you're pushing a single manifest - I'm not sure why you're seeing the weird result in GCR. Could one of them be the result of a previous run? Could you try with a different tag, to ensure they're separate?

If still the same, could you share the contents in the Manifest tab in GCR?

[pantelis-karamolegkos]

Let me delete both ghost images, use another tag and re-run the experiment. I will then share the manifest contents

[pantelis-karamolegkos]

btw I noticed (FWIF) that no matter the action version used, buildkit is always

moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.11.0 830288a71f447b46ad44ad5f7bd45148ec450d44

[jedevc]

btw I noticed (FWIF) that no matter the action version used, buildkit is always

Yes. This is intentional. If you want to pin the buildkit version, you can use the version field of the action to pin the version.

[pantelis-karamolegkos]

Same result

Manifest of the first image in the list (the one tagged 2.2.2)

{
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "schemaVersion": 2,
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:29de78dbd7dfdb97af5ee4a0e840e91e9337a9e74ec45c00c8c4c53fd35ba3f5",
         "size": 696,
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

Manifest of the 2nd image

{
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "digest": "sha256:f623c026bf51ce59c501d428786e33290f80477c63259cb114b8d8326035d67b",
      "size": 1082
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9",
         "size": 3370628
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
         "size": 32
      }
   ]
}

[pantelis-karamolegkos]

Here comes the coolest part...Whenever I delete these 2 images, a 3rd one appears (!!!)

Manifest

{
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "digest": "sha256:f623c026bf51ce59c501d428786e33290f80477c63259cb114b8d8326035d67b",
      "size": 1082
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9",
         "size": 3370628
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
         "size": 32
      }
   ]
}

[pantelis-karamolegkos]

Everything started about a week ago.

[pantelis-karamolegkos]

@jedevc I tried to change the buildkit version by using another buildx version

      - name: set up docker buildx
        id: setup-buildx
        uses: docker/[email protected]
        with:
          version: v0.9.1
          driver-opts: network=host

but still...

BuildKit version
builder-4fcd40b5-ef91-4e1d-9ab5-3c72b1b8a1490: moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.11.0 830288a71f447b46ad44ad5f7bd45148ec450d44

does this make sense?

[jedevc]

to pin the buildkit version you need:

      - name: set up docker buildx
        id: setup-buildx
        uses: docker/[email protected]
        with:
          version: v0.9.1
          driver-opts: network=host,image=moby/buildkit:v0.10.6

[jedevc]

Out of curiosity, does your Dockerfile have a #syntax= directive at the top? If so, could you share?

[pantelis-karamolegkos]

No, I have the simplest Dockerfile possible; created ad hoc just for the testing

FROM alpine:latest

RUN ls -al

[hadim]

I got the same issue using the v3 of that action (images of size 0 and off timestamp, etc). Using provenance: false was enough in my case (pushing on GCR + deploying on Cloud Run).

[enpatrik]

Same behaviour on AWS ECR (untagged image with size 0) with v3.3.0 release. Using provenance: false fixed the issue.

[zymotik]

Ideally this feature would have better been implemented in v4 or defaulted to false. Related to #778.

Would it be prudent to release a new sub version (ie: 3.4) with it defaulted to false? It would save a lot of headache for others.

[jedevc]

The change is not introduced in docker/build-push-action, the breaking change is actually introduced in buildx.

build-push-action only invokes buildx, using the defaults provided from that. You can pin buildx to a desired version, if you're concerned about breakage in-between versions. See the docs in the README for more information about how to do that.

Producing a new sub version that changes the default is not something that can be done in build-push-action - and it's not a likely change to happen in buildx either - breakage in the registry is a registry issue, not a builder issue.

[schongabor]

After using docker/build-push-action@v3 instead of v2, the provenance: false did the trick for me. 🤘

[crazy-max]

After using docker/build-push-action@v3 instead of v2, the provenance: false did the trick for me. 🤘

Oh right if you're still using docker/build-push-action@v2 then attestation provenance will always be created as we are not going to backport the provenance input to v2. Also if you're using v2 a warning should be shown if input doesn't not exist:

And as v2 is deprecated you should also have this warning in your pipeline:

Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: docker/build-push-action@v2. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.

[cosmith]

In which version is this supposed to be fixed? I still have the issue with docker/[email protected] and docker/setup-buildx-action@v3

without the provenance param:

with provenance: false:

[zekronium]

Still an issue at @v6 too

[mrclrchtr]

The same for me. If the problem is not fixed, why is the issue closed? Should a new one be opened?

[ap0h]

The same for me. If the problem is not fixed, why is the issue closed? Should a new one be opened?

Someone wrote above that the issue is in buildx and registry but using provenance: false resolved issue for others, I didn't try it yet but I will.

[mrclrchtr]

Someone wrote above that the issue is in buildx and registry but using provenance: false resolved issue for others, I didn't try it yet but I will.

Thank you, missed that.

In case anyone is waiting here. It seems to be solved with the current version.