Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker manifest does not work with output from buildx v0.10 #1509

Closed
debu99 opened this issue Jan 11, 2023 · 13 comments
Closed

docker manifest does not work with output from buildx v0.10 #1509

debu99 opened this issue Jan 11, 2023 · 13 comments

Comments

@debu99
Copy link

debu99 commented Jan 11, 2023

How to reproduce:
$ uname -a
Darwin xxxxxx 21.6.0 Darwin Kernel Version 21.6.0: Thu Sep 29 20:13:46 PDT 2022; root:xnu-8020.240.7~1/RELEASE_ARM64_T8101 arm64

$ docker buildx version
github.com/docker/buildx v0.10.0 8764628

$ docker buildx create --driver=docker-container --name=buildkit-builder --use
$ docker buildx inspect --bootstrap
$ docker buildx ls

$docker buildx build -t yourname/test:2-arm64 --platform=linux/arm64 --push .

$ docker manifest inspect yourname/test:2-arm64
no such manifest: docker.io/yourname/test:2-arm64
image

@crazy-max
Copy link
Member

Since buildx 0.10, a minimal provenance attestation is also pushed (unknown/unknown):

$ docker buildx build --tag crazymax/test --push .
...
#7 exporting to image
#7 exporting layers
#7 exporting layers 0.2s done
#7 exporting manifest sha256:b52fc2edb674b9a3fc945addd13e468984c91058eba02dd45c9827df5299dddb 0.0s done
#7 exporting config sha256:0355c7b04a8e614b6de7c038a04a063dab0655c77280e54a488979bf52b28b43 0.0s done
#7 exporting attestation manifest sha256:3032911c04a048a7e1d148e6505ed00d5690d8ae12fef878ad05ce77311afa4f 0.0s done
#7 exporting manifest list sha256:055420aed05f4dcfeab12c64dbb4dbcdfa28bd9f9647fe1a63e0813c3694e14f
#7 exporting manifest list sha256:055420aed05f4dcfeab12c64dbb4dbcdfa28bd9f9647fe1a63e0813c3694e14f 0.0s done
#7 pushing layers
#7 pushing layers 1.5s done
#7 pushing manifest for docker.io/crazymax/test:latest@sha256:055420aed05f4dcfeab12c64dbb4dbcdfa28bd9f9647fe1a63e0813c3694e14f
#7 pushing manifest for docker.io/crazymax/test:latest@sha256:055420aed05f4dcfeab12c64dbb4dbcdfa28bd9f9647fe1a63e0813c3694e14f 1.5s done
#7 DONE 3.3s

But don't think docker manifest supports OCI manifest. Debugging this command:

$ docker --debug manifest inspect crazymax/test
DEBU[0000] endpoints for docker.io/crazymax/test:latest: [{false https://registry-1.docker.io v2 false true true 0xc0003fd380}] 
DEBU[0000] not continuing on error (*errors.fundamental) docker.io/crazymax/test:latest is a manifest list 
DEBU[0000] endpoints for docker.io/crazymax/test:latest: [{false https://registry-1.docker.io v2 false true true 0xc00015ca80}]
DEBU[0002] continuing on error (errcode.Errors) manifest unknown: OCI manifest found, but accept header does not support OCI manifests 
no such manifest: docker.io/crazymax/test:latest

If you disable provenance it should work:

$ docker buildx build --tag crazymax/test --push --provenance=false .
#7 exporting to image
#7 exporting layers
#7 exporting layers done
#7 exporting manifest sha256:cf680f47b231ad1be1443c475cd45510213e7c05b839ee194d6593bb06ff8de9 0.0s done
#7 exporting config sha256:0355c7b04a8e614b6de7c038a04a063dab0655c77280e54a488979bf52b28b43 done
#7 pushing layers
#7 pushing layers 0.6s done
#7 pushing manifest for docker.io/crazymax/test:latest@sha256:cf680f47b231ad1be1443c475cd45510213e7c05b839ee194d6593bb06ff8de9
#7 pushing manifest for docker.io/crazymax/test:latest@sha256:cf680f47b231ad1be1443c475cd45510213e7c05b839ee194d6593bb06ff8de9 0.5s done
#7 DONE 1.1s
$ docker --debug manifest inspect crazymax/test
{
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "schemaVersion": 2,
        "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "digest": "sha256:0355c7b04a8e614b6de7c038a04a063dab0655c77280e54a488979bf52b28b43",
                "size": 1341
        },
        "layers": [
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "digest": "sha256:2461e8255644c236fb21f20d482f3ca7017d69b52f3177737502fbcfbdc82f55",
                        "size": 2588339
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "digest": "sha256:dd0d0c2c5c3d6127c4e2126db4658d49f9f0336f2f197f0ff48fb3ee89604719",
                        "size": 154
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "digest": "sha256:6d769fe3e59f51cf3f930102006edd896f257e4103cccd176fda77c5b02beaeb",
                        "size": 159
                }
        ]
}

cc @tonistiigi @jedevc

@jedevc
Copy link
Collaborator

jedevc commented Jan 11, 2023

Copying my workaround suggestions from another issue:

  1. Use docker buildx imagetools inspect --raw instead of docker manifest inspect - it should work similarly, and supports all the different media types in the registry. See the docs for docker buildx imagetools inspect..
    Hopefully, this will just work, and should be a drop-in replacement!
  2. Set oci-mediatypes=false in your --output flag (to use the docker distribution manifest list instead of an OCI index).
    This might cause some issues with the generated provenance, which means you'd probably prefer 3 instead.
  3. Set --provenance=false to not generate the provenance (which is what causes the multi-platform index to be generated, even for a single platform).
    Just remove the generated provenance entirely, this means that only a single manifest is created, no index needed, which sidesteps the problem.

@item4
Copy link

item4 commented Jan 11, 2023

If I use imagetools instead of default one, how can I use manifest push with imagetools?

@jedevc
Copy link
Collaborator

jedevc commented Jan 11, 2023

To merge manifests using imagetools you can use imagetools create to construct and push the manifest all in one step. If you're doing multi-platform builds, you can also use the new --platform flag to handle this automatically, without needing to have separate manifest merging logic, etc: https://docs.docker.com/build/building/multi-platform/.

@jedevc
Copy link
Collaborator

jedevc commented Jan 11, 2023

Looks related to moby/moby#43126 👀

@ddelange
Copy link

Hi 👋

I'd like to use docker manifest inspect -v to sum up layer sizes: https://stackoverflow.com/a/73108928/5511061

When --provenance is specified during build/push, I'll get a no such manifest error when querying docker manifest inspect -v for any image that includes an attestation manifest.

Additionally, docker buildx imagetools inspect --raw does not include the verbosity needed to sum up layer sizes.

Leaving this here as suggested in docker/build-push-action#764

@jedevc jedevc changed the title buildx 0.10.0 manifesh push error docker manifest does not work with output from buildx v0.11 Jan 20, 2023
@jedevc jedevc changed the title docker manifest does not work with output from buildx v0.11 docker manifest does not work with output from buildx v0.10 Jan 20, 2023
elrido added a commit to PrivateBin/docker-nginx-fpm-alpine that referenced this issue Jan 21, 2023
ddelange added a commit to ddelange/actions-runner-controller-releases that referenced this issue Jan 23, 2023
hoping it gets fixed upstream ref docker/buildx#1509 (comment)
kodiakhq bot pushed a commit to Azure/iotedge that referenced this issue Jan 24, 2023
Docker recently introduced [provenance attestation](https://docs.docker.com/build/attestations/slsa-provenance/) in buildx 0.10.0, which broke our multi-arch image builds. By default, `docker buildx build` now defaults to `--provenance true` which causes even single-architecture images to be built as a manifest list (aka multi-arch image consisting of one architecture, according to this GitHub issue [comment](docker/buildx#1509 (comment))). When we use our older manifest-tool to create a manifest from three single-arch images (amd64, arm32v7, and arm64v8), the tool fails because it can't create a manifest list that points to other manifest lists.

To mitigate, we'll disable provenance attestation for now. In the future we should look at updating how we create our multi-arch images.

To test, I ran the CI build and end-to-end tests to confirm the images build successfully and function as expected.

## Azure IoT Edge PR checklist:
phdelodder pushed a commit to deconz-community/deconz-docker that referenced this issue Jan 25, 2023
Workaround to opt-out of unsupported manifest until it's supported. Fix is from: docker/buildx#1509 (comment)

Watchtower autoupdater should work again: containrrr/watchtower#1529
damonbarry added a commit to damonbarry/iotedge that referenced this issue Jan 25, 2023
Docker recently introduced [provenance attestation](https://docs.docker.com/build/attestations/slsa-provenance/) in buildx 0.10.0, which broke our multi-arch image builds. By default, `docker buildx build` now defaults to `--provenance true` which causes even single-architecture images to be built as a manifest list (aka multi-arch image consisting of one architecture, according to this GitHub issue [comment](docker/buildx#1509 (comment))). When we use our older manifest-tool to create a manifest from three single-arch images (amd64, arm32v7, and arm64v8), the tool fails because it can't create a manifest list that points to other manifest lists.

To mitigate, we'll disable provenance attestation for now. In the future we should look at updating how we create our multi-arch images.

To test, I ran the CI build and end-to-end tests to confirm the images build successfully and function as expected.

## Azure IoT Edge PR checklist:
sunchao pushed a commit to sunchao/spark that referenced this issue Jun 2, 2023
… manifests

### What changes were proposed in this pull request?

This is found during Apache Spark 3.3.2 docker image publishing. It's not an Apache Spark but important for `docker-image-tool.sh` to provide backward compatibility during cross-building. This PR targets for all **future releases**, Apache Spark 3.4.0/3.3.3/3.2.4.

### Why are the changes needed?

Docker `buildx` v0.10.0 publishes OCI Manifests by default which is not supported by `docker manifest` command like the following.
docker/buildx#1509
```
$ docker manifest inspect apache/spark:v3.3.2
no such manifest: docker.io/apache/spark:v3.3.2
```

Note that the published images are working on both AMD64/ARM64 machines, but `docker manifest` cannot be used. For example, we cannot create `latest` tag.

### Does this PR introduce _any_ user-facing change?

This will fix the regression of Docker `buildx`.

### How was this patch tested?

Manually builds the multi-arch image and check `manifest`.

```
$ docker manifest inspect apache/spark:v3.3.2
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3444,
         "digest": "sha256:30ae5023fc384ae3b68d2fb83adde44b1ece05f926cfceecac44204cdc9e79cb",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3444,
         "digest": "sha256:aac13b5b5a681aefa91036d2acae91d30a743c2e78087c6df79af4de46a16e1b",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}
```

Closes apache#40051 from dongjoon-hyun/SPARK-42462.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 2ac70ae)
Signed-off-by: Dongjoon Hyun <[email protected]>
mkuf added a commit to mkuf/prind that referenced this issue Jun 10, 2023
snmvaughan pushed a commit to snmvaughan/spark that referenced this issue Jun 20, 2023
… manifests

### What changes were proposed in this pull request?

This is found during Apache Spark 3.3.2 docker image publishing. It's not an Apache Spark but important for `docker-image-tool.sh` to provide backward compatibility during cross-building. This PR targets for all **future releases**, Apache Spark 3.4.0/3.3.3/3.2.4.

### Why are the changes needed?

Docker `buildx` v0.10.0 publishes OCI Manifests by default which is not supported by `docker manifest` command like the following.
docker/buildx#1509
```
$ docker manifest inspect apache/spark:v3.3.2
no such manifest: docker.io/apache/spark:v3.3.2
```

Note that the published images are working on both AMD64/ARM64 machines, but `docker manifest` cannot be used. For example, we cannot create `latest` tag.

### Does this PR introduce _any_ user-facing change?

This will fix the regression of Docker `buildx`.

### How was this patch tested?

Manually builds the multi-arch image and check `manifest`.

```
$ docker manifest inspect apache/spark:v3.3.2
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3444,
         "digest": "sha256:30ae5023fc384ae3b68d2fb83adde44b1ece05f926cfceecac44204cdc9e79cb",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3444,
         "digest": "sha256:aac13b5b5a681aefa91036d2acae91d30a743c2e78087c6df79af4de46a16e1b",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}
```

Closes apache#40051 from dongjoon-hyun/SPARK-42462.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 2ac70ae)
Signed-off-by: Dongjoon Hyun <[email protected]>
mdesouky added a commit to uptick/actions that referenced this issue Jul 26, 2023
When trying to use the docker image built by the pipeline for lambda it fails with this error The image manifest or layer media type for the source image 305686791668.dkr.ecr.ap-southeast-2.amazonaws.com/auto-scheduler:main-e3745b8 is not supported.

this always happens on the main branch for some reason as it was not encountered with the test tag but nevertheless here is the issue on github:
docker/buildx#1509 (comment)
exercism/github-actions#113
eliteprodev added a commit to eliteprodev/rtc-simple-server that referenced this issue Sep 8, 2023
elsoa-invitech pushed a commit to elsoa-invitech/docker-nginx-fpm-alpine that referenced this issue Oct 31, 2023
tcitworld added a commit to framasoft/mobilizon that referenced this issue Nov 22, 2023
bobbypage added a commit to bobbypage/cadvisor that referenced this issue Mar 2, 2024
This is causing failure to combine the multi-arch images. See
docker/buildx#1509 for discussion

Signed-off-by: David Porter <[email protected]>
bobbypage added a commit to bobbypage/cadvisor that referenced this issue Mar 2, 2024
This is causing failure to combine the multi-arch images. See
docker/buildx#1509 for discussion

Signed-off-by: David Porter <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants